Spark - (Executor) Cache

About

Data - Cache in Spark.

Each executor has a cache.

Example

  • lines is recomputed
lines = sc.textFile("...", 4)
comments = lines.filter(isComment) 
print lines.count()
print comments.count() # lines is recomputed
  • lines is NOT recomputed but get from the cache
lines = sc.textFile("...", 4)
lines.cache() # save, lines is NOT recomputed when comments.count() is called
comments = lines.filter(isComment) 
print lines.count()
print comments.count()

Documentation / Reference


Powered by ComboStrap