Spark memory management

This memory pool is managed by Spark. This is responsible for storing intermediate state while doing task execution like joins or to store the broadcast variables. All the cached/persisted data will be stored in this segment, specifically in the storage memory of this segment.

1. How does spark memory work?
2. Can spark run out of memory?
3. How is memory divided in spark?
4. How do I tune my spark memory?

How does spark memory work?

Apache Spark is a cluster-computing platform that provides an API for distributed programming similar to the MapReduce model, but is designed to be fast for interactive queries and iterative algorithms. It primarily achieves this by caching data required for computation in the memory of the nodes in the cluster.

Can spark run out of memory?

Out of memory at the driver level

A driver in Spark is the JVM where the application's main control flow runs. More often than not, the driver fails with an OutOfMemory error due to incorrect usage of Spark.

How is memory divided in spark?

On Heap Memory

By default, Spark uses On-memory heap only. The On-heap memory area in the Executor can be roughly divided into the following four blocks: Storage Memory: It's mainly used to store Spark cache data, such as RDD cache, Unroll data, and so on.

How do I tune my spark memory?

Here are a few ways to do this:

1. In case the RAM size is less than 32 GB, the JVM flag should be set to –xx:+ UseCompressedOops. ...
2. Nested structures can be dodged by using several small objects as well as pointers.
3. Instead of using strings for keys you could use numeric IDs and enumerated objects.