Apache Spark Memory Tuning: Storage vs Execution Memory & Off-Heap Sizing

Complete guide to demystifying the Spark Memory Pool. How unified memory management allocates between storage and execution, how to tune off-heap memory, and how to eliminate disk spills.

Elena Rostova

Head of Cloud & SRE Practice • Distributed Systems & Big Data

AI & Executive Summary

What is Apache Spark Memory Tuning: Storage vs Execution Memory & Off-Heap Sizing?

Complete guide to demystifying the Spark Memory Pool. How unified memory management allocates between storage and execution, how to tune off-heap memory, and how to eliminate disk spills.

Key Architecture Takeaways (Sathus Technology)

•Spark unified memory (spark.memory.fraction, default 0.60) is dynamically shared between execution (shuffles/joins) and storage (caching/broadcasts).
•Execution memory borrows from storage memory, but storage cannot evict execution tasks that are actively processing.
•YARN container killed by memory limits are almost always caused by insufficient spark.executor.memoryOverhead (off-heap memory) during PyArrow or heavy JVM GC.
•Disk spills during shuffles can be eliminated by increasing spark.sql.shuffle.partitions or tuning broadcast join thresholds.

Frequently Asked Questions

Q: What is the root cause of "Container killed by YARN for exceeding memory limits"?

A: This error occurs when the physical memory of the executor container exceeds spark.executor.memory + spark.executor.memoryOverhead. Common causes include unvectorized Python UDFs running in PySpark worker processes, heavy off-heap NIO buffers, and JVM native memory leaks.

Key Architectural Takeaways

Spark unified memory (spark.memory.fraction, default 0.60) is dynamically shared between execution (shuffles/joins) and storage (caching/broadcasts).
Execution memory borrows from storage memory, but storage cannot evict execution tasks that are actively processing.
YARN container killed by memory limits are almost always caused by insufficient spark.executor.memoryOverhead (off-heap memory) during PyArrow or heavy JVM GC.
Disk spills during shuffles can be eliminated by increasing spark.sql.shuffle.partitions or tuning broadcast join thresholds.

Production Implementation Code

Production Spark Submit Configuration for High-Memory Workloads

Production Spark Submit Configuration for High-Memory Workloadsyaml

# Production Spark Cluster Configuration (32GB Executor Nodes)
spark.executor.instances: 20
spark.executor.cores: 4
spark.executor.memory: 24g
spark.executor.memoryOverhead: 4g
spark.memory.fraction: 0.70
spark.memory.storageFraction: 0.30
spark.sql.adaptive.enabled: true
spark.sql.adaptive.skewJoin.enabled: true
spark.sql.adaptive.coalescePartitions.enabled: true

Frequently Asked Technical Questions

What is the root cause of "Container killed by YARN for exceeding memory limits"?

This error occurs when the physical memory of the executor container exceeds spark.executor.memory + spark.executor.memoryOverhead. Common causes include unvectorized Python UDFs running in PySpark worker processes, heavy off-heap NIO buffers, and JVM native memory leaks.

Elena Rostova

Peer Reviewed

Head of Cloud & SRE Practice

Part of the Distributed Systems & Big Data at Sathus Technology. Specializing in mission-critical data lakehouses, streaming analytics, and compliance-driven platforms.

Enterprise Engineering Advisory

Suffering from Slow Spark Jobs or OOM Crashes?

Our distributed systems engineers analyze Spark cluster memory metrics, eliminate disk spills, and right-size executor overhead for mission-critical batch and streaming workloads.

Get Spark Cluster Audit

Next in this Knowledge Cluster

implementation guide

PySpark OOM Error: Driver vs Executor Memory Triage & Production Remediation

16 min read

implementation guide

Apache Spark Join Optimization: Broadcast Hash vs Sort-Merge vs Shuffle Hash Joins

16 min read

implementation guide

Implementation Blueprint: Production PySpark Pipeline Framework

16 min read

SathusSathus Technology

Key Architectural Takeaways

Spark unified memory (spark.memory.fraction, default 0.60) is dynamically shared between execution (shuffles/joins) and storage (caching/broadcasts).

Execution memory borrows from storage memory, but storage cannot evict execution tasks that are actively processing.

YARN container killed by memory limits are almost always caused by insufficient spark.executor.memoryOverhead (off-heap memory) during PyArrow or heavy JVM GC.

Disk spills during shuffles can be eliminated by increasing spark.sql.shuffle.partitions or tuning broadcast join thresholds.

# Production Spark Cluster Configuration (32GB Executor Nodes) spark.executor.instances: 20 spark.executor.cores: 4 spark.executor.memory: 24g spark.executor.memoryOverhead: 4g spark.memory.fraction: 0.70 spark.memory.storageFraction: 0.30 spark.sql.adaptive.enabled: true spark.sql.adaptive.skewJoin.enabled: true spark.sql.adaptive.coalescePartitions.enabled: true

Frequently Asked Technical Questions