Spark Executor Memory & Partition Sizing Calculator

Read Full Triage Guide: PySpark OOM Driver vs Executor Memory

Interactive Engineering Asset

Compute container allocation, off-heap buffers, and AQE partition targets based on Sathus distributed systems sizing formulas.

OOM Protection Formula Active

Stage Input Data Size:250 GB

10 GB500 GB1,000 GB2,000 GB

Worker Node Physical RAM:32 GB

Cores per Worker Node:8 Cores

Workload Execution Profile:

Tuned Spark Submit Sizing

spark.executor.memory

10 GB

JVM Heap per container

spark.executor.memoryOverhead

3584 MB

Off-heap Python/NIO safety buffer

spark.executor.cores

4 Cores

Optimal GC pause limit

spark.sql.shuffle.partitions

2000

~128MB per target partition

JVM Unified Memory Allocation

Exec (3G)

Storage (3G)

User (40%)

# Generated by Sathus Distributed Systems Calculator
spark.executor.instances: 16
spark.executor.cores: 4
spark.executor.memory: 10g
spark.executor.memoryOverhead: 3584m
spark.sql.shuffle.partitions: 2000
spark.sql.adaptive.enabled: true
spark.sql.adaptive.skewJoin.enabled: true
spark.sql.adaptive.coalescePartitions.enabled: true
spark.memory.fraction: 0.60
spark.memory.storageFraction: 0.50

How the Sathus Memory Sizing Formulas Work

1. Core Capping (4-5 Cores)

Assigning more than 5 cores to an executor leads to severe JVM stop-the-world garbage collection pauses. Keeping executor cores at 4 maximizes multi-threading while keeping GC pauses under 200ms.

2. PySpark Off-Heap Sizing

PySpark delegates DataFrame transformations to Python worker processes (PyArrow C++ memory, Pandas UDFs). These run outside the JVM. Without 20-25% memoryOverhead, YARN or K8s cgroups will send SIGKILL (Exit code 137).

3. 128MB Partition Rule

Spark tasks achieve optimal throughput when processing 100MB to 200MB per partition. Smaller partitions create metadata scheduling overhead; larger partitions (>2GB) crash the Spark internal ByteBuffer limit.

SathusSathus Technology

# Generated by Sathus Distributed Systems Calculator spark.executor.instances: 16 spark.executor.cores: 4 spark.executor.memory: 10g spark.executor.memoryOverhead: 3584m spark.sql.shuffle.partitions: 2000 spark.sql.adaptive.enabled: true spark.sql.adaptive.skewJoin.enabled: true spark.sql.adaptive.coalescePartitions.enabled: true spark.memory.fraction: 0.60 spark.memory.storageFraction: 0.50

How the Sathus Memory Sizing Formulas Work

1. Core Capping (4-5 Cores)

Assigning more than 5 cores to an executor leads to severe JVM stop-the-world garbage collection pauses. Keeping executor cores at 4 maximizes multi-threading while keeping GC pauses under 200ms.