Sathus Technical Domain Hub

Big Data & Distributed Systems

High-Throughput Spark Clusters, Flink Stream Processing & Memory Optimization

Distributed computing architecture, Apache Spark and PySpark performance tuning, shuffle optimization, Apache Flink stateful streams, and Kafka cluster topology.

Explore Big Data Platform Engineering Book Architecture Strategy Session

Core Technical Subclusters

Structured specialization tracks under the Big Data & Distributed Systems umbrella.

Apache Spark & PySpark

16+ specs

Driver/executor memory tuning, broadcast joins, and vectorized UDFs.

Real-Time Streaming Systems

11+ specs

Flink event time, RocksDB state backend, and Kafka consumer lag auto-scaling.

Distributed Storage & Serialization

8+ specs

Parquet, ORC, Avro, and object storage network throughput tuning.

Authoritative Guides, Benchmarks & Blueprints

Deep production guides authored by Sathus practice leads.

Implementation Code

16 min read

PySpark OOM Error: Driver vs Executor Memory Triage & Production Remediation

The definitive engineering triage guide to diagnosing and fixing Apache Spark OutOfMemory (OOM) crashes. Demystifying driver heap exhaustion from collect() vs executor container kills (Exit code 137), broadcast hash join thresholds, shuffle skew, Adaptive Query Execution (AQE), and off-heap memory overhead.

Key Highlight: Driver OOMs are client-side aggregation bottlenecks caused by .collect(), .toPandas(), or oversized broadcast hash tables exceeding the driver JVM.

By Elena RostovaRead Guide

Architecture Guide

15 min read

Apache Spark Memory Tuning: Storage vs Execution Memory & Off-Heap Sizing

Complete guide to demystifying the Spark Memory Pool. How unified memory management allocates between storage and execution, how to tune off-heap memory, and how to eliminate disk spills.

Key Highlight: Spark unified memory (spark.memory.fraction, default 0.60) is dynamically shared between execution (shuffles/joins) and storage (caching/broadcasts).

By Elena RostovaRead Guide

Implementation Code

16 min read

Apache Spark Join Optimization: Broadcast Hash vs Sort-Merge vs Shuffle Hash Joins

The definitive engineering guide to distributed join optimization in Apache Spark 3.5. Learn how physical join selection works under the Catalyst optimizer, how to eliminate disk spills and OOMs, and how to remediate severe data skew using Adaptive Query Execution (AQE) and key salting.

Key Highlight: Broadcast Hash Join (BHJ) completely eliminates shuffle write and network transfer, optimal when one side fits in executor memory.

By Elena RostovaRead Guide

SathusSathus Technology

Big Data & Distributed Systems

High-Throughput Spark Clusters, Flink Stream Processing & Memory Optimization

Distributed computing architecture, Apache Spark and PySpark performance tuning, shuffle optimization, Apache Flink stateful streams, and Kafka cluster topology.