Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Fundamentals

• Overview of the Big Data landscape and Spark's role in modern data platforms
• Exploring Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs, including when to use each approach
• Creating and configuring SparkSession and understanding the fundamentals of application settings

Module 2: PySpark DataFrames

• Ingesting and exporting data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
• Mastering PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and working with nested data structures
• Applying data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Essentials for Spark Users

• Understanding the relationship between Docker images, Kubernetes pods, and Spark driver/executor processes — a conceptual view for cluster consumers rather than administrators
• Insight into Spark job execution on Kubernetes: pod scheduling, resource requests/limits, and mapping session settings to these resources
• Distinguishing between what you control via Spark session parameters and what is managed by the platform team via cluster provisioning — and why this boundary is critical for tuning
• Hands-on exercise: Inspect a running Spark-on-Kubernetes job in a sandbox cluster to identify driver and executor pods

Module 4: Deep Dive into Spark Session Configuration on Kubernetes

• Analyzing executor count versus executor size: memory-versus-cores trade-offs and strategic reasoning
• Driver and executor memory allocation, overhead, and their translation into pod resources
• Dynamic allocation: behavior on Kubernetes, scenarios where it saves resources, and those where it does not
• Hands-on exercise: Execute the same job with varying executor and core configurations to compare runtime and resource utilization

Module 5: Scaling Behavior & Cost Optimization

• How adding or removing nodes impacts job behavior, completion time, and resource consumption
• Comparing many small executors versus fewer large ones: analyzing performance and cost trade-offs
• Shuffle behavior and partition tuning, including estimating the cost implications of configuration choices
• Hands-on exercise: Scale the sandbox cluster from five to ten nodes and observe the impact on job completion time and resource usage

Module 6: Efficient Data Ingestion & Partitioning

• The small-files problem: why sources split into many 1–5 MB Parquet files degrade performance and distort partitioning
• Strategies for repartitioning and coalescing data
• Managing partition sizes during both read and write operations
• Writing Parquet efficiently to prevent the recurrence of small-file issues downstream
• Hands-on exercise: Load a dataset comprising numerous small Parquet partitions, apply different repartitioning strategies, and compare pre- and post-optimization performance

Module 7: Pandas Memory Management & Failure Diagnosis

• Root causes of out-of-memory errors in Pandas and how to recognize symptoms
• Implementing memory-efficient conversion patterns between Spark and Pandas
• Avoiding memory spikes when exporting large datasets to CSV
• Utilizing chunked processing and dtype optimization in resource-constrained environments
• Hands-on exercise: Reproduce a typical Pandas out-of-memory scenario and resolve it using chunking and dtype optimization

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: performance characteristics, lazy evaluation, and memory behavior
• Understanding where Polars fits within modern cloud data stacks and migration roadmaps alongside PySpark and Pandas, including AWS environments
• Hands-on exercise: Rewrite a Pandas-heavy transformation in Polars and compare memory usage and processing speed

Module 9: Applying Optimization to ETL & ML Workloads

• Applying configuration, partitioning, and memory management principles across a realistic ETL pipeline
• Optimization considerations specific to machine learning workloads running on the same cluster
• Applying a practical tuning workflow to systematically diagnose cost and performance issues
• Hands-on exercise: Complete an end-to-end mini-project involving data loading, transformation, and a simple model training step, with participants tuning the Spark configuration themselves