Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Foundations

• Introduction to the Big Data ecosystem and Spark’s role in contemporary data platforms
• Examining Spark architecture: driver, executors, cluster manager, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining when to apply each approach
• Creating and configuring SparkSession while grasping the fundamentals of application configuration

Module 2: PySpark DataFrames

• Ingesting and exporting data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
• Manipulating PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and nested data management
• Enforcing data quality checks and crafting reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Fundamentals for Spark Users

• Conceptual overview of how Docker images and Kubernetes pods map to Spark driver and executor processes—focused on cluster consumers rather than administrators
• Exploring the mechanics of Spark jobs on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings
• Delineating the control boundaries between Spark session parameters and platform team cluster provisioning—and why this distinction is critical for tuning
• Hands-on exercise: Examine a live Spark-on-Kubernetes job in the sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — Deep Dive

• Analysing executor count versus executor size: memory-versus-cores trade-offs and reasoning frameworks
• Managing driver and executor memory allocation, overhead, and their translation into pod resources
• Dynamic allocation: Behaviour on Kubernetes, scenarios where it conserves resources, and those where it does not
• Hands-on exercise: Execute identical jobs under varying executor and core configurations to compare runtime and resource usage

Module 5: Scaling Behaviour & Cost Optimisation

• Evaluating how adding or removing nodes impacts job behaviour, completion time, and resource consumption
• Comparing the performance and cost trade-offs of many small executors versus fewer large ones
• Optimising shuffle behaviour and partition tuning, including estimating the cost impact of configuration choices
• Hands-on exercise: Scale the sandbox cluster from five to ten nodes to observe effects on job completion time and resource usage

Module 6: Efficient Data Ingestion & Partitioning

• Addressing the small-files problem: Why sources split across many 1–5 MB Parquet files degrade performance and distort partitioning
• Strategies for repartitioning and coalescing data
• Controlling partition sizes during read and write operations
• Writing Parquet efficiently to prevent downstream small-file proliferation
• Hands-on exercise: Load a dataset consisting of numerous small Parquet partitions, apply varied repartitioning strategies, and benchmark performance before and after optimisation

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying root causes of out-of-memory errors in Pandas and recognising associated symptoms
• Applying memory-efficient conversion patterns between Spark and Pandas
• Preventing memory blow-ups when exporting large datasets to CSV
• Leveraging chunked processing and dtype optimisation in resource-constrained environments
• Hands-on exercise: Replicate a typical Pandas out-of-memory scenario and resolve it using chunking and dtype optimisation

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: performance characteristics, lazy evaluation, and memory behaviour
• Understanding where Polars complements PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Hands-on exercise: Refactor a Pandas-centric transformation in Polars to compare memory usage and processing velocity

Module 9: Applying Optimisation to ETL & ML Workloads

• Implementing configuration, partitioning, and memory-management principles across a realistic ETL pipeline
• Considering optimisation factors specific to machine-learning workloads running on the same cluster
• Applying a practical tuning workflow to systematically diagnose cost and performance issues
• Hands-on exercise: Complete an end-to-end mini-project integrating data loading, transformation, and a simple model-training step, with participants tuning the Spark configuration independently