This intensive three-day practical course is dedicated to constructing and refining high-efficiency data-processing workloads by leveraging PySpark, Pandas, and Polars within Kubernetes-based environments.
Learners will gain a robust operational understanding of how Spark applications are executed on Kubernetes. They will explore how strategic application-level configuration decisions directly impact performance, scalability, resource efficiency, and overall cost. The curriculum addresses critical optimisation domains, such as executor sizing, memory allocation, dynamic resource allocation, partitioning strategies, shuffle mechanics, mitigating small-file issues, and maximising Parquet processing efficiency.
The course further tackles common operational challenges encountered with Pandas, such as memory constraints and out-of-memory exceptions, while introducing Polars as a high-performance alternative for specific data-processing tasks. Through practical exercises, participants will learn to diagnose performance and memory bottlenecks, evaluate various configuration strategies, and implement optimisation techniques in realistic ETL and machine learning contexts.
Throughout the training, the focus remains on practical decision-making: mastering the ability to identify performance bottlenecks, select the most suitable tool, configure Spark effectively, and balance performance gains against infrastructure resource consumption and cost.
Read more...
