PySpark and Machine Learning Training Course

PySpark & Machine Learning 

Module 1: Big Data & Spark Foundations

• An overview of the big data landscape and Spark’s role in modern data platforms
• Exploring Spark architecture, including drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs and selecting the appropriate approach for different use cases
• Setting up and configuring SparkSession, along with an understanding of application configuration basics

Module 2: PySpark DataFrames

• Ingesting and exporting data from various enterprise sources and formats (CSV, JSON, Parquet, Delta)
• Mastering PySpark DataFrames through transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced features like window functions, timestamp management, and nested data handling
• Incorporating data quality checks and writing clean, reusable, and maintainable PySpark code

Module 3: Processing Large Datasets Efficiently

• Grasping performance essentials, including partitioning strategies, shuffle mechanics, caching, and persistence
• Utilising optimisation techniques such as broadcast joins and analysing execution plans
• Processing large-scale data efficiently while adhering to best practices for scalable workflows
• Understanding schema evolution and the role of modern storage formats in enterprise contexts

Module 4: Feature Engineering at Scale

• Executing feature engineering via Spark MLlib, covering missing value imputation, categorical encoding, and feature scaling
• Creating reusable preprocessing steps and preparing datasets for integration into machine learning pipelines
• Introduction to feature selection techniques and strategies for managing imbalanced datasets

Module 5: Machine Learning with Spark MLlib

• Understanding the MLlib architecture and the Estimator/Transformer design pattern
• Training large-scale regression and classification models, including Linear Regression, Logistic Regression, Decision Trees, and Random Forests
• Comparing model performance and interpreting outcomes within distributed machine learning contexts

Module 6: End-to-End ML Pipelines

• Constructing comprehensive machine learning pipelines that integrate preprocessing, feature engineering, and modelling
• Implementing train/validation/test splitting strategies
• Conducting cross-validation and hyperparameter tuning using grid search and random search methods
• Structuring machine learning experiments to ensure reproducibility

Module 7: Model Evaluation & Practical ML Decision Making

• Selecting and applying appropriate evaluation metrics for regression and classification tasks
• Identifying signs of overfitting and underfitting to make informed model selection decisions
• Interpreting feature importance to understand model behaviour and insights

Module 8: Production & Enterprise Practices

• Saving and loading models within the Spark environment
• Implementing batch inference workflows suitable for large datasets
• Understanding the machine learning lifecycle as it applies to enterprise operations
• Introduction to model versioning, experiment tracking, and basic testing strategies

Practical Outcome

• Competence to work independently with PySpark
• Skill in processing large datasets with efficiency
• Ability to perform feature engineering at scale
• Capacity to build scalable machine learning pipelines