Course Outline
Foundations of the Databricks Platform and Lakehouse
- Core concepts of Databricks Lakehouse architecture and its components
- Strategies for organizing workspaces and catalogs
Databricks Workspace and Notebook Development
- Navigating the workspace and utilizing notebooks for development
- Designing code structures within reusable notebooks
Apache Spark Architecture and Execution Models
- Understanding Spark’s runtime architecture and execution mechanics
- Concepts of lazy evaluation and Directed Acyclic Graphs (DAG)
PySpark DataFrames and the DataFrame API
- Exploring DataFrame abstractions and schema definitions
- Executing core DataFrame operations and column expressions
Converting SQL to PySpark DataFrames
- Mapping core SQL clauses to equivalent DataFrame operations
- Implementing window functions and aggregations in PySpark
Data Ingestion and Ejection in Databricks
- Reading data from various file formats and database sources
- Writing and partitioning data within the Lakehouse
Delta Lake and Table Administration
- Managing Delta tables and ensuring ACID transactions
- Utilizing time travel features and schema evolution
Patterns for Data Cleaning and Transformation
- Techniques for data cleaning and type conversion
- Creating reusable transformation logic
User-Defined Functions and Modular Coding
- Implementing Python UDFs and pandas UDFs
- Refactoring procedural logic into modular functions
Performance Optimization and Tuning
- Applying partitioning and caching strategies
- Identifying bottlenecks using the Spark UI
Basics of Structured Streaming
- Differentiating between batch and streaming processing models
- Handling Streaming DataFrames and performing basic aggregations
Databricks Jobs and Workflow Orchestration
- Scheduling notebooks as individual jobs and tasks
- Constructing multi-step workflows with defined dependencies
Unity Catalog and Data Governance
- Understanding Unity Catalog architecture and namespace management
- Enforcing access control and tracking data lineage
Testing, Debugging, and Production Standards
- Unit testing logic written in PySpark
- Debugging techniques and maintaining code quality
Comprehensive Financial Services Applications
- Developing a complete banking ETL pipeline
- Converting legacy SQL processes into PySpark implementations
Migrating SQL Workloads to PySpark
- Defining migration strategies and planning patterns
- Executing incremental conversion of SQL workflows to PySpark
Requirements
- Familiarity with Python programming, including functions and data types
- Knowledge of SQL, covering joins, aggregations, and subqueries
- No previous experience with Databricks or PySpark is necessary
Target Audience
- Data engineers, data analysts, and other data professionals
- Teams in the process of migrating existing SQL-based workflows to Databricks and PySpark
Testimonials (2)
Showing different ways to optimize our PySpark queries.
Ivan Abrashev - UBB + DZI (KBC GROUP)
Course - Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.