Get in Touch
 Duration 35 hours

Course Outline

Foundations of the Databricks Platform and Lakehouse

  • Core concepts of Databricks Lakehouse architecture and its components
  • Strategies for organizing workspaces and catalogs

Databricks Workspace and Notebook Development

  • Navigating the workspace and utilizing notebooks for development
  • Designing code structures within reusable notebooks

Apache Spark Architecture and Execution Models

  • Understanding Spark’s runtime architecture and execution mechanics
  • Concepts of lazy evaluation and Directed Acyclic Graphs (DAG)

PySpark DataFrames and the DataFrame API

  • Exploring DataFrame abstractions and schema definitions
  • Executing core DataFrame operations and column expressions

Converting SQL to PySpark DataFrames

  • Mapping core SQL clauses to equivalent DataFrame operations
  • Implementing window functions and aggregations in PySpark

Data Ingestion and Ejection in Databricks

  • Reading data from various file formats and database sources
  • Writing and partitioning data within the Lakehouse

Delta Lake and Table Administration

  • Managing Delta tables and ensuring ACID transactions
  • Utilizing time travel features and schema evolution

Patterns for Data Cleaning and Transformation

  • Techniques for data cleaning and type conversion
  • Creating reusable transformation logic

User-Defined Functions and Modular Coding

  • Implementing Python UDFs and pandas UDFs
  • Refactoring procedural logic into modular functions

Performance Optimization and Tuning

  • Applying partitioning and caching strategies
  • Identifying bottlenecks using the Spark UI

Basics of Structured Streaming

  • Differentiating between batch and streaming processing models
  • Handling Streaming DataFrames and performing basic aggregations

Databricks Jobs and Workflow Orchestration

  • Scheduling notebooks as individual jobs and tasks
  • Constructing multi-step workflows with defined dependencies

Unity Catalog and Data Governance

  • Understanding Unity Catalog architecture and namespace management
  • Enforcing access control and tracking data lineage

Testing, Debugging, and Production Standards

  • Unit testing logic written in PySpark
  • Debugging techniques and maintaining code quality

Comprehensive Financial Services Applications

  • Developing a complete banking ETL pipeline
  • Converting legacy SQL processes into PySpark implementations

Migrating SQL Workloads to PySpark

  • Defining migration strategies and planning patterns
  • Executing incremental conversion of SQL workflows to PySpark

Requirements

  • Familiarity with Python programming, including functions and data types
  • Knowledge of SQL, covering joins, aggregations, and subqueries
  • No previous experience with Databricks or PySpark is necessary

Target Audience

  • Data engineers, data analysts, and other data professionals
  • Teams in the process of migrating existing SQL-based workflows to Databricks and PySpark

Number of participants


Price per participant (excl. VAT)

Testimonials (2)

Upcoming Courses

Related Categories