Get in Touch
 Duration 35 hours

Course Outline

Introduction to Diagnostic Fundamentals

  • Overview of common LLM system failure modes and Ollama-specific challenges
  • Setting up reproducible experiments and controlled testing environments
  • Essential debugging tools: local logs, request/response captures, and sandboxing

Reproducing and Isolating Issues

  • Methods for generating minimal failing examples and test seeds
  • Managing stateful versus stateless interactions to isolate context-related bugs
  • Controlling nondeterministic behavior through determinism and randomness management

Behavioral Assessment and Metrics

  • Quantitative measures: accuracy, ROUGE/BLEU variants, calibration, and perplexity proxies
  • Qualitative assessments: human-in-the-loop scoring and rubric development
  • Task-specific fidelity checks and defining acceptance criteria

Automated Testing and Regression Control

  • Unit tests for prompts and components, along with scenario and end-to-end testing
  • Building regression suites and establishing golden example baselines
  • Integrating Ollama model updates and automated validation gates into CI/CD

Observability and Monitoring

  • Implementing structured logging, distributed tracing, and correlation IDs
  • Tracking key operational metrics: latency, token consumption, error rates, and quality signals
  • Configuring alerts, dashboards, and SLIs/SLOs for model-driven services

Advanced Root Cause Analysis

  • Tracing complex flows including graphed prompts, tool calls, and multi-turn interactions
  • Conducting comparative A/B diagnostics and ablation studies
  • Investigating data provenance, debugging datasets, and resolving dataset-induced errors

Safety, Robustness, and Remediation Tactics

  • Applying mitigations such as filtering, grounding, retrieval augmentation, and prompt scaffolding
  • Using rollback, canary, and phased rollout patterns for model updates
  • Conducting post-mortems, documenting lessons learned, and implementing continuous improvement cycles

Summary and Future Directions

Requirements

  • Substantial experience in developing and deploying LLM applications
  • Proficiency with Ollama workflows and model hosting environments
  • Working knowledge of Python, Docker, and fundamental observability tools

Target Audience

  • AI engineers
  • ML Ops professionals
  • QA teams overseeing production LLM systems

Number of participants


Price per participant (excl. VAT)

Upcoming Courses

Related Categories