Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Fundamentals of LLMOps
- LLMOps vs MLOps: distinct operational challenges of LLMs
- The LLM application lifecycle: from prompting to evaluation, deployment, and monitoring
- Essential checklist for achieving production readiness in GenAI applications
Prompt Management and Version Control
- Prompt templating systems and dynamic variable injection
- Semantic versioning for prompts coupled with automated regression testing
- Prompt registries and streamlined collaboration workflows
Scalable LLM Evaluation
- Key evaluation dimensions: accuracy, relevance, safety, and groundedness
- Utilizing LLMs as judges and designing human evaluation pipelines
- Leveraging automated eval frameworks such as RAGAS, DeepEval, and custom evaluators
- Integrating quality gates into CI/CD pipelines for LLM releases
Safety Guardrails and Content Governance
- Implementing input and output guardrails using NeMo Guardrails and Guardrails AI
- PII detection, toxicity filtering, and defining topic boundaries
- Defending against jailbreaks and prompt injection attacks
- Conducting red-teaming exercises to ensure LLM safety
LLM Observability and Monitoring
- Telemetry collection: tracking token usage, latency, costs, and quality indicators
- Detecting drift in LLM outputs and embedding spaces
- Session-level tracing for complex multi-turn agent interactions
- Building dashboards and alerting systems with LangSmith, Arize, and OpenTelemetry
AI Gateway and Model Orchestration
- Multi-provider routing using tools like LiteLLM and Portkey
- Designing fallback strategies, retry logic, and circuit breakers
- Cost-effective model selection and intelligent load balancing
- Managing rate limits, quotas, and API key governance
Performance Optimization
- Implementing semantic caching via vector stores and exact-match strategies
- Enforcing structured outputs through constrained decoding
- Optimizing batching, streaming, and concurrency patterns
- Reducing latency across diverse model providers
Governance, Compliance, and Auditability
- Maintaining LLM audit trails: logging prompts, responses, and decision provenance
- Addressing data residency and privacy requirements for LLM APIs
- Implementing policy-as-code for internal LLM usage regulations
- Developing a comprehensive internal LLM operations playbook
Requirements
- Practical experience in developing or integrating LLM-powered solutions.
- Proficiency with Python and REST APIs.
- Fundamental knowledge of prompt engineering principles.
Target Audience
- ML engineers and MLOps specialists expanding their expertise into LLM operations.
- Platform engineers tasked with managing LLM infrastructure.
- Technical leaders overseeing production GenAI deployments.
Testimonials (2)
use of proper and effective prompt
Marses Pacaldo
Course - Generative AI and Prompt Engineering for Corporate Professionals
The interactive style, the exercises
