Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Strategies
- Trade-offs between Hosted APIs and self-hosted solutions
- Patterns for on-premises, edge, and air-gapped environments
- Single model vs. model portfolio approaches
- Cost analysis per decision
Hardware Sizing and Capacity Planning
- Memory footprint relative to parameter count
- CPU-only vs. GPU acceleration
- Planning for concurrency and throughput
- Reserving headroom for increasing task complexity
Serving Stacks
- Options including vLLM and SGLang
- llama.cpp and GGUF execution
- Ollama for local development
- Selecting stacks based on decision workload characteristics
Quantization While Preserving Calibration
- Precision formats and associated memory savings
- Measuring quality and calibration degradation
- Per-tensor and importance-aware quantization variants
- Acceptance testing post-quantization
Latency Engineering
- Median vs. tail latency
- Latency impact of context length
- Warm starts and model caching
- Batching multiple decision queries
Edge and Constrained Environments
- Running small decision models on CPUs
- Apple Silicon and embedded system deployment
- Offline operation and update strategies
- Distributing weights in air-gapped settings
Containerization and Scaling
- Designing images for model serving
- Managing volumes and caches
- Horizontal scaling and traffic routing
- Resource limits and GPU sharing
Observability
- Tracking request, latency, and error metrics
- Monitoring confidence distributions
- Logging decision outcomes and escalations
- Alerting on calibration drift
Versioning and Rollouts
- Version control for models and adapters
- Shadow and canary deployment strategies
- Pinning model files for audit purposes
- Rollback procedures
Security and Multi-Tenancy
- Authentication and rate limiting
- Isolation of tenant prompts and logs
- Network isolation and egress control
- Audit logging of decision processes
Requirements
- Experience with Linux administration and containers.
- Familiarity with model inference servers and GPU management.
- Foundational understanding of REST APIs, load balancing, and monitoring.
Audience
- MLOps and platform engineers managing inference infrastructure
- SRE teams providing support for AI services
- AI engineers deploying decision models on-premises
- Architects designing air-gapped or regulated AI deployments
