Self-Hosting Decision Models: Serving, Quantization, Latency, and Operations Schulung

Deployment Strategies

• Trade-offs between Hosted APIs and self-hosted solutions
• Patterns for on-premises, edge, and air-gapped environments
• Single model vs. model portfolio approaches
• Cost analysis per decision

Hardware Sizing and Capacity Planning

• Memory footprint relative to parameter count
• CPU-only vs. GPU acceleration
• Planning for concurrency and throughput
• Reserving headroom for increasing task complexity

Serving Stacks

• Options including vLLM and SGLang
• llama.cpp and GGUF execution
• Ollama for local development
• Selecting stacks based on decision workload characteristics

Quantization While Preserving Calibration

• Precision formats and associated memory savings
• Measuring quality and calibration degradation
• Per-tensor and importance-aware quantization variants
• Acceptance testing post-quantization

Latency Engineering

• Median vs. tail latency
• Latency impact of context length
• Warm starts and model caching
• Batching multiple decision queries

Edge and Constrained Environments

• Running small decision models on CPUs
• Apple Silicon and embedded system deployment
• Offline operation and update strategies
• Distributing weights in air-gapped settings

Containerization and Scaling

• Designing images for model serving
• Managing volumes and caches
• Horizontal scaling and traffic routing
• Resource limits and GPU sharing

Observability

• Tracking request, latency, and error metrics
• Monitoring confidence distributions
• Logging decision outcomes and escalations
• Alerting on calibration drift

Versioning and Rollouts

• Version control for models and adapters
• Shadow and canary deployment strategies
• Pinning model files for audit purposes
• Rollback procedures

Security and Multi-Tenancy

• Authentication and rate limiting
• Isolation of tenant prompts and logs
• Network isolation and egress control
• Audit logging of decision processes