Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Assessing cloud LLM risks: data retention policies, potential training on user inputs, and foreign jurisdiction constraints.
- Exploring Ollama's architecture: the model server, registry system, and OpenAI-compatible API.
- Evaluating Ollama against alternative tools like vLLM, llama.cpp, and Text Generation Inference.
- Navigating model licensing terms for Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Installing Ollama on Linux with support for CUDA and ROCm ecosystems.
- Implementing CPU-only fallback strategies and leveraging AVX/AVX2 optimizations.
- Deploying via Docker and managing persistent volume mappings.
- Configuring multi-GPU environments and defining VRAM allocation strategies.
Model Management
- Retrieving models from the Ollama registry, e.g., using 'ollama pull llama3'.
- Importing GGUF models from HuggingFace and TheBloke.
- Understanding quantization levels: analyzing tradeoffs between Q4_K_M, Q5_K_M, and Q8_0.
- Managing model switching and understanding concurrent model loading limits.
Custom Modelfiles
- Mastering Modelfile syntax: utilizing FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Optimizing generation parameters such as temperature, top_p, and repeat_penalty.
- Engineering system prompts to define role-specific behaviors.
- Creating custom models and publishing them to a local registry.
API Integration
- Utilizing the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enforcing JSON output modes.
- Integrating with frameworks like LangChain, LlamaIndex, and custom applications.
- Implementing authentication and rate limiting via reverse proxies.
Performance Optimization
- Managing context window sizing and KV cache efficiency.
- Optimizing batch inference and parallel request processing.
- Allocating CPU threads with NUMA awareness for better performance.
- Monitoring GPU utilization and managing memory pressure.
Security and Compliance
- Isolating network access for model serving endpoints.
- Implementing input filtering and output moderation pipelines.
- Maintaining audit logs for all prompts and completions.
- Verifying model provenance and integrity through hash validation.
Requirements
- Intermediate proficiency in Linux and container administration.
- A high-level understanding of machine learning and transformer architectures.
- Working familiarity with REST APIs and JSON structures.
Target Audience
- AI engineers and developers looking to migrate away from cloud LLM APIs.
- Organizations with strict data sensitivity policies that preclude the use of cloud models.
- Government and defense teams requiring air-gapped language model solutions.
