Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 21 hours
Course Outline
Infrastructure as Code for EXO
- Exploration of EXO deployment architectures, including single-node, multi-node, and RDMA-based clusters
- Automating dependency setup (Xcode, uv, Node.js, Rust) through configuration management tools
- Leveraging Nix flakes to ensure reproducible EXO builds and consistent developer environments
- Creating Ansible playbooks or shell scripts to enable unattended cluster provisioning
Reproducible Builds and CI Integration
- Pinning dependencies and compiling the dashboard within CI pipelines
- Executing EXO smoke tests on GitHub Actions or GitLab CI runners
- Developing golden images and snapshot-based rollback workflows for macOS and Linux virtual machines
- Versioning custom model cards in parallel with application code
Automating Cluster Discovery and Networking
- Setting up mDNS and static DNS records to ensure reliable libp2p node discovery
- Streamlining network profile creation and Thunderbolt bridge management on macOS systems
- Utilizing custom namespaces (EXO_LIBP2P_NAMESPACE) to isolate dev, staging, and prod clusters
- Establishing firewall rules and network segmentation for multi-tenant security
Storage and Model Lifecycle Management
- Architecting strategies for EXO_MODELS_DIRS and EXO_MODELS_READ_ONLY_DIRS
- Mounting NFS or SAN shares as read-only repositories to accelerate model provisioning
- Implementing garbage collection for obsolete caches and defining retention policies for versioned weights
- Automating pre-downloads of models and health checks prior to rolling updates
Monitoring and Alerting
- Routing EXO logs to centralized platforms such as ELK, Loki, or Splunk
- Constructing Grafana dashboards using data from EXO_TRACING_ENABLED output
- Triggering alerts for cluster membership changes, OOM events, and inference latency spikes
- Analyzing macmon hardware telemetry to identify model performance regressions
Updates, Rollbacks, and Disaster Recovery
- Piloting EXO binary updates on canary nodes before fleet-wide deployment
- Executing model-level rollbacks by switching quantized versions without re-downloading assets
- Backing up and restoring cluster state, custom namespaces, and cached weights
- Documenting runbooks for recovery scenarios involving total cluster rebuilds
Security Hardening and Compliance
- Enforcing TLS at the reverse proxy layer (nginx, traefik) for the dashboard and API
- Applying API rate limiting and IP whitelisting to secure EXO endpoints
- Isolating clusters using VLANs and zero-trust network policies
- Auditing access logs and maintaining a comprehensive inventory of deployed models and versions
Requirements
- Proficiency in DevOps practices, including CI/CD, IaC, and container orchestration
- Working knowledge of macOS or Linux system administration and package management
- Foundational understanding of networking, DNS, and storage concepts
Target Audience
- DevOps Engineers
- Infrastructure Architects
- SREs managing on-premise AI workloads
Testimonials (3)
Craig was extremely involved in the training, always making sure we are paying attention, adapted the examples to our day-to-day activities and always provided an answer when asked, even if the information was not added in the presentation.
Ecaterina Ioana Nicoale - BOOKING HOLDINGS ROMANIA SRL
Course - DevOps Foundation®
High level of commitment and knowledge of the trainer
Jacek - Softsystem
Course - DevOps Engineering Foundation (DOEF)®
real life examples
