EXO: End-to-End Local AI Cluster Deployment Training Course

Introduction to EXO and Local AI Clustering

• Overview of the EXO framework and the exo-explore ecosystem.
• Comparative analysis of centralized cloud inference versus distributed local inference.
• Architectural components: libp2p device discovery, MLX backend, dashboard, and API layers.
• Hardware prerequisites: Apple Silicon (M3 Ultra, M4 Pro/Max), Thunderbolt 5, and shared storage.

Installing EXO on macOS

• Configuring Xcode, Metal ToolChain, and necessary macOS prerequisites.
• Installing uv, Node.js, and the Rust nightly toolchain.
• Deploying the pinned macmon fork for Apple Silicon monitoring.
• Cloning the repository and building the dashboard using npm.
• Executing EXO from source and confirming the localhost:52415 dashboard.

Installing EXO on Linux

• Installing dependencies via apt or Homebrew on Linux systems.
• Setting up uv, Node.js 18+, and Rust nightly.
• Building the dashboard and running EXO in CPU-only mode.
• Understanding the directory layout, specifically XDG Base Directory paths for config, data, cache, and logs.

Automatic Device Discovery and Cluster Formation

• Comprehending libp2p-based auto-discovery across local networks.
• Setting up custom namespaces using EXO_LIBP2P_NAMESPACE for cluster isolation.
• Confirming node membership via the dashboard cluster view.
• Addressing discovery failures and network segmentation issues.

Enabling RDMA over Thunderbolt 5

• RDMA architecture details and the claim of 99 percent latency reduction.
• Activating RDMA in macOS Recovery mode using rdma_ctl.
• Reviewing cable requirements and port topology constraints on Mac Studio units.
• Ensuring consistent macOS versions across all cluster nodes.
• Resolving RDMA discovery and DHCP configuration issues.

Deploying Frontier Models

• Utilizing the dashboard to load and shard DeepSeek v3.1, Qwen3-235B, and Llama family models.
• Previewing instance placements through the /instance/previews API endpoint.
• Creating model instances utilizing pipeline or tensor-parallel sharding.
• Configuring custom model cards from the HuggingFace hub.

Monitoring and Troubleshooting

• Analyzing EXO logs and understanding distributed tracing mechanisms.
• Interpreting cluster health metrics in the dashboard cluster view.
• Diagnosing worker node failures and observing reconnection behavior.
• Leveraging EXO_TRACING_ENABLED for detailed performance bottleneck analysis.

Cluster Maintenance and Updates

• Procedures for updating EXO binaries and rebuilding the dashboard.
• Managing model caches and pre-downloaded models over NFS.
• Gracefully removing nodes and rebalancing workloads.