Running Distributed LLMs on SBC Clusters with Exo

In the rapidly evolving field of artificial intelligence, running Large Language Models (LLMs) typically demands substantial computational resources, often limiting accessibility for many enthusiasts and researchers. However, recent advancements have introduced innovative solutions to democratize AI capabilities.

One such development is the Exo software, a distributed LLM solution that enables the deployment of AI models across a cluster of devices, including single-board computers (SBCs) like the Indiedroid Nova and Raspberry Pi 5. This approach allows users to harness the collective power of multiple modest devices to run complex models that would otherwise require high-end hardware.

The Exo software operates by partitioning AI models dynamically based on the available network topology and device resources. This means that even devices without dedicated GPUs can contribute to the processing power needed for LLMs. The system supports various models, including LLaMA, Mistral, LlaVA, Qwen, and DeepSeek, making it versatile for different AI applications. 

For instance, the Indiedroid Nova, equipped with a 6 TOPS Neural Processing Unit (NPU), can be integrated into an Exo cluster to enhance AI processing capabilities. Similarly, the Raspberry Pi 5, known for its balance of performance and affordability, can serve as a node in such clusters, enabling users to experiment with AI models without significant financial investment. 

This distributed approach not only makes AI more accessible but also offers a scalable solution where users can add more devices to their cluster as needed. By leveraging existing hardware, such as smartphones, tablets, and various SBCs, the Exo software fosters a decentralized and cost-effective environment for AI development. 

For a practical demonstration of setting up and utilizing Exo software on a cluster of SBCs, you can refer to the following video:

This resource provides valuable insights into the process and showcases the potential of distributed AI using accessible hardware.