The Computer Dudes Inc

Let's start with a bit of History and the evolution of High Power Consumption Computers and Data Center as well as our research over the decades.

The Computer Dudes, Inc has been a leader in low power Single Board Computer research and design since the company's founding in 1997. At that time we were using Intel 486 single board computers that were about the size of an ISA Sound or Ethernet Card. The rest of the world was still using huge ISA type motherboards that required a "card" for each function such as Video Card, Sound, Ethernet, Modem, Printer, Hard Drive Controller or others.

As "Moore's Law" progressed Single Board Computers, SBC's, grew smaller, more powerful and required less power.

The advent of the Raspberry Pi series rapidly expanded our work in this area and by the Raspberry Pi 4 we were able to run the equivalent of "full sized computers" off a single RPI 4.

During the evolution of the Raspberry Pi series SBC's we worked on Clustering to increase power and to build servers that were essentially modular so more "nodes" could be added to increase power and performance. We also experimented with using Cloud Architecture to increase the speed, however, we quickly determined that "cloud architecture" was not secure.

In the late 2000's we began developing custom flat and then circular "computer cores" to hold each of the SBC's instead of standard tower or rack mounted server designs. The computer cores too up much less space, were lighter, and allowed for easier maintenance and repair.

In the 2010's we began designing custom computer cases and DC power supplies for our Computer Cores - including "Outdoor" solar and battery powered computers and super computers.

in the early 2020's we designed the worlds first Off Grid, Outdoor, No Building Required Computers - the "Solar One Monoliths." The Solar One Monoliths are made of Granite, Glass and Steel to hold the round 3D printed Computer Cores and were run off custom DC Solar and Battery Power Systems.

By 2022 we had built the worlds first "Monolithic Data Center" for off grid operation with multiple options for internet connectivity. These required no building and the first one is still in place at our headquarters / lab in Tallahassee, Florida, USA

Flash Forward to 2025 - High Power Consumption Data Centers with the "wrong architecture" and unbelievably high power consumption for Large Language Model, LLM, "Generative" AI's arose as a global problem. In our view the power consumption as well as cost in dollars is not sustainable.

To solve that problem we have now used the Raspberry Pi 5 series, in clusters, to create LLM Generative AI's. Below is our first publication on this work along with a "How To" guide. This document is "Open Source" and we have already progressed to the next level.

You can use the information below to build your own - or you can contact The Computer Dudes, Inc. (www.thecomputerdudesinc.com/index.php) to purchase a custom version or the newest version we are working on.

Now we are happy to present to you and the world - How to build a LLM Generative AI using clustered Raspberry Pi 5's and Alpaca 7B

Running Alpaca 7B on a Raspberry Pi 5 Beowulf cluster of 10 Pi 5s.

This example won’t give you true LLM-level performance like an A100 GPU, but it can distribute the workload using parallel processing without power hungry GPU's.

Here is how to setup a basic Generative LLM AI using only 10 clustered Raspberry Pi 5's. Build this version first then use it for testing and to design the next version using hundreds, or thousands, of Raspberry Pi 5 nodes or clusters.

Architecture:
Alpaca 7B (quantized, 4-bit GGUF) on a 10-node Raspberry Pi 5 cluster, with:
A master node (master controller)
9 worker nodes (inference or supporting roles)
Shared filesystem (via NFS or rsync)
Parallelized inference using llama.cpp or llm.cpp + custom sharding/load-balancing logic

Hardware & Network:
10x Raspberry Pi 5 (8GB)
10x MicroSD cards (32GB+)
1x Gigabit switch + Ethernet cables (avoid Wi-Fi bottlenecks)
1x USB SSD for model storage (shared or on each node)
Active cooling on all units (very important)
For the smallest footprint: PoE HATs + PoE switch for power and networking (like a backplane when used in a cluster)

Network Topology:
Assign static IPs to each node.
Node naming convention: pi-master, pi-node01, ..., pi-node09

Software Stack:
Component Description
OS Raspberry Pi OS 64-bit (Lite preferred)
MPI Message Passing Interface: OpenMPI
llama.cpp Inference engine: OpenBLAS
GGUF Model Alpaca 7B quantized (4-bit recommended)
Optional Python + Flask/Gradio for web API

SETUP Procedure:

STEP 1: Flash and Configure Raspberry Pi OS
Flash 64-bit Raspberry Pi OS Lite onto each Pi:
Set hostname: /etc/hostname
Set static IP: /etc/dhcpcd.conf
Enable SSH

Update:
How To:
bash
Copy
Edit
sudo apt update && sudo apt upgrade -y
Repeat for each Pi.

STEP 2: Set Up the Master Node
On pi-master
How To:
Install dependencies
bash
Copy
Edit
sudo apt install git build-essential cmake libopenblas-dev libomp-dev openmpi-bin openmpi-common libopenmpi-dev -y
Create SSH key and distribute it:
bash
Copy
Edit
ssh-keygen -t rsa
for i in {1..9}; do ssh-copy-id pi-node0$i; done

STEP 3: Set Up Shared Filesystem
Option A: NFS Share (on pi-master)
How To:
bash
Copy
Edit
sudo apt install nfs-kernel-server -y
sudo mkdir /mnt/share
sudo chown pi:pi /mnt/share
echo "/mnt/share *(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports
sudo exportfs -a
sudo systemctl restart nfs-kernel-server
Mount on worker nodes (pi-node01 → pi-node09)
bash
Copy
Edit
sudo apt install nfs-common -y
sudo mkdir /mnt/share
sudo mount pi-master:/mnt/share /mnt/share

STEP 4: Build llama.cpp with MPI Support on all nodes
How to:
bash
Copy
Edit
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake -DLLAMA_OPENBLAS=on ..
make -j4
OR for more rapid setup: create an automated deploy script using rsync from pi-master.

STEP 5: Download and Distribute Alpaca 7B GGUF on the master node:
How to:
Download from: https://huggingface.co/TheBloke/alpaca-7B-GGUF
Choose alpaca-7b-q4_0.gguf (4-bit quantized)
Store in /mnt/share/models/ or replicate to all nodes

STEP 6: MPI Configuration File
On pi-master, create hosts.txt:
How To:
text
Copy
Edit
pi-node01 slots=4
pi-node02 slots=4
pi-node03 slots=4
pi-node04 slots=4
pi-node05 slots=4
pi-node06 slots=4
pi-node07 slots=4
pi-node08 slots=4
pi-node09 slots=4
You can also include pi-master if you want it to join inference.

STEP 7: Run Distributed Inference - Cluster and command testing
Using OpenMPI to broadcast simple load across cluster:
How To:
bash
Copy
Edit
mpirun -np 40 --hostfile hosts.txt ./main -m /mnt/share/models/alpaca-7b-q4_0.gguf -n 64 -t 4 -p "Instruction: What is gravity?nnResponse:"
-np 40: Total CPU threads across nodes
-t 4: Threads per node
Please Note: llama.cpp does not support true model sharding yet. You're parallelizing at a token/request level, not at transformer layer level.

STEP 8: Build a Web API Interface for control and configuration. (experiment here)
Install Flask (on pi-master):
How To:
bash
Copy
Edit
sudo apt install python3-pip -y
pip install flask
Create a simple Flask app that:
Accepts prompts
Sends them to available nodes via SSH or socket
Returns the response
You can expand this to include:
Load balancing (round robin)
Node health check
Token-based rate limiting

Performance Issues and Solutions:
Issue                       Workaround
Sharding                  Use batching across requests
RAM pressure         Use 4-bit quantized models
Latency                   Optimize thread count, cooling
IO bottlenecks         Use SSD and avoid SD cards
Python bottlenecks  Use Rust/C++ for inference layer

Expectations:
10 Pi 5s won't give near-GPU speeds but they can handle the load by using multiple cores and parallel processing
You can handle small workloads in parallel.
Best used for offline chatbot, local research, or LLM experimentation and design
Performance: 5–15 tokens/sec depending on load & optimization.

10 Pi Version is for configuring and testing - then add Pi to nodes or additional clustered Pi Nodes to improve performance and capabilities.

Options:
A Python or Bash wrapper that uses SSH to distribute jobs
A Flask/Gradio-based frontend to your cluster
An auto-scaling job scheduler with Pi health monitoring

FINAL NOTE:  If you do not like the Raspberry Pi series SBC's or prefer an Intel Chipset you can do the same thing with just about any other brand of Single Board Computer.