Raspberry Pi Incus Cluster
Author: -fab- <fab@redterminal.org>
License: CC BY-SA 4.0
Published on: Fri, 01 Mar 2024 00:06:54 +0100
Last updated: Fri, 01 Mar 2024 00:06:54 +0100
I have build up an Incus cluster with 4 Raspberry Pi 4 (8GB) in a nice metal case. I build it mostly for tinkering with it. Let's see if it gets a serious purpose. For my NAS, SyncThing and other servers I have a Lenovo ThinkCentre M900 Tiny I7-6700T Proxmox host. I had 3 spare RPi4s and as the prices dropped with the arrival of the Pi5 I bought another RPi4 to complete my build.
4x Raspberry Pi Cluster in metal case (jpg)
The RPis are separately screwed in their slots together with their corresponding SSDs. Below the slots is a space for my 5 port managed switch to connect all the Pis with the router (or other switch).
Incus Manager
I chose Raspberry Pi OS with Incus as a system container and virtual machine manager contrary to LXD which is now owned by Canonical and I don't like to be bound to corporate interests. I don't want to need to buy a "Pro" version in future or what Canonical comes up with. LXD and Incus will diverge for sure but I think Incus is my best bet.
Incus on Linuxcontainers.org
The host systems are on VLAN50 and the containers are on VLAN55 (macvlan). With macvlans it's easy to add additional VLANs for special purposes.
Cluster List
Here is a list of the four Incus cluster hosts and they're fully operational.
$ incus cluster list
+-------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
+-------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| rpi01 | https://192.168.50.10:8443 | database-standby | aarch64 | default | | ONLINE | Fully operational |
+-------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| rpi02 | https://192.168.50.11:8443 | database-leader | aarch64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+-------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| rpi03 | https://192.168.50.12:8443 | database | aarch64 | default | | ONLINE | Fully operational |
+-------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| rpi04 | https://192.168.50.13:8443 | database | aarch64 | default | | ONLINE | Fully operational |
+-------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
Container List
For now I only have 2 containers running. They can be moved around the different hosts when stopped. Online moving doesn't work with containers reliably, so they must be stopped first and then after the move started again.
$ incus list
+-----------------+---------+-----------------------+------+-----------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+-----------------+---------+-----------------------+------+-----------+-----------+----------+
| ruling-sheep | RUNNING | 192.168.55.104 (eth0) | | CONTAINER | 0 | rpi02 |
+-----------------+---------+-----------------------+------+-----------+-----------+----------+
| ubuntu-template | RUNNING | 192.168.55.103 (eth0) | | CONTAINER | 0 | rpi01 |
+-----------------+---------+-----------------------+------+-----------+-----------+----------+
Remote controlling the cluster from my laptops/desktop
It's easy to set up Incus in a way that you can control the cluster as a "remote". With this feature you can launch/move/start/stop containers/VMs or evacuate a cluster member. You can control the cluster as if you were logged into one of it's hosts. All functions are provided. Very nice.
A lot of supported images
There are really a lot of downloadable and ready-to-use container and VM images for incus and arm64. From AlmaLinux to VoidLinux. You can get a complete list for the supported platform with the following command:
$ incus image list images: arm64|less
My own private Cloud
Although it's a toy project for now I have my own arm64 based cloud. I think I'll find some proper usage for it in the future. If you have any cool ideas write me an email. Any advice, comment or (constructive) criticism is appreciated.
All in all - Have fun!
-fab-
--
Back to index
Homepage