learning kubernetes with a cluster of raspberry pi 4's

I picked up an RPI4 right when it was released, as gigabit ethernet meant I could run pihole at school for cheap, without [slightly] throttling the internet for my roommates. It was indeed perfect for this use case, but I felt that I really wasn't utilizing it's potential. There it sat, under my desk, for months. Until now! I invested more money into something that I already wasn't utilizing to potential (smart, right?) and bought a whole bunch more stuff: 3 more RPI's, PoE hats for all, a PoE switch, and a case that could fit all four.

The goal: get a kubernetes cluster up and running, and install the dashboard. (Can you tell I'm a kubernetes beginner?).
cluster from the sidecluster from the top

The case I purchased didn't quite fit the PoE hats, so it's a little extra jampacked, but I avoided shorting anything, so hopefully we're okay.

OS Setup

I used the latest Raspian Lite, here was my process:
  1. Find the SD card with fdisk (it changed with different micro sd's).
  2. Burn image with dd, specifically: dd bs=4M if=2020-02-13-raspbian-buster-lite.img of=/dev/sdX conv=fsync.
  3. Mount /dev/sdX1 and enable SSH with cd /path/to/mnt/location && touch ssh.
  4. Mount /dev/sdX2 and enable static ip by editing the bottom of the /path/to/mnt/etc/dhcpcd.conf, and uncommenting the following lines at the bottom (mine are edited):
# Example static IP configuration:

interface eth0 static ip_address= static routers= static domain_name_servers=

Rinse and repeat this process 4 times, I chose, ...21, ...22, and ...23. I also reserved those IPs on my router, so they wouldn't get DHCP'd to another device if any of my Pi's go offline. After they're all up, I sudo raspi-config on each to set the hostnames (kmaster, kworker1, kworker2, kworker3) and set the timezones (unclear if this matters to me).

Installing k3s

The cluster is all up and running, so now I can install k3s, first on master ( After SSHing onto it:
-> curl -sfL https://get.k3s.io | sh - foo... bar... installing... -> sudo cat /var/lib/rancher/k3s/server/node-token K104c102328fkj9kw9a01a0422e5fe7ec320b0274f3e35a721cac588b8472e5d1dd::server:19238307020bdb828665190548c71c81

Now, to setup the other pi's as workers, you need to set the token, and URL of the master server. So, export the variables K3S_URL and K3S_TOKEN and install like before.
ssh pi@IP "export K3S_URL="" && export K3S_TOKEN="K104c102328fkj9kw9a01a0422e5fe7ec320b0274f3e35a721cac588b8472e5d1dd::server:19238307020bdb828665190548c71c81" && curl -sfL https://get.k3s.io | sh -"

That installs and starts the k3s-agent service, and connects them to master. With that, we should have a up and running k3s cluster, which can be controlled by kubectl from the master node.

-> sudo kubectl get nodes NAME STATUS ROLES AGE VERSION kworker2 Ready <none> 47h v1.17.4+k3s1 kworker3 Ready <none> 47h v1.17.4+k3s1 kworker1 Ready <none> 47h v1.17.4+k3s1 kmaster Ready master 47h v1.17.4+k3s1

Kubernetes dashboard

It's as simple as running this command: kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml

Then, following this post, you can grab the token like so:
sudo kubectl -n kubernetes-dashboard describe secret $(sudo kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}') | awk '/token:/{print $2}'

To actually connect, you need to use kubectl proxy, and if you run it from your master node, you need to proxy with SSH to view the page from your machine. ssh -L 8001:localhost:8001 pi@IP. Now you can navigate to the dashboard at http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/.

Note: this breaks on restart. The problem is something to do with iptables, seemingly related to docker, and can be fixed by flushing the iptables, I have a script for this that I run on restart, if broken.
#!/bin/sh sudo pkill kubectl sudo systemctl stop k3s sudo systemctl stop docker sudo iptables --flush sudo iptables -tnat --flush sudo systemctl start k3s sudo systemctl start docker

Future plans

I will rewrite the webstack for my site with microservices, so I can learn how to write microservices. I will hopefully get to play with OpenFaaS in that process. I would also like to get a pihole up and running again, which should be easy enough.


Alex Ellis' excellent guide helped on the installation of k3s and Jeff Geerling's pidramble inspired the physical build.