Using RKE2
I used to setup my homelab kubernetes cluster with Rancher and rke, both are products of Rancher. Honestly I haven’t spent too much time on both. Until recently I want to setup kubernetes cluster, again. So I goto Rancher’s website found that they have another product: RKE2.
It’s easy to setup just follow the RKE2 official installation guide. Here is summary of my installtion manually, I opened a tmux session with 4 panes, one for server, 3 for agents. Each has 4 cores, 8GB RAM 160GB storage, they are VMs running on a host of ESXi instance.
Server
# Add proxy only if you have such dependency
curl -sfL https://get.rke2.io | sudo https_proxy=http://192.168.3.1:8889 INSTALL_RKE2_CHANNEL=latest sh -
sudo bash -c 'cat > /etc/default/rke2-server <<EOF
CONTAINERD_HTTP_PROXY=http://192.168.3.1:8889
CONTAINERD_HTTPS_PROXY=http://192.168.3.1:8889
CONTAINERD_NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local
EOF'
# Or
sudo bash -c 'cat > /usr/local/lib/systemd/system/rke2-server.env <<EOF
HOME=/root
HTTP_PROXY=http://192.168.3.1:8889
HTTPS_PROXY=http://192.168.3.1:8889
NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local
EOF'
sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service
# sudo journalctl -u rke2-server -f
sudo cat /var/lib/rancher/rke2/server/node-token
# save /etc/rancher/rke2/rke2.yaml as token to communicate with cluster api
On Proxmox
Although it’s not good to install on Proxmox instead of VM inside it, for POC purpose I tried to installed it on Proxmox. Below line is required to add to /etc/network/interfaces
, under auto vmbr0
section.
Restart network with/etc/init.d/networking restart
or with systemd restart networking
.
up ip route add default via 192.168.3.1 dev vmbr0
Remote connect via frp
The default connection string is generated for LAN connection(of course), but when connecting from remote from home, you need to expose the API server https port, for example I am using frp.
/etc/rancher/rke2/config.yaml
tls-san:
- "<frp-server-id>"
frpc.ini
[kubernetes-api]
type = tcp
local_ip = 192.168.1.3
local_port = 6443
remote_port = 6443
rke2.yaml
# ...
server: https://120.24.177.213:16443
# ...
Agent
sudo apt update && sudo apt install -y nfs-common
curl -sfL https://get.rke2.io | sudo https_proxy=http://192.168.3.1:8889 INSTALL_RKE2_TYPE="agent" INSTALL_RKE2_CHANNEL=latest sh -
sudo systemctl enable rke2-agent.service
sudo mkdir -p /etc/rancher/rke2/
sudo bash -c 'cat > /etc/default/rke2-agent <<EOF
CONTAINERD_HTTP_PROXY=http://192.168.3.1:8889
CONTAINERD_HTTPS_PROXY=http://192.168.3.1:8889
CONTAINERD_NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local
EOF'
sudo bash -c 'cat > /etc/rancher/rke2/config.yaml <<EOF
server: https://192.168.3.10:9345
token: <token from server node>
EOF'
sudo systemctl start rke2-agent.service
StorageClass: NFS
https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
--set nfs.server=192.168.3.2 \
--set nfs.path=/k8s-pv
StorageClass: Longhorn
Just install according to the official guide, add ingress hostname if you want to access the dashboard by domain.
helm repo add longhorn https://charts.longhorn.io
helm repo update
helm install longhorn longhorn/longhorn \
--namespace longhorn-system \
--create-namespace
spec:
rules:
- host: longhorn.lan
http:
paths:
# ...
Installation
USER=<USERNAME_HERE>
PASSWORD=<PASSWORD_HERE>
echo "${USER}:$(openssl passwd -stdin -apr1 <<< ${PASSWORD})" >> auth
kubectl -n longhorn-system apply -f longhorn-ingress.yml
Install Jenkins
helm install jenkins jenkins/jenkins \
--set persistence.storageClassName="longhorn" \
--set persistence.size="16Gi" \
--set controller.ingress.enabled=true \
--set controller.ingress.hostName="jenkins.lan" \
--namespace jenkins \
--create-namespace