Skip to content

What to look out for when setting up a K3S cluster on PIs

Configure the nodes

  • static IP
  • DNS: disable systemd-resolved, or it will conflict with the embeded DNS server of K3S
  • storage:
    • nfs-kernel-sever, nfs-common
    • ZFS

Install K3S

1st master: curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --cluster-init

Other masters: curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --server https://<ip or hostname of server1>:6443

Workers: curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - agent --server https://<ip or hostname of server>:6443

Access

The kubectl config is located at /etc/rancher/k3s/k3s.yaml.

Install basic services

  • keepalived
  • storage

  • cert-manager 

    helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install \
cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

  • DNS: coreDNS

  • configure traefik 
    # /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
    - "--entryPoints.dnsudp.address=:53/udp"
    - "--entryPoints.dnstcp.address=:53/tcp"
    ...
    ports:
      dnsudp:
        port: 53
        exposedPort: 53
        expose:
          default: true
        protocol: UDP
      dnstcp:
        port: 53
        exposedPort: 53
        expose:
          default: true
        protocol: TCP
      ...
  • install registry
  • confgure registry 
    # /etc/rancher/k3s/registries.yaml
mirrors:
docker.io:
  endpoint:
    - https://registry-mirror.example.org/v2
configs:
  registry.example.org:
    auth:
      username: username
      password: password
  registry-mirror.example.org:
    auth:
      username: username
      password: password