Kubernets Installation From Scatch on rocky 9
Topics
- Downloading Release Binaries
- Configuring CA
- Installing ETCD
- Configuring API Server
- Configuring Controller Manager
- Configuring Scheduler
- Validating Cluster Status
- Worker Node Configuration
- Configuring Networking
- API to Kubelet RBAC
- Configuring DNS
- Kubelet Preferred Instance Type
Downloading Release Binaries
- Kubernetes Binaries
- Source Code
- Client Binaries
- Server Binaries
- Node Binaries
- Container Images
- Download release Binaries
- Download Etcd
- Install wget and unzip
yum install wget unzip -yDownload Server Binaries on master node
mkdir /root/binaries
cd /root/binaries
wget https://dl.k8s.io/v1.28.7/kubernetes-server-linux-amd64.tar.gz
tar -xzvf kubernetes-server-linux-amd64.tar.gz
cd /root/binaries/kubernetes/server/bin/- Check all master binaries
ls -l /root/binaries/kubernetes/server/bin- Download etcd in the same folder
cd /root/binaries/
wget https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-darwin-amd64.zip
unzip etcd-v3.5.12-darwin-amd64.zip
cd etcd-v3.5.12-darwin-amd64/- Check etcd relatated binaries
ls -l /root/binaries/tcd-v3.5.12-darwin-amd64/Download Node Binaries on Worker node
mkdir /root/binaries
cd /root/binaries
wget https://dl.k8s.io/v1.28.7/kubernetes-node-linux-amd64.tar.gz
tar -xzvf kubernetes-node-linux-amd64.tar.gz- Check all node binaries
ls -l /root/binaries/node/binConfiguring kubernetes Certificate Authority
Create base directory were all the certificates and keys will be stored
mkdir /root/certificates
cd /root/certificatesCreating a private key for Certificate Authority
openssl genrsa -out ca.key 2048Creating CSR
openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csrSelf-Sign the CSR
openssl x509 -req -in ca.csr -signkey ca.key -CAcreateserial -out ca.crt -days 1000See Contents of Certificate
openssl x509 -in ca.crt -text -nooutRemove CSR
rm -f ca.csrInstalling ETCD
Documentation Referred
Pre-Requisite: Set SERVER-IP variable
SERVER_IP=172.16.16.110 (change this to your IP)
echo $SERVER_IPConfigure the Certificates:
cd /root/certificates/
openssl genrsa -out etcd.key 2048Note: Replace the value associated with IP.1 in the below step.
cat > etcd.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = ${SERVER_IP}
IP.2 = 127.0.0.1
EOFopenssl req -new -key etcd.key -subj "/CN=etcd" -out etcd.csr -config etcd.cnf
openssl x509 -req -in etcd.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out etcd.crt -extensions v3_req -extfile etcd.cnf -days 1000Copy the Certificates and Key to /etc/etcd
mkdir /etc/etcd
cp etcd.crt etcd.key ca.crt /etc/etcdCopy the ETCD and ETCDCTL Binaries to the Path
cd /root/binaries/etcd-v3.5.12-darwin-amd64/
cp etcd etcdctl /usr/local/bin/Configure the systemd File
Create a service file:
cat <<EOF | sudo tee /etc/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos
[Service]
ExecStart=/usr/local/bin/etcd \\
--name master-1 \\
--cert-file=/etc/etcd/etcd.crt \\
--key-file=/etc/etcd/etcd.key \\
--peer-cert-file=/etc/etcd/etcd.crt \\
--peer-key-file=/etc/etcd/etcd.key \\
--trusted-ca-file=/etc/etcd/ca.crt \\
--peer-trusted-ca-file=/etc/etcd/ca.crt \\
--peer-client-cert-auth \\
--client-cert-auth \\
--initial-advertise-peer-urls https://${SERVER_IP}:2380 \\
--listen-peer-urls https://${SERVER_IP}:2380 \\
--listen-client-urls https://${SERVER_IP}:2379,https://127.0.0.1:2379 \\
--advertise-client-urls https://${SERVER_IP}:2379 \\
--initial-cluster-token etcd-cluster-0 \\
--initial-cluster master-1=https://${SERVER_IP}:2380 \\
--initial-cluster-state new \\
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOFStart Service:
systemctl start etcd
systemctl status etcd
systemctl enable etcdVerification Commands:
When we try with etcdctl –endpoints=https://127.0.0.1:2379 get foo, it gives unknown certificate authority.
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/etcd/ca.crt --cert=/etc/etcd/etcd.crt --key=/etc/etcd/etcd.key put test "Test"ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/etcd/ca.crt --cert=/etc/etcd/etcd.crt --key=/etc/etcd/etcd.key get testConfiguring API Server
Reference Documentation
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
PreRequisites
- Move the kube-apiserver binary to /usr/local/bin directory.
cd /root/binaries/kubernetes/server/bin/
cp kube-apiserver /usr/local/bin/Ensure that the SERVER_IP variable is still set.
Step 1. Generate Configuration File for CSR Creation.
cd /root/certificatescat <<EOF | sudo tee api.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
IP.1 = 127.0.0.1
IP.2 = ${SERVER_IP}
IP.3 = 10.0.2.15
EOFGenerate Certificates for API Server
openssl genrsa -out kube-api.key 2048
openssl req -new -key kube-api.key -subj "/CN=kube-apiserver" -out kube-api.csr -config api.conf
openssl x509 -req -in kube-api.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-api.crt -extensions v3_req -extfile api.conf -days 1000Generate Certificate for Service Account:
openssl genrsa -out service-account.key 2048
openssl req -new -key service-account.key -subj "/CN=service-accounts" -out service-account.csr
openssl x509 -req -in service-account.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out service-account.crt -days 100Copy the certificate files to /var/lib/kubernetes directory
mkdir /var/lib/kubernetes
cp etcd.crt etcd.key ca.crt kube-api.key kube-api.crt service-account.crt service-account.key /var/lib/kubernetes
ls /var/lib/kubernetesCreating Encryption key and Configuration
ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)cat > encryption-at-rest.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: ${ENCRYPTION_KEY}
- identity: {}
EOFcp encryption-at-rest.yaml /var/lib/kubernetes/encryption-at-rest.yamlCreating Systemd service file:
Systemd file:
cat <<EOF | sudo tee /etc/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-apiserver \
--advertise-address=${SERVER_IP} \
--allow-privileged=true \
--authorization-mode=Node,RBAC \
--client-ca-file=/var/lib/kubernetes/ca.crt \
--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
--enable-bootstrap-token-auth=true \
--etcd-cafile=/var/lib/kubernetes/ca.crt \
--etcd-certfile=/var/lib/kubernetes/etcd.crt \
--etcd-keyfile=/var/lib/kubernetes/etcd.key \
--etcd-servers=https://127.0.0.1:2379 \
--kubelet-client-certificate=/var/lib/kubernetes/kube-api.crt \
--kubelet-client-key=/var/lib/kubernetes/kube-api.key \
--service-account-key-file=/var/lib/kubernetes/service-account.crt \
--service-cluster-ip-range=10.32.0.0/24 \
--tls-cert-file=/var/lib/kubernetes/kube-api.crt \
--tls-private-key-file=/var/lib/kubernetes/kube-api.key \
--requestheader-client-ca-file=/var/lib/kubernetes/ca.crt \
--service-node-port-range=30000-32767 \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/var/log/kube-api-audit.log \
--bind-address=0.0.0.0 \
--event-ttl=1h \
--service-account-key-file=/var/lib/kubernetes/service-account.crt \
--service-account-signing-key-file=/var/lib/kubernetes/service-account.key \
--service-account-issuer=https://${SERVER_IP}:6443 \
--encryption-provider-config=/var/lib/kubernetes/encryption-at-rest.yaml \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOFStart the kube-api service:
systemctl start kube-apiserver
systemctl status kube-apiserver
systemctl enable kube-apiserverConfiguring Controller Manager
Generate Certificates:
cd /root/certificatesopenssl genrsa -out kube-controller-manager.key 2048
openssl req -new -key kube-controller-manager.key -subj "/CN=system:kube-controller-manager" -out kube-controller-manager.csr
openssl x509 -req -in kube-controller-manager.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-controller-manager.crt -days 1000Generating KubeConfig
cp /root/binaries/kubernetes/server/bin/kubectl /usr/local/binkubectl config set-cluster kubernetes-from-scratch \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://127.0.0.1:6443 \
--kubeconfig=kube-controller-manager.kubeconfigkubectl config set-cluster kubernetes-from-scratch \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://127.0.0.1:6443 \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \
--client-certificate=kube-controller-manager.crt \
--client-key=kube-controller-manager.key \
--embed-certs=true \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-from-scratch \
--user=system:kube-controller-manager \
--kubeconfig=kube-controller-manager.kubeconfigkubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfigCopying the files to kubernetes directory
cp kube-controller-manager.crt kube-controller-manager.key kube-controller-manager.kubeconfig ca.key /var/lib/kubernetes/Configuring SystemD service file:
cat <<EOF | sudo tee /etc/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
--bind-address=0.0.0.0 \\
--service-cluster-ip-range=10.32.0.0/24 \\
--cluster-cidr=10.200.0.0/16 \\
--kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
--authentication-kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
--authorization-kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
--leader-elect=true \\
--cluster-signing-cert-file=/var/lib/kubernetes/ca.crt \\
--cluster-signing-key-file=/var/lib/kubernetes/ca.key \\
--root-ca-file=/var/lib/kubernetes/ca.crt \\
--service-account-private-key-file=/var/lib/kubernetes/service-account.key \\
--use-service-account-credentials=true \\
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOFStart Controller Manager:
cp /root/binaries/kubernetes/server/bin/kube-controller-manager /usr/local/bin
systemctl start kube-controller-manager
systemctl status kube-controller-manager
systemctl enable kube-controller-managerConfiguring Scheduler
Generate Certificates:
cd /root/certificates
openssl genrsa -out kube-scheduler.key 2048
openssl req -new -key kube-scheduler.key -subj "/CN=system:kube-scheduler" -out kube-scheduler.csr
openssl x509 -req -in kube-scheduler.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-scheduler.crt -days 1000Generate Kubeconfig file:
{
kubectl config set-cluster kubernetes-from-scratch \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://127.0.0.1:6443 \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler \
--client-certificate=kube-scheduler.crt \
--client-key=kube-scheduler.key \
--embed-certs=true \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-from-scratch \
--user=system:kube-scheduler \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig
}Copy the scheduler kubeconfig
cp kube-scheduler.kubeconfig /var/lib/kubernetes/Configuring systemd service:
cat <<EOF | sudo tee /etc/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
--kubeconfig=/var/lib/kubernetes/kube-scheduler.kubeconfig \\
--authentication-kubeconfig=/var/lib/kubernetes/kube-scheduler.kubeconfig \\
--authorization-kubeconfig=/var/lib/kubernetes/kube-scheduler.kubeconfig \\
--bind-address=127.0.0.1 \\
--leader-elect=true
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOFVerification:
cp /root/binaries/kubernetes/server/bin/kube-scheduler /usr/local/bin
systemctl start kube-scheduler
systemctl status kube-scheduler
systemctl enable kube-schedulerValidating Cluster Status
Generate Certificate for Administrator User
cd /root/certificates
openssl genrsa -out admin.key 2048
openssl req -new -key admin.key -subj "/CN=admin/O=system:masters" -out admin.csr
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out admin.crt -days 1000Create KubeConfig file:
Note: Replace the IP address from the below snippet in line 5 with your IP address.
{
kubectl config set-cluster kubernetes-from-scratch \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://${SERVER_IP}:6443 \
--kubeconfig=admin.kubeconfig
kubectl config set-credentials admin \
--client-certificate=admin.crt \
--client-key=admin.key \
--embed-certs=true \
--kubeconfig=admin.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-from-scratch \
--user=admin \
--kubeconfig=admin.kubeconfig
kubectl config use-context default --kubeconfig=admin.kubeconfig
}Verify Cluster Status
kubectl get componentstatuses --kubeconfig=admin.kubeconfig
cp /root/certificates/admin.kubeconfig ~/.kube/config
kubectl get componentstatusesVerify Kubernetes Objects Creation
kubectl create namespace test
kubectl get namespace test -o yaml
kubectl create secret generic prod-secret --from-literal=username=admin --from-literal=password=password123
kubectl get secretWorker Node Configuration
Configure Container Runtime. (WORKER NODE)
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOFmodprobe overlay
modprobe br_netfiltercat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOFsysctl --systemsudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.tomlnano /etc/containerd/config.toml–> SystemdCgroup = true
systemctl restart containerdcat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOFsudo sysctl --systemPre-Requisites - 2: (WORKER NODE)
dnf install socat conntrack ipset
sysctl -w net.ipv4.conf.all.forwarding=1
cd /root/binaries/kubernetes/node/bin/
cp kube-proxy kubectl kubelet /usr/local/binGenerate Kubelet Certificate for Worker Node. (MASTER NODE)
Note:
- Replace the IP Address and Hostname field in the below configurations according to your enviornement.
- Run this in the Kubernetes Master Node
cd /root/certificatescat > openssl-worker.example.com.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = worker.example.com
IP.1 = 172.16.16.111
EOFopenssl genrsa -out worker.example.com.key 2048openssl req -new -key worker.example.com.key -subj "/CN=system:node:worker.example.com/O=system:nodes" -out worker.example.com.csr -config openssl-worker.example.com.cnf
openssl x509 -req -in worker.example.com.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out worker.example.com.crt -extensions v3_req -extfile openssl-worker.example.com.cnf -days 1000Step 2: Generate kube-proxy certificate: (MASTER NODE)
openssl genrsa -out kube-proxy.key 2048
openssl req -new -key kube-proxy.key -subj "/CN=system:kube-proxy" -out kube-proxy.csr
openssl x509 -req -in kube-proxy.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-proxy.crt -days 1000Step 3: Copy Certificates to Worker Node:
This can either be manual approach or via SCP. Certificates: kubelet, kube-proxy and CA certificate.
In-case you want to automate it, then following configuration can be used. In the demo, we had made used of manual way.
In-case, you want to transfer file from master to worker node, then you can make use of the following approach:
- Worker Node:
- Master Node:
- Copy certificate from master node to worker nodes
scp kube-proxy.crt kube-proxy.key worker.example.com.key worker.example.com.crt ca.crt vagrant@172.16.16.111:/tmp- Worker Node:
mkdir /root/certificates
cd /tmp
mv kube-proxy.crt kube-proxy.key worker.example.com.crt worker.example.com.key ca.crt /root/certificatesStep 4: Move Certificates to Specific Location. (WORKER NODE)
mkdir /var/lib/kubernetes
cd /root/certificates
cp ca.crt /var/lib/kubernetes
mkdir /var/lib/kubelet
mv worker.example.com.crt worker.example.com.key kube-proxy.crt kube-proxy.key /var/lib/kubelet/Step 5: Generate Kubelet Configuration YAML File: (WORKER NODE)
cat <<EOF | sudo tee /var/lib/kubelet/kubelet-config.yaml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
enabled: true
x509:
clientCAFile: "/var/lib/kubernetes/ca.crt"
authorization:
mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
- "10.32.0.10"
runtimeRequestTimeout: "15m"
cgroupDriver: systemd
EOFStep 6: Generate Systemd service file for kubelet: (WORKER NODE)
cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service
[Service]
ExecStart=/usr/local/bin/kubelet \
--config=/var/lib/kubelet/kubelet-config.yaml \
--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \
--kubeconfig=/var/lib/kubelet/kubeconfig \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOFStep 7: Generate the Kubeconfig file for Kubelet (WORKER NODE)
cd /var/lib/kubelet
cp /var/lib/kubernetes/ca.crt .
SERVER_IP=IP-OF-API-SERVER{
kubectl config set-cluster kubernetes-from-scratch \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://${SERVER_IP}:6443 \
--kubeconfig=worker.example.com.kubeconfig
kubectl config set-credentials system:node:worker.example.com \
--client-certificate=worker.example.com.crt \
--client-key=worker.example.com.key \
--embed-certs=true \
--kubeconfig=worker.example.com.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-from-scratch \
--user=system:node:worker.example.com \
--kubeconfig=worker.example.com.kubeconfig
kubectl config use-context default --kubeconfig=worker.example.com.kubeconfig
}mv worker.example.com.kubeconfig kubeconfigPart 2 - Kube-Proxy
Step 1: Copy Kube Proxy Certificate to Directory: (WORKER NODE)
mkdir /var/lib/kube-proxyStep 2: Generate KubeConfig file:
{
kubectl config set-cluster kubernetes-from-scratch \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://${SERVER_IP}:6443 \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials system:kube-proxy \
--client-certificate=kube-proxy.crt \
--client-key=kube-proxy.key \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-from-scratch \
--user=system:kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
}mv kube-proxy.kubeconfig /var/lib/kube-proxy/kubeconfigStep 3: Generate kube-proxy configuration file: (WORKER NODE)
cd /var/lib/kube-proxycat <<EOF | sudo tee /var/lib/kube-proxy/kube-proxy-config.yaml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
kubeconfig: "/var/lib/kube-proxy/kubeconfig"
mode: "iptables"
clusterCIDR: "10.200.0.0/16"
EOFStep 4: Create kube-proxy service file: (WORKER NODE)
cat <<EOF | sudo tee /etc/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-proxy \\
--config=/var/lib/kube-proxy/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOFWORKER NODE
systemctl start kubelet
systemctl start kube-proxy
systemctl enable kubelet
systemctl enable kube-proxyStep 6: (MASTER NODE)
kubectl get nodes- Check the log of kubelet
journalctl -u kubelet.service