【k8s-1.34.2安装部署】二.kubernets软件、证书、配置、脚本等文件准备

简介

本章节将安装k8s-1.34需要的二进制文件、证书配置文件，脚本，以及etcd、master端组件、worker端组件所用到的配置文件、启动脚本都准备好。同时客户可以自行按照"一.系统初始化及k8s集群规划" 中的软件都下载准备好。该篇文章中的所有配置文件和脚本可以在git地址获取:https://gitee.com/eeenet/k8s-install

== 文章导航===

【k8s-1.34.2安装部署】一.系统初始化及k8s集群规划
【k8s-1.34.2安装部署】二.kubernets软件、证书、配置、脚本等文件准备
【k8s-1.34.2安装部署】三.etcd-v3.6.6 TLS版集群安装
【k8s-1.34.2安装部署】四.kubernets master组件kube-apiserver，kube-controller-manager，kube-scheduler安装
【k8s-1.34.2安装部署】五.worker端containerd2.2.1、kubelet-1.34.2安装
【k8s-1.34.2安装部署】六.企业级部署cilium-1.18.4网络插件
【k8s-1.34.2安装部署】七.coredns-v1.13.1安装
【k8s-1.34.2安装部署】八.metric-server-0.8.0安装
【k8s-1.34.2安装部署】九.k8s管理平台xkube安装部署
【k8s-1.34.2安装部署】十.gateway Api v1.4.0安装
【k8s-1.34.2安装部署】十一.metallb-v0.15.2安装

一.软件下载及安装客户端工具

1.1.安装客户端工具

软件的下载地址参考上一章节:,将软件包解压提取出命令文件放到/usr/local/bin目录下并给予执行权限.
具体软件如下：

1.证书生成软件
 cfssl-certinfo_1.6.5_linux_amd64 #重命名为cfssl-certinfo 
 cfssljson_1.6.5_linux_amd64 #重命名为cfssljson
 cfssl_1.6.5_linux_amd64 #重命名为cfssl

2.k8s网络插件:
 cilium-linux-amd64.tar.gz   #这是cilium的客户端软件,解压提取出cilium
 cni-plugins-linux-amd64-v1.8.0.tgz

3.containerd 及运行时
 containerd-2.2.0-linux-amd64.tar.gz
 runc.amd64-1.3.3

4.etcd
 etcd-v3.6.6-linux-amd64.tar.gz

5.k8s包管理工具,安装cilium 
 helm-v4.0.0-linux-amd64.tar.gz #解压提取出helm

6. k8s软件包
kubernetes-server-linux-amd64.tar.gz  #解压提取出kubectl 和 kubectl-convert 和 kubeadm

从上边的安装包上解压提取出如下文件，并给予执行权限: chmod +x *,然后拷贝到/usr/local/bin目录下

drwxr-xr-x 2 root root      4096 Dec 16 11:59 ./
drwxr-xr-x 9 root root      4096 Dec 16 11:59 ../
-rwxr-xr-x 1 root root  11890840 May 10  2024 cfssl*
-rwxr-xr-x 1 root root   8413336 May 10  2024 cfssl-certinfo*
-rwxr-xr-x 1 root root   6205592 May 10  2024 cfssljson*
-rwxr-xr-x 1 root root 139694264 Oct 23 01:46 cilium*
-rwxr-xr-x 1 root root  65491128 Nov 12 19:39 helm*
-rwxr-xr-x 1 root root  74027192 Nov 12 03:26 kubeadm*
-rwxr-xr-x 1 root root  60559544 Nov 12 03:26 kubectl*
-rwxr-xr-x 1 root root  59642040 Nov 12 03:26 kubectl-convert*

二.准备证书配置文件并制作证书

2.1.证书配置文件列表如下

创建一个csr的目录，配置文件可以从git地址获取：https://gitee.com/eeenet/k8s-install
将如下文件都放在这个csr目录下,如果不从git下载，可以拷贝2.3-2.10的证书配置。

-rw-r--r-- 1 root root 245 Feb 24  2023 admin-csr.json
-rw-r--r-- 1 root root 330 Feb 23  2023 ca-config.json
-rw-r--r-- 1 root root 284 Nov 21 17:24 ca-csr.json
-rw-r--r-- 1 root root 410 Dec  5 12:16 etcd-csr.json
-rw-r--r-- 1 root root 458 Dec  5 12:15 kube-apiserver-csr.json
-rw-r--r-- 1 root root 292 Feb 23  2023 kube-controller-manager-csr.json
-rw-r--r-- 1 root root 274 Feb 23  2023 kube-scheduler-csr.json
-rw-r--r-- 1 root root 272 Feb 23  2023 proxy-client-csr.json

2.2.准备制作证书的脚本并制作证书：

脚本：create-cert.sh,拷贝并给予执行权限,和上边的csr文件夹在同一个目录下,准备好以上证书配置文件以后,执行该脚本，就会在当前目录创建cert目录，并将所有证书生成到cert下。
证书配置注意事项：1.证书中的hosts部分可以根据实际情况更换成自己需要的IP或域名。为了以后方便扩容，也可以规划一个泛域名。

#!/bin/sh

cert_dir="cert"

[ -d $cert_dir ] || mkdir -p $cert_dir

echo "create ca.pem ca-key.pem======="
cfssl gencert -initca csr/ca-csr.json | cfssljson -bare $cert_dir/ca
rm $cert_dir/ca.csr

echo "create etcd.pem etcd-key.pem======="
cfssl gencert -ca=$cert_dir/ca.pem -ca-key=$cert_dir/ca-key.pem -config=csr/ca-config.json -profile=kubernetes csr/etcd-csr.json | cfssljson -bare $cert_dir/etcd
rm -f $cert_dir/etcd.csr

echo "create kube-apiserver.pem kube-apiserver-key.pem======="
cfssl gencert -ca=$cert_dir/ca.pem -ca-key=$cert_dir/ca-key.pem -config=csr/ca-config.json -profile=kubernetes csr/kube-apiserver-csr.json | cfssljson -bare $cert_dir/kube-apiserver
rm -f $cert_dir/kube-apiserver.csr

echo "create kube-scheduler.pem kube-scheduler-key.pem======="
cfssl gencert -ca=$cert_dir/ca.pem -ca-key=$cert_dir/ca-key.pem -config=csr/ca-config.json -profile=kubernetes csr/kube-scheduler-csr.json | cfssljson -bare $cert_dir/kube-scheduler
rm -f $cert_dir/kube-scheduler.csr

echo "create kube-controller-manager.pem kube-controller-manager-key.pem======="
cfssl gencert -ca=$cert_dir/ca.pem -ca-key=$cert_dir/ca-key.pem -config=csr/ca-config.json -profile=kubernetes csr/kube-controller-manager-csr.json | cfssljson -bare $cert_dir/kube-controller-manager
rm -f $cert_dir/kube-controller-manager.csr

echo "create proxy-client.pem proxy-client-key.pem======="
cfssl gencert -ca=$cert_dir/ca.pem -ca-key=$cert_dir/ca-key.pem -config=csr/ca-config.json -profile=kubernetes csr/proxy-client-csr.json  | cfssljson -bare $cert_dir/proxy-client
rm -f $cert_dir/proxy-client.csr

echo "create admin.pem admin-key.pem======="
cfssl gencert -ca=$cert_dir/ca.pem -ca-key=$cert_dir/ca-key.pem -config=csr/ca-config.json -profile=kubernetes csr/admin-csr.json | cfssljson -bare $cert_dir/admin
rm -fv $cert_dir/admin.csr

2.3.ca-config.json

定义ca证书的过期时间,用于生成ca证书

{
    "signing": {
      "default": {
        "expiry": "175200h"
      },
      "profiles": {
        "kubernetes": {
           "expiry": "175200h",
           "usages": [
              "signing",
              "key encipherment",
              "server auth",
              "client auth"
          ]
        }
      }
    }
  }

2.4.ca-csr.json

定义ca证书的加密算法、地域及组织单位,用于生成ca证书

{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Guangzhou",
            "ST": "Guangdong",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

2.5.etcd-csr.json

定义etcd证书中的域名、IP、加密算法及组织单位,hosts中可以配置规划的etcd的主机名，或者etcd的vip、etcd的域名，如果考虑到以后的扩容问题，可以配置一个泛域名,例如：*.cluster.local

{
    "CN": "etcd",
    "hosts": [
        "etcd01.my-k8s.local",
        "etcd02.my-k8s.local",
        "etcd03.my-k8s.local",
        "*.my-k8s.local",
        "127.0.0.1"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Guangzhou",
            "ST": "Guangdong"
        }
    ]
}

2.6.kube-apiserver-csr.json

定义api-server证书中的域名、IP、加密算法及组织单位，hosts中的IP可以配置master的IP，apiserever的IP，以及规划的apiserver的vip，或调用api-server的域名、以及定义的service网段的第一个IP：10.96.0.1

{
  "CN": "kubernetes",
  "hosts": [
    "apiserver.my-k8s.local",
    "*.my-k8s.local",
    "127.0.0.1",
    "10.96.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Guangzhou",
      "ST": "Guangdong",
      "O": "k8s",
      "OU": "system"
    }
  ]
}

2.7.kube-controller-manager-csr.json

定义kube-controller-manager 证书中的api证书地址、节点IP、加密算法及组织单位

{
  "CN": "system:kube-controller-manager",
  "hosts": [
    "127.0.0.1"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:kube-controller-manager",
      "OU": "system"
    }
  ]
}

2.8.kube-scheduler-csr.json

定义kube-scheduler证书中的api证书地址、节点IP、加密算法及组织单位

{
  "CN": "system:kube-scheduler",
  "hosts": [
    "127.0.0.1"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:kube-scheduler",
      "OU": "system"
    }
  ]
}

2.9.admin-csr.json

该配置是用于生成k8s管理客户端kubectl所需的kubeconfig时需要公钥和私钥所必须的证书配置文件

{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:masters",             
      "OU": "system"
    }
  ]
}

2.10.proxy-client-csr.json

kube-apiserver 的另一种访问方式就是使用 kubectl proxy 来代理访问, 而该证书就是用来支持SSL代理访问的. 在该种访问模式下, 我们是以http的方式发起请求到代理服务的, 此时, 代理服务会将该请求发送给 kube-apiserver, 在此之前, 代理会将发送给 kube-apiserver 的请求头里加入证书信息

{
    "CN": "aggregator",
    "hosts": [],
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "Guangdong",
        "L": "Guangzhou",
        "O": "system:masters",
        "OU": "System"
      }
    ]
  }

三.制作kubeconfig文件

3.1.准备脚本:create-kubeconfig.sh

制作这一步的前提是需要上一步的证书文件已经生成好,修改KUBE_APISERVER 为自己定义好的apiserver的域名，然后执行该脚本，就会在当前目录创建kubeconfig，并将配置生成到该目录下,注意该脚本的位置需要和csr相同目录下。

#!/bin/bash

cert_dir="cert"
kube_dir="kubeconfig"
KUBE_APISERVER="https://apiserver.my-k8s.local:6443"

[ -d $kube_dir ] || mkdir -p $kube_dir


echo "create token ====="
cat > $kube_dir/token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:bootstrappers"
EOF


echo "create kube-controller-manager.kubeconfig ====="
kubectl config set-cluster kubernetes \
        --certificate-authority=$cert_dir/ca.pem \
        --embed-certs=true \
        --server=${KUBE_APISERVER} \
        --kubeconfig=$kube_dir/kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
        --client-certificate=$cert_dir/kube-controller-manager.pem \
        --client-key=$cert_dir/kube-controller-manager-key.pem \
        --embed-certs=true \
        --kubeconfig=$kube_dir/kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager \
        --cluster=kubernetes \
        --user=system:kube-controller-manager \
        --kubeconfig=$kube_dir/kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=$kube_dir/kube-controller-manager.kubeconfig


echo "create kube-scheduler.kubeconfig ====="
kubectl config set-cluster kubernetes \
        --certificate-authority=$cert_dir/ca.pem \
        --embed-certs=true \
        --server=${KUBE_APISERVER} \
        --kubeconfig=$kube_dir/kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
        --client-certificate=$cert_dir/kube-scheduler.pem \
        --client-key=$cert_dir/kube-scheduler-key.pem \
        --embed-certs=true \
        --kubeconfig=$kube_dir/kube-scheduler.kubeconfig

kubectl config set-context system:kube-scheduler \
        --cluster=kubernetes \
        --user=system:kube-scheduler \
        --kubeconfig=$kube_dir/kube-scheduler.kubeconfig

kubectl config use-context system:kube-scheduler --kubeconfig=$kube_dir/kube-scheduler.kubeconfig


echo "create kubelet-bootstrap.kubeconfig ====="
TOKEN=$(awk -F "," '{print $1}' $kube_dir/token.csv)
kubectl config set-cluster kubernetes \
          --certificate-authority=$cert_dir/ca.pem \
          --embed-certs=true \
          --server=${KUBE_APISERVER} \
          --kubeconfig=$kube_dir/kubelet-bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap \
          --token=${TOKEN} \
          --kubeconfig=$kube_dir/kubelet-bootstrap.kubeconfig

kubectl config set-context default \
          --cluster=kubernetes \
          --user=kubelet-bootstrap \
          --kubeconfig=$kube_dir/kubelet-bootstrap.kubeconfig

kubectl config use-context default --kubeconfig=$kube_dir/kubelet-bootstrap.kubeconfig

echo "create client kube.config ====="
kubectl config set-cluster kubernetes \
        --certificate-authority=$cert_dir/ca.pem \
        --embed-certs=true \
        --server=${KUBE_APISERVER} \
        --kubeconfig=$kube_dir/kube.kubeconfig

kubectl config set-credentials admin \
        --client-certificate=$cert_dir/admin.pem \
        --client-key=$cert_dir/admin-key.pem \
        --embed-certs=true \
        --kubeconfig=$kube_dir/kube.kubeconfig

kubectl config set-context kubernetes \
        --cluster=kubernetes \
        --user=admin \
        --kubeconfig=$kube_dir/kube.kubeconfig

kubectl config use-context kubernetes --kubeconfig=$kube_dir/kube.kubeconfig

四.准备etcd配置文件及启动脚步

4.1.etcd.conf

etcd01机器的 的配置，每台机不一样

#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/opt/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd01.my-k8s.local:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd01.my-k8s.local:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://etcd01.my-k8s.local:2380,etcd02=https://etcd02.my-k8s.local:2380,etcd03=https://etcd03.my-k8s.local:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-token-my-k8s"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_METRICS_URLS="http://0.0.0.0:2381"

etcd02的配置，每台机不一样

#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/opt/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd02.my-k8s.local:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd02.my-k8s.local:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://etcd01.my-k8s.local:2380,etcd02=https://etcd02.my-k8s.local:2380,etcd03=https://etcd03.my-k8s.local:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-token-my-k8s"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_METRICS_URLS="http://0.0.0.0:2381"

etcd03 的配置，每台机不一样

#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/opt/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd03.my-k8s.local:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd03.my-k8s.local:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://etcd01.my-k8s.local:2380,etcd02=https://etcd02.my-k8s.local:2380,etcd03=https://etcd03.my-k8s.local:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-token-my-k8s"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_METRICS_URLS="http://0.0.0.0:2381"

4.2.etcd启动脚本

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/conf/etcd.conf
WorkingDirectory=/opt/etcd/
ExecStart=/opt/etcd/bin/etcd \
  --cert-file=/opt/etcd/ssl/etcd.pem \
  --key-file=/opt/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/opt/etcd/ssl/ca.pem \
  --peer-cert-file=/opt/etcd/ssl/etcd.pem \
  --peer-key-file=/opt/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

五.准备master相关的配置文件

5.1.kube-apiserver.conf

注意配置中的文件、证书路径。注意以下几个参数：
--etcd-servers： 配置etcd的主机名，需要apiserver能解析此主机名，在/etc/hosts中加入映射。
--service-cluster-ip-range：配置service的网段，用章节一：文章中规划的网段

KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --secure-port=6443 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.96.0.0/16 \
  --token-auth-file=/opt/kubernetes/conf/token.csv \
  --service-node-port-range=30000-50000 \
  --tls-cert-file=/opt/kubernetes/ssl/kube-apiserver.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/opt/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/opt/kubernetes/ssl/kube-apiserver-key.pem \
  --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS \
  --service-account-issuer=https://kubernetes.default.svc \
  --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --etcd-cafile=/opt/etcd/ssl/ca.pem \
  --etcd-certfile=/opt/etcd/ssl/etcd.pem \
  --etcd-keyfile=/opt/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://etcd01.my-k8s.local:2379,https://etcd02.my-k8s.local:2379,https://etcd03.my-k8s.local:2379 \
  --allow-privileged=true \
  --audit-log-maxage=5 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/opt/kubernetes/logs/kube-apiserver-audit.log \
  --requestheader-allowed-names=aggregator \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --proxy-client-cert-file=/opt/kubernetes/ssl/proxy-client.pem \
  --proxy-client-key-file=/opt/kubernetes/ssl/proxy-client-key.pem \
  --v=4"

5.2.kube-controller-manager.conf

注意配置中的文件、证书路径。注意以下几个参数：
--service-cluster-ip-range： service 网段
--cluster-cidr： pod网段

KUBE_CONTROLLER_MANAGER_OPTS="--v=2 \
  --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig \
  --horizontal-pod-autoscaler-sync-period=10s \
  --service-cluster-ip-range=10.96.0.0/16 \
  --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.244.0.0/16 \
  --cluster-signing-duration=175200h \
  --root-ca-file=/opt/kubernetes/ssl/ca.pem \
  --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --tls-cert-file=/opt/kubernetes/ssl/kube-controller-manager.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/kube-controller-manager-key.pem \
  --use-service-account-credentials=true"

5.3.kube-scheduler.conf

注意配置中的文件路径

KUBE_SCHEDULER_OPTS="--kubeconfig=/opt/kubernetes/conf/kube-scheduler.kubeconfig \
--leader-elect=true \
--v=2"

5.4.kubelet.yaml

注意几个参数：
clusterDNS: 定义coredns的服务IP。/opt/kubernetes/ssl/ca.pem resolvConf：/run/systemd/resolve/resolv.conf为系统的resolved的dns配置路径，不配置此项会导致读取/etc/resolv.conf,而/etc/resolv.conf是/run/systemd/resolve/stub-resolv.conf的软连接，里面配置了本地缓存dns,127.0.0.1:53,会和k8s导致dns冲突

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: systemd 
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 2048000
maxPods: 200
resolvConf: /run/systemd/resolve/resolv.conf

4.5.containerd配置文件

containerd的配置文件，可以从containerd命令导出默认配置，并修改里面的镜像地址和SystemdCgroup 为true,导出命令参考如下：

containerd config default | sudo tee /etc/containerd/config.toml
sed -i 's#SystemdCgroup.*#SystemdCgroup = true#' /etc/containerd/config.toml
sed -i 's#sandbox_image.*#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.10.1"#' /etc/containerd/config.toml

config.toml 配置

version = 3
root = '/var/lib/containerd'
state = '/run/containerd'
temp = ''
disabled_plugins = []
required_plugins = []
oom_score = 0
imports = ['/etc/containerd/conf.d/*.toml']

[grpc]
  address = '/run/containerd/containerd.sock'
  tcp_address = ''
  tcp_tls_ca = ''
  tcp_tls_cert = ''
  tcp_tls_key = ''
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_tls_common_name = ''

[ttrpc]
  address = ''
  uid = 0
  gid = 0

[debug]
  address = ''
  uid = 0
  gid = 0
  level = ''
  format = ''

[metrics]
  address = ''
  grpc_histogram = false

[plugins]
  [plugins.'io.containerd.cri.v1.images']
    snapshotter = 'overlayfs'
    disable_snapshot_annotations = true
    discard_unpacked_layers = false
    max_concurrent_downloads = 3
    concurrent_layer_fetch_buffer = 0
    image_pull_progress_timeout = '5m0s'
    image_pull_with_sync_fs = false
    stats_collect_period = 10
    use_local_image_pull = false

    [plugins.'io.containerd.cri.v1.images'.pinned_images]
      sandbox = 'registry.aliyuncs.com/google_containers/pause:3.10.1'

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = '/etc/containerd/certs.d:/etc/docker/certs.d'

    [plugins.'io.containerd.cri.v1.images'.image_decryption]
      key_model = 'node'

  [plugins.'io.containerd.cri.v1.runtime']
    enable_selinux = false
    selinux_category_range = 1024
    max_container_log_line_size = 16384
    disable_apparmor = false
    restrict_oom_score_adj = false
    disable_proc_mount = false
    unset_seccomp_profile = ''
    tolerate_missing_hugetlb_controller = true
    disable_hugetlb_controller = true
    device_ownership_from_security_context = false
    ignore_image_defined_volumes = false
    netns_mounts_under_state_dir = false
    enable_unprivileged_ports = true
    enable_unprivileged_icmp = true
    enable_cdi = true
    cdi_spec_dirs = ['/etc/cdi', '/var/run/cdi']
    drain_exec_sync_io_timeout = '0s'
    ignore_deprecation_warnings = []

    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      default_runtime_name = 'runc'
      ignore_blockio_not_enabled_errors = false
      ignore_rdt_not_enabled_errors = false

      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          runtime_type = 'io.containerd.runc.v2'
          runtime_path = ''
          pod_annotations = []
          container_annotations = []
          privileged_without_host_devices = false
          privileged_without_host_devices_all_devices_allowed = false
          cgroup_writable = false
          base_runtime_spec = ''
          cni_conf_dir = ''
          cni_max_conf_num = 0
          snapshotter = ''
          sandboxer = 'podsandbox'
          io_type = ''

          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
            BinaryName = ''
            CriuImagePath = ''
            CriuWorkPath = ''
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            Root = ''
            ShimCgroup = ''
            SystemdCgroup = true

    [plugins.'io.containerd.cri.v1.runtime'.cni]
      bin_dir = ''
      bin_dirs = ['/opt/cni/bin']
      conf_dir = '/etc/cni/net.d'
      max_conf_num = 1
      setup_serially = false
      conf_template = ''
      ip_pref = ''
      use_internal_loopback = false

  [plugins.'io.containerd.differ.v1.erofs']
    mkfs_options = []
    enable_tar_index = false

  [plugins.'io.containerd.gc.v1.scheduler']
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = '0s'
    startup_delay = '100ms'

  [plugins.'io.containerd.grpc.v1.cri']
    disable_tcp_service = true
    stream_server_address = '127.0.0.1'
    stream_server_port = '0'
    stream_idle_timeout = '4h0m0s'
    enable_tls_streaming = false

    [plugins.'io.containerd.grpc.v1.cri'.x509_key_pair_streaming]
      tls_cert_file = ''
      tls_key_file = ''

  [plugins.'io.containerd.image-verifier.v1.bindir']
    bin_dir = '/opt/containerd/image-verifier/bin'
    max_verifiers = 10
    per_verifier_timeout = '10s'

  [plugins.'io.containerd.internal.v1.opt']
    path = '/opt/containerd'

  [plugins.'io.containerd.internal.v1.tracing']

  [plugins.'io.containerd.metadata.v1.bolt']
    content_sharing_policy = 'shared'
    no_sync = false

  [plugins.'io.containerd.monitor.container.v1.restart']
    interval = '10s'

  [plugins.'io.containerd.monitor.task.v1.cgroups']
    no_prometheus = false

  [plugins.'io.containerd.mount-handler.v1.erofs']

  [plugins.'io.containerd.nri.v1.nri']
    disable = false
    socket_path = '/var/run/nri/nri.sock'
    plugin_path = '/opt/nri/plugins'
    plugin_config_path = '/etc/nri/conf.d'
    plugin_registration_timeout = '5s'
    plugin_request_timeout = '2s'
    disable_connections = false

    [plugins.'io.containerd.nri.v1.nri'.default_validator]
      enable = false
      reject_oci_hook_adjustment = false
      reject_runtime_default_seccomp_adjustment = false
      reject_unconfined_seccomp_adjustment = false
      reject_custom_seccomp_adjustment = false
      reject_namespace_adjustment = false
      required_plugins = []
      tolerate_missing_plugins_annotation = ''

  [plugins.'io.containerd.runtime.v2.task']
    platforms = ['linux/amd64']

  [plugins.'io.containerd.service.v1.diff-service']
    default = ['walking']
    sync_fs = false

  [plugins.'io.containerd.service.v1.tasks-service']
    blockio_config_file = ''
    rdt_config_file = ''

  [plugins.'io.containerd.shim.v1.manager']
    env = []

  [plugins.'io.containerd.snapshotter.v1.blockfile']
    root_path = ''
    scratch_file = ''
    fs_type = ''
    mount_options = []
    recreate_scratch = false

  [plugins.'io.containerd.snapshotter.v1.btrfs']
    root_path = ''

  [plugins.'io.containerd.snapshotter.v1.devmapper']
    root_path = ''
    pool_name = ''
    base_image_size = ''
    async_remove = false
    discard_blocks = false
    fs_type = ''
    fs_options = ''

  [plugins.'io.containerd.snapshotter.v1.erofs']
    root_path = ''
    ovl_mount_options = []
    enable_fsverity = false
    set_immutable = false
    default_size = ''

  [plugins.'io.containerd.snapshotter.v1.native']
    root_path = ''

  [plugins.'io.containerd.snapshotter.v1.overlayfs']
    root_path = ''
    upperdir_label = false
    sync_remove = false
    slow_chown = false
    mount_options = []

  [plugins.'io.containerd.snapshotter.v1.zfs']
    root_path = ''

  [plugins.'io.containerd.tracing.processor.v1.otlp']

  [plugins.'io.containerd.transfer.v1.local']
    max_concurrent_downloads = 3
    concurrent_layer_fetch_buffer = 0
    max_concurrent_uploaded_layers = 3
    check_platform_supported = false
    config_path = ''
    max_concurrent_unpacks = 1

[cgroup]
  path = ''

[timeouts]
  'io.containerd.timeout.bolt.open' = '0s'
  'io.containerd.timeout.cri.defercleanup' = '1m0s'
  'io.containerd.timeout.metrics.shimstats' = '2s'
  'io.containerd.timeout.shim.cleanup' = '5s'
  'io.containerd.timeout.shim.load' = '5s'
  'io.containerd.timeout.shim.shutdown' = '3s'
  'io.containerd.timeout.task.state' = '2s'

[stream_processors]
  [stream_processors.'io.containerd.ocicrypt.decoder.v1.tar']
    accepts = ['application/vnd.oci.image.layer.v1.tar+encrypted']
    returns = 'application/vnd.oci.image.layer.v1.tar'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']

  [stream_processors.'io.containerd.ocicrypt.decoder.v1.tar.gzip']
    accepts = ['application/vnd.oci.image.layer.v1.tar+gzip+encrypted']
    returns = 'application/vnd.oci.image.layer.v1.tar+gzip'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']

4.6.kube-apiserver.service 启动脚本

kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service

[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

4.7.kube-controller-manager.service 启动脚本

kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

4.8.kube-scheduler.service 启动脚本

kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

4.9.kubelet.service 启动脚本

kubelet.service

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/opt/kubernetes/bin/kubelet \
  --hostname-override=node-hostname \ #此处需要配置正确的节点的主机名
  --bootstrap-kubeconfig=/opt/kubernetes/conf/kubelet-bootstrap.kubeconfig \
  --cert-dir=/opt/kubernetes/ssl \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig \
  --config=/opt/kubernetes/conf/kubelet.yaml \
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

4.10.containerd.service 启动脚本

containerd.service

# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target dbus.service

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

六.制作etcd、master、containerd、node的安装包

6.1. etcd安装包

1.创建目录etcd,解压etcd-v3.6.6-linux-amd64.tar.gz 将etcd、etcdctl、etcdutl 三个文件拷贝到etcd/bin下。
2.拷贝etcd.conf 到etcd/conf目录下,此处的配置文件，三台机分别拷贝etcd01.conf,etcd02.conf,etcd03.conf。
3.拷贝cert目录下生成好的证书到ssl目录下，注：该步骤是需要在前面已经在执行了create-cert.sh和create-kubeconfig.sh以后会生成，生成后不需要重复执行,不能将多次执行生成的cert和kubeconfig文件拷贝到不同的安装包内，一次执行，拷贝到不同的安装包。
4.etcd.service 是启动脚本，安装时需要拷贝到/usr/lib/systemd/system/

etcd目录结构如下:

├── bin
│   ├── etcd
│   ├── etcdctl
│   └── etcdutl
├── conf
│   └── etcd.conf
├── etcd.service
├── logs
└── ssl
    ├── ca-key.pem
    ├── ca.pem
    ├── etcd-key.pem
    └── etcd.pem

6.2. 制作master节点安装包

1.master除了安装kube-controller-manager、kube-scheduler、kube-apiserver，以外还需要安装containerd和kubelet。其中四个.service文件是启动脚本,安装时需要拷贝到/usr/lib/systemd/system/

1. 拷贝cert目录下生成好的证书到ssl目录下，需要拷贝的文件参考如下目录结构
2. kubeconfig文件拷贝到conf目录下,需要拷贝的文件参考如下目录结构

master节点目录结构

├── bin
│   ├── kube-apiserver
│   ├── kube-controller-manager
│   ├── kubelet
│   └── kube-scheduler
├── conf  #从kubeconfig目录拷贝kubeconfig文件
│   ├── kube-apiserver.conf
│   ├── kube-controller-manager.conf
│   ├── kube-controller-manager.kubeconfig
│   ├── kubelet-bootstrap.kubeconfig
│   ├── kubelet.yaml
│   ├── kube-scheduler.conf
│   ├── kube-scheduler.kubeconfig
│   └── token.csv  #在kubeconfig目录下
├── kube-apiserver.service
├── kube-controller-manager.service
├── kubelet.service
├── kube-scheduler.service
├── logs
└── ssl  #从cert目录拷贝
    ├── ca-key.pem
    ├── ca.pem
    ├── kube-apiserver-key.pem
    ├── kube-apiserver.pem
    ├── kube-controller-manager-key.pem
    ├── kube-controller-manager.pem
    ├── kube-scheduler-key.pem
    ├── kube-scheduler.pem
    ├── proxy-client-key.pem
    └── proxy-client.pem

6.3. node节点的安装包

1.node只需要安装containerd和kubelet。其中.service文件是启动脚本,安装时需要拷贝到/usr/lib/systemd/system/

1. 拷贝cert目录下生成好的证书到ssl目录下，需要拷贝的文件参考如下目录结构
2. kubeconfig文件拷贝到conf目录下,需要拷贝的文件参考如下目录结构

node节点的目录结构如下：

├── bin
│   ├── kubelet
│   └── kube-proxy
├── conf  #从kubeconfig目录拷贝kubeconfig文件
│   ├── kubelet-bootstrap.kubeconfig      
│   ├── kubelet.yaml
├── kubelet.service
├── logs
└── ssl  #从cert目录拷贝
    ├── ca-key.pem
    ├── ca.pem

6.4. containerd安装包

containerd2.0 开始安装时需要安装runc 和cni-plugins。该安装包是将runc 和 cni-plugins、containerd的执行文件、配置、启动脚本拷贝到一起。安装时：bin目录下的文件拷贝到/usr/local/bin下，cni目录拷贝到/opt/下，sbin/runc 拷贝到/usr/local/sbin 下，config.toml 拷贝到/etc/containerd，crictl.yaml 拷贝到/etc/ 下，containerd.service 拷贝到/etc/systemd/system/下，具体参考如下目录结构

下载地址:
https://gitee.com/eeenet/k8s-install
https://github.com/kubernetes-sigs/cri-tools/releases
https://github.com/containerd/containerd/releases
https://github.com/opencontainers/runc/releases
https://github.com/containernetworking/plugins/releases

├── etc
│   ├── containerd
│   │   └── config.toml
│   ├── crictl.yaml
│   └── systemd
│       └── system
│           └── containerd.service
├── opt
│   └── cni
│       └── bin
│           ├── bandwidth
│           ├── bridge
│           ├── dhcp
│           ├── dummy
│           ├── firewall
│           ├── host-device
│           ├── host-local
│           ├── ipvlan
│           ├── LICENSE
│           ├── loopback
│           ├── macvlan
│           ├── portmap
│           ├── ptp
│           ├── README.md
│           ├── sbr
│           ├── static
│           ├── tap
│           ├── tuning
│           ├── vlan
│           └── vrf
└── usr
    └── local
        ├── bin
        │   ├── containerd
        │   ├── containerd-shim-runc-v2
        │   ├── containerd-stress
        │   ├── crictl
        │   └── ctr
        └── sbin
            └── runc