一、简介
在阿里云服务器上部署k8s集群，一、3台k8s服务器，1个Master节点，2个工作节点，采用Calico网络技术。二、部署nginx服务到k8s集群，并验证nginx服务运行状态。
二、前置条件
1、需要准备3台服务器
172.21.173.7 k8s-master01 CPU：4C，内存：8G
172.21.173.8 k8s-work01 CPU：2C，内存：2G
172.21.173.9 k8s-work02 CPU：2C，内存：2G
2、安装软件版本
Ubuntu版本：Linux 6.8.0-40-generic
Kubernetes版本：v1.31.1
Docker版本：v27.3.1
Containerd版本：v1.7.22
Kubeadm版本：v1.31.1
Kubectl版本：v1.31.1
Kubelet版本：v1.31.1
Calico版本：v3.28.2
3、参考Kubernetes、Docker、Calico的官方文档
https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/
https://github.com/containerd/containerd/blob/main/docs/getting-started.md
https://docs.docker.com/engine/install/ubuntu/
https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
https://docs.tigera.io/calico/latest/getting-started/kubernetes/quickstart
三、安装containerd（所有服务器上）
1、卸载旧的服务
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
2、设置docker仓库地址
# Add Docker's official GPG key: sudo apt-get update sudo apt-get install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository to Apt sources: echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update
3、安装Docker
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin systemctl enable containerd
4、配置docker镜像地址
/etc/docker/daemon.json { "registry-mirrors" : ["https://o7klhlze.mirror.aliyuncs.com"] }
5、验证docker安装是否成功
sudo docker run hello-world
四、安装kubectl、kubeadm、kubelet（所有服务器上）
1、交换分区的配置
sudo swapoff -a
2、 更新 apt 包索引并安装使用 Kubernetes apt 仓库所需要的包
sudo apt-get update# apt-transport-https 可能是一个虚拟包（dummy package）；如果是的话，你可以跳过安装这个包sudo apt-get install -y apt-transport-https ca-certificates curl gpg
3、下载用于 Kubernetes 软件包仓库的公共签名密钥。所有仓库都使用相同的签名密钥，因此你可以忽略URL中的版本
# 如果 `/etc/apt/keyrings` 目录不存在，则应在 curl 命令之前创建它，请阅读下面的注释。# sudo mkdir -p -m 755 /etc/apt/keyringscurl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
4、添加 Kubernetes apt 仓库。 请注意，此仓库仅包含适用于 Kubernetes 1.31 的软件包； 对于其他 Kubernetes 次要版本，则需要更改 URL 中的 Kubernetes 次要版本以匹配你所需的次要版本 （你还应该检查正在阅读的安装文档是否为你计划安装的 Kubernetes 版本的文档）。
# 此操作会覆盖 /etc/apt/sources.list.d/kubernetes.list 中现存的所有配置。echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
5、更新 apt 包索引，安装 kubelet、kubeadm 和 kubectl
sudo apt-get updatesudo apt-get install -y kubelet kubeadm kubectlsudo apt-mark hold kubelet kubeadm kubectl systemctl enable kubelet
6、检查服务状态
systemctl status kubelet systemctl status containerd
7、配置主节点的kubeadm-config.yaml
kubeadm config print init-defaults > kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 172.21.173.7 bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/containerd/containerd.sock imagePullPolicy: IfNotPresent imagePullSerial: true name: k8s-master01 taints: null timeouts: controlPlaneComponentHealthCheck: 4m0s discovery: 5m0s etcdAPICall: 2m0s kubeletHealthCheck: 4m0s kubernetesAPICall: 1m0s tlsBootstrap: 5m0s upgradeManifests: 5m0s --- apiServer: {} apiVersion: kubeadm.k8s.io/v1beta4 caCertificateValidityPeriod: 87600h0m0s certificateValidityPeriod: 8760h0m0s certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: {} encryptionAlgorithm: RSA-2048 etcd: local: dataDir: /var/lib/etcd imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: 1.31.1 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 proxy: {} scheduler: {} --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd
8、配置work节点的kubeadm-config.yaml
kubeadm config print join-defaults > kubeadm-config.yaml
kubeadm-config.yaml的内容：
apiVersion: kubeadm.k8s.io/v1beta4 caCertPath: /etc/kubernetes/pki/ca.crt discovery: bootstrapToken: apiServerEndpoint: kube-apiserver:6443 token: abcdef.0123456789abcdef unsafeSkipCAVerification: true tlsBootstrapToken: abcdef.0123456789abcdef kind: JoinConfiguration nodeRegistration: criSocket: unix:///var/run/containerd/containerd.sock imagePullPolicy: IfNotPresent imagePullSerial: true name: k8s-work02 修改work名称 taints: null timeouts: controlPlaneComponentHealthCheck: 4m0s discovery: 5m0s etcdAPICall: 2m0s kubeletHealthCheck: 4m0s kubernetesAPICall: 1m0s tlsBootstrap: 5m0s upgradeManifests: 5m0s
9、配置域名
/etc/hosts 新增配置：
172.21.173.7 k8s-master01
172.21.173.8 k8s-work01
172.21.173.9 k8s-work02
172.21.173.7 kube-apiserver
hostnamectl set-hostname k8s-work02
10、配置containerd
containerd config default > /etc/containerd/config.toml
修改配置文件：
sandbox_image = "registry.k8s.io/pause:3.8"
修改为："registry.aliyuncs.com/google_containers/pause:3.10 "
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
BinaryName = ""
CriuImagePath = ""
CriuPath = ""
CriuWorkPath = ""
IoGid = 0
IoUid = 0
NoNewKeyring = false
NoPivotRoot = false
Root = ""
ShimCgroup = ""
SystemdCgroup = false 修改为true
重启服务：
systemctl restart containerd
11、主节点加入集群（仅主节点）
kubeadm init --config kubeadm-config.yaml --upload-certs --v=3 --ignore-preflight-errors=all
Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.21.173.7:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:135c555a101094be165b82b891fa14607e339725843cfb2d36c428266566d409
12、主节点配置.kube/config
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
13、work节点加入集群（仅work节点）
kubeadm join --config kubeadm-config.yaml --ignore-preflight-errors=all
[preflight] Running pre-flight checks [WARNING FileExisting-socat]: socat not found in system path [WARNING FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s [kubelet-check] The kubelet is healthy after 1.00277492s [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
14、work节点加入验证
kubectl get nodes
NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 20h v1.31.1 k8s-work01 Ready <none> 23h v1.31.1 k8s-work02 Ready <none> 45m v1.31.1
kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE coredns-855c4dd65d-rtdqw 1/1 Running 0 39h coredns-855c4dd65d-tbbxl 1/1 Running 0 39h etcd-k8s-master01 1/1 Running 9 (39h ago) 34h kube-apiserver-k8s-master01 1/1 Running 50 (39h ago) 34h kube-controller-manager-k8s-master01 1/1 Running 16 (39h ago) 34h kube-proxy-5hzvx 1/1 Running 0 39h kube-proxy-7dh52 1/1 Running 0 14h kube-proxy-7vt2k 1/1 Running 1 (13h ago) 37h kube-scheduler-k8s-master01 1/1 Running 27 (39h ago) 34h
五、安装calico（仅主节点）
1、下载operator与custom-resource
wget https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/tigera-operator.yaml wget https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/custom-resources.yaml
2、配置pod网段
修改：custom-resources.yaml
apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: # Configures Calico networking. registry: m.daocloud.io # 修改为该地址，默认为docker.io外网下载不了 calicoNetwork: ipPools: - name: default-ipv4-ippool blockSize: 26 cidr: 10.244.0.0/16 # 修改为主节点配置pod网段一样，podSubnet: 10.244.0.0/16 encapsulation: VXLANCrossSubnet natOutgoing: Enabled nodeSelector: all() --- # This section configures the Calico API server. # For more information, see: https://docs.tigera.io/calico/latest/reference/installation/api#operator.tigera.io/v1.APIServer apiVersion: operator.tigera.io/v1 kind: APIServer metadata: name: default spec: {}
3、安装calico
kubectl create -f tigera-operator.yaml kubectl create -f custom-resources.yaml
4、查看calico状态
kubectl get po -n calico-system
NAME READY STATUS RESTARTS AGE calico-kube-controllers-6549cbf949-9gnhh 1/1 Running 0 22h calico-node-g8df6 1/1 Running 1 (13h ago) 22h calico-node-jwdqz 1/1 Running 0 22h calico-node-wrkf6 1/1 Running 0 13h calico-typha-67659f7458-6nlmt 1/1 Running 1 (13h ago) 13h calico-typha-67659f7458-8zgw7 1/1 Running 0 22h csi-node-driver-6sq7m 2/2 Running 0 13h csi-node-driver-l57xv 2/2 Running 2 (13h ago) 22h csi-node-driver-r5tr8 2/2 Running 0 22h
六、发布nginx服务到k8s集群环境验证
kubectl apply -f nginx.yaml
kubectl get po NAME READY STATUS RESTARTS AGE nginx 1/1 Running 0 15m
