5. 部署kubernets集群

# 一、安装k8s工具

```
# disableexcludes=kubernetes：禁掉除了这个kubernetes之外的别的仓库
yum install -y kubelet-1.24.13 kubeadm-1.24.13 kubectl-1.24.13 --disableexcludes=kubernetes
systemctl enable kubelet
systemctl start kubelet
```

- kubelet 运行在集群所有节点上，用于启动Pod和容器等对象的工具
- kubeadm 用于初始化集群，启动集群的命令工具
- kubectl 用于和集群通信的命令行，通过kubectl可以部署和管理应用，查看各种资源，创建、删除和更新各种组件

默认安装最新版，也可以指定老版本安装
```
yum list kubeadm --showduplicates | sort -r
yum install -y kubeadm-1.23.6 kubectl-1.23.6 kubelet-1.23.6
```

如果使用containerd作为容器运行时，需要指定kubelet的容器runtime，如果使用docker作为容器runtime，则无需操作下面的步骤。
```
[root@k8s-master ~]# crictl config runtime-endpoint /run/containerd/containerd.sock
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl restart kubelet
```

# 二、kubeadm部署集群(v1.20前)

以下操作在master节点执行

## 2.1. 配置文件创建集群

获取默认的初始化参数文件
```
# kubeadm config print init-defaults > kubeadm-conf.yaml
```

配置kubeadm-conf.yaml初始化文件
```
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  # 1️⃣ master节点ip地址，如果 Master 有多个interface，建议明确指定
  advertiseAddress: 11.1.1.133
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  # 2️⃣ master节点主机名
  name: k8s-master
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
# 3️⃣ k8s安装版本
kubernetesVersion: v1.19.16
# 4️⃣ 将其指定为阿里云镜像地址
imageRepository: "registry.aliyuncs.com/google_containers"
networking:
  dnsDomain: cluster.local
  # 5️⃣ Kubernetes 支持多种网络方案，而且不同网络方案对--pod-network-cidr 有自己的要求，这里设置为 10.244.0.0/16 是因为我们将使用flannel 网络方案，必须设置成这个 CIDR。
  podSubnet: "10.244.0.0/16"
  serviceSubnet: 10.96.0.0/12
scheduler: {}
# 6️⃣ 增加ipvs的管理
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
```

>⚠️注意：
> 1. **在第五点中的pod子网配置中，确保配置正确，误配置为10.224会导致网络插件无法正常运行**
> 2. **在增加ipvs管理时,该方式已弃用，请勿使用：featureGates: SupportIPVSProxyMode: true**

指定配置创建k8s集群

```
kubeadm init --config=kubeadm-conf.yaml
```

## 2.2. 命令行初始化创建k8s集群

```
kubeadm init \
--apiserver-advertise-address=11.1.1.133 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.23.6 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16
```
- `–apiserver-advertise-address`
指明用 Master 的哪个 interface 与 Cluster 的其他节点通信。如果 Master 有多个interface，建议明确指定，如果不指定，kubeadm 会自动选择有默认网关的interface。

- `–pod-network-cidr`
指定 Pod 网络的范围。Kubernetes 支持多种网络方案，而且不同网络方案对–pod-network-cidr 有自己的要求，这里设置为 10.244.0.0/16 是因为使用flannel 网络方案，必须设置成这个 CIDR。

- `–image-repository`
Kubenetes默认Registries地址是 k8s.gcr.io，在国内并不能访问gcr.io，在1.13版本中可以增加–image-repository参数，默认值是k8s.gcr.io，将其指定为阿里云镜像地址：registry.aliyuncs.com/google_containers。

- `–kubernetes-version=v1.23.6`
关闭版本探测，因为它的默认值是stable-1，会导致从https://dl.k8s.io/release/stable-1.txt下载最新的版本号

执行完毕后控制台打印以下信息：
```
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 11.1.1.133:6443 --token 8csadg.9yoyb9rrmjzqq54z \
        --discovery-token-ca-cert-hash sha256:b79ac5eac322bc3719668f5d3ac506bce3c5310a73a660a7523a9c7758d0010a
```

> 如初始化过程中遇到错误，则在修复后错误后，可使用如下命令重置后重新完成初始化	
```
# 如遇错误可重置后重新初始化
kubeadm reset
```

## 2.3. 初始化kubectl

根据提示初始化kubectl
```
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

## 2.4. 启用kubectl命令自动补全

```
yum -y install bash-completion
echo "source <(kubectl completion bash)" >> ~/.bashrc
source ~/.bashrc
```

## 2.5. 测试kubectl

```
[root@k8s-master k8s-install]# kubectl get node
NAME         STATUS     ROLES                  AGE   VERSION
k8s-master   NotReady   control-plane,master   46s   v1.19.16
```

# 三、kubeadm部署集群(v1.20后)

## 3.1. 变化说明

>从1.20开始，开启ipvs配置字段发生了变化，访问官方查看最新版本ipvs开启的正确配置,通过https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md可以看到官方说明
>从1.22开始，推荐使用containerd作为容器运行时。

## 3.2. 初始化配置文件

```
# cat kubeadm-conf.yaml 
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.10.10  #修改为控制节点IP（VIP）
  bindPort: 6443
nodeRegistration:
  criSocket: /run/containerd/containerd.sock  #使用containerd为容器运行时
	# criSocket: /var/run/dockershim.sock  #使用docker为容器运行时
  imagePullPolicy: IfNotPresent
  name: k8s-master     #修改为控制节点主机名
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers  #修改为阿里镜像地址
kind: ClusterConfiguration
kubernetesVersion: 1.24.13  #版本
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16   #指定Pod网段
  serviceSubnet: 10.96.0.0/12  #指定Service网段
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind:  KubeProxyConfiguration
mode: ipvs
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
```

如果初始化时出现以下报错，先停止master节点的kubelet
>error execution phase preflight: [preflight] Some fatal errors occurred:
>       [ERROR Port-10250]: Port 10250 is in use

```
[root@k8s-master k8s-install]# systemctl stop kubelet
```

初始化失败解决办法：

如果因为配置文件异常导致集群初始化失败，可执行如下命令
```
# kubeadm reset 
# rm -rf $HOME/.kube/config 
```

# 四、启用基于flannel的Pod网络

项目地址：https://github.com/flannel-io/flannel
下载配置文件

```
[root@master ~]# wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
```

>⚠️注意：
> 1. **启用前请确保ipvs已正确安装，安装和验证请参考<基础环境准备>**

启用flannel
```
[root@master ~]# kubectl apply -f kube-flannel.yml
```

验证操作
```
[root@k8s-master k8s-install]# kubectl get pods -A
NAMESPACE      NAME                                 READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-5tjkb                1/1     Running   0          9m1s
kube-system    coredns-6d56c8448f-vnrqf             1/1     Running   0          19m
kube-system    coredns-6d56c8448f-x9q75             1/1     Running   0          19m
kube-system    etcd-k8s-master                      1/1     Running   0          20m
kube-system    kube-apiserver-k8s-master            1/1     Running   0          20m
kube-system    kube-controller-manager-k8s-master   1/1     Running   0          20m
kube-system    kube-proxy-9df97                     1/1     Running   0          19m
kube-system    kube-scheduler-k8s-master            1/1     Running   0          20m
```

# 五、其他node节点加入集群

将节点加入到集群
```
kubeadm join 11.1.1.133:6443 --token 8csadg.9yoyb9rrmjzqq54z \
  --discovery-token-ca-cert-hash \
  sha256:b79ac5eac322bc3719668f5d3ac506bce3c5310a73a660a7523a9c7758d0010a

```

>控制台信息被清理后的token获取方式
```
# token过期时，可重新创建
kubeadm token create

# 控制台token输出被清理后可通过该命令查看
kubeadm token list
```
```
# 获取discovery-token-ca-cert-hash值
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt |\
openssl rsa -pubin -outform der 2>/dev/null|\
openssl dgst -sha256 -hex|\
sed 's/^.* /sha256:/'
```

查看集群信息
```
[root@k8s-master ~]# kubectl get node
NAME         STATUS   ROLES                  AGE     VERSION
k8s-master   Ready    control-plane,master   5m26s   v1.24.13
k8s-work1    Ready    <none>                 3m39s   v1.24.13
k8s-work2    Ready    <none>                 3m48s   v1.24.13
[root@k8s-master ~]# kubectl get pod -A
NAMESPACE      NAME                                 READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-22hrr                1/1     Running   2          17m
kube-flannel   kube-flannel-ds-5tjkb                1/1     Running   2          36m
kube-flannel   kube-flannel-ds-kmtnk                1/1     Running   0          84s
kube-system    coredns-6d56c8448f-vnrqf             1/1     Running   3          47m
kube-system    coredns-6d56c8448f-x9q75             1/1     Running   2          47m
kube-system    etcd-k8s-master                      1/1     Running   2          47m
kube-system    kube-apiserver-k8s-master            1/1     Running   2          47m
kube-system    kube-controller-manager-k8s-master   1/1     Running   2          47m
kube-system    kube-proxy-6wmsl                     1/1     Running   2          17m
kube-system    kube-proxy-9df97                     1/1     Running   2          47m
kube-system    kube-proxy-fkkm6                     1/1     Running   0          84s
kube-system    kube-scheduler-k8s-master            1/1     Running   2          47m
```