6. 集群架构、安装和配置

# 访问控制（RBAC）

## 参考文档
[https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/rbac/](https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/rbac/)

## 例题
创建一个名为deployment-clusterrole且仅允许创建以下资源类型的新ClusterRole： Deployment StatefulSet DaemonSet 在现有的 namespace app-team1中创建一个名为cicd-token的新 ServiceAccount。 限于 namespace app-team1中，将新的ClusterRole deployment-clusterrole绑定到新的 ServiceAccount cicd-token。

## 解题思路
RBAC基本概念

- Role：角色，它其实是一组规则，定义了一组对 Kubernetes API 对象的操作权限。
- Subject：被作用者，既可以是“人”，也可以是“机器”，也可以使你在 Kubernetes 里定义的“用户”。
- RoleBinding：定义了“被作用者”和“角色”的绑定关系。

![image.png](/media/202406/18.6.124319802_image1.png) 解题步骤

- 创建ClusterRole，并指定操作权限操作资源以及名称
- 创建ServiceAccount，指定名称和名称空间
- 创建ClusterRoleBinding ，将ServiceAccount和ClusterRole绑定
- 验证权限

## 答案
使用yaml文件创建
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cicd-token
  namespace: app-team1
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployment-clusterrole
rules:
- apiGroups: [""]
  resources: ["deployment", "statefulset", "daemonset"]
  verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
	name: deployment-rolebinding
	namespace: app-team1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: deployment-clusterrole
subjects:
  - kind: ServiceAccount
    name: cicd-token
    namespace: app-team1
```
使用命令行创建
```yaml
# 创建clusterrole
[root@k8s-master ~]# kubectl create clusterrole deployment-clusterrole --verb=create --resource=deployment,statefulset,daemonset
clusterrole.rbac.authorization.k8s.io/deployment-clusterrole created
# 创建serviceaccount
[root@k8s-master ~]# kubectl create serviceaccount cicd-token -n app-team1
serviceaccount/cicd-token created  
# 创建rolebinding
[root@k8s-master ~]# kubectl create rolebinding cicd-token-rolebinding --serviceaccount=app-team1:cicd-token --clusterrole=deployment-clusterrole -n app-team1
rolebinding.rbac.authorization.k8s.io/cicd-token-rolebinding created
```
测试权限
```yaml
[root@k8s-master ~]# kubectl --as=system:serviceaccount:app-team1:cicd-token get pods -n app-team1
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:app-team1:cicd-token" cannot list resource "pods" in API group "" in the namespace "app-team1"
```

## 答案
```yaml
# 将节点停止调度
[root@k8s-master ~]# kubectl cordon k8s-work2 
# 驱逐节点
[root@k8s-master ~]# kubectl drain k8s-work2 
node/k8s-work2 cordoned
error: unable to drain node "k8s-work2" due to error:[cannot delete Pods declare no controller (use --force to override): default/test, cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-flannel/kube-flannel-ds-dq27j, kube-system/kube-proxy-dl77v], continuing command...
There are pending nodes to be drained:
 k8s-work2
cannot delete Pods declare no controller (use --force to override): default/test
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-flannel/kube-flannel-ds-dq27j, kube-system/kube-proxy-dl77v
# 根据提示添加选项
[root@k8s-master ~]# kubectl drain ek8s-node-1 --force --ignore-daemonsets
```

# 升级k8s版本

## 参考文档
[https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/](https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)

## 例题
现有的kubernetes集群正在运行版本：1.24.13。仅将主节点上（k8s-master）的所有kubernetes控制平面和节点组件升级到版本1.24.14 确保在升级之前drain主节点，并在升级后uncordon主节点。 另外，在主节点上升级kubelet和kubectl

## 解题思路
升级工作的基本流程如下：

1. 升级主控制平面节点
2. 升级其他控制平面节点
3. 升级工作节点

节点升级基本步骤如下：

1. 将节点设置为不可用
2. 安装指定版本的kubeadm
3. 执行upgrade命令查看升级计划
4. 执行upgrade升级k8s组件
5. 升级kubelet和kubectl
6. 重启kubelet
7. 将节点设置为可调度
8. 查看升级结果

## 答案
```bash
# 将节点设置为不可用
[root@k8s-master ~]# kubectl drain k8s-master
node/k8s-master cordoned
error: unable to drain node "k8s-master" due to error:cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-flannel/kube-flannel-ds-45rxr, kube-system/kube-proxy-fl2cq, continuing command...
There are pending nodes to be drained:
 k8s-master
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-flannel/kube-flannel-ds-45rxr, kube-system/kube-proxy-fl2cq
[root@k8s-master ~]# kubectl drain k8s-master --ignore-daemonsets

# 安装指定版本的kubeadm
# 实际考试中需要sudo -i，然后执行apt install kubeadm=1.24.14-00
[root@k8s-master ~]# dnf -y install kubeadm-1.24.14

# 查看升级计划
[root@k8s-master ~]# kubeadm upgrade plan
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT        TARGET
kubelet     3 x v1.24.13   v1.24.14

Upgrade to the latest version in the v1.24 series:

COMPONENT                 CURRENT    TARGET
kube-apiserver            v1.24.13   v1.24.14
kube-controller-manager   v1.24.13   v1.24.14
kube-scheduler            v1.24.13   v1.24.14
kube-proxy                v1.24.13   v1.24.14
CoreDNS                   v1.8.6     v1.8.6
etcd                      3.5.6-0    3.5.6-0

You can now apply the upgrade by executing the following command:

kubeadm upgrade apply v1.24.14

# 执行升级操作，可以指定哪些组件不需要升级,例如etcd
[root@k8s-master ~]# kubeadm upgrade apply v1.24.14 --etcd-upgrade=false
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.24.14". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven\'t already done so.

# 升级kubelet和kubectl
[root@k8s-master ~]# dnf -y install kubelet-1.24.14 kubectl-1.24.14

# 重启kubelet
[root@k8s-master ~]# systemctl restart kubelet
[root@k8s-master ~]# systemctl status kubelet

# 设置节点为可调度
[root@k8s-master ~]# kubectl uncordon k8s-master
node/k8s-master uncordoned

# 验证升级结果
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane 4d21h v1.24.14
k8s-work1 Ready <none> 4d21h v1.24.13
k8s-work2 Ready <none> 4d21h v1.24.13
```

# etcd备份与恢复

## 参考文档
[https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/configure-upgrade-etcd/](https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/configure-upgrade-etcd/)

## 例题
为运行在https://127.0.0.1:2379上的现有etcd实例创建快照并将快照保存到/data/backup/etcd-snapshot.db 然后还原位于/data/backup/etcd-snapshot-previous.db的现有先前快照
> 提供了以下TLS证书和密钥，以通过etcdctl连接到服务器
> CA证书：/etc/etcd/pki/ca.pem
> 客户端证书：/etc/etcd/pki/client.pem
> 客户端密钥：/etc/etcd/pki/client-key.pem

## 解题思路
执行备份和恢复命令时，一定要指定etcdctl_api版本为3，否则报错。 在恢复etcd数据时，先停止etcd服务，然后查看数据目录和用户组，然后将原来的数据移走，最后再恢复数据启动服务。

## 答案
```bash
# 备份
[root@k8s-master ~]# ETCDCTL_API=3 etcdctl snapshot save /data/backup/etcd-snapshot.db --endpoints=https://127.0.0.1:2379 --cacert=/etc/etcd/pki/ca.pem --cert=/etc/etcd/pki/client.pem --key=/etc/etcd/pki/client-key.pem
{"level":"info","ts":"2023-05-16T10:56:39.347+0800","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/data/backup/etcd-snapshot.db.part"}
{"level":"info","ts":"2023-05-16T10:56:39.496+0800","logger":"client","caller":"v3@v3.5.6/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-05-16T10:56:39.497+0800","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-05-16T10:56:40.384+0800","logger":"client","caller":"v3@v3.5.6/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-05-16T10:56:40.449+0800","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"5.1 MB","took":"1 second ago"}
{"level":"info","ts":"2023-05-16T10:56:40.449+0800","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/data/backup/etcd-snapshot.db"}
Snapshot saved at /data/backup/etcd-snapshot.db
[root@k8s-master ~]# ls -lh /data/backup/etcd-snapshot.db
-rw------- 1 root root 4.9M 5月  16 10:56 /data/backup/etcd-snapshot.db

# 恢复 
[root@k8s-work1 ~]# systemctl stop etcd 
[root@k8s-work1 ~]# systemctl cat etcd # 确认下数据目录（--data-dir值） 
[root@k8s-work1 ~]# mv /var/lib/etcd /var/lib/etcd.bak 
[root@k8s-work1 ~]# ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/etcd/pki/ca.pem --cert=/etc/etcd/pki/client.pem --key=/etc/etcd/pki/client-key.pem snapshot restore --data-dir=/var/lib/etcd /data/backup/etcd-snapshot-previous.db
{"level":"info","ts":1684658606.325091,"caller":"snapshot/v3_snapshot.go:296","msg":"restoring snapshot","path":"/data/backup/etcd-snapshot-previous.db","wal-dir":"/var/lib/etcd/member/wal","data-dir":"/var/lib/etcd","snap-dir":"/var/lib/etcd/member/snap"}
{"level":"info","ts":1684658606.4937744,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"0","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":1684658606.5958042,"caller":"snapshot/v3_snapshot.go:309","msg":"restored snapshot","path":"/data/backup/etcd-snapshot-previous.db","wal-dir":"/var/lib/etcd/member/wal","data-dir":"/var/lib/etcd","snap-dir":"/var/lib/etcd/member/snap"}
# 考试是所属组可能不一致，记得调整
[root@k8s-work1 ~]# chown -R etcd:etcd /var/lib/etcd 
[root@k8s-work1 ~]# systemctl start etcd
```

# 设置节点不可用

## 参考文档
[https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/safely-drain-node/](https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/safely-drain-node/)

## 例题
将名为k8s-work2的node设置为不可用，并重新调度该node上所有运行的pods

## 解题思路
在对节点执行维护（例如内核升级、硬件维护等）之前， 可以先使用kubectl cordon命令，将节点停止调度，然后执行kubectl drain 从节点安全地逐出所有 Pod。 如果执行drain提示错误，根据提示再加上选项，例如--delete-local-data --force drain的参数

- –force：当一些pod不是经 ReplicationController, ReplicaSet, Job, DaemonSet 或者 StatefulSet 管理的时候就需要用–force来强制执行 (例如:kube-proxy)
- –ignore-daemonsets：无视DaemonSet管理下的Pod
- –delete-local-data：如果有mount local volumn的pod，会强制杀掉该pod并把料清除掉

## 答案
```bash
[root@k8s-master ~]# kubectl cordon k8s-work2
node/k8s-work2 cordoned
[root@k8s-master ~]# kubectl drain k8s-work2
node/k8s-work2 already cordoned
error: unable to drain node "k8s-work2" due to error:[cannot delete Pods declare no controller (use --force to override): default/filebeat, default/kucc4, default/web-server, cannot delete Pods with local storage (use --delete-emptydir-data to override): default/legacy-app, cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-flannel/kube-flannel-ds-dq27j, kube-system/kube-proxy-4xbgn], continuing command...
There are pending nodes to be drained:
 k8s-work2
cannot delete Pods declare no controller (use --force to override): default/filebeat, default/kucc4, default/web-server
cannot delete Pods with local storage (use --delete-emptydir-data to override): default/legacy-app
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-flannel/kube-flannel-ds-dq27j, kube-system/kube-proxy-4xbgn
[root@k8s-master ~]# kubectl drain k8s-work2 --force --delete-emptydir-data --ignore-daemonsets
```