15. 部署Harbor私有镜像仓库

> 说明⚠️： 独立部署和helm部署到k8s集群按实际需要均可部署，二选一部署即可

Harbor 是一个开源的容器镜像仓库，提供了企业级的镜像管理功能。

# 一、 独立部署harbor

## 1.1. 下载软件包

安装docker
- 略

安装docker-compose
- 略

下载harbor离线安装包：[GitHub仓库地址](https://github.com/goharbor/harbor "GitHub仓库地址")
```
[root@opt ~]# wget https://github.com/goharbor/harbor/releases/download/v2.10.1/harbor-offline-installer-v2.10.1.tgz
[root@opt ~]# tar -xvf harbor-offline-installer-v2.10.1.tgz 
```

## 1.2. 修改配置
```
[root@harbor ~]# cp harbor.yml.tmpl harbor.yml
[root@harbor ~]# vim harbor.yml
# 设置域名
hostname: harbor.local.com
# 注释https相关配置
# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80
```
>⚠️注意：示例未包含https访问，需自行配置

创建systemd服务管理脚本
```
tee >/lib/systemd/system/harbor.service<<EOF
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor

[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/docker-compose -f /opt/harbor/docker-compose.yml up
ExecReload=/usr/local/bin/docker-compose -f /opt/harbor/docker-compose.yml restart
ExecStop=/usr/local/bin/docker-compose -f /opt/harbor/docker-compose.yml down

[Install]
WantedBy=multi-user.target
EOF
```

```
systemctl enable harbor --now
```

## 1.3. 执行安装
运行install.sh脚本执行安装

```
[root@harbor  harbor]# ./install.sh 
[Step 5]: starting Harbor ...
[+] Building 0.0s (0/0)                                                                                                                                            
[+] Running 10/10
 ✔ Network harbor_harbor        Created                                                                                                                       0.3s 
 ✔ Container harbor-log         Started                                                                                                                       1.8s 
 ✔ Container harbor-portal      Started                                                                                                                      12.8s 
 ✔ Container registry           Started                                                                                                                      13.1s 
 ✔ Container redis              Started                                                                                                                      13.2s 
 ✔ Container registryctl        Started                                                                                                                      11.4s 
 ✔ Container harbor-db          Started                                                                                                                      12.0s 
 ✔ Container harbor-core        Started                                                                                                                      14.3s 
 ✔ Container nginx              Started                                                                                                                      18.2s 
 ✔ Container harbor-jobservice  Started                                                                                                                      18.1s 
 ✔ ----Harbor has been installed and started successfully.----
```

## 1.4. 访问登陆

初始用户名: `admin`
初始密码:   `Harbor12345`

![](/media/202407/2024-07-15_175837_3366160.4245474364360151.png)

# 二、helm部署harbor

## 2.1、添加 Helm 仓库

添加 Harbor 的 Helm 仓库：

```sh
helm repo add harbor https://helm.goharbor.io
```

## 2.2、下载并解压 Harbor
这将会在当前目录下生成一个 `harbor` 文件夹，包含所有需要的文件。
下载 Harbor Chart 并解压，以便进行必要的配置：

```sh
helm pull harbor/harbor --untar
```

## 2.3、配置 Harbor 服务的暴露方式

通过修改 `values.yaml` 文件来配置 Harbor 服务的暴露方式。以下是几种常见的暴露方式：

- **Ingress**: 需要在 Kubernetes 集群中安装 Ingress 控制器。如果禁用了 TLS，在拉取/推送镜像时必须在命令中包含端口。
- **ClusterIP**: 在集群内部的 IP 地址上暴露服务，使得服务只能在集群内部访问。
- **NodePort**: 在每个节点的 IP 地址上的静态端口暴露服务，可以通过 `NodeIP:NodePort` 从集群外部访问。
- **LoadBalancer**: 使用云提供商的负载均衡器对外暴露服务。

在 `values.yaml` 文件中找到以下配置部分，并根据您的需求进行修改：

```yaml
expose:
  type: ingress  # 可以是 ingress、clusterIP、nodePort 或 loadBalancer
  tls:
    enabled: true  # 如果使用 TLS，请设置为 true
  ingress:
    hosts:
      core: harbor.local.com  # 替换为您的域名
    className: "nginx" # 指定ingress使用nginx类
```

## 2.4、配置外部 URL

外部 URL 用于：

- 填充在门户中显示的 docker/helm 命令
- 填充返回给 docker 客户端的 token 服务 URL

格式为：`protocol://domain[:port]`。通常情况下：

- 如果通过 Ingress 暴露服务，域名应为 `expose.ingress.hosts.core` 的值。
- 如果通过 ClusterIP 暴露服务，域名应为 `expose.clusterIP.name` 的值。
- 如果通过 NodePort 暴露服务，域名应为一个 Kubernetes 节点的 IP 地址。
- 如果通过 LoadBalancer 暴露服务，设置为您的域名，并添加 CNAME 记录将域名映射到您从云提供商获得的域名。

在 `values.yaml` 文件中找到以下配置部分，并进行修改：

```yaml
externalURL: https://harbor.local.com  # 替换为您的外部 URL
```

## 2.5、配置数据持久化方式

您可以选择不同的数据持久化方式：

- **Disable**: 数据不会在 Pod 终止后保存。
- **Persistent Volume Claim (默认)**: 需要 Kubernetes 集群中有一个默认的 StorageClass 来动态地提供卷。您可以在 `values.yaml` 文件中指定另一个 StorageClass 或设置 `existingClaim` 使用已有的持久化卷。
- **External Storage**: 对于镜像和 chart，支持使用 Azure、GCS、S3、Swift 和 OSS 作为外部存储。

>本文示例采用NFS存储： StorageClass名称：nfs-storage
⚠️提示：如未存在StorageClass默认存储类，请参见[ **<部署NFS文件存储>** ](/doc/478/ "部署NFS文件存储")进行创建

创建 PersistentVolumeClaim供harbor使用
注意⚠️：
1. 每个pv只能由一个pvc绑定，Harbor需要创建多个pv和pvc
2. 根据实际需要确定pv和pvc大小
3. 确保指定的共享目录权限为777

```
tee > harbor/persistentVolumeClaim-Harbor.yaml <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-harbor-registry
  namespace: harbor
spec:
  capacity:
    storage: 50Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-storage
  nfs:
    server: 192.168.88.62
    path: /data/nfs/harbor-storage
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-harbor-jobservice
  namespace: harbor
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-storage
  nfs:
    server: 192.168.88.62
    path: /data/nfs/harbor-storage
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-harbor-database
  namespace: harbor
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-storage
  nfs:
    server: 192.168.88.62
    path: /data/nfs/harbor-storage
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-harbor-redis
  namespace: harbor
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-storage
  nfs:
    server: 192.168.88.62
    path: /data/nfs/harbor-storage
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-harbor-trivy
  namespace: harbor
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-storage
  nfs:
    server: 192.168.88.62
    path: /data/nfs/harbor-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-registry
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: nfs-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-jobservice
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: nfs-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-database
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-redis
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-trivy
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: nfs-storage
EOF
```

在 `values.yaml` 文件中找到以下配置部分，并进行修改：

```yaml
persistence:
  persistentVolumeClaim:
    registry:
      existingClaim: "harbor-registry" # 指定创建的pvc名称
      size: 50Gi  # 根据需要调整大小，和创建的pvc大小一致
    jobservice:
      jobLog:
        existingClaim: "harbor-jobservice" # 指定创建的pvc名称
        size: 1Gi
    database:
      existingClaim: "harbor-database"
      size: 1Gi
    redis:
      existingClaim: "harbor-redis"
      size: 1Gi
    trivy:
      existingClaim: "harbor-trivy"
      size: 5Gi
harborAdminPassword: "admin" # (可选)指定管理员密码，默认Harbor12345
```

## 2.6、安装 Harbor

创建harbor命名空间
```
kubectl create ns harbor
```

创建pvc
```
kubectl apply -f harbor/persistentVolumeClaim-Harbor.yaml
```

完成配置后，使用以下命令安装 Harbor到harbor命名空间下：

```sh
helm -n harbor install harbor ./harbor -f ./harbor/values.yaml
```

这将在 Kubernetes 集群中安装 Harbor，并使用您在 `values.yaml` 文件中配置的设置。

# 三、授权harbor仓库

## 3.1、 docker授权harbor仓库

docker配置文件私有仓库设置
```
[root@master ~]# vim /etc/docker/daemon.json
{
    "registry-mirrors": [
        "https://mirror.ccs.tencentyun.com",
        "https://o2j0mc5x.mirror.aliyuncs.com"
    ],
    "insecure-registries": [
        "https://harbor.local.com"
    ]
}
```
重启docker
```
systemctl daemon-reload 
systemctl restart docker
```

master节点登陆测试
```
[root@tiaoban ~]# docker login harbor.local.com -u admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
```

推送镜像测试
```
[root@tiaoban ~]# docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
5cc84ad355aa: Pull complete 
Digest: sha256:5acba83a746c7608ed544dc1533b87c737a0b0fb730301639a0179f9344b1678
Status: Downloaded newer image for busybox:latest
docker.io/library/busybox:latest
[root@tiaoban ~]# docker tag busybox:latest harbor.local.com/library/busybox:latest
[root@tiaoban ~]# docker push harbor.local.com/library/busybox:latest
The push refers to repository [harbor.local.com/library/busybox]
01fd6df81c8e: Pushed 
latest: digest: sha256:62ffc2ed7554e4c6d360bce40bbcf196573dd27c4ce080641a2c59867e732dee size: 527
```

## 3.2、Containerd授权访问harbor仓库

修改配置文件
```
[root@master1 ~]# mkdir -p /etc/containerd/certs.d/harbor.local.com
[root@master1 ~]# cat > /etc/containerd/certs.d/harbor.local.com/hosts.toml << EOF
server = "https://harbor.local.com"

[host."https://harbor.local.com"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = true
EOF
```

重启containerd
```
[root@master1 ~]# systemctl restart containerd 
```

登陆测试
```
[root@master1 ~]# nerdctl login -u admin -p Harbor12345 --insecure-registry harbor.local.com
WARN[0000] WARNING! Using --password via the CLI is insecure. Use --password-stdin. 
WARN[0000] skipping verifying HTTPS certs for "harbor.local.com" 
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
```

推送镜像测试
```
[root@master1 ~]# nerdctl pull busybox
[root@master1 ~]# nerdctl tag busybox:latest harbor.local.com/library/busybox:latest
[root@master1 ~]# nerdctl push --insecure-registry harbor.local.com/library/busybox:latest
```

# 四、kubernets访问harbor仓库

由于harbor采用了用户名密码认证，所以在镜像下载时，如果仓库类型为私有，则需要配置sercet才可以拉取镜像。

创建认证secret
```
kubectl create secret docker-registry registry-secret --namespace=default --docker-server=harbor.local.com --docker-username=admin --docker-password=Harbor12345
```

查看secret
```
# kubectl get secrets 
NAME              TYPE                             DATA   AGE
registry-secret   kubernetes.io/dockerconfigjson   1      9s
```

使用相应的私有registry中镜像的Pod资源的定义，即可通过imagePullSecrets字段使用此Secret对象
```
	apiVersion: v1
	kind: Pod 
	metadata:
	  name: secret-imagepull-demo
	  namespace: default
	spec:
	  imagePullSecrets:
	  - name: registry-secret
	  containers:
	  - image: 192.168.10.14/k8s/nginx:v1
	    name: myapp
```