本文记录使用Velero Restic快速完成云原生应用迁移至ACK集群的实践过程。

0. 实践步骤概览

（1）创建GKE集群（或自建Kubernetes集群）
（2）在GKE集群上部署示例应用Jenkins Application并执行一个构建任务
（3）创建ACK集群
（4）在ACK集群中部署Minio Service用于应用迁移时数据中转服务
（5）在GKE上部署Velero并备份整个Jenkins Application
（6）把Jenkins Application使用的容器镜像同步到阿里云容器镜像仓库
（7）在ACK上创建Jenkins Application所使用的StorageClass
（8）在ACK上部署Velero并恢复整个Jenkins Application
（9）在ACK上替换Jenkins Application所使用的容器镜像
（10）验证Jenkins Application能否正常提供服务及其构建任务是否存在

1. 环境物料清单

GKE集群（或自建Kubernetes集群）
ACK集群

2. 创建GKE集群并部署示例应用Jenkins Application

(1)在MarketPlace中找到Jenkins应用并配置部署到GKE集群的jenkins命名空间下：

(2)应用正在完成部署：

(3)查看Ingress并访问jenkins服务：

(4)Jenkins应用的初始化配置：

(5)Jenkin应用上创建一个名为gke-to-ack的任务并执行构建：

（6）Jenkins应用的k8s资源列表如下

$ kubectl -n jenkins get deploy
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
jenkins-jenkins-deployment   1         1         1            1           177m
$ kubectl -n jenkins get svc
NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
jenkins-jenkins-agents-connector   ClusterIP   10.39.241.75    <none>        50000/TCP        177m
jenkins-jenkins-ui                 NodePort    10.39.253.141   <none>        8080:31759/TCP   177m
$ kubectl -n jenkins get ing
NAME                 HOSTS   ADDRESS        PORTS     AGE
jenkins-jenkins-ui   *       xx.xx.xx.xx   80, 443   177m
$ kubectl -n jenkins get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
jenkins-jenkins-pvc   Bound    pvc-e64eacc4-a482-11e9-bfa9-42010a8000da   8Gi        RWO            standard       177m
$ kubectl -n jenkins get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                       STORAGECLASS   REASON   AGE
pvc-e64eacc4-a482-11e9-bfa9-42010a8000da   8Gi        RWO            Delete           Bound    jenkins/jenkins-jenkins-pvc                                 standard                177m

3. 创建ACK集群并部署Minio对象存储应用

(1) 自定义替换minio-deploy.yaml中的 MINIO_ACCESS_KEY 和 MINIO_SECRET_KEY值并部署：

kubectl apply -f minio-deploy.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: minio

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  namespace: minio
  name: minio
  labels:
    component: minio
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        component: minio
    spec:
      volumes:
      - name: storage
        emptyDir: {}
      - name: config
        emptyDir: {}
      containers:
      - name: minio
        image: minio/minio:latest
        imagePullPolicy: IfNotPresent
        args:
        - server
        - /storage
        - --config-dir=/config
        env:
        - name: MINIO_ACCESS_KEY
          value: "<your MINIO_ACCESS_KEY>"
        - name: MINIO_SECRET_KEY
          value: "your MINIO_SECRET_KEY"
        ports:
        - containerPort: 9000
        volumeMounts:
        - name: storage
          mountPath: "/storage"
        - name: config
          mountPath: "/config"

---
apiVersion: v1
kind: Service
metadata:
  namespace: minio
  name: minio
  labels:
    component: minio
spec:
  # ClusterIP is recommended for production environments.
  # Change to NodePort if needed per documentation,
  # but only if you run Minio in a test/trial environment, for example with Minikube.
  type: LoadBalancer
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    component: minio

(2) 查看minio LoadBalancer信息并登陆应用：

$ kubectl -n minio get svc
NAME    TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)          AGE
minio   LoadBalancer   172.21.2.216   xxx.xxx.xxx.xx   9000:30912/TCP   2d4h

(3) 替换minio-job.yaml中的minio_server_url minino_access_key minio_secret_key并运行job在minio中创建名为velero的bucket：

kubectl apply -f minio-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  namespace: minio
  name: minio-setup
  labels:
    component: minio
spec:
  template:
    metadata:
      name: minio-setup
    spec:
      restartPolicy: OnFailure
      volumes:
      - name: config
        emptyDir: {}
      containers:
      - name: mc
        image: minio/mc:latest
        imagePullPolicy: IfNotPresent
        command:
        - /bin/sh
        - -c
        - "mc --config-dir=/config config host add velero <your minio_server_url> <your minino_access_key> <your minio_secret_key> && mc --config-dir=/config mb -p velero/velero"
        volumeMounts:
        - name: config
          mountPath: "/config"

(4) 查看bucket是否创建成功：

4. 在GKE和ACK上部署velero

(1) 安装velero client：
请从official release下载最新版本的velero客户端：

(2) 安装velero server：
创建credentials-velero文件并设置aws_access_key_id 与 aws_secret_access_key的值

[default]
aws_access_key_id = <your minio_access_key>
aws_secret_access_key = <your minio_secret_key>

ACK集群中请替换minio_server_url的值并指定image参数部署velero server

velero install     --provider aws    --image registry.cn-hangzhou.aliyuncs.com/acs/velero:latest  --bucket velero     --secret-file ./credentials-velero     --use-volume-snapshots=false     --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=<your minio_server_url> --use-restic     --wait

GKE中请替换minio_server_url的值并部署velero server

velero install     --provider aws    --bucket velero     --secret-file ./credentials-velero     --use-volume-snapshots=false     --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=<your minio_server_url> --use-restic     --wait

5. 在GKE集群中备份Jenkins Application

(1)在备份带volume信息的pod之前，我们要通过给pod加annotation来告诉velero哪些pod需要包含volume数据， 查看jenkins应用的Deployment资源：

$ kubectl -n jenkins get deploy jenkins-jenkins-deployment -oyaml

(2)为pod加annotation:

$ kubectl -n jenkins get po
NAME                                          READY   STATUS      RESTARTS   AGE
jenkins-deployer-zh5p4                        0/1     Completed   0          109m
jenkins-jenkins-deployment-7df86c64d4-tqqlr   1/1     Running     0          109m
$ kubectl -n jenkins annotate pod/jenkins-jenkins-deployment-7df86c64d4-tqqlr backup.velero.io/backup-volumes=jenkins-jenkins-pvc
pod/jenkins-jenkins-deployment-7df86c64d4-tqqlr annotated

(3)创建备份

$ velero backup create gcloud-jenkins-backup-restic --include-namespaces jenkins --wait
Backup request "gcloud-jenkins-backup-restic" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
.......................
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe gcloud-jenkins-backup-restic` and `velero backup logs gcloud-jenkins-backup-restic`.

(4) 查看备份

$ ./velero backup get
NAME                           STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
gcloud-jenkins-backup-restic   Completed   2019-07-12 18:48:48 +0800 +08   29d       default            <none>

（5）登录Minio Server查看

5. 同步（或批量）迁移容器镜像

把Jenkins应用使用的容器镜像导入到阿里云容器镜像仓库，导入到的地址为：

registry.cn-hangzhou.aliyuncs.com/haoshuwei/jenkins:2.150.3

批量迁移可参考：https://github.com/AliyunContainerService/sync-repo.git

6. 在ACK集群中创建Jenkins应用所使用的StorageClass standard

$ kubectl apply -f storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: standard
provisioner: alicloud/disk
parameters:
  type: cloud
reclaimPolicy: Delete

7. 在ACK集群中恢复Jenkins Application

查看备份信息：

$ velero  backup get
NAME                           STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
gcloud-jenkins-backup-restic   Completed   2019-07-12 18:48:48 +0800 CST   29d       default            <none>

恢复应用：

$ velero restore create --from-backup gcloud-jenkins-backup-restic

此处需要编辑deployment修改image为registry.cn-hangzhou.aliyuncs.com/haoshuwei/jenkins:2.150.3

查看restore的状态：

$ velero restore describe gcloud-jenkins-backup-restic-20190712190536
$ velero restore logs gcloud-jenkins-backup-restic-20190712190536

restore完成后的状态为：

$ velero restore get
NAME                                          BACKUP                         STATUS      WARNINGS   ERRORS   CREATED                         SELECTOR
gcloud-jenkins-backup-restic-20190712190536   gcloud-jenkins-backup-restic   Completed   0          0        2019-07-12 19:05:36 +0800 CST   <none>

8. 查看ACK集群上jenkins应用的ingress并访问服务进行验证

$ kubectl -n jenkins get ing
NAME                 HOSTS   ADDRESS          PORTS     AGE
jenkins-jenkins-ui   *       xx.xx.xx.xx   80, 443   56m

至此，一个带pv存储的jenkins应用被完整地从GKE迁移到ACK上。

参考链接：
https://velero.io/docs/v1.0.0/get-started/
https://velero.io/docs/v1.0.0/restic/#troubleshooting
https://github.com/heptio/velero
https://github.com/AliyunContainerService/velero-plugin