WorkingTipsOnGpu

1. 环境配置信息

整个验证环境的配置信息如下:

gpumaster: 10.168.100.2	4核16G
gpunode1: 10.168.100.3	4核16G PCI直通B5:00 Tesla V100
gpunode2: 10.168.100.4	4核16G PCI直通B2:00 Tesla V100

节点的操作系统配置如下, CentOS 7.6最小化安装方式:

# uname -a
Linux gpumaster 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

其中master节点上外挂了一块500G 的数据盘，需要手动挂载至/dcos目录:

[root@gpumaster ~]# df -h | grep dcos
/dev/vdb1                493G   73M  467G   1% /dcos
[root@gpumaster ~]# cat /etc/fstab | grep dcos
/dev/vdb1        /dcos                       ext4       defaults        0 0

3个节点依次关闭selinux/firewalld:

# vi /etc/selinux/config
...
SELINUX=disabled
...
# systemctl disable firewalld
# reboot

2. 部署CCSE集群

依次添加节点:

新增一个名为gpucluster的集群:

集群创建完毕后，新增两个GPU节点：

添加完成后，检查集群状态:

[root@gpumaster ~]# kubectl get node
NAME           STATUS   ROLES    AGE     VERSION
10.168.100.2   Ready    master   6m19s   v1.17.3
10.168.100.3   Ready    node     78s     v1.17.3
10.168.100.4   Ready    node     78s     v1.17.3

3. 升级内核

在三个节点上，依次执行以下操作以升级内核。

配置离线软件库:

# cd /etc/yum.repos.d
# mkdir back
# mv CentOS-* back
# vi nvidia.repo
[nvidia]
name=nvidia
baseurl=http://10.168.100.144:8200/repo/x86_64/nvidiarpms
gpgcheck=0
enabled=1
proxy=_none_
# yum install -y kernel-ml

配置grub启动:

# vi /etc/default/grub
...
GRUB_DEFAULT=0
...
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet rd.driver.blacklist=nouveau nouveau.modeset=0"
...
# grub2-mkconfig -o /boot/grub2/grub.cfg

完全禁用系统自带的nouveau驱动:

# echo 'install nouveau /bin/false' >> /etc/modprobe.d/nouveau.conf

执行完上述操作后需重启机器并验证内核是否更改成功:

# uname -a
Linux gpunode2 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

4. gpu-operator文件准备

Harbor中预上传的镜像文件列表如下(nvcr.io及nvidia):

从10.168.100.1上scp以下目录到所有节点:

$ scp -r docker@10.168.100.1:/home/docker/nvidia_items .

预Load nfd镜像:

# docker load<quay.tar
...
Loaded image: quay.io/kubernetes_incubator/node-feature-discovery:v0.6.0

5. 安装NVIDIA/gpu-operator

登录到gpumaster节点，从文件创建一个部署charts时需用到的configmap:

# cat ccse.repo
[ccse-k8s]
name=Centos local yum repo for k8s
baseurl=http://10.168.100.144:8200/repo/x86_64/k8s-offline-pkgs
gpgcheck=0
enabled=1
proxy=_none_

[ccse-centos7-base]
name=Centos local yum repo for k8s
baseurl=http://10.168.100.144:8200/repo/x86_64/centos7-base
gpgcheck=0
enabled=1
proxy=_none_

[fuck]
name=Centos local yum repo for k8s 111
baseurl=http://10.168.100.144:8200/repo/x86_64/nvidiarpms
gpgcheck=0
enabled=1
proxy=_none_
# kubectl create namespace gpu-operator-resources
namespace/gpu-operator-resources created
# kubectl create configmap repo-config -n gpu-operator-resources --from-file=ccse.repo
configmap/repo-config created

现在创建gpu-operator实例:

# cd gpu-operator/
#  helm install --generate-name . -f values.yaml

检查实例运行情况:

# kubectl get po
NAME                                                              READY   STATUS    RESTARTS   AGE
chart-1618803326-node-feature-discovery-master-655c6997cd-fp465   1/1     Running   0          65s
chart-1618803326-node-feature-discovery-worker-7flft              1/1     Running   0          65s
chart-1618803326-node-feature-discovery-worker-mkqm7              1/1     Running   0          65s
chart-1618803326-node-feature-discovery-worker-w2d44              1/1     Running   0          65s
gpu-operator-945878fff-l22vc                                      1/1     Running   0          65s

给GPU节点手动添加标签，gpu-operator-resources命名空间下的实例运行情况:

使能GPU驱动安装:

# kubectl label nodes 10.168.100.3 nvidia.com/gpu.deploy.driver=true       
node/10.168.100.3 labeled
# kubectl label nodes 10.168.100.4 nvidia.com/gpu.deploy.driver=true       
node/10.168.100.4 labeled

检查GPU驱动编译情况:

# kubectl  get po -n gpu-operator-resources
NAME                            READY   STATUS    RESTARTS   AGE
nvidia-driver-daemonset-w6d2q   1/1     Running   0          86s
nvidia-driver-daemonset-zmf9l   1/1     Running   0          86s
# kubectl logs po nvidia-driver-daemonset-zmf9l -n gpu-operator-resources
Installation of the kernel module for the NVIDIA Accelerated Graphics Driver for Linux-x86_64 (version 460.32.03) is now complete.

Loading IPMI kernel module...
Loading NVIDIA driver kernel modules...
Starting NVIDIA persistence daemon...
Mounting NVIDIA driver rootfs...
Done, now waiting for signal

使能device-plugin, dcgm-exporter等：

# kubectl label nodes 10.168.100.4 nvidia.com/gpu.deploy.container-toolkit=true
# kubectl label nodes 10.168.100.4 nvidia.com/gpu.deploy.device-plugin=true
# kubectl label nodes 10.168.100.4 nvidia.com/gpu.deploy.dcgm-exporter=true
# kubectl label nodes 10.168.100.4 nvidia.com/gpu.deploy.gpu-feature-discovery=true

# kubectl label nodes 10.168.100.3 nvidia.com/gpu.deploy.container-toolkit=true
# kubectl label nodes 10.168.100.3 nvidia.com/gpu.deploy.device-plugin=true
# kubectl label nodes 10.168.100.3  nvidia.com/gpu.deploy.dcgm-exporter=true
# kubectl label nodes 10.168.100.3 nvidia.com/gpu.deploy.gpu-feature-discovery=true

检查toolkit-daemonset运行情况，会发现Init:ImagePullBackOff报错信息:

# kubectl get po -n gpu-operator-resources
NAME                                       READY   STATUS                  RESTARTS   AGE
nvidia-container-toolkit-daemonset-6kqq5   0/1     Init:ImagePullBackOff   0          2m16s
nvidia-container-toolkit-daemonset-cbww2   0/1     Init:ImagePullBackOff   0          4m1s
# kubectl logs nvidia-container-toolkit-daemonset-cbww2 -n gpu-operator-resources
  Normal   BackOff         3m31s (x7 over 4m46s)  kubelet, 10.168.100.4  Back-off pulling image "10.168.100.144:8021/nvcr.io/nvidia/k8s/cuda@sha256:ed723a1339cddd75eb9f2be2f3476edf497a1b189c10c9bf9eb8da4a16a51a59"
  Warning  Failed          3m31s (x7 over 4m46s)  kubelet, 10.168.100.4  Error: ImagePullBackOff
  Normal   Pulling         3m20s (x4 over 4m48s)  kubelet, 10.168.100.4  Pulling image "10.168.100.144:8021/nvcr.io/nvidia/k8s/cuda@sha256:ed723a1339cddd75eb9f2be2f3476edf497a1b189c10c9bf9eb8da4a16a51a59"

这是因为pod拉取的镜像tag不对所导致，需要手动修改image的tag:

# kubectl get ds -n gpu-operator-resources
NAME                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                  AGE
nvidia-container-toolkit-daemonset   2         2         0       2            0           nvidia.com/gpu.deploy.container-toolkit=true   133m
nvidia-driver-daemonset              2         2         2       2            2           nvidia.com/gpu.deploy.driver=true              135m
# kubectl edit ds nvidia-container-toolkit-daemonset -n gpu-operator-resources
        #image: 10.168.100.144:8021/nvcr.io/nvidia/k8s/cuda@sha256:ed723a1339cddd75eb9f2be2f3476edf497a1b189c10c9bf9eb8da4a16a51a59
        image: 10.168.100.144:8021/nvcr.io/nvidia/cuda:11.2.1-base-ubi8

刷新pod运行情况，可以看到nvidia-container-toolkit-daemonset及nvidia-device-plugin-daemonset运行正常，而nvidia-device-plugin-validation则Init:CreashLoopBackOff失败:

# kubectl get po -n gpu-operator-resources
NAME                                       READY   STATUS                  RESTARTS   AGE
nvidia-container-toolkit-daemonset-27qj8   1/1     Running                 0          52s
nvidia-container-toolkit-daemonset-g5ndb   1/1     Running                 0          51s
nvidia-device-plugin-daemonset-sqfdc       1/1     Running                 0          26s
nvidia-device-plugin-daemonset-wldkd       1/1     Running                 0          26s
nvidia-device-plugin-validation            0/1     Init:CrashLoopBackOff   1          9s
nvidia-driver-daemonset-m4xjv              1/1     Running                 0          137m
nvidia-driver-daemonset-vkrz5              1/1     Running                 5          137m

定位该validation所在的节点名(此例中为10.168.100.3):

# kubectl get po nvidia-device-plugin-validation -n  gpu-operator-resources -o wide
NAME                              READY   STATUS                  RESTARTS   AGE     IP              NODE           NOMINATED NODE   READINESS GATES
nvidia-device-plugin-validation   0/1     Init:CrashLoopBackOff   4          2m55s   172.26.222.10   10.168.100.3   <none>           <none>

获取启动失败原因:

# kubectl describe po nvidia-device-plugin-validation -n gpu-operator-resources
......
  Warning  Failed            56s (x5 over 2m21s)   kubelet, 10.168.100.3  Error: failed to start container "device-plugin-validation-init": Error response from daemon: linux runtime spec devices: error gathering device information while adding custom device "/dev/nvidiactl": no such file or directory

登录10.168.100.3节点，获取/dev下驱动程序设备名:

# docker ps | grep nvidia-device-plugin-daemonset | grep -v pause
abbea480fdf2        10.168.100.144:8021/nvcr.io/nvidia/k8s-device-plugin       "nvidia-device-plugin"   6 minutes ago       Up 6 minutes                            k8s_nvidia-device-plugin-ctr_nvidia-device-plugin-daemonset-sqfdc_gpu-operator-resources_b9988b02-82a6-4637-a7f0-fdee5a448d60_0
# docker exec -it k8s_nvidia-device-plugin-ctr_nvidia-device-plugin-daemonset-sqfdc_gpu-operator-resources_b9988b02-82a6-4637-a7f0-fdee5a448d60_0 /bin/bash
[root@nvidia-device-plugin-daemonset-sqfdc /]# ls /dev/nvidia* -l -h
crw-rw-rw- 1 root root 195, 254 Apr 19 03:52 /dev/nvidia-modeset
crw-rw-rw- 1 root root 237,   0 Apr 19 06:08 /dev/nvidia-uvm
crw-rw-rw- 1 root root 237,   1 Apr 19 06:08 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 Apr 19 03:52 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Apr 19 03:52 /dev/nvidiactl
[root@nvidia-device-plugin-daemonset-sqfdc /]# exit

在主机级别(10.168.100.3)上手动创建/dev/nvidiactl文件, 依据同样步骤在10.168.100.4上查找到相应的设备驱动号也添加/dev/nvidiactl文件:

[root@gpunode1 ~]# mknod -m 666 /dev/nvidiactl c 195 255
[root@gpunode1 ~]# ls /dev/nvidiactl -l
crw-rw-rw- 1 root root 195, 255 Apr 19 02:19 /dev/nvidiactl

delete掉nvidia-device-plugin-validation这个pod后，kubelet将重新拉起一个，此时报错信息有变化，提示缺少/dev/nvidia-uvm设备驱动文件:

  Warning  Failed     10s (x2 over 11s)  kubelet, 10.168.100.4  Error: failed to start container "device-plugin-validation-init": Error response from daemon: linux runtime spec devices: error gathering device information while adding custom device "/dev/nvidia-uvm": no such file or directory

按照上面创建/dev/nvidiactl的方法创建/dev/nvidia-uvm驱动文件，注意设备号与容器中保持一致:

# mknod -m 666 /dev/nvidia-uvm c 237 0

删除pod后重新拉起，报错信息为缺少/dev/nvidia-uvm-tools:

  Warning  Failed     9s (x2 over 10s)  kubelet, 10.168.100.4  Error: failed to start container "device-plugin-validation-init": Error response from daemon: linux runtime spec devices: error gathering device information while adding custom device "/dev/nvidia-uvm-tools": no such file or directory

手动创建nvidia-uvm-tools设备文件后删除pod等待kubelet重新拉起pod:

# mknod -m 666 /dev/nvidia-uvm-tools c 237 1
  Warning  Failed     12s (x2 over 12s)  kubelet, 10.168.100.3  Error: failed to start container "device-plugin-validation-init": Error response from daemon: linux runtime spec devices: error gathering device information while adding custom device "/dev/nvidia-modeset": no such file or directory

手动创建nvidia-modeset设备文件后删除pod等待kubelet重新拉起pod:

# mknod -m 666 /dev/nvidia-modeset c 195 254
  Warning  Failed     13s (x2 over 14s)  kubelet, 10.168.100.4  Error: failed to start container "device-plugin-validation-init": Error response from daemon: linux runtime spec devices: error gathering device information while adding custom device "/dev/nvidia0": no such file or directory

手动创建nvidia0设备文件后删除pod等待kubelet重新拉起pod:

# mknod -m 666 /dev/nvidia0 c 195 0
# kubectl  get po -A | grep device-plugin-validation
gpu-operator-resources   nvidia-device-plugin-validation                                   0/1     Completed   0          2m26s

此时kubelet将继续拉起剩余的nvidia资源，最终状态应该是:

# kubectl  get po -A
NAMESPACE                NAME                                                              READY   STATUS      RESTARTS   AGE
default                  chart-1618804240-node-feature-discovery-master-5f446799f4-sk7vg   1/1     Running     0          163m
default                  chart-1618804240-node-feature-discovery-worker-5sllh              1/1     Running     1          163m
default                  chart-1618804240-node-feature-discovery-worker-86w4w              1/1     Running     0          163m
default                  chart-1618804240-node-feature-discovery-worker-fl52v              1/1     Running     0          163m
default                  gpu-operator-945878fff-88thn                                      1/1     Running     0          163m
gpu-operator-resources   gpu-feature-discovery-p6zqs                                       1/1     Running     0          53s
gpu-operator-resources   gpu-feature-discovery-x88v4                                       1/1     Running     0          53s
gpu-operator-resources   nvidia-container-toolkit-daemonset-27qj8                          1/1     Running     0          26m
gpu-operator-resources   nvidia-container-toolkit-daemonset-g5ndb                          1/1     Running     0          26m
gpu-operator-resources   nvidia-dcgm-exporter-c9vht                                        1/1     Running     0          74s
gpu-operator-resources   nvidia-dcgm-exporter-mz7rh                                        1/1     Running     0          74s
gpu-operator-resources   nvidia-device-plugin-daemonset-sqfdc                              1/1     Running     0          25m
gpu-operator-resources   nvidia-device-plugin-daemonset-wldkd                              1/1     Running     0          25m
gpu-operator-resources   nvidia-device-plugin-validation                                   0/1     Completed   0          2m47s
gpu-operator-resources   nvidia-driver-daemonset-m4xjv                                     1/1     Running     0          163m
gpu-operator-resources   nvidia-driver-daemonset-vkrz5                                     1/1     Running     5          163m
....

6. 测试GPU

gpu-operator目录下预置了一个test.yaml文件，直接创建:

[root@gpumaster gpu-operator]# kubectl create -f test.yaml
pod/dcgmproftester created
[root@gpumaster gpu-operator]# kubectl  get po -o wide | grep dcgmproftester
dcgmproftester                                                    1/1     Running   0          103s   172.26.243.149   10.168.100.4   <none>           <none>

找寻到10.168.100.4上的nvidia-device-plugin-daemonset的pod， 观察该节点上gpu的功耗及显存占用情况，可以看到该工作负载确实使用了gpu中的运算单元:

# kubectl exec nvidia-device-plugin-daemonset-wldkd -n gpu-operator-resources nvidia-smi
nvidia 33988608 269 nvidia_modeset,nvidia_uvm, Live 0xffffffffa05dd000 (PO)
Mon Apr 19 06:39:26 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  On   | 00000000:00:08.0 Off |                  Off |
| N/A   61C    P0   208W / 250W |    493MiB / 32510MiB |     84%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

测试完毕后该pod将处于completed状态，观察其输出:

# kubectl  get po -o wide | grep dcgm
dcgmproftester                                                    0/1     Completed   0          4m12s   172.26.243.149   10.168.100.4   <none>           <none>
# kubectl  logs dcgmproftester
.....
TensorEngineActive: generated ???, dcgm 0.000 (74380.8 gflops)
TensorEngineActive: generated ???, dcgm 0.000 (75398.9 gflops)
TensorEngineActive: generated ???, dcgm 0.000 (75787.6 gflops)
TensorEngineActive: generated ???, dcgm 0.000 (77173.9 gflops)
TensorEngineActive: generated ???, dcgm 0.000 (75669.5 gflops)
Skipping UnwatchFields() since DCGM validation is disabled

WorkingTipsOnShrinkingCCSE

Repository Shrinking

Create a new vm via following command:

# cd /var/lib/libvirt/qemu/save
### Following is for creating a new vm for saving rpms
# virsh dumpxml node1>example.xml
# vim example.xml 
# qemu-img create -f qcow2 -b ccsebaseimage.qcow2 saverpms.qcow2 
Formatting 'saverpms.qcow2', fmt=qcow2 size=536870912000 backing_file=ccsebaseimage.qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
# virsh define example.xml 
Domain nodetmp defined from example.xml
# virsh start nodetmp
Domain nodetmp started
# virsh net-dhcp-leases default
### Getting the ip address for nodetmp(10.17.18.199)
# scp ./ccse-offline-files.tar.gz root@10.17.18.199:/home/
# ssh root@10.17.18.199

Following is on 10.17.18.199:

[root@first ~]# cd /home/
[root@first home]# tar xzvf ccse-offline-files.tar.gz
# vi /etc/yum.conf
keepcache=1
### Add a new vm disk (vdb)
# fdisk /dev/vdb
# mkfs.ext4 /dev/vdb1
# mkdir /dcos
# mount /dev/vdb1 /dcos
# vi /etc/fstab 
/dev/vdb1 /dcos                   ext4     defaults        0 0
# mount -a
# exit

Bug-fix(lsof):

# scp ./Packages/lsof-4.87-6.el7.x86_64.rpm root@10.17.18.199:/root/

Backup the vm disks on host machine:

# virsh destroy nodetmp
# mv saverpms.qcow2 saverpms1.qcow2
# qemu-img create -f qcow2 -b saverpms1.qcow2 saverpms.qcow2
# virsh start nodetmp
# ssh root@10.17.18.199

Re-login, and run:

#  rpm -ivh /root/lsof-4.87-6.el7.x86_64.rpm 
# cd /home/ccse-xxxxxxxx
# vi config/config.yaml
common:
  # 控制台和/或Harbor所在的主机IP
  host: 10.17.18.199
# vim ./files/offline-repo/ccse-centos7-base.repo
	#[ccse-centos7-base]
	#name=ccse-offline-repo
	#baseurl=file://{centos7_base_repo_dir}
	#enabled=1
	#gpgcheck=0
	[ccse-centos7-base]
	name=Centos local yum repo for k8s
	baseurl=http://10.17.18.2:8200/repo/x86_64/centos7-base
	gpgcheck=0
	enabled=1
	proxy=_none_
# ./deploy.sh install all 2>&1 | sudo tee install-log_`date "+%Y%m%d%H%M"`

Notice, 10.17.18.2 is for existing ccse console.
After deployment, the cached rpms is listed as:

# find /var/cache | grep rpm$
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/audit-2.8.5-4.el7.x86_64.rpm
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/audit-libs-2.8.5-4.el7.x86_64.rpm
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/audit-libs-python-2.8.5-4.el7.x86_64.rpm
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/checkpolicy-2.5-8.el7.x86_64.rpm
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/libsemanage-python-2.5-14.el7.x86_64.rpm
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/policycoreutils-2.5-34.el7.x86_64.rpm
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/policycoreutils-python-2.5-34.el7.x86_64.rpm
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/python-IPy-0.75-6.el7.noarch.rpm
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/setools-libs-3.3.8-4.el7.x86_64.rpm
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/libcgroup-0.41-21.el7.x86_64.rpm
/var/cache/yum/x86_64/7/ccse-centos7-base/packages/unzip-6.0-21.el7.x86_64.rpm

Now enable the visit for ccse console(web ui):

# systemctl stop firewalld
# systemctl disable firewalld
# setenforce 0
# vi /etc/selinux/config 
SELINUX=disabled

ccse webui:

Create a new vm and added it on ccse webui, in newly added vm do following command:

# vi /etc/yum.conf 
keepcached
# systemctl stop firewalld
# systemctl disable firewalld
# setenforce 0
# vi /etc/selinux/config 
SELINUX=disabled

Create a new cluster, and fetch the new vm’s rpm cache:

[root@first cache]# find . | grep rpm$
./yum/x86_64/7/ccse-centos7-base/packages/audit-2.8.5-4.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/audit-libs-2.8.5-4.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/checkpolicy-2.5-8.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/audit-libs-python-2.8.5-4.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/libsemanage-python-2.5-14.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/libcgroup-0.41-21.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/policycoreutils-2.5-34.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/python-IPy-0.75-6.el7.noarch.rpm
./yum/x86_64/7/ccse-centos7-base/packages/setools-libs-3.3.8-4.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/policycoreutils-python-2.5-34.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/conntrack-tools-1.4.4-7.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/socat-1.7.3.2-2.el7.x86_64.rpm
./yum/x86_64/7/ccse-centos7-base/packages/libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm
./yum/x86_64/7/ccse-k8s/packages/container-selinux-2.119.1-1.c57a6f9.el7.noarch.rpm
./yum/x86_64/7/ccse-k8s/packages/docker-ce-18.09.9-3.el7.x86_64.rpm
./yum/x86_64/7/ccse-k8s/packages/containerd.io-1.2.13-3.2.el7.x86_64.rpm
./yum/x86_64/7/ccse-k8s/packages/docker-ce-cli-18.09.9-3.el7.x86_64.rpm
./yum/x86_64/7/ccse-k8s/packages/3f1db71d0bb6d72bc956d788ffee737714e5717c629b26355a2dcf1dba4ad231-kubelet-1.17.3-0.x86_64.rpm
./yum/x86_64/7/ccse-k8s/packages/548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm
./yum/x86_64/7/ccse-k8s/packages/35625b6ab1da6c58ce4946742181c0dcf9ac9b6c2b5bea2c13eed4876024c342-kubectl-1.17.3-0.x86_64.rpm

harbor shrinking

Save the harbor images:

[root@first ~]# docker save -o harbor.tar goharbor/chartmuseum-photon:v0.9.0-v1.8.6 goharbor/harbor-migrator:v1.8.6 goharbor/redis-photon:v1.8.6 goharbor/clair-photon:v2.1.0-v1.8.6 goharbor/notary-server-photon:v0.6.1-v1.8.6 goharbor/notary-signer-photon:v0.6.1-v1.8.6 goharbor/harbor-registryctl:v1.8.6 goharbor/registry-photon:v2.7.1-patch-2819-v1.8.6 goharbor/nginx-photon:v1.8.6 goharbor/harbor-log:v1.8.6 goharbor/harbor-jobservice:v1.8.6 goharbor/harbor-core:v1.8.6 goharbor/harbor-portal:v1.8.6 goharbor/harbor-db:v1.8.6 goharbor/prepare:v1.8.6
[root@first ~]# ls -l -h harbor.tar 
-rw-------. 1 root root 1.5G Apr 11 23:31 harbor.tar
[root@first ~]# cp harbor.tar harbor.tar.back
[root@first ~]# xz -T4 harbor.tar
[root@first ~]# ls -l -h harbor.tar.*
-rw-------. 1 root root 1.5G Apr 11 23:31 harbor.tar.back
-rw-------. 1 root root 428M Apr 11 23:31 harbor.tar.xz

rpm

rpms combine:

[root@first rpms]# ls -l -h | wc -l
12
##### After transferring from working node
#########################################
[root@first rpms]# cp /tmp/rpms/* .
cp: overwrite ‘./audit-2.8.5-4.el7.x86_64.rpm’? y
cp: overwrite ‘./audit-libs-2.8.5-4.el7.x86_64.rpm’? y
cp: overwrite ‘./audit-libs-python-2.8.5-4.el7.x86_64.rpm’? y
cp: overwrite ‘./checkpolicy-2.5-8.el7.x86_64.rpm’? y
cp: overwrite ‘./libcgroup-0.41-21.el7.x86_64.rpm’? y
cp: overwrite ‘./libsemanage-python-2.5-14.el7.x86_64.rpm’? y
cp: overwrite ‘./policycoreutils-2.5-34.el7.x86_64.rpm’? y
cp: overwrite ‘./policycoreutils-python-2.5-34.el7.x86_64.rpm’? y
cp: overwrite ‘./python-IPy-0.75-6.el7.noarch.rpm’? y
cp: overwrite ‘./setools-libs-3.3.8-4.el7.x86_64.rpm’? y
[root@first rpms]# ls -l -h | wc -l
17

WorkingTipsOnKata

Install & Configuration

Install kata on archlinux, first install snapd:

$ yaourt snapd
$ sudo systemctl enable --now snapd.socket

Using snapd for installing kata:

$ sudo snap install kata-containers --classic

Check the kata-container runtimes:

$ kata-containers.runtime --version
kata-runtime  : 1.12.1
   commit   : b967088a667018b7468a9f93d48cb81650e0dfa4
   OCI specs: 1.0.1-dev
$ which kata-containers.runtime
/var/lib/snapd/snap/bin/kata-containers.runtime

Add the kata container runtime for docker-ce:

$ sudo mkdir -p /etc/systemd/system/docker.service.d
$ sudo vim /etc/systemd/system/docker.service.d/kata-containers.conf 
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/snap/kata-containers/current/usr/bin/kata-runtime
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

Check the docker info:

$ docker info | grep Runtime
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux kata-runtime runc
 Default Runtime: runc

Testing

Run a busybox using kata-runtime:

$ sudo docker run -ti --runtime kata-runtime busybox sh

Checking the docker hardware(qemu):

/ # free -m
              total        used        free      shared  buff/cache   available
Mem:           1993          26        1965           0           2        1948
Swap:             0           0           0
/ # uname -a
Linux 172144f42ad4 5.4.60.container #1 SMP Wed Jan 20 17:43:09 UTC 2021 x86_64 GNU/Linux

Comparing to runc busybox:

$ sudo docker run -it busybox /bin/sh
/ # free -m
              total        used        free      shared  buff/cache   available
Mem:          23932        3759       12883        1003        7289       18795
Swap:          2047           0        2047
/ # uname -a
Linux 7d484813ddd3 5.10.16-arch1-1 #1 SMP PREEMPT Sat, 13 Feb 2021 20:50:18 +0000 x86_64 GNU/Linux

Get the running qemu :

# ps -ef | grep qemu
root      130733  130681  0 14:41 ?        00:00:03 /var/lib/snapd/snap/kata-containers/716/usr/bin/qemu-system-x86_64 -name sandbox-172144f42ad4130671d2f3282f84be7d33f17ec9f308234d9172162f6dac8a1f -uuid 07ebc86a-91a7-4180-accd-c9d1dbd3ac29 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/
.....

Useful tips

Get the kata env:

$ kata-containers.runtime kata-env

See if the system is ready for running kata:

$ sudo kata-containers.runtime kata-check

WorkingTipsOnRPIXC

目标

基于树莓派的arm64快速验证平台。

系统准备

Ubuntu官网下载到Ubuntu 20.04.2 LTS 64位镜像:

https://ubuntu.com/download/raspberry-pi

插入tf卡，烧写img文件至tf卡中, Linux 命令为:

 $ sudo dd if=./ubuntu-20.04.2-preinstalled-server-arm64+raspi.img of=/dev/sdd bs=1M && sudo sync
记录了3108+1 的读入
记录了3108+1 的写出
3259499520字节（3.3 GB，3.0 GiB）已复制，79.1856 s，41.2 MB/s

Windows下可自行下载rpi镜像烧写软件以完成烧写操作。

初次登陆

如果具备mini-hdmi转接线及鼠标，则可外接显示器用于登陆RPI。
如果不具备显示器，则可以通过接入网络来远程登陆RPI，登陆到局域网段的路由器上查看rpi获取到的Ip地址即可。

例如， 在路由器上获取到RPI IP地址:

$ cat /var/lib/misc/dnsmasq.leases
1615556943 dc:a6:32:e2:0b:44 10.137.149.171 ubuntu ff:dc:6b:56:57:00:02:00:00:ab:11:b2:87:a0:99:d0:fb:b1:ce

而后用ssh登陆, 默认用户名/密码为ubuntu/ubuntu，初次登陆后会强制要求更改:

$ ssh ubuntu@10.137.149.171
。。。。

WARNING: Your password has expired.
You must change your password now and login again!
Changing password for ubuntu.
Current password: 
New password: 
Retype new password: 
passwd: password updated successfully
Connection to 10.137.149.171 closed.
$ ssh ubuntu@10.137.149.171
ubuntu@10.137.149.171's password: 
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-1028-raspi aarch64)

虚拟机开发环境准备

安装基本桌面, 选择lightdm:

$ sudo apt-get install lxde virt-manager tightvncserver net-tools

开启vnc:

$ vncserver
You will require a password to access your desktops.

Password: 
Verify:   
Would you like to enter a view-only password (y/n)? y
Password: 
Verify:   

New 'X' desktop is ubuntu:1

Creating default startup script /home/ubuntu/.vnc/xstartup
Starting applications specified in /home/ubuntu/.vnc/xstartup
Log file is /home/ubuntu/.vnc/ubuntu:1.log

此时需要先关闭正在运行的vncserver, 配置默认桌面环境后再重新创建:

ubuntu@ubuntu:~$ vncserver -kill :1
Killing Xtightvnc process ID 30773
ubuntu@ubuntu:~$ vim ~/.vnc/xstartup 
#!/bin/sh
exec startlxde

ubuntu@ubuntu:~$ vncserver

New 'X' desktop is ubuntu:1

Starting applications specified in /home/ubuntu/.vnc/xstartup
Log file is /home/ubuntu/.vnc/ubuntu:1.log

此时可以使用vncviewer登陆:

开启libvirtd权限:

$ sudo systemctl enable libvirtd
$ sudo systemctl start libvirtd
$  sudo usermod -a -G kvm,libvirt  ubuntu

此时需重启rpi后，重新登陆/开启vnc后，验证virt-manager的可用性:

虚拟机系统安装

进入到安装界面后(All-In-One安装)：

安装过程:

安装完毕后可以备份一下初始化镜像以便后续使用:

$ sudo virsh dumpxml ubuntu20.04 | grep qcow2
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/ubuntu20.04.qcow2'/>
ubuntu@ubuntu:~$ sudo cp /var/lib/libvirt/images/ubuntu20.04.qcow2 .

后续开始部署RONG:

部署完毕后资源占用情况:

root@node:/home/test/Rong# free -m
              total        used        free      shared  buff/cache   available
Mem:           5909        1790         132           2        3987        4205
Swap:             0           0           0
root@node:/home/test/Rong# df -h
Filesystem               Size  Used Avail Use% Mounted on
udev                     2.9G     0  2.9G   0% /dev
tmpfs                    591M  2.5M  589M   1% /run
/dev/mapper/vgnode-root   24G   13G   12G  52% /

XC适配

选择国产操作系统ISO用于安装：

安装界面：

外面ping:

ubuntu@ubuntu:~$ ping 192.168.122.30
PING 192.168.122.30 (192.168.122.30) 56(84) bytes of data.
64 bytes from 192.168.122.30: icmp_seq=1 ttl=64 time=0.941 ms
64 bytes from 192.168.122.30: icmp_seq=2 ttl=64 time=0.499 ms

8-> 设置root password.

如果无其他设置，则直接安装

安装过程:

安装完毕后，登陆:

$ ssh root@192.168.122.30
The authenticity of host '192.168.122.30 (192.168.122.30)' can't be established.
ECDSA key fingerprint is SHA256:wC8hcKiDjbz1+l9MAIUWMZju0evX4ZAIQjz+GPzEL4I.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.30' (ECDSA) to the list of known hosts.

Authorized users only. All activities may be monitored and reported.
root@192.168.122.30's password: 

Authorized users only. All activities may be monitored and reported.
Web console: https://localhost:9090/ or https://192.168.122.30:9090/

Last failed login: Thu Apr  2 03:46:11 CST 2020 from 192.168.122.1 on ssh:notty
There was 1 failed login attempt since the last successful login.
[root@localhost ~]# 

部署完RONG后，检查:

[root@localhost Rong]# kubectl get node
NAME     STATUS   ROLES    AGE     VERSION
test01   Ready    master   9m48s   v1.18.8
[root@localhost Rong]# uname -a
Linux localhost.localdomain 4.19.90-17.ky10.aarch64 #1 SMP Sun Jun 28 14:27:40 CST 2020 aarch64 aarch64 aarch64 GNU/Linux
[root@localhost Rong]# cat /etc/issue

Authorized users only. All activities may be monitored and reported.

pod对应的指标:

[root@localhost ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-f874b4f5f-846zc       0/1     CrashLoopBackOff    12         24m
kube-system   calico-node-4d869                             0/1     CrashLoopBackOff    12         25m
kube-system   coredns-dff8fc7d-f2n26                        0/1     ContainerCreating   0          23m
kube-system   dns-autoscaler-7b85bccb5f-264cz               0/1     ContainerCreating   0          23m
kube-system   kube-apiserver-test01                         1/1     Running             1          28m
kube-system   kube-controller-manager-test01                1/1     Running             1          28m
kube-system   kube-proxy-q6mnw                              1/1     Running             1          28m
kube-system   kube-scheduler-test01                         1/1     Running             1          28m
kube-system   kubernetes-dashboard-674bb5ff47-mw97w         0/1     ContainerCreating   0          23m
kube-system   kubernetes-metrics-scraper-54fbb4d595-64b6j   0/1     ContainerCreating   0          23m
kube-system   metrics-server-757968d55d-62czd               0/2     ContainerCreating   0          21m
kube-system   tiller-deploy-75dc954ffd-psj68                0/1     ContainerCreating   0          22m

同样的vm在华为的HI1616机器上表现正常。可见YINHE 麒麟 V10的系统bug较多，可能未曾适配过完整的硬件列表。

WorkingTIPSOnHA

目的: 设置RONG服务器的高可用性.

Install ipvsadm for every node:

# apt-get install -y ipvsadm