WorkingTipsOnKata

Install & Configuration

Install kata on archlinux, first install snapd:

$ yaourt snapd
$ sudo systemctl enable --now snapd.socket

Using snapd for installing kata:

$ sudo snap install kata-containers --classic

Check the kata-container runtimes:

$ kata-containers.runtime --version
kata-runtime  : 1.12.1
   commit   : b967088a667018b7468a9f93d48cb81650e0dfa4
   OCI specs: 1.0.1-dev
$ which kata-containers.runtime
/var/lib/snapd/snap/bin/kata-containers.runtime

Add the kata container runtime for docker-ce:

$ sudo mkdir -p /etc/systemd/system/docker.service.d
$ sudo vim /etc/systemd/system/docker.service.d/kata-containers.conf 
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/snap/kata-containers/current/usr/bin/kata-runtime
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

Check the docker info:

$ docker info | grep Runtime
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux kata-runtime runc
 Default Runtime: runc

Testing

Run a busybox using kata-runtime:

$ sudo docker run -ti --runtime kata-runtime busybox sh

Checking the docker hardware(qemu):

/ # free -m
              total        used        free      shared  buff/cache   available
Mem:           1993          26        1965           0           2        1948
Swap:             0           0           0
/ # uname -a
Linux 172144f42ad4 5.4.60.container #1 SMP Wed Jan 20 17:43:09 UTC 2021 x86_64 GNU/Linux

Comparing to runc busybox:

$ sudo docker run -it busybox /bin/sh
/ # free -m
              total        used        free      shared  buff/cache   available
Mem:          23932        3759       12883        1003        7289       18795
Swap:          2047           0        2047
/ # uname -a
Linux 7d484813ddd3 5.10.16-arch1-1 #1 SMP PREEMPT Sat, 13 Feb 2021 20:50:18 +0000 x86_64 GNU/Linux

Get the running qemu :

# ps -ef | grep qemu
root      130733  130681  0 14:41 ?        00:00:03 /var/lib/snapd/snap/kata-containers/716/usr/bin/qemu-system-x86_64 -name sandbox-172144f42ad4130671d2f3282f84be7d33f17ec9f308234d9172162f6dac8a1f -uuid 07ebc86a-91a7-4180-accd-c9d1dbd3ac29 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/
.....

Useful tips

Get the kata env:

$ kata-containers.runtime kata-env

See if the system is ready for running kata:

$ sudo kata-containers.runtime kata-check

WorkingTipsOnRPIXC

目标

基于树莓派的arm64快速验证平台。

系统准备

Ubuntu官网下载到Ubuntu 20.04.2 LTS 64位镜像:

https://ubuntu.com/download/raspberry-pi

/images/2021_03_12_09_45_02_1212x332.jpg

插入tf卡,烧写img文件至tf卡中, Linux 命令为:

 $ sudo dd if=./ubuntu-20.04.2-preinstalled-server-arm64+raspi.img of=/dev/sdd bs=1M && sudo sync
记录了3108+1 的读入
记录了3108+1 的写出
3259499520字节(3.3 GB,3.0 GiB)已复制,79.1856 s,41.2 MB/s

Windows下可自行下载rpi镜像烧写软件以完成烧写操作。

初次登陆

如果具备mini-hdmi转接线及鼠标,则可外接显示器用于登陆RPI。
如果不具备显示器,则可以通过接入网络来远程登陆RPI,登陆到局域网段的路由器上查看rpi获取到的Ip地址即可。

例如, 在路由器上获取到RPI IP地址:

$ cat /var/lib/misc/dnsmasq.leases
1615556943 dc:a6:32:e2:0b:44 10.137.149.171 ubuntu ff:dc:6b:56:57:00:02:00:00:ab:11:b2:87:a0:99:d0:fb:b1:ce

而后用ssh登陆, 默认用户名/密码为ubuntu/ubuntu,初次登陆后会强制要求更改:

$ ssh ubuntu@10.137.149.171
。。。。

WARNING: Your password has expired.
You must change your password now and login again!
Changing password for ubuntu.
Current password: 
New password: 
Retype new password: 
passwd: password updated successfully
Connection to 10.137.149.171 closed.
$ ssh ubuntu@10.137.149.171
ubuntu@10.137.149.171's password: 
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-1028-raspi aarch64)

虚拟机开发环境准备

安装基本桌面, 选择lightdm:

$ sudo apt-get install lxde virt-manager tightvncserver net-tools

/images/2021_03_12_10_12_41_512x417.jpg

开启vnc:

$ vncserver
You will require a password to access your desktops.

Password: 
Verify:   
Would you like to enter a view-only password (y/n)? y
Password: 
Verify:   

New 'X' desktop is ubuntu:1

Creating default startup script /home/ubuntu/.vnc/xstartup
Starting applications specified in /home/ubuntu/.vnc/xstartup
Log file is /home/ubuntu/.vnc/ubuntu:1.log

此时需要先关闭正在运行的vncserver, 配置默认桌面环境后再重新创建:

ubuntu@ubuntu:~$ vncserver -kill :1
Killing Xtightvnc process ID 30773
ubuntu@ubuntu:~$ vim ~/.vnc/xstartup 
#!/bin/sh
exec startlxde

ubuntu@ubuntu:~$ vncserver

New 'X' desktop is ubuntu:1

Starting applications specified in /home/ubuntu/.vnc/xstartup
Log file is /home/ubuntu/.vnc/ubuntu:1.log

此时可以使用vncviewer登陆:

/images/2021_03_12_10_29_49_737x234.jpg

开启libvirtd权限:

$ sudo systemctl enable libvirtd
$ sudo systemctl start libvirtd
$  sudo usermod -a -G kvm,libvirt  ubuntu

此时需重启rpi后,重新登陆/开启vnc后,验证virt-manager的可用性:

/images/2021_03_12_10_38_23_792x442.jpg

虚拟机系统安装

/images/2021_03_12_10_46_47_632x356.jpg

/images/2021_03_12_10_47_23_497x497.jpg

/images/2021_03_12_10_47_34_386x250.jpg

/images/2021_03_12_10_47_50_428x281.jpg

/images/2021_03_12_10_48_06_531x525.jpg

进入到安装界面后(All-In-One安装):

/images/2021_03_12_10_48_56_657x395.jpg

安装过程:

/images/2021_03_12_10_54_08_682x525.jpg

安装完毕后可以备份一下初始化镜像以便后续使用:

$ sudo virsh dumpxml ubuntu20.04 | grep qcow2
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/ubuntu20.04.qcow2'/>
ubuntu@ubuntu:~$ sudo cp /var/lib/libvirt/images/ubuntu20.04.qcow2 .

后续开始部署RONG:

/images/2021_03_12_11_57_01_966x589.jpg

部署完毕后资源占用情况:

root@node:/home/test/Rong# free -m
              total        used        free      shared  buff/cache   available
Mem:           5909        1790         132           2        3987        4205
Swap:             0           0           0
root@node:/home/test/Rong# df -h
Filesystem               Size  Used Avail Use% Mounted on
udev                     2.9G     0  2.9G   0% /dev
tmpfs                    591M  2.5M  589M   1% /run
/dev/mapper/vgnode-root   24G   13G   12G  52% /

XC适配

选择国产操作系统ISO用于安装:

/images/2021_03_12_12_09_23_546x515.jpg

/images/2021_03_12_12_10_04_505x320.jpg

/images/2021_03_12_12_10_19_501x256.jpg

/images/2021_03_12_12_10_37_507x382.jpg

安装界面:

/images/2021_03_12_12_11_29_648x177.jpg

/images/2021_03_12_12_13_08_754x293.jpg

/images/2021_03_12_12_13_25_669x176.jpg

/images/2021_03_12_12_16_03_704x343.jpg

/images/2021_03_12_12_16_21_676x294.jpg

/images/2021_03_12_12_16_36_678x192.jpg

/images/2021_03_12_12_16_49_686x245.jpg

/images/2021_03_12_12_17_00_672x242.jpg

/images/2021_03_12_12_17_19_692x218.jpg

/images/2021_03_12_12_17_39_684x297.jpg

/images/2021_03_12_12_17_53_637x171.jpg

/images/2021_03_12_12_18_08_647x302.jpg

/images/2021_03_12_12_18_26_676x147.jpg

/images/2021_03_12_12_18_49_698x344.jpg

/images/2021_03_12_12_19_03_659x170.jpg

外面ping:

ubuntu@ubuntu:~$ ping 192.168.122.30
PING 192.168.122.30 (192.168.122.30) 56(84) bytes of data.
64 bytes from 192.168.122.30: icmp_seq=1 ttl=64 time=0.941 ms
64 bytes from 192.168.122.30: icmp_seq=2 ttl=64 time=0.499 ms

8-> 设置root password.

如果无其他设置,则直接安装

/images/2021_03_12_12_21_21_705x347.jpg

安装过程:

/images/2021_03_12_12_22_16_651x345.jpg

安装完毕后,登陆:

$ ssh root@192.168.122.30
The authenticity of host '192.168.122.30 (192.168.122.30)' can't be established.
ECDSA key fingerprint is SHA256:wC8hcKiDjbz1+l9MAIUWMZju0evX4ZAIQjz+GPzEL4I.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.30' (ECDSA) to the list of known hosts.

Authorized users only. All activities may be monitored and reported.
root@192.168.122.30's password: 

Authorized users only. All activities may be monitored and reported.
Web console: https://localhost:9090/ or https://192.168.122.30:9090/

Last failed login: Thu Apr  2 03:46:11 CST 2020 from 192.168.122.1 on ssh:notty
There was 1 failed login attempt since the last successful login.
[root@localhost ~]# 

部署完RONG后,检查:

[root@localhost Rong]# kubectl get node
NAME     STATUS   ROLES    AGE     VERSION
test01   Ready    master   9m48s   v1.18.8
[root@localhost Rong]# uname -a
Linux localhost.localdomain 4.19.90-17.ky10.aarch64 #1 SMP Sun Jun 28 14:27:40 CST 2020 aarch64 aarch64 aarch64 GNU/Linux
[root@localhost Rong]# cat /etc/issue

Authorized users only. All activities may be monitored and reported.

pod对应的指标:

[root@localhost ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-f874b4f5f-846zc       0/1     CrashLoopBackOff    12         24m
kube-system   calico-node-4d869                             0/1     CrashLoopBackOff    12         25m
kube-system   coredns-dff8fc7d-f2n26                        0/1     ContainerCreating   0          23m
kube-system   dns-autoscaler-7b85bccb5f-264cz               0/1     ContainerCreating   0          23m
kube-system   kube-apiserver-test01                         1/1     Running             1          28m
kube-system   kube-controller-manager-test01                1/1     Running             1          28m
kube-system   kube-proxy-q6mnw                              1/1     Running             1          28m
kube-system   kube-scheduler-test01                         1/1     Running             1          28m
kube-system   kubernetes-dashboard-674bb5ff47-mw97w         0/1     ContainerCreating   0          23m
kube-system   kubernetes-metrics-scraper-54fbb4d595-64b6j   0/1     ContainerCreating   0          23m
kube-system   metrics-server-757968d55d-62czd               0/2     ContainerCreating   0          21m
kube-system   tiller-deploy-75dc954ffd-psj68                0/1     ContainerCreating   0          22m

同样的vm在华为的HI1616机器上表现正常。可见YINHE 麒麟 V10的系统bug较多,可能未曾适配过完整的硬件列表。

WorkingTIPSOnHA

目的: 设置RONG服务器的高可用性.

Install ipvsadm for every node:

# apt-get install -y ipvsadm

LinuxTips12

1. vagrant-libvirt

vagrant 2.2.14版中的一个包依赖BUG导致几乎所有的插件都没法装。解决方案是回归到旧版本的vagrant后安装vagrant-libvirt:

$ sudo pacman -U /var/cache/pacman/pkg/vagrant-2.2.10-2-x86_64.pkg.tar.zst
$ vagrant --version
Vagrant 2.2.10

Install vagrant-libvirt via:

$ vagrant plugin install vagrant-libvirt  --plugin-clean-sources --plugin-source  https://mirrors.tuna.tsinghua.edu.cn/rubygems/ --debug
$ vagrant plugin install vagrant-mutate  --plugin-clean-sources --plugin-source  https://mirrors.tuna.tsinghua.edu.cn/rubygems/ --debug

Examine the installed vagrant plugins:

$ vagrant plugin list
vagrant-libvirt (0.3.0, global)
vagrant-mutate (1.2.0, global)

2. Ignore PKG globally

For ignoring some package in archlinux’s pacman upgrade, do following:

$ sudo vim /etc/pacman.conf
IgnorePkg   = vagrant

Then in next pacman -Syu --noconfirm we will ignore vagrant.

3. Enable dmesg

Enable dmesg for normal user:

# sudo sysctl kernel.dmesg_restrict=0

4. docker.io启动panic

现象: systemctl restart docker报错无法启动,查看原因怀疑是containerd问题,

# journalctl -xeu containerd>kkkk.txt
# cat kkkk.txt

在输出的日志中发现invalid page type: xx: xx问题,此时直接apt-get purge掉所有和docker/containerd相关的包,而后删除/var/lib/containerd目录, 问题得以解决。

5. System Installation Time

Detect the system installation time via:

# ls -lact --full-time /etc |tail

6. disable fedora initial-setup

via:

# systemctl stop initial-setup && systemctl disable initial-setup

7. python3’s SimpleHTTPServer

via:

$ python3 -m http.server 8888

8. curl and tar xzvf

via:

curl www.xxxxx.com/kkk.tar.gz | tar xzvf

WorkingTipsOnRPISharingDesktop

无线接入点配置

1. 基本配置

树莓派4B 8G版本,刷入了Ubuntu 20.04.2 arm64版本:

ubuntu@rpi1:~$ cat /etc/issue
Ubuntu 20.04.2 LTS \n \l

ubuntu@rpi1:~$ uname -a
Linux rpi1 5.4.0-1028-raspi #31-Ubuntu SMP PREEMPT Wed Jan 20 11:30:45 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

2. 网络配置

默认Ubuntu20.04采用netplan作为网络配置方式,一般情况下满足网络配置需求,然而在配置无线接入点的时候,需要固定wlan0 IP地址的情况下,netplan配置就不能成功,因为它在配置wlan0 固定IP地址时需要配置ssid。因而我们采用传统的ifupdown作为网络配置手段:

关闭netplan配置:

# mv /etc/netplan/50-cloud-init.yaml /root

安装必要的软件:

# apt-get install -y resolvconf netctl ifupdown hostapd dnsmasq

配置网络:

# vim /etc/network/interfaces
    # Include files from /etc/network/interfaces.d:
    source-directory /etc/network/interfaces.d
    
    source /etc/network/interfaces.d/*
    
    auto lo
    iface lo inet loopback
    
    
    auto eth0
    iface eth0 inet dhcp
    
    auto wlan0
    iface wlan0 inet static  
        address 10.0.70.1
            netmask 255.255.255.0
# chmod 777 /etc/network/interfaces

现在重新启动树莓派,发现eth0配置成功,然而wlan0尚未配置, 我们通过在crontab中配置定时任务的方法来配置wlan0:

# crontab -e 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

@reboot sleep 120 && /usr/bin/systemctl restart networking 

使能crontab:

# systemctl enable cron

3. hostapd配置

配置hostapd(/etc/hostapd/hostapd.conf):

# the interface used by the AP
interface=wlan0
driver=nl80211
# "g" simply means 2.4GHz band
hw_mode=g
# the channel to use
channel=10
# limit the frequencies used to those allowed in the country
ieee80211d=1
# the country code
country_code=CN
# 802.11n support
ieee80211n=1
# QoS support
wmm_enabled=1
# the name of the AP
ssid=rpiwifi
macaddr_acl=0
# 1=wpa, 2=wep, 3=both
auth_algs=1
ignore_broadcast_ssid=0
# WPA2 only
wpa=2
wpa_passphrase=xxxxxxxxxxxxx
wpa_key_mgmt=WPA-PSK
#wpa_pairwise=TKIP
rsn_pairwise=CCMP

编辑文件/etc/default/hostapd, 更改含有DAEMON_CONF的行为: DAEMON_CONF="/etc/hostapd/hostapd.conf".

然而此时hostapd在启动以后并不会重新启动,我们需要在crontab中添加其自动启动.

# crontab -e
@reboot sleep 120 && /usr/bin/systemctl restart networking  && systemctl restart hostapd

4. IP地址配置

此时hostapd无法给客户端配置IP地址,为此我们需要配置dnsmasq(/etc/dnsmasq.conf):

#配置监听地址
listen-address=127.0.0.1,10.0.70.1
#配置DHCP分配段
dhcp-range=10.0.70.50,10.0.70.150,12h
dhcp-option=3,10.0.70.1

5. iptables配置

编辑/etc/sysctl.conf并取消这一行的注释:

net.ipv4.ip_forward=1

为eth0出站流量添加伪装:

# sudo iptables -t nat -A  POSTROUTING -o eth0 -j MASQUERADE

我们调节crontab为:

@reboot sleep 30 && /usr/bin/systemctl restart networking  && systemctl restart hostapd && /usr/sbin/iptables -t nat -A  POSTROUTING -o eth0 -j MASQUERADE

到现在为止,我们应该可以配置出了一个随时可以访问internet的rpi接入点。

后续需要注意点

以该rpi为接入点,接入到某个网络中,然而该网络中的Internet是通过另台rpi的WIFI所共享的。

另台RPI上的无线连接通过wifi-menu来配置:

# apt-get install -y netctl