WorkingTipsOnPVESRIOV

Change the repository for using chinese site:

$ cat /etc/apt/sources.list
deb http://mirrors.ustc.edu.cn/debian bullseye main contrib

deb http://mirrors.ustc.edu.cn/debian bullseye-updates main contrib

# security updates
deb http://mirrors.ustc.edu.cn/debian-security bullseye-security main contrib

Update to 5.19 kernel and linux-header:

deb http://mirrors.ustc.edu.cn/debian bullseye main contrib

deb http://mirrors.ustc.edu.cn/debian bullseye-updates main contrib

# security updates
deb http://mirrors.ustc.edu.cn/debian-security stable-security main contrib

# no-subscription
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

Install kernel(via download):

scp ./pve-headers-5.19.7-2-pve_5.19.7-2_amd64.deb pve-kernel-5.19.7-2-pve_5.19.7-2_amd64.deb root@192.168.1.52:~
root@xxxx:~# dpkg -i pve-headers-5.19.7-2-pve_5.19.7-2_amd64.deb pve-kernel-5.19.7-2-pve_5.19.7-2_amd64.deb 
reboot

Install dkms, then install i915-sriov-dkms.

tar -xvf i915-sriov-dkms.tar
mv i915-sriov-dkms /usr/src
dkms install -m i915-sriov -v dkms
# dkms status
i915-sriov, dkms, 5.19.7-2-pve, x86_64: installed

Install sysfsutils:

apt install sysfsutils -y
echo "devices/pci0000:00/0000:00:02.0/sriov_numvfs = 7" > /etc/sysfs.conf

NoVNCOperationOn4ANetwork

因4A速度实在太慢,通过bmc远程安装系统基本上不可行。故采用在跳板机上启动novnc加持的桌面session的方式用来部署操作系统。

登陆跳板机192.168.xx.xx后,运行以下命令建立一个监听跳板机6080端口的lxde桌面环境(因跳板机是arm64环境,需要指定arm64后缀的镜像):

sudo docker run -p 6080:80 -e USER=doro -e PASSWORD=password -v /dev/shm:/dev/shm dorowu/ubuntu-desktop-lxde-vnc:focal-arm64

通过xshell和proxifier建立全局代理后,在浏览器里直接访问192.168.xx.xx:6080即可访问到创建的桌面环境:

/images/2022_12_26_21_21_39_1048x730.jpg

在该桌面环境里可直接调用firefox下载相关镜像,并使用bmc挂载该镜像用于安装操作系统:

/images/2022_12_26_21_24_22_1240x783.jpg

因为安装介质都在同一数据中心内,所以安装时不会出明显卡顿,大约15~30分钟内即可安装成功。

安装完后的bmc界面:

/images/2022_12_26_21_23_41_1018x495.jpg

VirglHeadlessMode

1. 目的

验证在headless GPU Server上启用virgl加速的可行性

2. 环境

硬件: nvidia gtx1060 6G独显笔记本(附带intel集显):

$ sudo lspci | grep -i vga
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-H GT2 [UHD Graphics 630]
01:00.0 VGA compatible controller: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile] (rev a1)
$ sudo nvidia-smi 
Fri Dec  9 15:13:58 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.11    Driver Version: 525.60.11    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   52C    P8     7W /  N/A |    203MiB /  6144MiB |     17%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

系统及软件(arch下依赖qemu/qemu-ui-egl-headless):

$ uname -r
6.0.12-arch1-1
$ cat /etc/issue
Arch Linux \r (\l)
$ qemu-system-x86_64 --version
QEMU emulator version 7.1.0
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
$ sudo pacman -Q | grep -i qemu | grep egl
qemu-ui-egl-headless 7.1.0-11

VM环境:

$ cat /etc/issue
Ubuntu 20.04.5 LTS \n \l
$ uname -r
5.15.0-56-generic

3. 启动VM

纯命令行下启动(无x11登陆):

sudo qemu-system-x86_64 -name  ubuntu2204 -drive file=`pwd`/ubuntudesktop2004_spice.qcow2,if=virtio -m 8192 -enable-kvm -cpu host -smp 8,sockets=1,cores=8,threads=1   -spice unix=on,addr=/tmp/spice.sock,disable-ticketing=on,plaintext-channel=default,seamless-migration=on,image-compression=off,jpeg-wan-compression=never,zlib-glz-wan-compression=never,streaming-video=off,playback-compression=off \
-netdev user,id=vnet,hostfwd=:127.0.0.1:2278-:22,hostfwd=tcp::5555-:5555,hostfwd=tcp::5556-:5556,hostfwd=tcp::5557-:5557,hostfwd=tcp::5800-:5800,hostfwd=tcp::3389-:3389,hostfwd=tcp::14000-:4000 -device virtio-net-pci,netdev=vnet \
-display egl-headless,rendernode=/dev/dri/renderD129 \
-device virtio-vga-gl,max_outputs=1 \
-device virtio-serial-pci \
-chardev spicevmc,id=charchannel1,name=vdagent \
-device virtserialport,chardev=charchannel1,id=channel1,name=com.redhat.spice.0

其中rendernode为nvidia gtx1060.

4. 验证

另起一终端,ssh到vm后验证:

$ export DISPLAY=:0
$ glxinfo | grep -i render
direct rendering: Yes
    GLX_MESA_query_renderer, GLX_MESA_swap_control, GLX_NV_float_buffer, 
    GLX_MESA_copy_sub_buffer, GLX_MESA_query_renderer, GLX_MESA_swap_control, 
Extended renderer info (GLX_MESA_query_renderer):
    Renderbuffer free memory - total: 21 MB, largest block: 21 MB
    Renderbuffer free aux. memory - total: 0 MB, largest block: 0 MB
OpenGL renderer string: virgl
    GL_ARB_conditional_render_inverted, GL_ARB_conservative_depth, 
    GL_NV_conditional_render, GL_NV_copy_image, GL_NV_depth_clamp, 
    GL_ARB_compute_shader, GL_ARB_conditional_render_inverted, 
    GL_NV_conditional_render, GL_NV_copy_image, GL_NV_depth_clamp, 
    GL_EXT_render_snorm, GL_EXT_robustness, GL_EXT_sRGB_write_control, 
    GL_NV_conditional_render, GL_NV_draw_buffers, GL_NV_fbo_color_attachments, 
    GL_OES_element_index_uint, GL_OES_fbo_render_mipmap, 
$ ls /dev/dri/
by-path  card0  renderD128
$ sudo dmesg | grep -i virgl
[    2.257506] [drm] features: +virgl +edid -resource_blob -host_visible
$ glmark2
=======================================================
    glmark2 2021.02
=======================================================
    OpenGL Information
    GL_VENDOR:     Mesa/X.org
    GL_RENDERER:   virgl
    GL_VERSION:    3.1 Mesa 21.2.6
=======================================================
[build] use-vbo=false: FPS: 5102 FrameTime: 0.196 ms

.........


[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 6301 FrameTime: 0.159 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 5922 FrameTime: 0.169 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 5785 FrameTime: 0.173 ms
=======================================================
                                  glmark2 Score: 3996 
=======================================================

最终跑分为接近4000分,由此可见headless server下virgl的2d/3d均可用, 且使用了nvidia卡的加速能力。

MigrationFromCentOS76ToRocky

1. 初始环境(CentOS76)

使用vagrant创建CentOS7.6.1810虚拟机一台:

# vim Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "bento/centos-7.6"
    config.vm.provider "virtualbox" do |v|
        v.memory = 4096
        v.cpus = 4
    end
end
# vagrant up 
# vagrant ssh

登陆入系统后检查初始环境信息:

[vagrant@localhost ~]$ cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 
[vagrant@localhost ~]$ uname -a
Linux localhost.localdomain 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

2. CentOS7.6升级到CentOS8.1

2.1 升级前准备

安装EPEL(Extra Package for Enterprise Linux)库,因升级中需用到该库, 并创建该库缓存:

# yum install -y epel-release.noarch
# yum makecache fast

安装升级CentOS 7到8过程中需要用到的rpmconf包及yum-utils包:

# yum install -y yum-utils rpmconf

计算重复/无用包及配置:

# rpmconf -a
安装过程中,选择 N:  
*** aliases (Y/I/N/O/D/M/Z/S) [default=N] ? 
Your choice: N

升级yumdnf,安装完毕后,删除不再需要的yum及相关包:

# yum install -y dnf
# dnf remove -y yum yum-metadata-parser

删除现有的repo配置并重新生成仓库缓存, 并使用dnf升级为最新状态:

# rm -Rf /etc/yum
#  dnf makecache
Extra Packages for Enterprise Linux 7 - x86_64   5.9 MB/s |  16 MB     00:02    
CentOS-7 - Base                                  4.1 MB/s |  10 MB     00:02    
CentOS-7 - Updates                               7.6 MB/s |  21 MB     00:02    
CentOS-7 - Extras                                165 kB/s | 332 kB     00:02    
Metadata cache created.
# dnf upgrade -y

2.2 升级8仓库准备

跨大版本升级可能会衍生出一些问题,因而我们先使用8.1版本的CentOS8系统包作为中间状态升级:

# dnf upgrade -y  https://mirrors.ustc.edu.cn/centos-vault/8.1.1911//BaseOS/x86_64/os/Packages/{centos-release-8.1-1.1911.0.8.el8.x86_64.rpm,centos-gpg-keys-8.1-1.1911.0.8.el8.noarch.rpm,centos-repos-8.1-1.1911.0.8.el8.x86_64.rpm}

升级EPEL仓库包从EL 7到EL 8:

# dnf upgrade -y epel-release

因跨大版本升级可能会导致问题,切换BaseOSAppStream库为8.1.1911:

# cd /etc/yum.repos.d
# mkdir back
# mv CentOS-* back/
# sudo tee CentOS-Linux-BaseOS.repo<<EOM
[baseos]
name=CentOS Linux \$releasever - BaseOS
baseurl=http://vault.centos.org/8.1.1911/BaseOS/\$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
EOM

# sudo tee CentOS-Linux-AppStream.repo<<EOM
[appstream]
name=CentOS Linux \$releasever - AppStream
baseurl=http://vault.centos.org/8.1.1911/AppStream/\$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
EOM
# sudo cp CentOS-Linux-BaseOS.repo /root/CentOS-Linux-BaseOS.repo
# sudo cp CentOS-Linux-AppStream.repo /root/CentOS-Linux-AppStream.repo
# dnf makecache
Extra Packages for Enterprise Linux 8 - x86_64                4.7 MB/s |  13 MB     00:02    
CentOS-8 - Base                                               1.1 MB/s | 4.6 MB     00:04    
Extra Packages for Enterprise Linux Modular 8 - x86_64        197 kB/s | 733 kB     00:03    
CentOS-8 - AppStream                                          1.3 MB/s | 8.4 MB     00:06    
CentOS-8 - Extras                                             4.3 kB/s |  10 kB     00:02    
Module yaml error: Unexpected key in data
Module yaml error: Unexpected key in data
Module yaml error: Unexpected key in data
Module yaml error: Unexpected key in data
Module defaults error: Unexpected key in data
Module defaults error: Unexpected key in data
Module defaults error: Unexpected key in data
Module defaults error: Unexpected key in data
Metadata cache created.

2.3 升级到CentOS 8

删除所有安装过的kernel:

# rpm -e `rpm -q kernel`

删除sysvinit-tools等冲突包:

# rpm -e --nodeps sysvinit-tools sysvinit-tools python36-rpmconf 

执行以下命令,升级到CentOS 8:

# dnf -y --releasever=8 --allowerasing --setopt=deltarpm=false distro-sync

升级成功后,/etc/yum.repos.d目录下会被覆盖,回归到8.1版本

# cd /etc/yum.repos.d
# mkdir back1
# mv CentOS-* back1/
# sudo cp /root/CentOS-Linux-BaseOS.repo .
# sudo cp /root/CentOS-Linux-AppStream.repo .

安装内核包:

# dnf makecache
# dnf install -y kernel-core

安装minimalCore包组合:

# mv /etc/yum/protected.d /etc/yum/protected.d.back
# dnf -y groupupdate "Core" "Minimal Install"

2.4 升级CentOS8.5

回归原来的包配置:

# cd /etc/yum.repos.d/
# rm -f CentOS-*
# mv back1/* .
# CENTOS_BASE=http://vault.centos.org
# sed -i -e "s|mirrorlist=|#mirrorlist=|g" /etc/yum.repos.d/CentOS-* 
# sed -i -e 's|#baseurl=http://mirror.centos.org|baseurl='$CENTOS_BASE'|g' /etc/yum.repos.d/CentOS-*
# dnf update

更新完毕后,重启,而后检查内核及操作系统版本:

# uname -a
Linux localhost.localdomain 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release 
CentOS Linux release 8.5.2111

3. CentOS 8 -> Rocky Linux 8

下载Rocky Linux升级脚本:

# dnf -y install wget
# wget https://raw.githubusercontent.com/rocky-linux/rocky-tools/main/migrate2rocky/migrate2rocky.sh
# chmod a+x migrate2rocky.sh

升级到Rocky Linux 8.6:

./migrate2rocky.sh -r

/images/2022_11_02_16_04_40_865x689.jpg

等待一段时间(取决于网速及机器配置,vagrant虚拟机约15分钟),等待升级完成:

/images/2022_11_02_16_11_03_707x457.jpg

4. 验证

重启该机器后,验证升级情况:

[vagrant@localhost ~]$ uname -a
Linux localhost.localdomain 4.18.0-372.32.1.el8_6.x86_64 #1 SMP Thu Oct 27 15:18:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[vagrant@localhost ~]$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

WindowsVirtualBoxUCC

Env Preparation

Install VirtualBox 7.0.2:

/images/2022_10_26_09_19_41_455x301.jpg

View version:

/images/2022_10_26_09_20_32_843x589.jpg

Disable the dhcp server for default Host-only Networks:

/images/2022_10_26_09_27_37_627x467.jpg

VM Creation

Create a new vm(type Linux->Other Linux):

/images/2022_10_26_09_29_24_778x373.jpg

2 Cores, 2048MB Memory:

/images/2022_10_26_09_29_56_643x203.jpg

Specify centos85.vdi file for hard disk:

/images/2022_10_26_09_30_38_648x273.jpg

Configure the vm to specify its networking:

/images/2022_10_26_09_31_24_804x391.jpg

Visit https://192.168.56.199 should reach the ucc server web admin page:

/images/2022_10_26_09_34_44_771x540.jpg

UCC Client Testing

Create a new machine(type Windows 10):

/images/2022_10_26_09_36_04_747x326.jpg

2 Cores, 3072 MB Memory, Enable EFI:

/images/2022_10_26_09_36_49_648x238.jpg

Create 50 GB disk:

/images/2022_10_26_09_37_09_646x267.jpg

Configure this VM, select ICH9 as its chipset:

/images/2022_10_26_09_38_41_620x481.jpg

Enable Network as its boot sequence:

/images/2022_10_26_09_40_04_594x370.jpg

Change Network to host-only type:

/images/2022_10_26_09_42_16_713x344.jpg

System will boot into EFI shell, input exit to bios:

/images/2022_10_26_09_40_52_603x279.jpg

Select Boot Manage, press enter:

/images/2022_10_26_09_41_13_582x236.jpg

Select UEFI PXEv4 for booting:

/images/2022_10_26_09_41_34_638x346.jpg

pxe will boot into register name, input name ucctestwin10 for continue:

/images/2022_10_26_09_44_19_1152x854.jpg

Click setup button for initialization this node:

/images/2022_10_26_09_45_11_958x620.jpg

Initialize the disk using the default parameters:

/images/2022_10_26_09_46_25_1706x560.jpg

Now return to login page, the disk and network indication should be green, enter username/password for continue:

/images/2022_10_26_09_49_10_470x605.jpg

Click TCI for downloading the package and boot into windows 10:

/images/2022_10_26_09_49_59_1529x871.jpg

windows 10 in virtualbox:

/images/2022_10_26_09_59_21_909x816.jpg

Don’t try idv, cause in virtualbox it will run into error:

/images/2022_10_26_10_00_56_1024x531.jpg

Ubuntu Verification

Upload the image and copy it into corresponding directory:

/images/2022_10_26_10_30_02_651x127.jpg

Register image in UCC Server admin pages:

/images/2022_10_26_10_31_04_714x559.jpg

Create a new machine for testing Ubuntu:

/images/2022_10_26_10_31_44_633x307.jpg

2Core, 2048 MB , enable uefi:

/images/2022_10_26_10_32_08_663x260.jpg

Select ICH9 for chipset, and enable the network boot:

/images/2022_10_26_10_32_57_681x426.jpg

Change the network:

/images/2022_10_26_10_33_13_568x366.jpg

exit and into the bios:

/images/2022_10_26_10_34_06_616x290.jpg

Boot Manager and select UEFI PXEv4:

/images/2022_10_26_10_34_33_600x419.jpg

Register client:

/images/2022_10_26_10_35_33_658x218.jpg

Initialize the disk:

/images/2022_10_26_10_36_03_1413x407.jpg

Select Ubuntu for downloading and boot into system:

/images/2022_10_26_10_37_38_1001x301.jpg

Ubuntu desktop:

/images/2022_10_26_10_41_47_809x720.jpg