gravatar

很惭愧，就做了一点微小的工作

Dash

WorkingTipsOn3060vfioPassthrough

Jan 19, 2024
Technology

Host Preparation

Edit grub configuration:

# vim /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt"

# update-grub

Edit vfio items:

# vim /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2508

Edit initramfs items:

# vim /etc/initramfs-tools/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Bios configuration, disable igpu, and select PEG:

PEG is technically just "PCI Express Graphics" your 16x lane slot to the CPU.

Add kvm items:

# vim /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1

Get the device via devicehunt.com:

GA106 [GeForce RTX 3050 OEM]
Type	Information
ID	2508
Vendor Details
NVIDIA Corporation
Type	Information
ID	10DE

Use gpu-z under windows to fetch the rom.

Edit the file using bless under linux, find 55AA:

/images/2024_01_19_15_02_58_1227x693.jpg

Delete everything before this 55AA :

/images/2024_01_19_15_05_02_1248x606.jpg

Create the qcow2, specify its backup file:

qemu-img create -f qcow2 -b win10_pure_with_rdp_open.qcow2 -F qcow2 win10_nvidia_3050.qcow2

Create the machine:

/images/2024_01_19_15_30_00_494x489.jpg

Specify its cpus and memory:

/images/2024_01_19_15_30_26_424x168.jpg

Specify its name:

/images/2024_01_19_15_31_04_495x322.jpg

Custom its chipset and firmware:

/images/2024_01_19_15_31_31_572x194.jpg

Add a new network(macvtap) for rdp:

/images/2024_01_19_15_32_24_570x305.jpg

Using qxl and spice for verifying its working, then add nvidia 3050.

Adding GPU items

Getting the gpu card related:

$ sudo lspci -nn  | grep 03:00
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2508] (rev a1)
03:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:228e] (rev a1)

Verify its iommu groups:

$ for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort
......

/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.1

......

Edit the grub parameters:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt kvm.ignore_msrs=1 vfio-pci.ids=10de:2508,10de:228e"

update grub then reboot. verify kernel driver in use via:

# lspci -kn | grep -A 2 03:00
03:00.0 0300: 10de:2508 (rev a1)
	Subsystem: 1028:c97c
	Kernel driver in use: vfio-pci
--
03:00.1 0403: 10de:228e (rev a1)
	Subsystem: 1028:c97c
	Kernel driver in use: vfio-pci

Add the pci devices :

/images/2024_01_19_15_45_32_836x497.jpg

The modified content is listed as :

<hostdev mode="subsystem" type="pci" managed="yes">
  <source>
    <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
  </source>
  <rom file="/usr/share/OVMF/GA106.rom"/>
  <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0" multifunction="on"/>
</hostdev>

将显卡从第二槽位到第一槽位，解决了vfio问题。

/images/2024_01_19_16_26_23_499x188.jpg

/images/2024_01_19_16_29_16_590x433.jpg

添加usb设备:

/images/2024_01_19_16_34_00_227x132.jpg

去掉qxl及spice, 只使用实际的物理显卡设备来进行测试. 成功，但是uefi启动的时候无tiano core的画面。

DistCCBuildingInLXC

Jan 16, 2024
Technology

Create a profile named bridgeprofile:

$ lxc profile create bridgeprofile
$  lxc profile show bridgeprofile
config: {}
description: Bridged networking LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
name: bridgeprofile

Steps:

$ cat bridge
config: {}
description: Bridged networking LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
name: bridgeprofile
$ cat bridge | lxc profile edit bridgeprofile

Then create a lxc named distcc:

$ lxc launch -p default -p bridgeprofile ubuntu:22.04 distcc

Edit the netplan confguration in lxc instance(distcc):

root@distcc:~# cat /etc/netplan/50-cloud-init.yaml 
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: false
            addresses: [192.168.1.9/24]
            gateway4: 192.168.1.33
root@distcc:~# netplan

Install distcc via:

apt install -y distcc build-essential

Configure the distcc:

STARTDISTCC="true"
ALLOWEDNETS="192.168.1.0/24"
LISTENER="0.0.0.0"
JOBS="12"

Buildlibvirt8OnUbuntu1804

Jan 16, 2024
Technology

build steps for building libvirt on ubuntu18.04:

  275  tar xJvf libvirt-8.0.0.tar.xz 
  276  cd libvirt-8.0.0/
  277  ls
  278  meson build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_interface=enabled -Ddriver_libvirtd=enabled -Ddriver_remote=enabled -Ddriver_network=enabled --prefix=/usr
  279  apt-cache search xsltproc
  280  sudo apt install -y xsltproc
  281  meson build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_interface=enabled -Ddriver_libvirtd=enabled -Ddriver_remote=enabled -Ddriver_network=enabled --prefix=/usr
  282  apt-cache search rst2html
  283  apt-cache search rst
  284  meson build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_interface=enabled -Ddriver_libvirtd=enabled -Ddriver_remote=enabled -Ddriver_network=enabled --prefix=/usr
  285  sudo apt-get install python3-docutils
  286  meson build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_interface=enabled -Ddriver_libvirtd=enabled -Ddriver_remote=enabled -Ddriver_network=enabled --prefix=/usr
  287  apt-cache search libtirpc
  288  sudo apt install -y libtirpc-dev
  289  meson build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_interface=enabled -Ddriver_libvirtd=enabled -Ddriver_remote=enabled -Ddriver_network=enabled --prefix=/usr
  290  apt-cache search gnutls
  291  sudo apt install -y libgnutls28-dev
  292  meson build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_interface=enabled -Ddriver_libvirtd=enabled -Ddriver_remote=enabled -Ddriver_network=enabled --prefix=/usr
  293  apt-cache search libxml-2.0
  294  apt-cache search libxml
  295  sudo apt install -y libxml2-dev
  296  meson build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_interface=enabled -Ddriver_libvirtd=enabled -Ddriver_remote=enabled -Ddriver_network=enabled --prefix=/usr
  297  apt-cache search pciaccess
  298  sudo apt install -y libpciaccess-dev
  299  meson build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_interface=enabled -Ddriver_libvirtd=enabled -Ddriver_remote=enabled -Ddriver_network=enabled --prefix=/usr
  300  apt-cache search YAJL
  301  sudo apt install -y libyajl-dev
  302  meson build -Dsystem=true -Ddriver_qemu=enabled -Ddriver_interface=enabled -Ddriver_libvirtd=enabled -Ddriver_remote=enabled -Ddriver_network=enabled --prefix=/usr
  303  history

WorkingTipsOnRongHe

Jan 15, 2024
Technology

Tips

crontab items:

@reboot /usr/bin/execpipe.sh

execpipe content:

$ cat /usr/bin/execpipe.sh 
#!/bin/bash
while true; do eval "$(cat  /mypipe)" &> /mypipeoutput.txt;done
#while true; do eval "$(cat  /mypipe)";done

Create the pipe via:

$ ls / | grep mypipe
mypipe
mypipeoutput.txt

Kernel Building(VB)

Build the kernel via:

apt install -y git fakeroot build-essential ncurses-dev xz-utils libssl-dev bc flex libelf-dev bison rsync kmod cpio unzip
unzip kernel-config.zip
cp kernel-config/x86_64_defconfig .config
./scripts/config --disable DEBUG_INFO
echo "" | make ARCH=x86_64 olddefconfig
make ARCH=x86_64 -j16 LOCALVERSION=-lts2021-iotg  bindeb-pkg

Kernel patch backport:

drivers/gpu/drm/i915/display/intel_fbc.c, line 1029, not equal to tc's implementation

/drivers/gpu/drm/i915# vim i915_driver.c, 存在较大不同

OnBuildxpumanager

Dec 13, 2023
Technology

environment

vagrant vm, for using this vm we could reach out the internet(through gfw).

Steps

Get the source code and prepare the code changes:

# apt install -y git build-essential
# git clone  https://github.com/intel/xpumanager.git
# cd xpumanager
# vim ./core/src/vgpu/precheck.cpp +72    
    } else if (cmdRes.output().find("vmx") != std::string::npos) {
        /*
        *   VMX flag detected by lscpu
        */
        result->vmxFlag = true;
    } else {
        result->vmxFlag = true;
        //result->vmxFlag = false;
        //std::string msg = "No VMX flag, Please ensure Intel VT enabled in BIOS";
        //strncpy(result->vmxMessage, msg.c_str(), msg.size() + 1);
    }
# vim builder/Dockerfile.builder-ubuntu
    make -j && make install && \
---->
    make -j8 && make install && \

Build the build docker image, save the iidfile:

$ sudo docker build --build-arg BASE_VERSION=$BASE_VERSION --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --iidfile /tmp/xpum_builder_ubuntu_$BASE_VERSION.iid -f builder/Dockerfile.builder-ubuntu .
$ sudo docker images
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
<none>       <none>    8beea6fc722f   9 minutes ago   1.92GB
ubuntu       22.04     b6548eacb063   11 days ago     77.8MB
$ cp /tmp/xpum_builder_ubuntu_22.04.iid ~

Using this docker image for building the deb file(xpumanager):

sudo docker run --rm \
    -v $PWD:$PWD \
    -u $UID \
    -e CCACHE_DIR=$PWD/.ccache \
    -e CCACHE_BASEDIR=$PWD \
    $(cat /tmp/xpum_builder_ubuntu_$BASE_VERSION.iid) $PWD/build.sh
cp /home/vagrant/xpumanager/build/xpumanager_1.2.25_20231213.023315.251edc28~u22.04_amd64.deb ~

Build the xpu-smi:

rm -fr build
sudo docker run --rm \
    -v $PWD:$PWD \
    -u $UID \
    -e CCACHE_DIR=$PWD/.ccache \
    -e CCACHE_BASEDIR=$PWD \
    $(cat /tmp/xpum_builder_ubuntu_$BASE_VERSION.iid) $PWD/build.sh -DDAEMONLESS=ON
cp /home/vagrant/xpumanager/build/xpu-smi_1.2.25_20231213.023748.251edc28~u22.04_amd64.deb ~

verification

Install:

# sudo apt-get install -y ./xpu-smi_1.2.25_20231213.023748.251edc28~u22.04_amd64.deb
# ls /dev/dri/
by-path  card0  card1  renderD128  renderD129