SRIOVBACKPORTKERNELFORUBUNTU2204

Hardware/OS information:

dash@alder:~$ uname -a
Linux alder 5.15.0-92-generic #102-Ubuntu SMP Wed Jan 10 09:33:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
dash@alder:~$ lscpu | grep -i model
Model name:                         12th Gen Intel(R) Core(TM) i3-12100
Model:                              151

Install prerequisite packages:

sudo apt install dkms make debhelper devscripts build-essential flex bison mawk

Checkout the firmware:

git clone https://github.com/intel-gpu/intel-gpu-firmware
cd intel-gpu-firmware
sudo mkdir -p /lib/firmware/updates/i915/
sudo cp firmware/*.bin /lib/firmware/updates/i915/

Checkout the src code:

cd Code
git clone https://github.com/intel-gpu/intel-gpu-i915-backports/
cd intel-gpu-i915-backports/
git checkout backport/main
make i915dkmsdeb-pkg
sudo dpkg -i ../intel-i915-dkms_1.23.9.11.231003.15+i1-1_all.deb

Check the dkms status:

$ dkms status
intel-i915-dkms/1.23.9.11.231003.15, 5.15.0-92-generic, x86_64: installed
$ sudo reboot
$ sudo dmesg |grep -i backport
[    1.336656] COMPAT BACKPORTED INIT
[    1.337218] Loading modules backported from I915-23.9.11
[    1.337702] Backport generated by backports.git I915_23.9.11_PSB_231003.15
[    1.380192] [drm] I915 BACKPORTED INIT 
[    1.421899]  __init_backport+0x47/0xfa [i915]

Add kernel options:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on i915.enable_guc=3 i915.max_vfs=7"

Got the unstable conditions(once the dkms is installed, then cannot reboot).

WorkingTipsOnCasaOS

Install:

curl -fsSL https://get.casaos.io | sudo bash

Issue:

[FAILED] rclone.service is not running, Please reinstall.

Solved via:

cp /bin/mkdir /usr/bin/mkdir
cp /bin/rm /usr/bin/rm

Re-run:

curl -fsSL https://get.casaos.io | sudo bash

Using docker for running redroid:

version: "3"
services:
  redroid:
    image: redroid12:latest
    stdin_open: true
    tty: true
    privileged: true
    ports:
      - "5555:5555"
    volumes:
    # 資料存放在目前目錄下
      - ./redroid-12-data:/data
    command:
    # 禁用GPU硬體加速
      - androidboot.redroid_gpu_mode=guest
      - androidboot.use_memfd=1

  scrcpy:
    image: emptysuns/scrcpy-web:v0.1
    privileged: true
    ports:
      - "48000:8000"
    volumes:
    # 資料存放在目前目錄下
      - ./scrcpy-data:/data
    links:
    # always using myphone1
      - redroid:myphone1

Connect via:

docker exec -it redroid_scrcpy_1 adb connect myphone1:5555

做到casaos的编排里。

WorkingTipsOnUsingimgFileInDocker

准备Img文件:

$ wget https://mirrors.ustc.edu.cn/raspberry-pi-os-images/raspios_lite_arm64/images/raspios_lite_arm64-2023-12-11/2023-12-11-raspios-bookworm-arm64-lite.img.xz
$ unxz 2023-12-11-raspios-bookworm-arm64-lite.img.xz 
$ ls
2023-12-11-raspios-bookworm-arm64-lite.img

以准备好的img文件设置loop挂载点, 而后检查挂载点:

$ sudo losetup -fP --show  2023-12-11-raspios-bookworm-arm64-lite.img
/dev/loop17
$ sudo fdisk -l /dev/loop17
Disk /dev/loop17: 2.55 GiB, 2738880512 bytes, 5349376 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4e639091

Device        Boot   Start     End Sectors  Size Id Type
/dev/loop17p1         8192 1056767 1048576  512M  c W95 FAT32 (LBA)
/dev/loop17p2      1056768 5349375 4292608    2G 83 Linux

将/dev/loop17p2分区挂载到/mnt9后，而后透传给容器:

$  sudo mount /dev/loop17p2 /mnt9/
$ sudo docker run -it -v /mnt9:/raspbian rockylinux:9 bash

在容器中检查/raspbian挂载点所在的位置，分区大小为2G:

[root@34e78094dace /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         1.8T  1.5T  259G  86% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/loop17p2   2.0G  1.6G  330M  83% /raspbian
/dev/nvme0n1p2  1.8T  1.5T  259G  86% /etc/hosts
tmpfs            63G     0   63G   0% /proc/asound
tmpfs            63G     0   63G   0% /proc/acpi
tmpfs            63G     0   63G   0% /proc/scsi
tmpfs            63G     0   63G   0% /sys/firmware

WorkingTipsOn3060vfioPassthrough

Host Preparation

Edit grub configuration:

# vim /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt"

# update-grub

Edit vfio items:

# vim /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2508

Edit initramfs items:

# vim /etc/initramfs-tools/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Bios configuration, disable igpu, and select PEG:

PEG is technically just "PCI Express Graphics" your 16x lane slot to the CPU.

Add kvm items:

# vim /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1

Get the device via devicehunt.com:

GA106 [GeForce RTX 3050 OEM]
Type	Information
ID	2508
Vendor Details
NVIDIA Corporation
Type	Information
ID	10DE

Use gpu-z under windows to fetch the rom.

Edit the file using bless under linux, find 55AA:

Delete everything before this 55AA :

Create the qcow2, specify its backup file:

qemu-img create -f qcow2 -b win10_pure_with_rdp_open.qcow2 -F qcow2 win10_nvidia_3050.qcow2

Create the machine:

Specify its cpus and memory:

Specify its name:

Custom its chipset and firmware:

Add a new network(macvtap) for rdp:

Using qxl and spice for verifying its working, then add nvidia 3050.

Adding GPU items

Getting the gpu card related:

$ sudo lspci -nn  | grep 03:00
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2508] (rev a1)
03:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:228e] (rev a1)

Verify its iommu groups:

$ for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort
......

/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.1

......

Edit the grub parameters:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt kvm.ignore_msrs=1 vfio-pci.ids=10de:2508,10de:228e"

update grub then reboot. verify kernel driver in use via:

# lspci -kn | grep -A 2 03:00
03:00.0 0300: 10de:2508 (rev a1)
	Subsystem: 1028:c97c
	Kernel driver in use: vfio-pci
--
03:00.1 0403: 10de:228e (rev a1)
	Subsystem: 1028:c97c
	Kernel driver in use: vfio-pci

Add the pci devices :

The modified content is listed as :

<hostdev mode="subsystem" type="pci" managed="yes">
  <source>
    <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
  </source>
  <rom file="/usr/share/OVMF/GA106.rom"/>
  <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0" multifunction="on"/>
</hostdev>

将显卡从第二槽位到第一槽位，解决了vfio问题。

添加usb设备:

去掉qxl及spice, 只使用实际的物理显卡设备来进行测试. 成功，但是uefi启动的时候无tiano core的画面。

DistCCBuildingInLXC

Create a profile named bridgeprofile:

$ lxc profile create bridgeprofile
$  lxc profile show bridgeprofile
config: {}
description: Bridged networking LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
name: bridgeprofile

Steps:

$ cat bridge
config: {}
description: Bridged networking LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
name: bridgeprofile
$ cat bridge | lxc profile edit bridgeprofile

Then create a lxc named distcc:

$ lxc launch -p default -p bridgeprofile ubuntu:22.04 distcc

Edit the netplan confguration in lxc instance(distcc):

root@distcc:~# cat /etc/netplan/50-cloud-init.yaml 
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: false
            addresses: [192.168.1.9/24]
            gateway4: 192.168.1.33
root@distcc:~# netplan

Install distcc via:

apt install -y distcc build-essential

Configure the distcc:

STARTDISTCC="true"
ALLOWEDNETS="192.168.1.0/24"
LISTENER="0.0.0.0"
JOBS="12"