gravatar

很惭愧，就做了一点微小的工作

Dash

WorkingTipsOnIncusBasedFogServer

May 8, 2024
Technology

1. dhcpd服务器考虑

1.1 dhcpd在容器内出现的问题

因为incus内部使用的是私有网络，因此一旦将isc-dhcp-server启动在容器内，则只能监听容器内部的地址，而一旦更改子网定义为主机网络侧，则会出现dhcpd.conf检查错误:

5月 08 08:58:44 fogincuschinese dhcpd[2532]: No subnet declaration for eth0 (10.147.148.100).
5月 08 08:58:44 fogincuschinese dhcpd[2532]: ** Ignoring requests on eth0.  If this is not what
5月 08 08:58:44 fogincuschinese dhcpd[2532]:    you want, please write a subnet declaration
5月 08 08:58:44 fogincuschinese dhcpd[2532]:    in your dhcpd.conf file for the network segment
5月 08 08:58:44 fogincuschinese dhcpd[2532]:    to which interface eth0 is attached. **
5月 08 08:58:44 fogincuschinese dhcpd[2532]: 
5月 08 08:58:44 fogincuschinese dhcpd[2532]: 
5月 08 08:58:44 fogincuschinese dhcpd[2532]: Not configured to listen on any interfaces!

出现上述问题的原因在于：容器内eth0为10.147.148.100, 而主机侧为192.168.1.0.24, 无法写如下的配置文件(diff文件更改了默认dhcpd.conf中的监听网段) :

root@fogincuschinese:~# diff /etc/dhcp/dhcpd.conf /etc/dhcp/dhcpd.conf.back 
24c24
< subnet 192.168.1.0 netmask 255.255.255.0{
---
> subnet 10.147.148.0 netmask 255.255.255.0{
26c26
<     range dynamic-bootp 192.168.1.50 192.168.1.90;
---
>     range dynamic-bootp 10.147.148.10 10.147.148.254;
29c29
<     option routers 192.168.1.33;
---
>     option routers 10.147.148.1;
31c31
<     next-server 192.168.1.40;
---
>     next-server 10.147.148.100;

1.2 dhcpd在主机上的实现

安装:

# apt install -y isc-dhcp-server

配置，使能:

$ sudo vim /etc/dhcp/dhcpd.conf 
subnet 192.168.1.0 netmask 255.255.255.0{
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 192.168.1.50 192.168.1.90;
    default-lease-time 21600;
    max-lease-time 43200;
    option routers 192.168.1.33;
    option domain-name-servers 223.5.5.5;
    next-server 192.168.1.40;
}
$ sudo systemctl daemon-reload
$ sudo systemctl start isc-dhcp-server
$ sudo systemctl enable isc-dhcp-server

需要对齐fogserver中的配置.

2. tftpd服务

如果直接使用inpus中的容器，则因为tftp的通信协商机制中，会因为使用1024~65535的随机端口和客户端通信，而容器本身比较难搞定proxy, 而导致通信失败。
因此我们需要将tftpd也从里面搞出来， 在主机上实现。

但，如果做了这么多以后，还是容器吗？

所以上面的方式，未必适合

3. macvlan网

直接用macvlan直接覆盖。

注意替换IP：

Change FOG Server IP Address
Procedural Steps
Follow appropriate steps for your Linux distribution to change the OS’s IP address.

Update the ipaddress= field (and other fields if necessary) inside the /opt/fog/.fogsettings file. The .fogsettings file.

Rerun the installer, you’ll need to use –recreate-CA and –recreate-keys keys as the installer provides a certificate with a Common Name based on the ip which will be shipped in the iPxe kernel and failed to load any https resources as the certificate isn’t valid anymore.

Update the IP address inside /tftpboot/default.ipxe (look for the chain line i.e chain https://x.x.x.x/fog/service/ipxe/boot.php##params)

Update the IP address for the storage node on the FOG system where you changed the IP address Web Interface -> Storage Management

Update the IP address on a any master storage node that may reference this FOG server Web Interface -> Storage Management

(For master server) Update the FOG_WEB_HOST value Web Interface -> FOG Configuration -> FOG Settings -> Web Server -> FOG_WEB_HOST

(For master server) Update the FOG_TFTP_HOST value Web Interface -> FOG Configuration -> FOG Settings -> TFTP Server -> FOG_TFTP_HOST

Optionaly if you have configured a dhcpd:

Update IP addresses (fog and gateway) inside the /etc/dhcp/dhcpd.conf.

Don’t forgot to check your /etc/export for nfs server as well as your apache2 configuration as the installer override it.

incus 快速启动流程

  1. 用户需要安装好incus.
  2. 导入镜像，配置网路。
  3. 快速开出实例，作为Pxe和部署服务器使用。

Steps:

dash@server:~$ tar xzvf incusdebs.tar.gz
$ sudo chmod 777 -R incuddebs
$ sudo apt install -y incus
sudo adduser dash incus-admin

init yaml:

config:
  images.auto_update_interval: "0"
networks: []
storage_pools:
- config: {}
  description: ""
  name: default
  driver: dir
profiles:
- config: {}
  description: ""
  devices:
    root:
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null

quickly init:

# cat init.sh 
cat <<EOF | incus admin init --preseed
config:
  images.auto_update_interval: "0"
networks: []
storage_pools:
- config: {}
  description: ""
  name: default
  driver: dir
profiles:
- config: {}
  description: ""
  devices:
    root:
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null
EOF

Create incusbr0(macvlan):

 incus network create incusbr0 --type=macvlan parent=enp7s0

Edit profile default:

cat default.yaml | incus profile edit default

Add new profile:

incus profile create nfs-server
    cat nfs-server-profile.yaml | incus profile edit nfs-server

另外的机器上导出镜像:

# incus publish fogincuschinese --alias fogAuto
Instance published with fingerprint: 0a4a4299661f19d880a203031b4e7df88996a99be110979975633baf9504b1dc
# incus image export fogAuto .
Image exported successfully!

开始导入镜像:

# ls -l -h 0a4a4299661f19d880a203031b4e7df88996a99be110979975633baf9504b1dc.tar.gz 
-rw-rw-r-- 1 dash dash 857M May  8 09:06 0a4a4299661f19d880a203031b4e7df88996a99be110979975633baf9504b1dc.tar.gz
root@server:/home/dash# incus image import 0a4a4299661f19d880a203031b4e7df88996a99be110979975633baf9504b1dc.tar.gz --alias fogAuto
Image imported with fingerprint: 0a4a4299661f19d880a203031b4e7df88996a99be110979975633baf9504b1dc
root@server:/home/dash# incus image list
+---------+--------------+--------+----------------------------------------+--------------+-----------+-----------+----------------------+
|  ALIAS  | FINGERPRINT  | PUBLIC |              DESCRIPTION               | ARCHITECTURE |   TYPE    |   SIZE    |     UPLOAD DATE      |
+---------+--------------+--------+----------------------------------------+--------------+-----------+-----------+----------------------+
| fogAuto | 0a4a4299661f | no     | Debian bookworm amd64 (20240506_05:24) | x86_64       | CONTAINER | 856.42MiB | 2024/05/08 09:08 UTC |
+---------+--------------+--------+----------------------------------------+--------------+-----------+-----------+----------------------+

开始开出第一个实例:

# LANG=zh_CN.UTF-8 incus launch fogAuto fogInstance  -p nfs-server -p default -c security.privileged=true -c raw.apparmor="mount fstype=rpc_pipefs, mount fstype=nfsd,"
Launching fogInstance
root@server:/home/dash# incus  list        
+-------------+---------+---------------------+------+-----------+-----------+
|    NAME     |  STATE  |        IPV4         | IPV6 |   TYPE    | SNAPSHOTS |
+-------------+---------+---------------------+------+-----------+-----------+
| fogInstance | RUNNING | 192.168.1.44 (eth0) |      | CONTAINER | 0         |
+-------------+---------+---------------------+------+-----------+-----------+
root@server:/home/dash# incus exec fogInstance bash
root@fogInstance:~#

进入到实例里开始重新部署为192.168.1.46:

root@fogInstance:~# cd regen/
root@fogInstance:~/regen# ls
1_regen.sh  2_reinstall.sh  cn-fogproject-master  inventoy.ini	mysql.sh  regen.yml  templates
root@fogInstance:~/regen# vim inventoy.ini 
root@fogInstance:~/regen# ./1_regen.sh 

PLAY [Write fogsettings] *********************************************************************************************************************************************************************

...
这里需要重启一次容器实例，非常快:   

root@server:/home/dash# incus exec fogInstance bash
root@fogInstance:~# cd regen/
root@fogInstance:~/regen# ./2_reinstall.sh

push 镜像到相应位置:

incus file push -pr idvnext/ fogInstance/images/idvnext/

接下来就可以愉快的玩耍了。

综合

传递文件:

$ scp -r hostincus/ dash@192.168.1.38:~

incusAdjustNetworking

May 8, 2024
Technology

The default networking bridge incusbr0 enabled the dhcp by default, that’s not good for using dhcpd service in containers, so I have to remove the default behavior of the incusbr0, and add a new behavior for it.

Directly delete the bridge will get an error:

$ incus network delete incusbr0
Error: The network is currently in use

Show this network’s usage:

$ incus network show incusbr0
config:
  ipv4.address: 10.147.148.1/24
  ipv4.nat: "true"
  ipv6.address: none
description: ""
name: incusbr0
type: bridge
used_by:
- /1.0/instances/fogincuschinese
- /1.0/instances/foginlxc
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

Edit its profile:

$ incus profile edit default
config: {}
description: Default Incus profile
devices:
-  eth0:
-    name: eth0
-    network: incusbr0
-    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
- /1.0/instances/foginlxc
- /1.0/instances/fogincuschinese

Now you could delete this networking via:

$ incus network delete incusbr0
Network incusbr0 deleted

RE-create the networking via following command(dhcpv4/v6 disabled):

$ incus network create incusbr0 ipv4.dhcp=false ipv6.dhcp=false ipv4.address=10.147.148.1/24
Network incusbr0 created

Check this networking:

$ ip a show incusbr0
10: incusbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:c9:c4:e6 brd ff:ff:ff:ff:ff:ff
    inet 10.147.148.1/24 scope global incusbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:1515:fb8e:9dab::1/64 scope global 
       valid_lft forever preferred_lft forever

RE-Add the networking profile into default:

$ incus profile edit default
...
description: Default Incus profile
devices:
+  eth0:
+    name: eth0
+    network: incusbr0
+    type: nic
  root:
...

Re-lauch the previously stopped container instance:

$ incus start fogincuschinese
$ incus list
+-----------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
|      NAME       |  STATE  |         IPV4          |                     IPV6                      |   TYPE    | SNAPSHOTS |
+-----------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| fogincuschinese | RUNNING | 10.147.148.100 (eth0) | fd42:1515:fb8e:9dab:216:3eff:fef3:8307 (eth0) | CONTAINER | 0         |
+-----------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+

Until now you could enable the dhcpd in container and then use forwarding rules for redirect to host.

Final command:

incus network create incusbr0 ipv4.dhcp=false ipv4.address=10.147.148.1/24 ipv4.nat=true ipv6.address=none

InstallFogServerInIncus

Apr 28, 2024
Technology

Bridge ways:

Create incus instance:

$ incus launch images:debian/12 fogserver
$ incus list                   
+-----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
|   NAME    |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+-----------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| fogserver | RUNNING | 10.238.20.219 (eth0) | fd42:9396:9aca:8842:216:3eff:fe04:5da6 (eth0) | CONTAINER | 0         |
+-----------+---------+----------------------+-----------------------------------------------+-----------+-----------+

Configure the incus instance:

dash@hope:~$ incus exec fogserver bash
root@fogserver:~# vi /etc/apt/sources.list
root@fogserver:~# cat /etc/apt/sources.list
# Generated by distrobuilder
deb http://mirrors.ustc.edu.cn/debian bookworm main
deb http://mirrors.ustc.edu.cn/debian bookworm-updates main
deb http://mirrors.ustc.edu.cn/debian-security/ bookworm-security main
root@fogserver:~# apt update -y && apt install -y git

Fetch the source code:

root@fogserver:~# git clone https://github.com/fogproject/fogproject.git fogproject-master

Cloning into 'fogproject-master'...
remote: Enumerating objects: 158214, done.
remote: Counting objects: 100% (4216/4216), done.
remote: Compressing objects: 100% (1073/1073), done.
remote: Total 158214 (delta 3135), reused 4011 (delta 3023), pack-reused 153998
Receiving objects: 100% (158214/158214), 848.89 MiB | 14.41 MiB/s, done.
Resolving deltas: 100% (112946/112946), done.
root@fogserver:~# du -hs fogproject-master/
890M	fogproject-master/

Using fogserver in container is not a good idea.

InstallInpusOnUbuntu2004

Apr 28, 2024
Technology

1. Install/Configuration

Import keyring:

mkdir -p /etc/apt/keyrings/
curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc

update repository and install inpus:

apt update -y
apt install -y incus
 apt install -y incus-ui-canonical
incus config set core.https_address :8443

Generate cert file in browser:

/images/2024_04_28_09_32_56_1083x755.jpg

In incus server:

root@hope:~# cp /home/dash/Downloads/incus-ui.crt ./Downloads/
root@hope:~# incus config trust add-certificate Downloads/incus-ui.crt

In browser:

/images/2024_04_28_09_34_37_1035x384.jpg

The import file should be:

/images/2024_04_28_09_34_57_414x98.jpg

Then back to browser windows, confirm the imported cert:

/images/2024_04_28_09_35_08_630x331.jpg

Your UI would be looks like:

/images/2024_04_28_09_35_46_945x645.jpg

Add your user into incus group:

sudo adduser dash incus-admin

Init the incus:

$ incus admin init
Would you like to use clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (dir, lvm, lvmcluster, btrfs) [default=btrfs]: dir
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=incusbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: no
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]:

Create the first instance:

incus launch images:ubuntu/22.04 first

Finally we could remove lxd:

$ sudo snap remove lxd

2. images

list image:

$ incus image list
+-------+--------------+--------+----------------------------------------+--------------+-----------+-----------+----------------------+
| ALIAS | FINGERPRINT  | PUBLIC |              DESCRIPTION               | ARCHITECTURE |   TYPE    |   SIZE    |     UPLOAD DATE      |
+-------+--------------+--------+----------------------------------------+--------------+-----------+-----------+----------------------+
|       | 8b2691953577 | no     | Debian bookworm amd64 (20240424_05:24) | x86_64       | CONTAINER | 94.50MiB  | 2024/04/28 02:09 UTC |
+-------+--------------+--------+----------------------------------------+--------------+-----------+-----------+----------------------+
|       | 479d8812eada | no     | Ubuntu jammy amd64 (20240427_07:42)    | x86_64       | CONTAINER | 120.93MiB | 2024/04/28 01:55 UTC |
+-------+--------------+--------+----------------------------------------+--------------+-----------+-----------+----------------------+

incus remote show images repositories:

$ incus remote list
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
|      NAME       |                URL                 |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC | GLOBAL |
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
| images          | https://images.linuxcontainers.org | simplestreams | none        | YES    | NO     | NO     |
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
| local (current) | unix://                            | incus         | file access | NO     | YES    | NO     |
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+

search images:

$ incus image list images: bookworm
+--------------------------------+--------------+--------+----------------------------------------+--------------+-----------------+-----------+----------------------+
|             ALIAS              | FINGERPRINT  | PUBLIC |              DESCRIPTION               | ARCHITECTURE |      TYPE       |   SIZE    |     UPLOAD DATE      |
+--------------------------------+--------------+--------+----------------------------------------+--------------+-----------------+-----------+----------------------+
| debian/12 (7 more)             | 2b5e45154f58 | yes    | Debian bookworm amd64 (20240424_05:24) | x86_64       | VIRTUAL-MACHINE | 349.13MiB | 2024/04/24 00:00 UTC |
+--------------------------------+--------------+--------+----------------------------------------+--------------+-----------------+-----------+----------------------+
| debian/12 (7 more)             | 8b2691953577 | yes    | Debian bookworm amd64 (20240424_05:24) | x86_64       | CONTAINER       | 94.50MiB  | 2024/04/24 00:00 UTC |
+--------------------------------+--------------+--------+----------------------------------------+--------------+-----------------+-----------+----------------------+
| debian/12/arm64 (3 more)       | dbba0a514259 | yes    | Debian bookworm arm64 (20240424_05:24) | aarch64      | CONTAINER       | 91.50MiB  | 2024/04/24 00:00 UTC |
+--------------------------------+--------------+--------+----------------------------------------+--------------+-----------------+-----------+----------------------+
| debian/12/arm64 (3 more)       | e2fc3247a569 | yes    | Debian bookworm arm64 (20240424_05:24) | aarch64      | VIRTUAL-MACHINE | 338.21MiB | 2024/04/24 00:00 UTC |
+--------------------------------+--------------+--------+----------------------------------------+--------------+-----------------+-----------+----------------------+
....

launch:

incus launch -p default -p bridgeprofile images:debian/12 kissdebian

safebootloaderTips

Apr 25, 2024
Technology

From Makefile:

O ?= ./build
......
$O/bootx64.efi: $O/chainload/loader.efi $O/vmlinuz $O/initrd.cpio.xz
	$O/chainload/unify-kernel $@ \
		linux=$O/vmlinuz \
		initrd=$O/initrd.cpio.xz \
		cmdline=config/cmdline-5.4.117.txt

file content:

kkk@kkk:~/safeboot-loader$ ls build/chainload/loader.efi 
build/chainload/loader.efi
kkk@kkk:~/safeboot-loader$ file build/chainload/loader.efi 
build/chainload/loader.efi: PE32+ executable (EFI application) x86-64 (stripped to external PDB), for MS Windows
kkk@kkk:~/safeboot-loader$ ls build/chainload/loader.efi  -l -h
-rwxrwxr-x 1 idv idv 52K  4月 18 14:32 build/chainload/loader.efi
kkk@kkk:~/safeboot-loader$ vim build/chainload/loader.efi 
kkk@kkk:~/safeboot-loader$ ls build/vmlinuz 
build/vmlinuz
kkk@kkk:~/safeboot-loader$ ls build/vmlinuz  -l -h
-rw-rw-r-- 1 idv idv 2.5M  4月 18 10:12 build/vmlinuz
kkk@kkk:~/safeboot-loader$ ls build/initrd.cpio.xz -l -h
-rw-rw-r-- 1 idv idv 13M  4月 18 14:32 build/initrd.cpio.xz
kkk@kkk:~/safeboot-loader$ ls config/cmdline-5.4.117.txt 
config/cmdline-5.4.117.txt
kkk@kkk:~/safeboot-loader$ cat config/cmdline-5.4.117.txt 
earlyprintk=serial,ttyS0,115200 console=tty0 console=ttyS0,115200 noefi acpi=of