gravatar

很惭愧，就做了一点微小的工作

Dash

Usingdockerforbuildingkubeadm

Jan 21, 2021
Technology

X86 version kubeadm building process:

# docker run -it ubuntu:18.04 /bin/bash
# cat /etc/issue
Ubuntu 18.04.5 LTS \n \l
# apt-get update -y
# apt-get install -y wget unzip vim build-essential rsync
# wget https://github.com/kubernetes/kubernetes/archive/v1.19.7.zip
# wget https://golang.org/dl/go1.15.7.linux-amd64.tar.gz
# tar -C /usr/local -xzf go1.15.7.linux-amd64.tar.gz
# export PATH=$PATH:/usr/local/go/bin
# go version
go version go1.15.7 linux/amd64
# cd kubernetes-v1.19.7
# vim cmd/kubeadm/app/constants/constants.go
CertificateValidity = time.Hour * 24 * 365 * 100
#  vim vendor/k8s.io/client-go/util/cert/cert.go
func NewSelfSignedCACert
NotAfter: 	now.Add(duration365d * 100).UTC(),
func GenerateSelfSignedCertKeyWithFixtures
maxAge := 100 * time.Hour * 24 * 365
# make all WHAT=cmd/kubeadm
# _output/bin/kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"archive", BuildDate:"2021-01-21T07:15:36Z", GoVersion:"go1.15.7", Compiler:"gc", Platform:"linux/amd64"}
# cp _output/bin/kubeadm ..

Arm64 version kubeadm building process:

Edit files and make changes:

# vim hack/make-rules/cross.sh
    make all WHAT="${KUBE_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_SERVER_PLATFORMS[*]}"
    
    #make all WHAT="${KUBE_NODE_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_NODE_PLATFORMS[*]}"
    #
    #make all WHAT="${KUBE_CLIENT_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_CLIENT_PLATFORMS[*]}"
    #
    #make all WHAT="${KUBE_TEST_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_PLATFORMS[*]}"
    #
    #make all WHAT="${KUBE_TEST_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_SERVER_PLATFORMS[*]}"
#  vim hack/lib/golang.sh
    readonly KUBE_SUPPORTED_SERVER_PLATFORMS=(
    #  linux/amd64
    #  linux/arm
      linux/arm64
    #  linux/s390x
    #  linux/ppc64le
    )
    
    //.............
    
    kube::golang::server_targets() {
      local targets=(                       
      #  cmd/kube-proxy                                                        
      #  cmd/kube-apiserver                 
      #  cmd/kube-controller-manager                                 
      #  cmd/kubelet 
        cmd/kubeadm
      #  cmd/kube-scheduler
      #  vendor/k8s.io/apiextensions-apiserver
      #  cluster/gce/gci/mounter
      )

Build:

# make cross

v1.20.7 update

Have to change go version to v1.17 for building, also notice the memory usage.

apt-get update -y && apt-get install -y wget unzip vim build-essential rsync
wget https://github.com/kubernetes/kubernetes/archive/v1.20.7.zip
wget https://golang.org/dl/go1.17.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.17.linux-amd64.tar.gz 
export PATH=$PATH:/usr/local/go/bin
go version
unzip v1.20.7.zip 
cd kubernetes-1.20.7/
vim cmd/kubeadm/app/constants/constants.go
vim vendor/k8s.io/client-go/util/cert/cert.go
make all WHAT=cmd/kubeadm
cp _output/bin/kubeadm ..

WorkingTipsOnRongV7.0

Jan 20, 2021
Technology

用于记录基于kubespray v2.15.0离线化过程。

包准备

Download kubespray v2.15.0 source code via:

# wget https://github.com/kubernetes-sigs/kubespray/archive/v2.15.0.tar.gz

使用RongRobots得到离线包以便替换。

$ ls -l -h RobotSon.tar.gz 
-rw-r--r-- 1 dash root 882M Jan 20 15:58 RobotSon.tar.gz

准备的目录如下:

# mkdir RobotSon
# tar xzvf RobotSon.tar.gz -C RobotSon/
# ls 
kubespray-2.15.0.tar.gz  Origin  RobotSon  RobotSon.tar.gz  Rong

代码修改

替换静态文件:

# rm -f Rong/pre-rong/rong_static/for_cluster/calicoctl 
# rm -f Rong/pre-rong/rong_static/for_cluster/cni-plugins-linux-amd64-v0.8.7.tgz 
# rm -f Rong/pre-rong/rong_static/for_cluster/kube*
# cp RobotSon/release/calicoctl Rong/pre-rong/rong_static/for_cluster/
# cp RobotSon/release/cni-plugins-linux-amd64-v0.9.0.tgz Rong/pre-rong/rong_static/for_cluster/
# cp RobotSon/release/kube* Rong/pre-rong/rong_static/for_cluster/

创建离线docker镜像包并替代原有离线镜像包:

# cd RobotSon/data
# tar czvf docker.tar.gz docker/
# cd ../../
# rm -f Rong/pre-rong/rong_static/for_master0/docker.tar.gz
# mv RobotSon/data/docker.tar.gz Rong/pre-rong/rong_static/for_master0/

更改rong/1_preinstall/roles/preinstall/tasks/main.yml, 更改为新的静态包.

替换rong/3_k8s目录:

# tar xzvf kubespray-2.15.0.tar.gz
# rm -rf rong/3_k8s/
# mv kubespray-2.15.0/* rong/3_k8s/

更改bootstrap角色:

# cp ./rong/3_k8s/roles/bootstrap-os/tasks/main.yml ./rong/3_k8s/roles/bootstrap-os/task/main_main.yml
# cp /run/media/dash/aa3eda99-dc11-4c07-a5f1-d00eb0acc850/Rong_V7.0.0/Origin/rong/3_k8s/roles/bootstrap-os/tasks/main_kfz.yml ./rong/3_k8s/roles/bootstrap-os/tasks/
# cp /run/media/dash/aa3eda99-dc11-4c07-a5f1-d00eb0acc850/Rong_V7.0.0/Origin/rong/3_k8s/roles/bootstrap-os/tasks/main.yml ./rong/3_k8s/roles/bootstrap-os/tasks/

更改container-engine/docker角色，与上差不多的步骤。

更改rong-vars.yml里的相关定义:

kubeadm_download_url:
kubelet_download_url:
kubectl_download_url:
helm_download_url:

helm_enabled: true
#helm_version: "v2.16.1"
helm_skip_refresh: true

containerd_version: '1.2.13'

ArchInstallOnUSB

Jan 5, 2021
Technology

Hardware

320G usb disk, laptop(running archlinux already).

Steps

fdisk the usb disk and create with following partitions:

$ sudo fdisk -l /dev/sdc
Disk /dev/sdc：298.09 GiB，320072933376 字节，625142448 个扇区
磁盘型号：Storage         
单元：扇区 / 1 * 512 = 512 字节
扇区大小(逻辑/物理)：512 字节 / 512 字节
I/O 大小(最小/最佳)：512 字节 / 512 字节
磁盘标签类型：dos
磁盘标识符：0x112a2f3d

设备       启动    起点      末尾      扇区   大小 Id 类型
/dev/sdc1          2048   1050623   1048576   512M ef EFI (FAT-12/16/32)
/dev/sdc2       1050624 625142447 624091824 297.6G 83 Linux

Format the disk:

$ $ sudo mkfs.fat -F32 /dev/sdc1
mkfs.fat 4.1 (2017-01-24)
$ sudo mkfs.ext4 /dev/sdc2

Install arch-install-scripts on archlinux. Then mount the disk to install point:

$ sudo mount /dev/sdc2 /mnt
$ sudo mkdir -p /mnt/boot
$ sudo mount /dev/sdc1 /mnt/boot

Now use pacstrap for installing basic system onto usb disk:

$ sudo pacstrap -c /mnt base linux linux-firmware base-devel

Generate /etc/fstab:

# genfstab -U /mnt >> /mnt/etc/fstab
# vim /mnt/etc/fstab
comment the swap partition

chroot into /mnt:

# arch-chroot /mnt

# ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
# pacman -S vim
# vim /etc/locale.gen
en_US.UTF-8 UTF-8  
en_US ISO-8859-1  
zh_CN.GB18030 GB18030  
zh_CN.GBK GBK  
zh_CN.UTF-8 UTF-8  
zh_CN GB2312 
# locale-gen
# vim /etc/locale.conf
LANG=en_US.UTF-8
# vim /etc/hostname
archusb
# vim /etc/hosts 
    # Static table lookup for hostnames.
    # See hosts(5) for details.
    127.0.0.1	localhost
    ::1		localhost
    127.0.1.1	archusb
# pacman -S net-tools tcpdump iotop dhcpcd openssh dosfstools ntfs-3g amd-ucode intel-ucode grub efibootmgr
# systemctl enable sshd
# cat /etc/mkinitcpio.conf | grep block
    #    HOOKS=(base udev autodetect block filesystems)
    #    HOOKS=(base udev block filesystems)
    #    HOOKS=(base udev block mdadm encrypt filesystems)
    #    HOOKS=(base udev block lvm2 filesystems)
    HOOKS=(base udev block keyboard autodetect modconf filesystems fsck)
#  mkinitcpio -P
# passwd

Make grub configuration:

# grub-install --target=i386-pc /dev/sdc --recheck
# grub-install --target=x86_64-efi --efi-directory=/boot/efi --removable --recheck

Or(uefi mode):

grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB 
grub-mkconfig -o /boot/grub/grub.cfg

Support generic gpu:

# pacman -S xf86-video-vesa xf86-video-ati xf86-video-intel xf86-video-amdgpu xf86-video-nouveau xf86-video-fbdev

Network configuration:

# pacman -S networkmanager
# systemctl enable NetworkManager
# grub-mkconfig -o /boot/grub/grub.cfg

Now you could use usb disk for booting up the system, enjoy it.

libvirt configuration

Install iptables, etc.

# pacman -S ebtables iptables dnsmasq

Configure bridge networking using network manager:

$ nmcli connection add type bridge ifname br0 stp no
$ nmcli connection add type bridge-slave ifname enp30s0 master br0

Case static ip address:

nmcli conn add type bridge ifname br0 ipv4.method manual ipv4.address "10.137.149.5/24" ipv4.gateway "10.137.149.1" ipv4.dns 223.5.5.5 
nmcli connection add type bridge-slave ifname eth0 master br0

Case dhcp(notice the bridge name):

nmcli connection modify bridge-br0 ipv4.method auto

Change mtu to 9000:

# nmcli connection modify bridge-slave-eth0 802-3-ethernet.mtu 9000
# nmcli connection show bridge-slave-eth0 | grep mtu
802-3-ethernet.mtu:                     9000

iptables for libvirt:

# iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT
# iptables-save -f /etc/iptables/iptables.rules
# systemctl enable iptables.service

Then your bridge could be use.

TurnToRong

Dec 29, 2020
Technology

现场安装时，因为某些不可控的原因，可能无法安装定制化操作系统，此时可使用以下步骤，从最小化安装的Ubuntu18.04 转换为RONG节点:

以下操作以Ubuntu18.04.5为例说明，默认操作用户为安装时创建的用户kkk, 现场需要根据情况灵活调整。

  1. 上传ISO到机器:
# scp ./ubuntu-18.04.5-server-amd64-auto-xfs.iso kkk@192.168.122.32:/home/kkk
kkk@192.168.122.32's password:
  1. 在机器上挂载iso:
kkk@ubuntu:~$ sudo mount ubuntu-18.04.5-server-amd64-auto-xfs.iso /media/cdrom
[sudo] password for kkk: 
mount: /mnt: WARNING: device write-protected, mounted read-only.
  1. 使用iso作为本地安装源:
# rm -f /etc/apt/sources.list
# apt-cdrom -m -d=/media/cdrom add
# cat /etc/apt/sources.list
deb cdrom:[Ubuntu-Server 18.04.5 LTS _Bionic Beaver_ - Release amd64 (20200810)]/ bionic main restricted
  1. 此时apt-get更新源并安装对应的包:
# apt-get update 
# apt-get install nfs-common openssh-server update-motd parted build-essential telnet tcpdump python

安装完毕后程序会自动umount /media/cdrom下挂载的ISO， 如果提示需要重新mount /media/cdrom的时候，则在另一终端重新mount iso至/media/cdrom下则可。

  1. 注入root免登录密钥
$ sudo su
#  ssh-keygen 
一路按回车，创建公钥私钥
# vim  /root/.ssh/authorized_keys
粘贴以下内容, 此内容在rong ISO的preseed/auto.seed中可以找到, 开头为"ssh-rsa", 结尾为"DashSSD"标识.

ssh-rsa owaugowugouwoguwougowuoguwougouwogwe例子例子例子例子例子例子**************= dash@DashSSD
  1. 此时可以进行RONG的正常部署, 不一定需要使用test用户登录。

WorkingTipsOnVNCRemote

Dec 11, 2020
Technology

Hardware & OS

Hardware configuration:

# lscpu
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
32Core
# free -g
              total        used        free      shared  buff/cache   available
Mem:             62          19          10           0          33          42
Swap:             0           0           0
# df -h
/dev/mapper/vg-root  1.7T  1.1T  538G  66% /

OS Configuration:

# cat /etc/issue
Ubuntu 16.04.4 LTS \n \l

AIM

To use this server as the vagrant environment.

vagrant-libvirt

use docker for running vagrant:

# docker pull vagrantlibvirt/vagrant-libvirt:latest

Install libvirtd related:

# apt-get install -y virt-manager
# systemctl status libvirt-bin qemu

Desktop

Use awesome as the default desktop:

# apt-get install -y i3
# /usr/lib/apt/apt-helper download-file https://debian.sur5r.net/i3/pool/main/s/sur5r-keyring/sur5r-keyring_2020.02.03_all.deb keyring.deb SHA256:c5dd35231930e3c8d6a9d9539c846023fe1a08e4b073ef0d2833acd815d80d48
# dpkg -i ./keyring.deb
# echo "deb http://debian.sur5r.net/i3/ $(grep '^DISTRIB_CODENAME=' /etc/lsb-release | cut -f2 -d=) universe" >> /etc/apt/sources.list.d/sur5r-i3.list
# apt-get update -y
# apt install i3
# apt-get install -y tigervncserver
# vncpasswd
# vncserver -localhost -nolisten tcp
# vim ~/.vnc/xstartup
#!/bin/bash
i3 &

Change to lxde4:

cat ~/.vnc/xstartup
	#/etc/X11/Xsession
	exec startlxde

client

Enable the ssh transfering:

$ ssh -p 62022 -L 127.0.0.1:5901:localhost:5901 root@xxx.xxx.xxx.xxx

then viewer localhost:5901 you could see the desktop