gravatar

很惭愧，就做了一点微小的工作

Dash

RPICluster

Feb 5, 2021
Technology

Flashing

Flash with following os:

root@ubuntu:/home/ubuntu# uname -a
Linux ubuntu 5.4.0-1028-raspi #31-Ubuntu SMP PREEMPT Wed Jan 20 11:30:45 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
root@ubuntu:/home/ubuntu# cat /etc/issue
Ubuntu 20.04.2 LTS \n \l

Disable auto update:

# systemctl stop apt-daily.timer;systemctl disable apt-daily.timer ; systemctl stop apt-daily-upgrade.timer ; systemctl disable apt-daily-upgrade.timer; systemctl stop apt-daily.service;  systemctl mask apt-daily.service; systemctl daemon-reload

Configure hostname:

# hostnamectl set-hostname rpi1
# hostnamectl set-hostname rpi2

Configure cn repository:

# vim /etc/apt/sources.list
deb http://mirrors.ustc.edu.cn/ubuntu-ports/ focal main restricted universe multiverse
# deb-src http://mirrors.ustc.edu.cn/ubuntu-ports/ focal main main restricted universe multiverse
deb http://mirrors.ustc.edu.cn/ubuntu-ports/ focal-updates main restricted universe multiverse
# deb-src http://mirrors.ustc.edu.cn/ubuntu-ports/ focal-updates main restricted universe multiverse
deb http://mirrors.ustc.edu.cn/ubuntu-ports/ focal-backports main restricted universe multiverse
# deb-src http://mirrors.ustc.edu.cn/ubuntu-ports/ focal-backports main restricted universe multiverse
deb http://mirrors.ustc.edu.cn/ubuntu-ports/ focal-security main restricted universe multiverse
# deb-src http://mirrors.ustc.edu.cn/ubuntu-ports/ focal-security main restricted universe multiverse
# apt-get update -y

Install some packages:

# apt-get install -y net-tools virt-manager lxde tightvncserver sshpass ssh-askpass
# usermode -a -G kvm,libvirt  ubuntu

Configure vnc

# vncserver
# vim ~/.vnc/xstartup
	#!/bin/sh
	exec startlxde
# vncserver -kill :1
# vncserver

In vnc we could use virt-manager.

Install new os:

/images/2021_02_05_16_53_45_633x380.jpg

KubeQuickRun

Feb 3, 2021
Technology

Via following tips we could quickly run apps in k8s:

# kubectl create deployment xxxx --image=xxxx:16.04 -- sleep 3000
# kubectl get deployment
# kubectl get deployment xxxx -oyaml --export>kkk.yaml
# vim kkk.yaml
command: ["sleep"]
args: ["3600000"]
# kubectl delete -f kkk.yaml
# kubectl create -f kkk.yaml
# kubectl scale deployment xxxx -replicas=4

DirectlyEditDEB

Jan 25, 2021
Technology

项目中有时需要对DEB包做些许修改，譬如，在离线场景下某些DEB包的postinst中有需要用在线方式下载可执行文件的情况，这种情况下可以借助dpkg-deb的解包/压缩命令来临时修改出一个含有所有离线文件的包，举opennebula-node-firecracker的包为例说明。

opennebula-node-firecracker是用于在opennebula上运行轻量级VM firecracker的包，在线安装时它需要从github上拉取相应版本的firecracker及 jailer可执行文件，这让它在离线场景下无法安装成功。同时，因为它默认作为runtime的一种，其配置文件与opennebula-node-kvm里的某些配置文件相同，导致安装时无法继续，我们可以通过如下的方式修改此包，让其适配离线化安装。

准备

在线场景下下载该包，并拷贝到工作目录, 解压:

# cp /var/cache/apt/archives/opennebula-node-firecracker_5.12.0.3-1.ce_amd64.deb .
# mkdir tmp
# dpkg-deb -R opennebula-node-firecracker_5.12.0.3-1.ce_amd64.deb tmp/

主要目录结构如下:

├── DEBIAN
│   ├── conffiles
│   ├── control
│   ├── md5sums
│   ├── postinst
│   └── postrm
├── etc
│   ├── cron.d
│   │   └── opennebula-node
│   ├── sudoers.d
│   │   └── opennebula-node-firecracker
│   └── sysctl.d
│       └── bridge-nf-call.conf
├── srv
│   └── jailer
│       └── firecracker
└── usr
    ├── bin
    │   └── svncterm_server
    ├── sbin
    │   ├── install-firecracker
    │   ├── one-clean-firecracker-domain
    │   └── one-prepare-firecracker-domain
    └── share
        └── doc
            └── opennebula-node-firecracker
                ├── changelog.Debian.gz
                ├── copyright
                └── NEWS.Debian.gz

我们需要修改的要点如下:

1. DEBIAN/conffiles, 含有此包需写入的配置文件。
2. DEBIAN/md5sums, 含有可执行文件的md5校验码。
3. DEBIAN/postinst, 包安装完成后需执行的脚本。
4. etc/ 下是包安装后在主机上需添加的配置文件。
5. usr/bin, usr/sbin，主机上需拷入的可执行文件。

修改

观察DEBIAN/postinst中含有以下条目:

# cat DEBIAN/postinst 
#!/bin/sh

set -e

ONE_USER=oneadmin

if [ "$1" = "configure" ]; then
    # Install Firecracker + jailer
    /usr/sbin/install-firecracker

打开install-firecracker文件后观察其下载脚本为:

# cat usr/sbin/install-firecracker 
#!/bin/sh
....

# Download version version of Firecracker
curl -LOJ https://github.com/firecracker-microvm/firecracker/releases/download/${version}/firecracker-${version}-$(uname -m)
mv firecracker-${version}-$(uname -m) /usr/bin/firecracker
chmod +x /usr/bin/firecracker


# Download version version of jailer
curl -LOJ https://github.com/firecracker-microvm/firecracker/releases/download/${version}/jailer-${version}-$(uname -m)
mv jailer-${version}-$(uname -m) /usr/bin/jailer
chmod +x /usr/bin/jailer

这里我们直接干掉所有的curl及mv脚本，把预先下载好的firecracker/jailer文件拷贝到安装目录即可。

# tree ../tmp/usr
../tmp/usr
├── bin
+++ │   ├── firecracker
+++ │   ├── jailer
│   └── svncterm_server
├── sbin
│   ├── install-firecracker
.....

此外需要修改md5sums及干掉etc下重复的配置文件。而后压缩包。

压缩包

一条命令:

# dpkg-deb -b tmp opennebula-node-firecracker_5.12.0.3-1.ce_fixed_amd64.deb

检查新生成包的大小:

# ls -l -h *.deb
-rw-r--r-- 1 root root  24K Jan 25 09:06 opennebula-node-firecracker_5.12.0.3-1.ce_amd64.deb
-rw-r--r-- 1 root root 1.2M Jan 25 09:11 opennebula-node-firecracker_5.12.0.3-1.ce_fixed_amd64.deb

使用fixed后的包安装，此时可忽略internet下载过程，且解决了包安装时的冲突问题。

Usingdockerforbuildingkubeadm

Jan 21, 2021
Technology

X86 version kubeadm building process:

# docker run -it ubuntu:18.04 /bin/bash
# cat /etc/issue
Ubuntu 18.04.5 LTS \n \l
# apt-get update -y
# apt-get install -y wget unzip vim build-essential rsync
# wget https://github.com/kubernetes/kubernetes/archive/v1.19.7.zip
# wget https://golang.org/dl/go1.15.7.linux-amd64.tar.gz
# tar -C /usr/local -xzf go1.15.7.linux-amd64.tar.gz
# export PATH=$PATH:/usr/local/go/bin
# go version
go version go1.15.7 linux/amd64
# cd kubernetes-v1.19.7
# vim cmd/kubeadm/app/constants/constants.go
CertificateValidity = time.Hour * 24 * 365 * 100
#  vim vendor/k8s.io/client-go/util/cert/cert.go
func NewSelfSignedCACert
NotAfter: 	now.Add(duration365d * 100).UTC(),
func GenerateSelfSignedCertKeyWithFixtures
maxAge := 100 * time.Hour * 24 * 365
# make all WHAT=cmd/kubeadm
# _output/bin/kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"archive", BuildDate:"2021-01-21T07:15:36Z", GoVersion:"go1.15.7", Compiler:"gc", Platform:"linux/amd64"}
# cp _output/bin/kubeadm ..

Arm64 version kubeadm building process:

Edit files and make changes:

# vim hack/make-rules/cross.sh
    make all WHAT="${KUBE_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_SERVER_PLATFORMS[*]}"
    
    #make all WHAT="${KUBE_NODE_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_NODE_PLATFORMS[*]}"
    #
    #make all WHAT="${KUBE_CLIENT_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_CLIENT_PLATFORMS[*]}"
    #
    #make all WHAT="${KUBE_TEST_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_PLATFORMS[*]}"
    #
    #make all WHAT="${KUBE_TEST_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_SERVER_PLATFORMS[*]}"
#  vim hack/lib/golang.sh
    readonly KUBE_SUPPORTED_SERVER_PLATFORMS=(
    #  linux/amd64
    #  linux/arm
      linux/arm64
    #  linux/s390x
    #  linux/ppc64le
    )
    
    //.............
    
    kube::golang::server_targets() {
      local targets=(                       
      #  cmd/kube-proxy                                                        
      #  cmd/kube-apiserver                 
      #  cmd/kube-controller-manager                                 
      #  cmd/kubelet 
        cmd/kubeadm
      #  cmd/kube-scheduler
      #  vendor/k8s.io/apiextensions-apiserver
      #  cluster/gce/gci/mounter
      )

Build:

# make cross

v1.20.7 update

Have to change go version to v1.17 for building, also notice the memory usage.

apt-get update -y && apt-get install -y wget unzip vim build-essential rsync
wget https://github.com/kubernetes/kubernetes/archive/v1.20.7.zip
wget https://golang.org/dl/go1.17.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.17.linux-amd64.tar.gz 
export PATH=$PATH:/usr/local/go/bin
go version
unzip v1.20.7.zip 
cd kubernetes-1.20.7/
vim cmd/kubeadm/app/constants/constants.go
vim vendor/k8s.io/client-go/util/cert/cert.go
make all WHAT=cmd/kubeadm
cp _output/bin/kubeadm ..

WorkingTipsOnRongV7.0

Jan 20, 2021
Technology

用于记录基于kubespray v2.15.0离线化过程。

包准备

Download kubespray v2.15.0 source code via:

# wget https://github.com/kubernetes-sigs/kubespray/archive/v2.15.0.tar.gz

使用RongRobots得到离线包以便替换。

$ ls -l -h RobotSon.tar.gz 
-rw-r--r-- 1 dash root 882M Jan 20 15:58 RobotSon.tar.gz

准备的目录如下:

# mkdir RobotSon
# tar xzvf RobotSon.tar.gz -C RobotSon/
# ls 
kubespray-2.15.0.tar.gz  Origin  RobotSon  RobotSon.tar.gz  Rong

代码修改

替换静态文件:

# rm -f Rong/pre-rong/rong_static/for_cluster/calicoctl 
# rm -f Rong/pre-rong/rong_static/for_cluster/cni-plugins-linux-amd64-v0.8.7.tgz 
# rm -f Rong/pre-rong/rong_static/for_cluster/kube*
# cp RobotSon/release/calicoctl Rong/pre-rong/rong_static/for_cluster/
# cp RobotSon/release/cni-plugins-linux-amd64-v0.9.0.tgz Rong/pre-rong/rong_static/for_cluster/
# cp RobotSon/release/kube* Rong/pre-rong/rong_static/for_cluster/

创建离线docker镜像包并替代原有离线镜像包:

# cd RobotSon/data
# tar czvf docker.tar.gz docker/
# cd ../../
# rm -f Rong/pre-rong/rong_static/for_master0/docker.tar.gz
# mv RobotSon/data/docker.tar.gz Rong/pre-rong/rong_static/for_master0/

更改rong/1_preinstall/roles/preinstall/tasks/main.yml, 更改为新的静态包.

替换rong/3_k8s目录:

# tar xzvf kubespray-2.15.0.tar.gz
# rm -rf rong/3_k8s/
# mv kubespray-2.15.0/* rong/3_k8s/

更改bootstrap角色:

# cp ./rong/3_k8s/roles/bootstrap-os/tasks/main.yml ./rong/3_k8s/roles/bootstrap-os/task/main_main.yml
# cp /run/media/dash/aa3eda99-dc11-4c07-a5f1-d00eb0acc850/Rong_V7.0.0/Origin/rong/3_k8s/roles/bootstrap-os/tasks/main_kfz.yml ./rong/3_k8s/roles/bootstrap-os/tasks/
# cp /run/media/dash/aa3eda99-dc11-4c07-a5f1-d00eb0acc850/Rong_V7.0.0/Origin/rong/3_k8s/roles/bootstrap-os/tasks/main.yml ./rong/3_k8s/roles/bootstrap-os/tasks/

更改container-engine/docker角色，与上差不多的步骤。

更改rong-vars.yml里的相关定义:

kubeadm_download_url:
kubelet_download_url:
kubectl_download_url:
helm_download_url:

helm_enabled: true
#helm_version: "v2.16.1"
helm_skip_refresh: true

containerd_version: '1.2.13'