Building k8s-dns-node-cache-arm64

Jul 3, 2019
Technology

Steps

Clone the source code from github:

# git clone https://github.com/kubernetes/dns.git
# git checkout tags/1.15.1 -b local_1.15.1

Modify the source code:

# vim Makefile
BINARIES := \
    node-cache
#    e2e \
#    ginkgo \
#    sidecar-e2e


CONTAINER_BINARIES := \
    node-cache

ARCH ?= arm64

Now building via:

# make build
# make containers

Will get error, now copy the generated node-cache file into destination directory:

# cp ./.go/bin/node-cache bin/arm64/

Edit the dnsmasq’s Makefile via:

# vim images/dnsmasq/Makefile

/images/2019_07_03_14_47_54_635x612.jpg

make will also get error, manually compile the dnsmasq:

# cd ./images/dnsmasq/_output/arm64/dnsmasq-2.78
# make
# cp src/dnsmasq ../docker

Now go the project root directory and make containers:

# make containers
# docker images  | grep dns
staging-k8s.gcr.io/k8s-dns-dnsmasq-arm64      1.15.1-dirty         fbb04ccb60e6        About an hour ago   3.63MB
staging-k8s.gcr.io/k8s-dns-node-cache-arm64   1.15.1-dirty         bf6131745b5e        About an hour ago   71.9MB

Now replace the dns-node-cache default to our own build-out version we could enable node-cache working on arm64.

Arm64ISOCustomization

Jul 2, 2019
Technology

Make working directory:

#  mkdir Rong1907iso
# cd Rong1907iso/
# cp ../ubuntu-18.04.2-server-arm64.iso .
# cp -r ./iso/* ./newISO
# cp -r ./iso/.disk ./newISO
# umount ./iso
# rm -f ubuntu-18.04.2-server-arm64.iso 
# rm -rf iso/

Add seed files under preseed directory, then edit the grub files:

root@arm02:/home/test/Rong1907iso/newISO# ls preseed/
cli.seed      hwe-ubuntu-server-minimal.seed    hwe-ubuntu-server.seed  fuck_auto.seed        ubuntu-server-minimal.seed    ubuntu-server.seed
hwe-cli.seed  hwe-ubuntu-server-minimalvm.seed  fuck.seed            fuck_auto_multi.seed  ubuntu-server-minimalvm.seed
root@arm02:/home/test/Rong1907iso/newISO# ls boot/grub/grub.cfg 
boot/grub/grub.cfg

Edit the grub file like following:

set menu_color_normal=white/black
set menu_color_highlight=black/yellow

insmod gzio

set timeout=10
menuentry "Auto Install Ubuntu Server(Manual-Partition)" {
        set gfxpayload=keep
        linux   /install/vmlinuz auto-install/enable=true file=/cdrom/preseed/fuck.seed quiet ---
        initrd  /install/initrd.gz
}
menuentry "Auto Install Ubuntu Server(Auto-Partition-AllInOne)" {
        set gfxpayload=keep
        linux   /install/vmlinuz auto-install/enable=true file=/cdrom/preseed/fuck_auto.seed quiet ---
        initrd  /install/initrd.gz
}
menuentry "Auto Install Ubuntu Server(Auto-Partition-Seperate)" {
        set gfxpayload=keep
        linux   /install/vmlinuz auto-install/enable=true file=/cdrom/preseed/fuck_auto_multi.seed quiet ---
        initrd  /install/initrd.gz
}
menuentry "Install Ubuntu Server" {
        set gfxpayload=keep
        linux   /install/vmlinuz  file=/cdrom/preseed/ubuntu-server.seed quiet ---
        initrd  /install/initrd.gz
}

Make the iso via following command:

# xorriso -as mkisofs -r -checksum_algorithm_iso md5,sha1 -V 'Server 18.04.2 LTS arm64' -o ./fuck_ubuntu180402_arm64.iso -J -joliet-long -cache-inodes -e boot/grub/efi.img  -no-emul-boot -append_partition 2 0xef newISO/boot/grub/efi.img  -partition_cyl_align all newISO/
root@arm02:/home/test/Rong1907iso# ls
newISO  fuck_ubuntu180402_arm64.iso

Using the fuck_ubuntu180402_arm64.iso you could install systme on arm64 based server.

arm64KubesprayOfflineTips

Jun 28, 2019
Technology

Folder structure

Compare the downloaded source code to our offlined edition, make some changes.

cluster.yml should added kube-deploy related items.   
ansible.cfg should be modified.
Added role/kube-deploy folder. 
scale.yml/upgrade-cluster.yml should be modified. 
Added deploy.key for easy deployment. 
roles/kubernetes-apps/ansible/defaults/main.yml, modified dashboard_skip_login condition
roles/kubernetes-apps/ansible/templates/dashboard.yml.j2: NodePort modification
roles/kubespray-defaults/defaults/main.yaml: enable_nodelocaldns:false(TBD)
roles/download/defaults/main.yml: download position, for example hyperkube/kubeadm/cni/calicoctl etc. 
/roles/kubernetes/master/templates/kubeadm-config.v1alpha3.yaml.j2: controllerManager listening port to 0.0.0.0
roles/kubernetes/master/tasks/kubeadm-upgrade.yml: upgrade items to --force

(Todo) bootstrap.sh

Change the installation of ansible from apt-get to pip-cache

#!/bin/sh
## 
OS_ID=`cat /etc/os-release | grep VERSION_CODENAME | awk -F '=' {'print $2'}`
echo $OS_ID
# xenial use 1604, bionic use 1804
if [ "$OS_ID" = "xenial" ]; then
	sudo tar xJvf ./roles/kube-deploy/files/1604debs.tar.xz -C /usr/local/
else
	sudo tar xJvf ./roles/kube-deploy/files/1804debs.tar.xz -C /usr/local/
	sudo tar xJvf ./roles/kube-deploy/files/pip_ansible.tar.xz -C /usr/local/
fi
sudo echo "deb [trusted=yes] file:///usr/local/static ./">/etc/apt/sources.list
sudo apt-get update -y
# Install pip so we could use pip for installing ansible
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python-pip
# Install ansible via ansible(version 2.8.1)
sudo pip install --no-index --find-links /usr/local/pip_ansible ansible
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python-netaddr

kube-deploy role

Added the offline role, and replace the files.

Replace:

nginx-autoindex.tar.xz 
kubeadm(arm version)
hyperkube(arm version)
cni-plugins-arm64-v0.6.0.tgz(arm version)
calicoctl(arm version)

nginx-autoindex

Find the Dockerfile, and build the arm64 based docker images via following commands:

# mkdir -p ~/code/autoindex
# vim Dockerfile
FROM nginx
MAINTAINER Jason Kingsbury

RUN sed -i 'N; s/root   \/usr\/share\/nginx\/html;\n        index  index.html index.htm;/root   \/usr\/share\/nginx\/html;\n        autoindex on;/' /etc/nginx/conf.d/default.conf
# sudo docker build -t jrelva/nginx-autoindex:latest .
# sudo docker  --name docker-nginx -p 7888:80 -d --restart=always -v `pwd`:/usr/share/nginx/html jrelva/nginx-autoindex

Save and xz the docker images:

# sudo docker save jrelva/nginx-autoindex:latest>autoindex.tar; sudo xz autoindex.tar

Transfer the autoindex.tar.xz to folder.

➜  files ls -l -h | grep autoindex
-rwxr-xr-x  1 dash dash  26M 5月   7 16:40 autoindex.tar.xz
➜  files pwd
/home/dash/Code/kubsprayarm64/roles/kube-deploy/files
➜  files ls -l -h | grep autoindex
-rwxr-xr-x  1 dash dash  23M 6月  28 10:59 autoindex.tar.xz

secureregistryserver

Change the docker-compose file, also pull the arm based docker images:

# docker pull registry:2
# docker pull nginx:latest
# docker save registry:2>regsitry.tar; xz registry.tar
# docker save nginx:latest>nginx19.tar; xz nginx19.tar

Verification

Create a new virtual machine(18.04.2):

$ qemu-img create -f qcow2 pure1804.qcow2 200G
```'
Install the system:    

![/images/2019_06_28_12_10_54_469x237.jpg](/images/2019_06_28_12_10_54_469x237.jpg)


### harbor offline
With the harbor-offline-installer-1.7.0-arm64.tgz we could quickly setup the offline harbor environment:

ls

common docker-compose.clair.yml docker-compose.yml harbor.cfg LICENSE prepare
docker-compose.chartmuseum.yml docker-compose.notary.yml harbor.1.7.0-arm64.tar.gz install.sh open_source_license

vim harbor.cfg

./install.sh –with-chartmuseum

docker ps

…..


Login:    

![/images/2019_06_28_12_19_39_475x380.jpg](/images/2019_06_28_12_19_39_475x380.jpg)

Create user kubespray:    

![/images/2019_06_28_12_19_59_447x368.jpg](/images/2019_06_28_12_19_59_447x368.jpg)

Fill in user info:    

![/images/2019_06_28_12_20_28_555x469.jpg](/images/2019_06_28_12_20_28_555x469.jpg)

Create project:    

![/images/2019_06_28_12_21_45_549x348.jpg](/images/2019_06_28_12_21_45_549x348.jpg)

Projects:    

![/images/2019_06_28_12_22_02_667x296.jpg](/images/2019_06_28_12_22_02_667x296.jpg)

Add kubespray to kubesprayns as administrator:    

![/images/2019_06_28_12_22_42_718x295.jpg](/images/2019_06_28_12_22_42_718x295.jpg)

Now you could login with kubespray user:    

![/images/2019_06_28_12_23_52_936x419.jpg](/images/2019_06_28_12_23_52_936x419.jpg)

Now in docker-compose folder we just `docker-compose down` all of the service and backup our environment:

docker-compose down

Stopping nginx … done Stopping harbor-jobservice … done Stopping harbor-portal … done Stopping harbor-core … done Stopping redis … done Stopping harbor-adminserver … done Stopping harbor-db … done Stopping registry … done Stopping registryctl … done Stopping harbor-log …

docker save -o harbor.tar goharbor/chartmuseum-photon:v0.7.1-1.7.0-arm64 goharbor/redis-photon:1.7.0-arm64 goharbor/clair-photon:v2.0.7-1.7.0-arm64 goharbor/notary-server-photon:v0.6.1-1.7.0-arm64 goharbor/notary-signer-photon:v0.6.1-1.7.0-arm64 goharbor/harbor-registryctl:1.7.0-arm64 goharbor/registry-photon:v2.6.2-1.7.0-arm64 goharbor/nginx-photon:1.7.0-arm64 goharbor/harbor-log:1.7.0-arm64 goharbor/harbor-jobservice:1.7.0-arm64 goharbor/harbor-core:1.7.0-arm64 goharbor/harbor-portal:1.7.0-arm64 goharbor/harbor-adminserver:1.7.0-arm64 goharbor/harbor-db:1.7.0-arm64

xz harbor.tar

RaspberryPIArm64

Jun 26, 2019
Technology

AIM

Replace the armhf(raspbain) with arm64 system.
Refers to:

/home/dash/Code/blogsource-master/src/content/post/2019-06-26-RaspberryPIArm64.md

Installation

Unxz the images and write to the tf card:

# unxz ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img.xz
# sudo dd if=./ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img of=/dev/sdd bs=1M && sudo sync

Configuration

Configure the repository :

# vim /etc/apt/sources.list
deb https://mirrors.ustc.edu.cn/ubuntu-ports/ bionic main restricted universe multiverse
# deb-src https://mirrors.ustc.edu.cn/ubuntu-ports/ bionic main main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu-ports/ bionic-updates main restricted universe multiverse
# deb-src https://mirrors.ustc.edu.cn/ubuntu-ports/ bionic-updates main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu-ports/ bionic-backports main restricted universe multiverse
# deb-src https://mirrors.ustc.edu.cn/ubuntu-ports/ bionic-backports main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu-ports/ bionic-security main restricted universe multiverse
# deb-src https://mirrors.ustc.edu.cn/ubuntu-ports/ bionic-security main restricted universe multiverse
# sudo apt-get update -y
# sudo apt-get upgrade -y
# sudo hostnamectl set-hostname arm64
# sudo apt-get install -y build-essential

Install docker-ce for arm64:

#  offline packages. 
# apt-get install -y docker-ce
# docker version
# cd /var/lib
# cp -r docker /media/sda/
# mv docker docker.back
# ln -s /media/sda/docker .
# ls -l -h | grep docker
lrwxrwxrwx  1 root      root        17 Jun 26 04:16 docker -> /media/sda/docker
drwxr-xr-x  2 root      root      4.0K Jun 26 04:13 docker-engine
drwx--x--x 14 root      root      4.0K Jun 26 04:13 docker.back
# systemctl start docker

Building harbor:

# apt-get install -y docker-compose

dns issue, install stubby:

# apt-get install -y stubby
# vim /etc/resolv.conf
nameserver 127.0.0.1

TipsOnARMVirtManager

Jun 25, 2019
Technology

Install following packages:

# apt-get install -y virt-manager 
# sudo apt-get install -y qemu-efi-aarch64 qemu-efi-arm ovmf

很惭愧，就做了一点微小的工作