May 22, 2019
TechnologyPurpose
ZFS On proxmox, performance issue.
Steps
1. megaclisas-status
The proxmox is in offline environment, so need to get the megaclisas-status
packages installed ready.
# sudo docker run -it debian:9.4 /bin/bash
root@f427df462cbd:/# cat /etc/debian_version
9.4
root@f427df462cbd:/# apt-get install -y vim
root@f427df462cbd:/# vim apt.conf.d/docker-clean
Comment all
root@f427df462cbd:/# apt-get install -y wget gnupg2
# wget -O - https://hwraid.le-vert.net/debian/hwraid.le-vert.net.gpg.key | apt-key add -
# cat /etc/apt/sources.list
deb http://deb.debian.org/debian stretch main
deb http://security.debian.org/debian-security stretch/updates main
deb http://deb.debian.org/debian stretch-updates main
deb http://hwraid.le-vert.net/debian stretch main
# apt-get update -y
# apt-get install megaclisas-status
# cd /var/cache/
# find . | grep deb$ | xargs -I % cp % /root/deb/
Transfer the debs into the promox machine, install it via:
# cd /root/deb/
# dpkg -i daemon_0.6.4-1+b2_amd64.deb megacli*
root@ks:~/deb# megaclisas-status
-- Controller information --
-- ID | H/W Model | RAM | Temp | BBU | Firmware
c0 | SAS3108 | 1024MB | 64C | Absent | FW: 24.7.0-0057
-- Array information --
-- ID | Type | Size | Strpsz | Flags | DskCache | Status | OS Path | CacheCade |InProgress
c0u0 | RAID-1 | 558G | 256 KB | RA,WT | Default | Optimal | /dev/sda | None |None
c0u1 | RAID-5 | 7271G | 256 KB | ADRA,WT | Default | Optimal | /dev/sdb | None |None
c0u2 | RAID-5 | 7271G | 256 KB | ADRA,WT | Default | Optimal | /dev/sdc | None |None
c0u3 | RAID-5 | 7271G | 256 KB | ADRA,WT | Default | Optimal | /dev/sdd | None |None
c0u4 | RAID-5 | 7271G | 256 KB | ADRA,WT | Default | Optimal | /dev/sde | None |No
Change the properties:
root@ks:~/deb# megacli -LDGetProp -Cache -LALL -a0
Adapter 0-VD 0(target id: 0): Cache Policy:WriteThrough, ReadAhead, Direct, No Write Cache if bad BBU
Adapter 0-VD 1(target id: 1): Cache Policy:WriteThrough, ReadAdaptive, Direct, No Write Cache if bad BBU
Adapter 0-VD 2(target id: 2): Cache Policy:WriteThrough, ReadAdaptive, Direct, No Write Cache if bad BBU
Adapter 0-VD 3(target id: 3): Cache Policy:WriteThrough, ReadAdaptive, Direct, No Write Cache if bad BBU
Adapter 0-VD 4(target id: 4): Cache Policy:WriteThrough, ReadAdaptive, Direct, No Write Cache if bad BBU
Exit Code: 0x00
root@ks:~/deb# megacli -LDSetProp NORA -LALL -a0
Set Read Policy to NoReadAhead on Adapter 0, VD 0 (target id: 0) success
Set Read Policy to NoReadAhead on Adapter 0, VD 1 (target id: 1) success
Set Read Policy to NoReadAhead on Adapter 0, VD 2 (target id: 2) success
Set Read Policy to NoReadAhead on Adapter 0, VD 3 (target id: 3) success
Set Read Policy to NoReadAhead on Adapter 0, VD 4 (target id: 4) success
Exit Code: 0x00
root@ks:~/deb# megacli -LDGetProp -Cache -LALL -a0
Adapter 0-VD 0(target id: 0): Cache Policy:WriteThrough, ReadAheadNone, Direct, No Write Cache if bad BBU
Adapter 0-VD 1(target id: 1): Cache Policy:WriteThrough, ReadAheadNone, Direct, No Write Cache if bad BBU
Adapter 0-VD 2(target id: 2): Cache Policy:WriteThrough, ReadAheadNone, Direct, No Write Cache if bad BBU
Adapter 0-VD 3(target id: 3): Cache Policy:WriteThrough, ReadAheadNone, Direct, No Write Cache if bad BBU
Adapter 0-VD 4(target id: 4): Cache Policy:WriteThrough, ReadAheadNone, Direct, No Write Cache if bad BBU
Exit Code: 0x00
Hope this will greately improve performance.
Notice(For disk cache):
Optional toppings for most LD configurations:
WT : WriteThrough safer. Only returns once data is written to disk.
WB: WriteBack faster. Returns as soon as data is is written to cache
NORA : No Read Ahead vs
RA: ReadAhead vs
ADRA : Adaptive ReadAhead where if the previous two requests were sequential, it pre-loads the next in sequence.
Cached: Cache reads.
Direct: Only the previous read is cached.
-strpszM : Stripe size so -strpsz64 means 64kb stripe size.
Hsp[E0:S0] : Choose this drive to be a hot-spare
May 21, 2019
TechnologyStartPoint
Working directory:
# /home/xxxx/Code/vagrant/ai_k8s/RONG/package/files/Rong
# vagrant status
Current machine states:
outnode-1 running (libvirt)
A running k8s cluster:
[root@outnode-1 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)
[root@outnode-1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
outnode-1 Ready master 32m v1.14.1
Configure repository:
# mount /dev/sr0 /mnt
# cd /etc/yum.repos.d
# mv *.repo /root/
# vim cdrom.repo
[local]
name=local
baseurl=file:///mnt
enabled=1
gpgcheck=0
# yum makecache
# yum install -y vim git nfs-utils rpcbind
Configure nfs server:
# mkdir -p /opt/nfs
# vim /etc/exports
/opt/nfs *(rw,async,no_root_squash,no_subtree_check)
# service rpcbind start
# service nfs start
# systemctl enable nfs-server
# systemctl start nfs-server
# systemctl enable nfs.service
# systemctl enable rpcbind
Configure helm via:
# helm repo add stable https://kubernetes-charts.storage.googleapis.com
# helm install stable/nfs-client-provisioner --set nfs.server=10.142.108.191 --set nfs.path=/opt/nfs
# kubectl get sc
nfs-client cluster.local/righteous-condor-nfs-client-provisioner 7m8s
# kubectl edit sc nfs-client
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
Install
Clone the source code from github:
# git clone https://github.com/IBM/FfDL.git
#
TOO many errors here. To be continue.
May 21, 2019
Technology上午想了一下关于开发后续要努力的方向。
K8S去中心化
早先做的关于Rong的方案都是中心化的，各个组件依赖于一个中心化的kube-deploy节点，当然这个节点也是我对于kubespray项目的一个拓展吧，把dns服务器/secure-registry/harbor服务都落地于一个中心化的节点来做。好处是做到了中心化管理，同时通过欺骗各个节点签名档的方式，无缝衔接了docker.io/gcr.io/quay.io等的pull/push请求。坏处是无法做到高可用，或者说如果做高可用，该如何来设计这个节点呢？
思路：
1. registry on k8s?
2. harbor on k8s?
AI on K8s
有不错的起点，就是Ffdl这个平台。
但问题是我需要一个去中心化后的节点来做为起步。
眼下不管是Rong的内网版或者是去中心化的外网版本看来都不是特别理想的实现平台。
clearLinux
这个可以作为后续的起点系统，深入研究。
May 13, 2019
TechnologyBackground
For enabling local storage provision on kubespray, and make use of the local
disk for pod storage usage.
Enable
Enable the local storage pool via:
# vim inventory/sample/group_vars/k8s-cluster/addons.yml
# Rancher Local Path Provisioner
local_path_provisioner_enabled: true
# Local volume provisioner deployment
local_volume_provisioner_enabled: true
local_volume_provisioner_namespace: kube-system
local_volume_provisioner_storage_classes:
local-storage:
host_dir: /mnt/disks
mount_dir: /mnt/disks
fast-disks:
host_dir: /mnt/fast-disks
mount_dir: /mnt/fast-disks
block_cleaner_command:
- "/scripts/shred.sh"
- "2"
volume_mode: Filesystem
fs_type: ext4
Prepare
Prepare the local storage via:
#
# mkdir -p /mnt/fast-disks/vol-alertmanager-res-alertmanager-0
# mkdir -p /mnt/fast-disks/vol-prometheus-res-prometheus-0
# mkdir -p /mnt/fast-disks/es-data-es-data-efk-cluster-default-0
# mkdir -p /mnt/fast-disks/es-data-es-master-efk-cluster-default-0
# truncate /mnt/vol-alertmanager-res-alertmanager-0 --size 20G
# truncate /mnt/vol-prometheus-res-prometheus-0 --size 20G
# truncate /mnt/es-data-es-data-efk-cluster-default-0 --size 10G
# truncate /mnt/es-data-es-master-efk-cluster-default-0 --size 10G
# mkfs.ext4 /mnt/vol-alertmanager-res-alertmanager-0
# mkfs.ext4 /mnt/vol-prometheus-res-prometheus-0
# mkfs.ext4 /mnt/es-data-es-data-efk-cluster-default-0
# mkfs.ext4 /mnt/es-data-es-master-efk-cluster-default-0
Edit the /etc/fstab for mounting them automatically:
/mnt/vol-alertmanager-res-alertmanager-0 /mnt/fast-disks/vol-alertmanager-res-alertmanager-0 ext4 rw 0 1
/mnt/vol-prometheus-res-prometheus-0 /mnt/fast-disks/vol-prometheus-res-prometheus-0 ext4 rw 0 1
/mnt/es-data-es-data-efk-cluster-default-0 /mnt/fast-disks/es-data-es-data-efk-cluster-default-0 ext4 rw 0 1
/mnt/es-data-es-master-efk-cluster-default-0 /mnt/fast-disks/es-data-es-master-efk-cluster-default-0 ext4 rw 0 1
Usage
I prepare the storage for I use them in helm/charts, and helm/charts
automatically request the storage from storage class, thus I have to make
/mnt/fast-disks as the default storage class.
# kubectl edit sc fast-disks
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"fast-disks"},"provisioner":"kubernetes.io/no-provisioner","volumeBindingMode":"WaitForFirstConsumer"}
+ storageclass.kubernetes.io/is-default-class: "true"
verify:
root@localnode-1:/mnt# kubectl get pvc --all-namespaces
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
logging es-data-es-data-efk-cluster-default-0 Bound local-pv-7d48bf57 20Gi RWO fast-disks 4h17m
logging es-data-es-master-efk-cluster-default-0 Bound local-pv-64a35d15 20Gi RWO fast-disks 4h17m
monitoring vol-alertmanager-res-alertmanager-0 Bound local-pv-24ed6560 20Gi RWO fast-disks 4h21m
monitoring vol-prometheus-res-prometheus-0 Bound local-pv-e998c4c2 20Gi RWO fast-disks 4h21m
TBD
- Now to enlarge the disk?
May 8, 2019
TechnologyIn a minimum installed system, do following:
# mount /dev/sr0 /mnt
# vim /etc/yum.repos.d/local.repo
[local]
name=local
baseurl=file:///mnt
enabled=1
gpgcheck=0
# yum makecache
# cat /mnt/repodata/764ce0e938d43b3e9cb1bcca13cf71630aac0c44149f4a61548f642df3c70858-comps-Server.x86_64.xml
Around Line 937-1031, get all of the packges:
# cat list1.txt | awk -F '>' {'print $2'} | awk -F '<' {'print $1'}
According to the output, install all of the packges:
yum install -y Red_Hat_Enterprise_Linux-Release_Notes-7-en-US
yum install -y audit
.......
yum install -y tboot
yum install -y NetworkManager-config-server
Now do following for getting all of the packages:
# for i in `rpm -qa`; do echo $i.rpm|xargs -I % cp /mnt/Packages/% /root/pkgs; done
Added Base group:
# sed -n 483,632p /mnt/repodata/764ce0e938d43b3e9cb1bcca13cf71630aac0c44149f4a61548f642df3c70858-comps-Server.x86_64.xml | awk -F '>' {'print $2'} | awk -F '<' {'print $1'} | xargs -I % yum install -y %
# for i in `rpm -qa`; do echo $i.rpm|xargs -I % cp /mnt/Packages/% /root/pkgs; done
Upload the Packages to building server, then:
# mv repodata/ /root/whole_repodata
# mv Packages/ /root/whole_Packages
# mv /root/thinrepodata/ repodata
# mv /root/pkgs/ Packages
# cp repodata/764ce0e938d43b3e9cb1bcca13cf71630aac0c44149f4a61548f642df3c70858-comps-Server.x86_64.xml .
# createrepo -g 764ce0e938d43b3e9cb1bcca13cf71630aac0c44149f4a61548f642df3c70858-comps-Server.x86_64.xml .
# mkisofs -o /root/new.iso -b isolinux/isolinux.bin -c isolinux/boot.cat --no-emul-boot --boot-load-size 4 --boot-info-table -J -R -V "RHEL-7.4\x20Server.x86_64" .
Your /root/new.iso will be much more smaller, cause it only contains Core
and Base group packages.
