NodeExporterOnCentOS6

Jul 25, 2018
Technology

从 https://github.com/prometheus/node_exporter/releases, 下载适合机器架构的node_exporter版本，服务器上，通常下载为amd64的tar.gz包，而后解压。

下载daemonize的rpm包安装，我们需要这个包来保证node_exporter的daemon运行.

# wget https://forensics.cert.org/centos/cert/6/x86_64//daemonize-1.7.3-7.el6.x86_64.rpm
# rpm -ivh daemonize-1.7.3-7.el6.x86_64.rpm

解压node_exporter至/usr/bin目录下，现在开始撰写初始化脚本。

# vim /etc/init.d/node_exporter

    #!/bin/bash
    #
    #	/etc/rc.d/init.d/node_exporter
    #
    # chkconfig: 2345 80 80
    #
    # config: /etc/prometheus/node_exporter.conf
    # pidfile: /var/run/prometheus/node_exporter.pid
    
    # Source function library.
    . /etc/init.d/functions
    
    
    RETVAL=0
    PROG="node_exporter"
    DAEMON_SYSCONFIG=/etc/sysconfig/${PROG}
    DAEMON=/usr/bin/${PROG}
    PID_FILE=/var/run/prometheus/${PROG}.pid
    LOCK_FILE=/var/lock/subsys/${PROG}
    LOG_FILE=/var/log/prometheus/node_exporter.log
    DAEMON_USER="prometheus"
    FQDN=$(hostname --long)
    GOMAXPROCS=$(grep -c ^processor /proc/cpuinfo)
    
    . ${DAEMON_SYSCONFIG}
    
    start() {
      if check_status > /dev/null; then
        echo "node_exporter is already running"
        exit 0
      fi
    
      echo -n $"Starting node_exporter: "
      daemonize -u ${DAEMON_USER} -p ${PID_FILE} -l ${LOCK_FILE} -a -e ${LOG_FILE} -o ${LOG_FILE} ${DAEMON} ${ARGS}
      RETVAL=$?
      echo ""
      return $RETVAL
    }
    
    stop() {
        echo -n $"Stopping node_exporter: "
        killproc -p ${PID_FILE} -d 10 ${DAEMON}
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && rm -f ${LOCK_FILE} ${PID_FILE}
        return $RETVAL
    }  
    
    check_status() {
        status -p ${PID_FILE} ${DAEMON}
        RETVAL=$?
        return $RETVAL
    }
    
    case "$1" in
        start)
            start
            ;;
        stop)
            stop
            ;;
        status)
    	check_status
            ;;
        reload|force-reload)
            reload
            ;;
        restart)
            stop
            start
            ;;
        *)
            N=/etc/init.d/${NAME}
            echo "Usage: $N {start|stop|status|restart|force-reload}" >&2
            RETVAL=2
            ;;
    esac
    
    exit ${RETVAL}

同时在/etc/sysconfig目录下创建node_exporter文件，默认的我们不添加任何初始化参数（后面你可以添加自定义的启动参数用于控制node_exporter的启动）:

# vim /etc/sysconfig/node_exporter
ARGS=""

我们的初始化脚本默认使用了prometheus用户，因而我们需要在系统中手动创建该用户，并创建对应的目录，以承载pid文件及存放log等:

# groupadd -r prometheus
# mkdir -p /usr/hostonnet/prometheus/
# useradd -r -g prometheus -s /sbin/nologin -d /usr/hostonnet/prometheus/ -c "prometheus Daemons" prometheus
# chown -R prometheus:prometheus /usr/hostonnet/prometheus/
# chown prometheus:prometheus /var/log/prometheus.log
# mkdir -p /var/run/prometheus/
# mkdir -p /var/log/prometheus/
# touch /var/log/prometheus/node_exporter.log
# chmod 777 /var/log/prometheus/node_exporter.log 
# chown prometheus:prometheus /var/log/prometheus/node_exporter.log
# /etc/init.d/node_exporter start
# chkconfig node_exporter on

检查，打开浏览器，查看http://ipaddress:9100/metrics是否可以看到对应导出的metrics即可。
prometheus的配置文件中，添加上该节点的metrics导出信息即可。

MonocularChinese

Jul 25, 2018
Technology

Moncular汉化过程，记录如下:

# git clone https://github.com/kubernetes-helm/monocular.git
# cd monocular
# vim dev_env/ui/Dockerfile
    FROM bitnami/node:8
    
    # Install yarn
    RUN install_packages gnupg2
    RUN install_packages apt-transport-https && \
      curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
      #echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
      echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list 
    RUN install_packages yarn

全程需要翻墙。
其中需要下载一大堆镜像，运行成功后用浏览器访问http://localhost:4200即可。在目录下编辑对应的文件，更改中文即可看到效果:

/images/2018_07_25_09_10_33_735x531.jpg

GrafanaChinese

Jul 24, 2018
Technology

公司内部要用到Grafana肯定会有各种阻力，最大阻力恐怕来自于很多同事英文不好。因而参考了网上的一个教程做grafana的汉化。实际的汉化我也没有做完，无非是做一个框架在这里，以后如果研发压力大，就直接铺人在上头做做翻译。

环境部署:

$ git clone git://github.com/grafana/grafana
$ git checkout tags/v5.2.1 -b grafana_local

安装cnpm开发环境:

$ sudo npm cache clean --force
$ sudo npm install -g cnpm --registry=https://registry.npm.taobao.org
$ which cnpm
/usr/bin/cnpm

开发环境初始化:

$ cnpm install
$ npm start

此时，生成的代码将运行于 localhost:3333端口，我们需要一个本地运行的3000端口的grafana服务器来查看更改。

需要拷贝出已经运行容器里的/usr/share/grafana下的内容:

$ sudo docker cp 66daaf6770b9:/usr/share/grafana .

而后更改conf/defaults.ini下的条目:

app_mode = development
static_root_path = /public

现在重新启动该容器，注意需要挂载新的盘符至其中:

sudo docker run -d -p 3000:3000 -e "GF_SECURITY_ADMIN_PASSWORD=admin_password" -v /media/sda5/grafana_db:/var/lib/grafana -v /home/xxxx/grafana/grafana:/usr/share/grafana -v /home/xxxx/Code/grafana/grafana/public:/public grafana/grafana

此时对代码的任一修改都可以很快体现在最终的http://localhost:3000网站中。

修改的一个截图如下:

/images/2018_07_24_14_32_30_586x522.jpg

PrometheusTips

Jul 23, 2018
Technology

AIM

Setup prometheus monitoring system in inner intranet.
OS: CentOS/Ubuntu/RHEL, etc.

Prometheus Server

Run server in following commands:

# vim /xxxxxx/prometheus/prometheus.yml
    global:
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).
    
      # Attach these labels to any time series or alerts when communicating with
      # external systems (federation, remote storage, Alertmanager).
      external_labels:
        monitor: 'kkkkk-monitor'
    
    # Alertmanager configuration
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          # - alertmanager:9093
    
    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      # - "first_rules.yml"
      # - "second_rules.yml"
    
    # A scrape configuration containing exactly one endpoint to scrape:
    # Here it's Prometheus itself.
    scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'prometheus'
    
        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.
    
        static_configs:
            - targets: ['localhost:9090', '192.192.189.1:9100']
    
# docker run -d -p 9090:9090 -v /xxxxxx/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /media/sda5/prometheus:/prometheus prom/prometheus

node-export

In every node, run following:

# docker run -d  --net="host"  --pid="host"  quay.io/prometheus/node-exporter

grafana

Run grafana server via:

# docker run -d -p 3000:3000 -e "GF_SECURITY_ADMIN_PASSWORD=admin_password" -v /media/sda5/grafana_db:/var/lib/grafana grafana/grafana

Add grafana id of 1860(Node Exporter Full) or 159(Prometheus system), then you could view the grafana displayed dashboard of node-exporter of the system.

grafana configuration

Initial picture:

/images/2018_08_23_15_15_39_620x438.jpg

Add datasource:

/images/2018_08_23_15_16_36_600x632.jpg

Add redis templates:

# wget https://raw.githubusercontent.com/oliver006/redis_exporter/master/contrib/grafana_prometheus_redis_dashboard_alias.json

Import:

/images/2018_08_23_15_18_26_951x381.jpg

Detailed:

/images/2018_08_23_15_18_58_584x566.jpg

Corresponding data source:

/images/2018_08_23_15_19_39_358x467.jpg

CustomizeDockerISO

Jul 20, 2018
Technology

单位里一些同事需要一个开箱即用的Docker环境，以下是制作自启动Docker的ISO制作过程。

准备

准备一台新安装的Ubuntu16.04机器，在其中安装docker, docker load需要定制的镜像。而后，保存/var/lib/docker/目录下的条目，简而言之，就是将/var/lib/docker压包。

定制化ISO

从基础镜像起步，之前我已经定制了1604_pure.iso，里面已经安装docker/docker-compose，并内置了用于运行portus（一个容器镜像仓库）所需的镜像文件，现在只需要从其中替换掉镜像文件即可。

很惭愧，就做了一点微小的工作