MultipleOfflineRegistryServer
Jan 29, 2019
Technology
Suppose your kubernetes clusters are in totally offline environment, you don’t
have any connection to Internet, how to deal with the docker pull request
within the cluster? Following are steps for solving this problem.
Docker-Composed Registry
This step is pretty simple, I mainly refers to:
Following the steps in article, I got a running docker-compose triggered docker registry server.
Following is my folder structure:
# tree .
├── data
├── docker-compose.yml
├── nginx
│ ├── openssl.cnf
│ ├── registry.conf
│ ├── registry.password
│ ├── server.crt
│ ├── server.csr
│ └── server.key
Docker-compose file is listed as:
# cat docker-compose.yml
nginx:
image: "nginx:1.9"
ports:
- 443:443
links:
- registry:registry
volumes:
- ./nginx/:/etc/nginx/conf.d:ro
registry:
image: registry:2
ports:
- 127.0.0.1:5000:5000
environment:
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
volumes:
- ./data:/data
Handle SSL
The above steps will only create a single hostname ssl enabled docker registry server, we want to enable multi-domain self-signed SSL certificate.
Create a file called openssl.cnf with the following details, notice that we
have set docker.io, gcr.io, quay.io and elastic.co.
# vim openssl.cnf
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
[req_distinguished_name]
countryName = SL
countryName_default = SL
stateOrProvinceName = Western
stateOrProvinceName_default = Western
localityName = Colombo
localityName_default = Colombo
organizationalUnitName = ABC
organizationalUnitName_default = ABC
commonName = *.docker.io
commonName_max = 64
[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.docker.io
DNS.2 = *.gcr.io
DNS.3 = *.quay.io
DNS.4 = *.elastic.co
DNS.5 = docker.io
DNS.6 = gcr.io
DNS.7 = quay.io
DNS.8 = elastic.co
Create the Private key.
# rm -f server.key
# sudo openssl genrsa -out server.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
.....................+++++
......................................+++++
e is 65537 (0x010001)
Create Certificate Signing Request (CSR).
# rm -f server.csr
# sudo openssl req -new -out server.csr -key server.key -config openssl.cnf
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
SL [SL]:
Western [Western]:
Colombo [Colombo]:
ABC [ABC]:
*.docker.io []:
*.docker.io is from the above defined parts in openssl.cnf.
Sign the SSL Certificate.
# rm -f server.crt
# sudo openssl x509 -req -days 36500 -in server.csr -signkey server.key -out server.crt -extensions v3_req -extfile openssl.cnf
Signature ok
subject=C = SL, ST = Western, L = Colombo, OU = ABC
Getting Private key
You can check the ssl certificated content via:
# openssl x509 -in server.crt -noout -text
Certificate:
Data:
....
Validity
Not Before: Jan 29 08:30:20 2019 GMT
Not After : Jan 5 08:30:20 2119 GMT
....
X509v3 Subject Alternative Name:
DNS:*.docker.io, DNS:*.gcr.io, DNS:*.quay.io, DNS:*.elastic.co, DNS:docker.io, DNS:gcr.io, DNS:quay.io, DNS:elastic.co
nginx conf
Add the domain items into registry.conf:
upstream docker-registry {
server registry:5000;
}
server {
listen 443;
server_name docker.io gcr.io quay.io elastic.co;
# SSL
ssl on;
ssl_certificate /etc/nginx/conf.d/server.crt;
ssl_certificate_key /etc/nginx/conf.d/server.key;
dns configuration
Install bind9 on ubuntu, or install dnsmasq on centos system. take bind9’s configuration for example, add following configurations:
# ls db*
db.docker.io db.gcr.io db.quay.io
The named.conf.default-zones is listed as following:
# cat named.conf.default-zones
// prime the server with knowledge of the root servers
zone "." {
type hint;
file "/etc/bind/db.root";
};
// be authoritative for the localhost forward and reverse zones, and for
// broadcast zones as per RFC 1912
zone "localhost" {
type master;
file "/etc/bind/db.local";
};
zone "127.in-addr.arpa" {
type master;
file "/etc/bind/db.127";
};
zone "0.in-addr.arpa" {
type master;
file "/etc/bind/db.0";
};
zone "255.in-addr.arpa" {
type master;
file "/etc/bind/db.255";
};
zone "docker.io" {
type master;
file "/etc/bind/db.docker.io";
};
zone "quay.io" {
type master;
file "/etc/bind/db.quay.io";
};
zone "gcr.io" {
type master;
file "/etc/bind/db.gcr.io";
};
zone "elastic.co" {
type master;
file "/etc/bind/db.elastic.co";
};
Take elastic.co for example, show its content:
# cat db.elastic.co
$TTL 604800
@ IN SOA elastic.co. root.localhost. (
1 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS localhost.
@ IN A 192.168.122.154
elastic.co IN NS 192.168.122.154
docker IN A 192.168.122.154
Using the docker registry
Point the nodes’ dns server to your registry nodes, so if you ping elastic.co,
you got reply from 192.168.122.154.
Then add the crt into your docker’s trusted sites:
# cp server.crt /usr/local/share/ca-certificates
# update-ca-certificates
# systemctl restart docker
# docker login -u dddd -p gggg docker.io
# docker login -u dddd -p gggg quay.io
# docker login -u dddd -p gggg gcr.io
# docker login -u dddd -p gggg k8s.gcr.io
Now you can freely push/pull images from offline registry server, as if you are on Internet.