Glusterfs Howto

I want to expand my storage size on DigitalOcean, the droplet I have on DO one have 11G, and the other have 15G size, so if I could combine them together, I could do much more development on it. Following is how-to.

Glusterfs Setup

Install it under Ubuntu via:

# apt-get install glusterfs-server

In both node, install the same software, and then add following lines into your /etc/hosts:

10.17.17.195    Gluster2
10.17.17.194    Gluster1

In Gluster1, probe Gluster2:

root@Gluster1:~# gluster peer probe Gluster2
peer probe: success

Then view the gluster peer status:

root@Gluster1:~# gluster peer status
Number of Peers: 1

Hostname: Gluster2
Port: 24007
Uuid: 881dedb8-6cd4-4127-8c96-223daef081f5
State: Peer in Cluster (Connected)

Create the volumn via:

root@Gluster1:~# gluster volume create vol_replica transport tcp Gluster2:/home/glustervms Gluster1:/home/glustervms force
volume create: vol_replica: success: please start the volume to access data

Start the created vol:

root@Gluster1:~# gluster volume start vol_replica
volume start: vol_replica: success

View volumn info:

root@Gluster1:~# gluster volume info 
 
Volume Name: vol_replica
Type: Distribute
Volume ID: 953456f3-0c46-4d07-ac41-591d1e398be6
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: Gluster2:/home/glustervms
Brick2: Gluster1:/home/glustervms

Now create the folder and mount the glusterfs via:

root@Gluster1:/home# mkdir glustervmsmnt
root@Gluster1:/home# mount -t glusterfs Gluster1:/vol_replica /home/glustervmsmnt/

View the disk filesystem info:

Gluster1:/vol_replica   39G  4.7G   32G  13% /home/glustervmsmnt

Glusterfs Volumn deletion

The replica is not the one we want, for combine two partitions, I need the Distributed stripped mode, which is the one described in:
http://www.gluster.org/community/documentation/index.php/Gluster_3.2:_Creating_Distributed_Striped_Volumes
So first I have to delete the one I’ve created in the above part.
First umount the one I’ve created:

root@Gluster1:/home/glustervms# umount /home/glustervmsmnt 

checked via mount or df -h we could see the one we have mounted has been umounted.

Second, stop the volumes we’ve created:

root@Gluster1:/home/glustervms# gluster volume stop vol_replica
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: vol_replica: success

Third, delete volumn:

root@Gluster1:/home/glustervms# gluster volume delete vol_replica
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: vol_replica: success

Check the volume status:

root@Gluster1:/home/glustervms# gluster volume info
No volumes present

Create the distributed stripped volume

Create the bigvolume:

root@Gluster1:/home/glustervms# gluster volume create bigvolume transport tcp Gluster2:/home/glustervms Gluster1:/home/glustervms force

Start the volume:

root@Gluster1:/home/glustervms# gluster volume start bigvolume
volume start: bigvolume: success

View the status of the volume:

root@Gluster1:/home/glustervmsmnt# gluster volume info 
 
Volume Name: bigvolume
Type: Distribute
Volume ID: 3e09f074-4675-46d3-873f-f00ef13fb509
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: Gluster2:/home/glustervms
Brick2: Gluster1:/home/glustervms

Mount it via following command:

# mount -t glusterfs Gluster1:/bigvolume /home/glustervmsmnt/

Trouble-Shooting

root@Gluster1:/home/glustervms# gluster volume create bigvolume transport tcp Gluster2:/home/glustervms Gluster1:/home/glustervms force
volume create: bigvolume: failed: /home/glustervms or a prefix of it is already part of a volume

Resolve it via:

#  apt-get install attr
#  setfattr -x trusted.glusterfs.volume-id /home/glustervms
#  setfattr -x trusted.gfid /home/glustervms
#  rm -rf /home/glustervms/.glusterfs/

Re-run the gluster volome create command it will create the volume which combines two folders.

Digital Ocean Scenario

My DO droplet runs Ubuntu and CentOS, their version is Trusty(14.04) and CentOS7, so do following:

CentOS # wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo
CentOS # yum -y install glusterfs glusterfs-fuse glusterfs-server
CentOS # systemctl start glusterd
CentOS # systemctl enable glusterd
Trusty # apt-get install glusterfs-server

Add each other’s name and ip address into /etc/hosts, make sure they could ping each other and get responsible:

1xx.xx.xxx.xxx	CentOS
1xx.xx.xxx.xx	Trusty

Use Trusty as the server, so on the Trusty machine, detect the CentOS’s glusterd configuration as:

Trusty # gluster peer probe CentOS
peer probe: success

Check the status:

Trusty # gluster peer status
Number of Peers: 1

Hostname: CentOS
Port: 24007
Uuid: xxxxxxxx
State: Peer in Cluster (Connected)

Create the bigvolume, and mount it into your own directory via:

Trusty # gluster volume create bigvolume transport tcp CentOS:/home/glustervms Trusty:/home/glustervms force
volume create: bigvolume: success: please start the volume to access data
Trusty # gluster volume start bigvolume
volume start: bigvolume: success
Trusty # gluster volume info
 
Volume Name: bigvolume
Type: Distribute
Volume ID: xxxxxxxxxxxxxxxxxxxxx
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: CentOS:/home/glustervms
Brick2: Trusty:/home/glustervms
Trusty # mkdir /home/glustervmsmnt/
Trusty # mount -t glusterfs Trusty:/bigvolume /home/glustervmsmnt/
Trusty # df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/vda1                20G  9.4G  9.2G  51% /
none                    4.0K     0  4.0K   0% /sys/fs/cgroup
udev                    235M  8.0K  235M   1% /dev
tmpfs                    50M  396K   49M   1% /run
none                    5.0M     0  5.0M   0% /run/lock
none                    246M  1.1M  244M   1% /run/shm
none                    100M     0  100M   0% /run/user
Trusty:/bigvolume       40G   14G   24G  37% /home/glustervmsmnt

Now you could operate under the /home/glustervmsmnt and you have 24G size partion of the disk. Enjoy them!!!

Trouble-Shooting 1

If you met File change as we read it in tar something, do following things:

Trusty # gluster volume set bigvolume performance.stat-prefetch off
volume set: success

Build fuel icehouse iso

Fuel6.0 didn’t support icdhouse by default, so we have to build it manually, the steps are listed as following:

apt-get install git
mkdir ~/fuel
cd ~/fuel
git clone https://github.com/stackforge/fuel-main.git
cd fuel-main
 ./prepare-build-env.sh
export MIRROR_BASE=http://mirror.fuel-infra.org/fwm/6.0-icehouse
make iso

After making the iso which have icehouse will be available.

TroubleShooting

Some modifications should be made before we make them:

Trusty@ubuntu1204:~/code/fuel6.0/fuel-main$ git checkout stable/6.0
Branch stable/6.0 set up to track remote branch stable/6.0 from origin.
Switched to a new branch 'stable/6.0'
Trusty@ubuntu1204:~/code/fuel6.0/fuel-main$ git branch
  master
* stable/6.0

Quickly play puppet

Server and Client

In server, install following packages:

$ sudo apt-get install puppet puppetmaster

In client, only install puppet is enough.
After installation, added the each other’s name into /etc/hosts, let them ping each other via name rather than via ip address.

Sign

In client, do following:

$ clouder@pc121:/etc/puppet$ puppet agent --test --server=pc119
Exiting; no certificate found and waitforcert is disabled

Then in server, listed all of the certification request:

root@pc119:~/Code/herokublog# sudo puppet cert list
  "pc121" (SHA256) 28:23:36:3C:E4:8B:3A:15:D2:B0:8C:A2:BC:E9:A1:E5:6A:6F:76:0E:40:73:29:1F:8F:8C:D4:83:1F:92:4F:C7

sign the cert:

root@pc119:~/Code/herokublog# sudo puppet cert sign pc121
Notice: Signed certificate request for pc121
Notice: Removing file Puppet::SSL::CertificateRequest pc121 at '/var/lib/puppet/ssl/ca/requests/pc121.pem'

Verify it in the client:

clouder@pc121:/etc/puppet$ puppet agent --test
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: getaddrinfo: Name or service not known
Info: Retrieving plugin
Error: /File[/home/clouder/.puppet/var/lib]: Failed to generate additional resources using 'eval_generate': getaddrinfo: Name or service not known

Configuration

TBD

Fuel Build Issues

I started to deploy OpenContrail use fuel, so following are some tips for building the plugins.

Fule-Plugin-Builder

I encounter following errors when building the plugins of the contrail:

# fuel-plugin-builder --build ./
Unexpected error
Wrong package version "2.0.0"

This is because the FPB on PyPI is too old for building the 2.0.0 version of the fuel-plugins.
Work-around is we manually create the fpb via following steps:

# git clone https://github.com/stackforge/fuel-plugins
# cd fuel-plugins/fuel_plugin_builder
# python setup.py sdist
# pip install dist/fuel-plugin-builder-2.0.0.dev.tar.gz

Now your fpb could build 2.0.0 version of the packages.

Build Fuel Contrail plugin

First we install following packages on Ubuntu.

# sudo apt-get install createrepo rpm dpkg-dev
# easy_install pip
# pip install fuel-plugin-builder

But notice the fuel-plugin-builder is too old in pip repository, so use the git version instead.
Or in CentOS, you could install fuel-plugin-build via:

# yum install createrepo rpm dpkg-devel
# pip install fuel-plugin-builder

Retrive the source code and build it via:

# git clone https://github.com/stackforge/fuel-plugin-contrail
# cd fuel-plugin-contrail/
# fpb --build . --debug
[root:/home/juju/Code/fuel-plugin-contrail]# ls
contrail-1.0-1.0.0-0.noarch.rpm  LICENSE         repositories
deployment_scripts               metadata.yaml   specs
environment_config.yaml          pre_build_hook  tasks.yaml
install.sh                       README.md

Once the contrail-1.0xxxx.rpm is available it indicates the plugin is available for fuel to use.
Notice this plugin should be compatible for Mirantis Fuel 6.1 and Juniper Contrail 2.01.

Using Fuel For Deploying OpenStack

Network Configuration

Fuel network configuration is listed as following pictures:
PXE network, for using fuel controller to control all of the nodes, 10.20.0.0/24:

Public network, or floating ip network 172.16.0.0/24
Admin network, 192.168.0.0/24:

Fuel Controller Installation

Create a virtual machine, which have 2-Core, 3072MB Memory, and 100G Hard-disk, 3 ethernet port available for using, startup using the iso file, and then beging installing.