Gluster filesystem mount issue in LXC container

I was setting up a Gluster cluster file system in 2 LXC containers where I ran into an error.

Implementation process

1. Install gluster-server,

in both containers:

$ sudo apt-get install glusterfs-server

2. Peer the gluster nodes,

in container01:

$ sudo gluster peer probe container02
peer probe: success

in container02:

$ sudo gluster peer probe container01
peer probe: success

3. Create the glusterfs manager folder,

in both containers:

$ sudo mkdir /gluster_data

4. Create vol1 volume,

in containers01:

$ sudo gluster volume create vol1 replica 2 transport tcp container01:/gluster_data container02:/gluster_data force
volume create: vol1: success: please start the volume to access data

5. Start the volume,

in container01:

$ sudo gluster volume start vol1
volume start: vol1: success

6. Mount the newly created shared storage vol1:

$ sudo mkdir /pool1

$ sudo mount /pool1 

And here, when I tried to mount the vol1, i got following error:

[2015-12-12 21:19:37.277176] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.2 (/usr/sbin/glusterfs --volfile-id=/vol1 --volfile-server=container01 /pool1)

[2015-12-12 21:19:37.277756] E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (No such file or directory)

[2015-12-12 21:19:37.277768] E [xlator.c:390:xlator_init] 0-fuse: Initialization of volume 'fuse' failed, review your volfile again

As per error it seems gluster looking for fuse device /dev/fuse which apparently isn't exist.

$ sudo ls /dev/fuse

ls: cannot access /dev/fuse: No such file or directory

Solution

To create the fuse device manually,

in both containers:

$ sudo mknod /dev/fuse c 10 229

Now try to mount again:

$ sudo mount /pool1 

$ df -h /pool1

Filesystem            Size  Used Avail Use% Mounted on

container01:/vol1     30G   26G  2.5G  92% /pool1

And if you want to make the mount boot persistent, add the following line in the /etc/fstab for each container respectively: 

$ cat /etc/fstab

# UNCONFIGURED FSTAB FOR BASE SYSTEM

container01:/vol1 /pool1 glusterfs defaults,_netdev 0 0 

Create Snappy Ubuntu as a Docker Image

Snappy ubuntu core is the latest member of ubuntu family that specifically designed to run on Linux containers. To put it simple it's a stripped down ubuntu with some advanced features such as transactional upgrades/rollback to bring more stability, security with AppArmor and new snappy package manager instead of apt-get.

I wanted to run snappy ubuntu on a docker container, however I couldn't find the base snappy image on the docker hub to pull. So I decided to make and push it myself to docker hub.

First we need to download the Snappy image. It can be downloaded from https://developer.ubuntu.com/en/snappy/start/#try-x86

# wget http://releases.ubuntu.com/15.04/ubuntu-15.04-snappy-amd64-generic.img.xz

It comes as a XZ compressed IMG filesystem dumped image. we will unzx it:

# unzx ubuntu-15.04-snappy-amd64-generic.img.xz

For ISO images we simply can loop mount and access them:

# mkdir /mnt/iso
# mount -o loop image.iso /mnt/iso

And proceed to read the mounted directory as tar and create the docker image from STDIN:

# tar -C /mnt/iso -c . | docker import - yourrepo/yourimage:anytag


However, for IMG dump images it's a bit hassle-y:

1. Load the nbd kernel module with max_parts option if you are in a debian-based machine:

# modprobe nbd max_parts=16

2. Mount the image to /dev/nbd0 with qemu:

# qemu-nbd -c /dev/nbd0 ubuntu-15.04-snappy-amd64-generic.img

3. Ask kernel to re-read the partition table:

# partprobe /dev/nbd0


We can see the filsystem within this image via fdisk:

# fdisk -l /dev/nbd0

Disk /dev/nbd0: 3,6 GiB, 3899999744 bytes, 7617187 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 374F5ED9-98FE-47B5-B430-A8983D7CB41B

Device Start End Sectors Size Type

/dev/nbd0p1 8192 16383 8192 4M BIOS boot
/dev/nbd0p2 16384 278527 262144 128M EFI System
/dev/nbd0p3 278528 2375679 2097152 1G Linux filesystem
/dev/nbd0p4 2375680 4472831 2097152 1G Linux filesystem
/dev/nbd0p5 4472832 7614463 3141632 1,5G Linux filesystem


Our desired contents are located in nbd0p3, so we will mount it:

# mount /dev/nbd0p3 /mnt/snappy/

There we are! We have all that is required to create our docker image:

# tar -C /mnt/snappy/ -c . | docker import - arvinep/ubuntu:snappy

To check the newly created image, let's list the local docker images:

# docker images

REPOSITORY            TAG    IMAGE ID      CREATED        VIRTUAL SIZE

arvinep/ubuntu-snappy snappy 2e796faa9adc  9 minutes ago  604.8 MB

ubuntu                latest d55e68e6cc9c  26 hours ago   187.9 MB

Cool! Let's run it and access to our docker container:

# docker run -it arvinep/ubuntu:snappy /bin/bash

It's up and running alongside with other docker containers:

# docker ps

CONTAINER ID   IMAGE                 COMMAND      CREATED             STATUS              PORTS  NAMES
25d60fa0238d   arvinep/ubuntu:snappy "/bin/bash"  About a minute ago  Up About a minute          sleepy_banach       
1cff710b5604   ubuntu:latest         "/bin/bash"  2 hours ago         Up 2 hours                 angry_davinci       
afe31fbcbf7d   ubuntu:latest         "/bin/bash"  2 hours ago         Up 2 hours                 serene_lovelace     
74a46b4b203f   ubuntu:latest         "/bin/bash"  2 hours ago         Up 2 hours                 tender_bartik    

I have pushed the snappy ubuntu to docker hub so you can pull and enjoy it:

# docker run -it arvinep/ubuntu-snappy /bin/bash

OpenSSH client access issues after patching to version 7

After OpenSSH has been patched from vulnerable version 5 to the latest secure version 7.1p, we have encountered some connection issues with some of the clients.

Error:

# tail -f /var/log/messages 

...

fatal: Unable to negotiate with 213.61.200.74: no matching cipher found. 

Their offer: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,arcfour [preauth]

Root Cause:

Based on the version 7.1 release note, many ciphers have been disabled due to security issues:

OpenSSH 7.1 release note: 

 * Several ciphers will be disabled by default: blowfish-cbc,

   cast128-cbc, all arcfour variants and the rijndael-cbc aliases

   for AES.

Solution:

Need to add legacy ciphers to sshd_config in order to support the ssh client:

# vim /etc/ssh/sshd_config

...

Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,blowfish-cbc,aes128-cbc,3des-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,arcfour

Error:

After adding the ciphers and restarting daemon, same client encounter different error:

# tail -f /var/log/messages 

...

fatal: Unable to negotiate with 213.61.200.74: no matching key exchange method found. Their offer: 

diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1 [preauth]

Root Cause:

Based on the version 7.0 release note, some of the key exchange methods have been disabled

OpenSSH 7.0 release note: 

 * Support for the 1024-bit diffie-hellman-group1-sha1 key exchange

   is disabled by default at run-time. It may be re-enabled using

   the instructions at http://www.openssh.com/legacy.html

 * ssh(1), sshd(8): extend Ciphers, MACs, KexAlgorithms,

   HostKeyAlgorithms, PubkeyAcceptedKeyTypes and HostbasedKeyTypes

   options to allow appending to the default set of algorithms

   instead of replacing it. Options may now be prefixed with a '+'

   to append to the default, e.g. "HostKeyAlgorithms=+ssh-dss".

Solution:

To add the legacy MAC and key exchange algorithms back:

# vim /etc/ssh/sshd_config

...

MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-sha1-96,hmac-md5-96

KexAlgorithms +diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1

Building RPM OpenSSH 7.1p1 on RHEL/CentOS 6.5

After I've implemented the OpenVAS vulnerability assessment system, I've made a complete vulnerability testing on the environment for both Linux and Wintel servers.

The result for the all Linux servers were Red :) Severity 10.0(High). The reason was the OpenSSH version 5.

Test result:

High (CVSS: 8.5)

NVT: OpenSSH Multiple Vulnerabilities (OID: 1.3.6.1.4.1.25623.1.0.806052)

Product detection result: cpe:/a:openbsd:openssh:5.3 by SSH Server type and version (OID: 1.3.6.1.4.1.25623.1.0.10267)

SummaryThis host is running OpenSSH and is prone to multiple vulnerabilities.

Vulnerability Detection Result

Installed version: 5.3
Fixed version:     7.0

ImpactSuccessful exploitation will allow an attacker to gain privileges, to conduct impersonation attacks, to conduct brute-force attacks or cause a denial of service.
Impact Level: Application

SolutionUpgrade to OpenSSH 7.0 or later. For updates refer to http://www.openssh.com

Affected Software/OSOpenSSH versions before 7.0

Vulnerability InsightMultiple flaws are due to: - Use-after-free vulnerability in the 'mm_answer_pam_free_ctx' function in monitor.c in sshd. - Vulnerability in 'kbdint_next_device' function in auth2-chall.c in sshd. - vulnerability in the handler for the MONITOR_REQ_PAM_FREE_CTX request.

Vulnerability Detection MethodGet the installed version with the help of detect NVT and check the version is vulnerable or not.
Details: OpenSSH Multiple Vulnerabilities (OID: 1.3.6.1.4.1.25623.1.0.806052)
Version used: $Revision: 1784 $

Product Detection Result

Product:	cpe:/a:openbsd:openssh:5.3
Method:	SSH Server type and version (OID: 1.3.6.1.4.1.25623.1.0.10267)

References

CVE:	CVE-2015-6564, CVE-2015-6563, CVE-2015-5600
CERT:	DFN-CERT-2015-1679 , DFN-CERT-2015-1644 , DFN-CERT-2015-1632 , DFN-CERT-2015-1591 , DFN-CERT-2015-1443 , DFN-CERT-2015-1406 , DFN-CERT-2015-1263 , DFN-CERT-2015-1259 , DFN-CERT-2015-1252 , DFN-CERT-2015-1239 , DFN-CERT-2015-1161 , DFN-CERT-2015-1159
Other:	http://seclists.org/fulldisclosure/2015/Aug/54
	http://openwall.com/lists/oss-security/2015/07/23/4

As per above table, solution is to upgrade the OpenSSH to version 7.

Being in CentOS 6.5, it is not possible to use default YUM repo in order to upgrade the package as latest package version 7 is not exist in the official repositories.

Installing the OpenSSH 7 from the RPM package also was not an option as you will run into the dependency hell as jumping from OpenSSH 5 of CentOS 6.5 to OpenSSH 7 of CentOS 7 requires lots of package dependencies to be resolved.

My only remaining option was to build the OpenSSH from the latest source code.

I ran through building the package and faced some issues. Following is how I did and how solved the issues.


1. Download the latest source code from openssh website and extract the package:

# wget http://ftp.spline.de/pub/OpenBSD/OpenSSH/portable/openssh-7.1p1.tar.gz

# tar xzf openssh-7.1p1.tar.gz


2. Copy the specification file to specs folder:

Make sure folder is exist:

# mkdir -p /usr/src/redhat/SPECS/

# cp openssh-7.1p1/contrib/redhat/openssh.spec /usr/src/redhat/SPECS/


3. Copy the downloaded source code to sources directory:

# cp openssh-7.1p1.tar.gz /root/rpmbuild/SOURCES/


4. We need to make some changes in our spec file. The x11-askpass and gnome-askpass needs to be disabled as they belong to graphical environment. So we will change the default value of 0 to 1 in below:

# vim /usr/src/redhat/SPECS/openssh.spec
...
# Do we want to disable building of x11-askpass? (1=yes 0=no)
%define no_x11_askpass 1

# Do we want to disable building of gnome-askpass? (1=yes 0=no)
%define no_gnome_askpass 1

5. Now we are ready to build our RPM from the source:

# cd /usr/src/redhat/SPECS

# rpmbuild -bb openssh.spec

If it went smooth then you should be greeted with "exit 0" success code and 3 newly built RPMs:

# ll /root/rpmbuild/RPMS/x86_64/

total 2052

-rw-r--r-- 1 root root 725882 Nov 25 17:05 openssh-7.1p1-1.x86_64.rpm

-rw-r--r-- 1 root root 915295 Nov 25 17:05 openssh-clients-7.1p1-1.x86_64.rpm

-rw-r--r-- 1 root root 452869 Nov 25 17:05 openssh-server-7.1p1-1.x86_64.rpm

However, if you ran into errors like me, then you were not the luckiest person.

Issue 1:

During the first run, following error came up:

RPM build errors:

    line 92: buildprereq is deprecated: BuildPreReq: glibc-devel, pam

    Bad exit status from /var/tmp/rpm-tmp.cxuIJZ (%build)

Solution:
comment the deprecated "BuildPreReq" in line 92.


Issue 2:
The second round of built ran into below error:

--Next error:
checking whether OpenSSL has NID_secp384r1... yes
checking whether OpenSSL has NID_secp521r1... yes
checking if OpenSSL's NID_secp521r1 is functional... no
checking for arc4random... no
checking for arc4random_buf... no
checking for arc4random_stir... no
checking for arc4random_uniform... no
checking for ia_openinfo in -liaf... no
checking whether OpenSSL's PRNG is internally seeded... yes
configure: error: PAM headers not found
error: Bad exit status from /var/tmp/rpm-tmp.mMkQeT (%build)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.mMkQeT (%build)

Solution:
As stated in the output, the PAM headers not found, so it needs to be installed:

# yum install pam-devel

After you have successfully built the OpenSSH RPMs, you can go ahead and install them on top of your current OpenSSH.

First make sure you create a backup from your current ssh client and server configs:

# mv /etc/ssh/ssh_config /etc/ssh/ssh_config.before
# mv /etc/ssh/sshd_config /etc/ssh/sshd_config.before

# rpm -Uvh openssh-7.1p1-1.x86_64.rpm openssh-server-7.1p1-1.x86_64.rpm openssh-clients-7.1p1-1.x86_64.rpm

To confirm:

# yum info openssh

Installed Packages

Name        : openssh

Arch        : x86_64

Version     : 7.1p1

Release     : 1

Size        : 1.9 M

Repo        : installed

Summary     : The OpenSSH implementation of SSH protocol versions 1 and 2.

URL         : http://www.openssh.com/portable.html

License     : BSD

OpenSSH server needs to be resatrted for changes to take effect. However, please make sure you have a proper console access to your machine as restarting sshd daemon might disconnect your ssh connection:

# /etc/init.d/sshd restart

Stopping sshd:                                         [  OK  ]

Starting sshd:                                         [  OK  ]

Side Note:

If you are using a local YUM repository as I do, then you might want to add your new RPMs to your repo and make them accessible to other servers as well. 

First copy your new rpms to your repo packages path:

# cp /root/rpmbuild/RPMS/x86_64/openssh-* /repo_pub/pub/LocalYumRepoRHEL6.5/Packages/

Then update your repo database: 

# createrepo --update /repo_pub/pub/LocalYumRepoRHEL6.5/

Now from client side you should be able to update your OpenSSH with yum command,

First cleanup the repo data:

# yum clean all

Then update the yum:

# yum update

puppet-deps                                                                                                                                  | 2.5 kB     00:00

puppet-deps/primary_db                                                                                                                       |  27 kB     00:00

puppet-products                                                                                                                              | 2.5 kB     00:00

puppet-products/primary_db                                                                                                                   | 155 kB     00:00

rhel-local                                                                                                                                   | 2.9 kB     00:00

rhel-local/primary_db                                                                                                                        | 3.2 MB     00:00

Setting up Update Process

Resolving Dependencies

--> Running transaction check

---> Package openssh.x86_64 0:5.3p1-94.el6 will be updated

---> Package openssh.x86_64 0:7.1p1-1 will be an update

---> Package openssh-clients.x86_64 0:5.3p1-94.el6 will be updated

---> Package openssh-clients.x86_64 0:7.1p1-1 will be an update

---> Package openssh-server.x86_64 0:5.3p1-94.el6 will be updated

---> Package openssh-server.x86_64 0:7.1p1-1 will be an update

--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================================================

 Package                                  Arch                            Version                             Repository                           Size

====================================================================================================================================================================

Updating:

 openssh                                      x86_64                              7.1p1-1                             rhel-local                              709 k

 openssh-clients                              x86_64                              7.1p1-1                             rhel-local                              894 k

 openssh-server                               x86_64                              7.1p1-1                             rhel-local                              442 k

Transaction Summary

====================================================================================================================================================================

Upgrade       3 Package(s)

Total download size: 2.0 M

Is this ok [y/N]:y

Note:

After I've restarted the sshd daemon, I couldn't connect to some of the servers,

I received following error in messages logs while try to establish a new connection after upgraded OpenSSH:

# tail -f /var/log/messages 

...

error: sshd error Could not get shadow information for user

Solution:

I've noticed this pattern occurs on the servers with SELinux Enforcing!

# getenforce

Enforcing

You can temporary disable the SELinux with "setenforce":

# setenforce 0

And to make it permanent, disable SELinux by setting the "disabled" in its config file, which requires server reboot:

# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux