2 December 2015

Building RPM OpenSSH 7.1p1 on RHEL/CentOS 6.5

After I've implemented the OpenVAS vulnerability assessment system, I've made a complete vulnerability testing on the environment for both Linux and Wintel servers.

The result for the all Linux servers were Red :) Severity 10.0(High). The reason was the OpenSSH version 5.

Test result:

High (CVSS: 8.5)
NVT: OpenSSH Multiple Vulnerabilities (OID: 1.3.6.1.4.1.25623.1.0.806052)
Product detection result: cpe:/a:openbsd:openssh:5.3 by SSH Server type and version (OID: 1.3.6.1.4.1.25623.1.0.10267)
SummaryThis host is running OpenSSH and is prone to multiple vulnerabilities.
Vulnerability Detection Result
Installed version: 5.3
Fixed version:     7.0
ImpactSuccessful exploitation will allow an attacker to gain privileges, to conduct impersonation attacks, to conduct brute-force attacks or cause a denial of service.
Impact Level: Application
SolutionUpgrade to OpenSSH 7.0 or later. For updates refer to http://www.openssh.com
Affected Software/OSOpenSSH versions before 7.0
Vulnerability InsightMultiple flaws are due to: - Use-after-free vulnerability in the 'mm_answer_pam_free_ctx' function in monitor.c in sshd. - Vulnerability in 'kbdint_next_device' function in auth2-chall.c in sshd. - vulnerability in the handler for the MONITOR_REQ_PAM_FREE_CTX request.
Vulnerability Detection MethodGet the installed version with the help of detect NVT and check the version is vulnerable or not.
Details: OpenSSH Multiple Vulnerabilities (OID: 1.3.6.1.4.1.25623.1.0.806052)
Version used: $Revision: 1784 $
Product Detection Result
Product:cpe:/a:openbsd:openssh:5.3
Method:SSH Server type and version (OID: 1.3.6.1.4.1.25623.1.0.10267)
References
CVE:CVE-2015-6564, CVE-2015-6563, CVE-2015-5600
CERT:DFN-CERT-2015-1679 , DFN-CERT-2015-1644 , DFN-CERT-2015-1632 , DFN-CERT-2015-1591 , DFN-CERT-2015-1443 , DFN-CERT-2015-1406 , DFN-CERT-2015-1263 , DFN-CERT-2015-1259 , DFN-CERT-2015-1252 , DFN-CERT-2015-1239 , DFN-CERT-2015-1161 , DFN-CERT-2015-1159
Other:http://seclists.org/fulldisclosure/2015/Aug/54
http://openwall.com/lists/oss-security/2015/07/23/4


As per above table, solution is to upgrade the OpenSSH to version 7.

Being in CentOS 6.5, it is not possible to use default YUM repo in order to upgrade the package as latest package version 7 is not exist in the official repositories.

Installing the OpenSSH 7 from the RPM package also was not an option as you will run into the dependency hell as jumping from OpenSSH 5 of CentOS 6.5 to OpenSSH 7 of CentOS 7 requires lots of package dependencies to be resolved.

My only remaining option was to build the OpenSSH from the latest source code.

I ran through building the package and faced some issues. Following is how I did and how solved the issues.


1. Download the latest source code from openssh website and extract the package:

# wget http://ftp.spline.de/pub/OpenBSD/OpenSSH/portable/openssh-7.1p1.tar.gz

# tar xzf openssh-7.1p1.tar.gz


2. Copy the specification file to specs folder:

Make sure folder is exist:

# mkdir -p /usr/src/redhat/SPECS/

# cp openssh-7.1p1/contrib/redhat/openssh.spec /usr/src/redhat/SPECS/


3. Copy the downloaded source code to sources directory:

# cp openssh-7.1p1.tar.gz /root/rpmbuild/SOURCES/


4. We need to make some changes in our spec file. The x11-askpass and gnome-askpass needs to be disabled as they belong to graphical environment. So we will change the default value of 0 to 1 in below:

# vim /usr/src/redhat/SPECS/openssh.spec
...
# Do we want to disable building of x11-askpass? (1=yes 0=no)
%define no_x11_askpass 1

# Do we want to disable building of gnome-askpass? (1=yes 0=no)
%define no_gnome_askpass 1


5. Now we are ready to build our RPM from the source:

# cd /usr/src/redhat/SPECS

# rpmbuild -bb openssh.spec


If it went smooth then you should be greeted with "exit 0" success code and 3 newly built RPMs:

# ll /root/rpmbuild/RPMS/x86_64/
total 2052
-rw-r--r-- 1 root root 725882 Nov 25 17:05 openssh-7.1p1-1.x86_64.rpm
-rw-r--r-- 1 root root 915295 Nov 25 17:05 openssh-clients-7.1p1-1.x86_64.rpm
-rw-r--r-- 1 root root 452869 Nov 25 17:05 openssh-server-7.1p1-1.x86_64.rpm


However, if you ran into errors like me, then you were not the luckiest person.

Issue 1:
During the first run, following error came up:

RPM build errors:
    line 92: buildprereq is deprecated: BuildPreReq: glibc-devel, pam
    Bad exit status from /var/tmp/rpm-tmp.cxuIJZ (%build)


Solution:
comment the deprecated "BuildPreReq" in line 92.


Issue 2:
The second round of built ran into below error:

--Next error:
checking whether OpenSSL has NID_secp384r1... yes
checking whether OpenSSL has NID_secp521r1... yes
checking if OpenSSL's NID_secp521r1 is functional... no
checking for arc4random... no
checking for arc4random_buf... no
checking for arc4random_stir... no
checking for arc4random_uniform... no
checking for ia_openinfo in -liaf... no
checking whether OpenSSL's PRNG is internally seeded... yes
configure: error: PAM headers not found
error: Bad exit status from /var/tmp/rpm-tmp.mMkQeT (%build)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.mMkQeT (%build)

Solution:
As stated in the output, the PAM headers not found, so it needs to be installed:

# yum install pam-devel


After you have successfully built the OpenSSH RPMs, you can go ahead and install them on top of your current OpenSSH.

First make sure you create a backup from your current ssh client and server configs:

# mv /etc/ssh/ssh_config /etc/ssh/ssh_config.before
# mv /etc/ssh/sshd_config /etc/ssh/sshd_config.before

# rpm -Uvh openssh-7.1p1-1.x86_64.rpm openssh-server-7.1p1-1.x86_64.rpm openssh-clients-7.1p1-1.x86_64.rpm


To confirm:

# yum info openssh
Installed Packages
Name        : openssh
Arch        : x86_64
Version     : 7.1p1
Release     : 1
Size        : 1.9 M
Repo        : installed
Summary     : The OpenSSH implementation of SSH protocol versions 1 and 2.
URL         : http://www.openssh.com/portable.html
License     : BSD


OpenSSH server needs to be resatrted for changes to take effect. However, please make sure you have a proper console access to your machine as restarting sshd daemon might disconnect your ssh connection:

# /etc/init.d/sshd restart
Stopping sshd:                                         [  OK  ]
Starting sshd:                                         [  OK  ]


Side Note:
If you are using a local YUM repository as I do, then you might want to add your new RPMs to your repo and make them accessible to other servers as well. 

First copy your new rpms to your repo packages path:

# cp /root/rpmbuild/RPMS/x86_64/openssh-* /repo_pub/pub/LocalYumRepoRHEL6.5/Packages/


Then update your repo database: 

# createrepo --update /repo_pub/pub/LocalYumRepoRHEL6.5/


Now from client side you should be able to update your OpenSSH with yum command,

First cleanup the repo data:

# yum clean all


Then update the yum:

# yum update
puppet-deps                                                                                                                                  | 2.5 kB     00:00
puppet-deps/primary_db                                                                                                                       |  27 kB     00:00
puppet-products                                                                                                                              | 2.5 kB     00:00
puppet-products/primary_db                                                                                                                   | 155 kB     00:00
rhel-local                                                                                                                                   | 2.9 kB     00:00
rhel-local/primary_db                                                                                                                        | 3.2 MB     00:00
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package openssh.x86_64 0:5.3p1-94.el6 will be updated
---> Package openssh.x86_64 0:7.1p1-1 will be an update
---> Package openssh-clients.x86_64 0:5.3p1-94.el6 will be updated
---> Package openssh-clients.x86_64 0:7.1p1-1 will be an update
---> Package openssh-server.x86_64 0:5.3p1-94.el6 will be updated
---> Package openssh-server.x86_64 0:7.1p1-1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================================================
 Package                                  Arch                            Version                             Repository                           Size
====================================================================================================================================================================
Updating:
 openssh                                      x86_64                              7.1p1-1                             rhel-local                              709 k
 openssh-clients                              x86_64                              7.1p1-1                             rhel-local                              894 k
 openssh-server                               x86_64                              7.1p1-1                             rhel-local                              442 k

Transaction Summary
====================================================================================================================================================================
Upgrade       3 Package(s)

Total download size: 2.0 M
Is this ok [y/N]:y


Note:
After I've restarted the sshd daemon, I couldn't connect to some of the servers,
I received following error in messages logs while try to establish a new connection after upgraded OpenSSH:

# tail -f /var/log/messages 
...
error: sshd error Could not get shadow information for user


Solution:
I've noticed this pattern occurs on the servers with SELinux Enforcing!

# getenforce
Enforcing

You can temporary disable the SELinux with "setenforce":

# setenforce 0

And to make it permanent, disable SELinux by setting the "disabled" in its config file, which requires server reboot:

# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux





22 comments:

  1. Very nicely explained article. worked very well.

    ReplyDelete
  2. only issue i am facing is after closing the session and connecting again as root user, i am facing permission denied error. any idea why this is happening now?

    ReplyDelete
    Replies
    1. That could be due to the upgraded sshd_config file which disables the direct root login by default. You can enable it in /etc/ssh/sshd_config:
      PermitRootLogin yes

      Delete
  3. I have added the line PermitRootLogin yes but it didnt work, i added the line AllowUsers root than also it didnt work. Can you please help me debugging it.
    my other sudo users are working but somehow i want to enable root user login for a while.

    ReplyDelete
    Replies
    1. 1. Did you restart the sshd after you made the change?
      #service sshd restart

      2. You should only have one "PermitRootLogin" in sshd_config file, and it should be set to "yes", then restart ssh daemon.

      3. Is SELinux enabled in your machine? If yes, it needs to be "permissive"!
      # getenforce

      4. You may paste your /var/log/messages log entries here while trying to login with root to your machine.
      # tail -f /var/log/messages

      Delete
  4. Im unable to ssh to the host at all once I have updated the package... any ideas ? My ultimate requirement is to have users ssh with dsa key only, but currently I cannot ssh with PermitRootLogin yes. PasswordAuthentication yes. In other words this package build renders ssh useless..

    ReplyDelete
  5. you can paste your /var/log/messages log entries while try to ssh to the box.

    ReplyDelete
  6. Hi Arvin - we have WHM and cpanel in our centos 7. When I tried to Copy the downloaded source code to sources directory, it states "cp: cannot create regular file '/root/rpmbuild/SOURCES/': No such file or directory". I imagine the source code is elsewhere, but I don't know where it is or how to look for this. Could you please help?

    ReplyDelete
    Replies
    1. Hi there, make sure the directory /root/rpmbuild/SOURCES is exist, if not you can create it:
      # ls -ld /root/rpmbuild/SOURCES
      # mkdir -p /root/rpmbuild/SOURCES
      then you can copy your downloaded source code to SOURCES dir and continue the process.
      If you are not sure where did you download the source code, you can search for it:
      # find / -iname '*openssh-7.1p1.tar.gz*'
      Hope it helps,

      Delete
    2. I managed to finish the install - luckily no errors. Now, I think I may have missed the step of removing the other SSH. When typing "yum list installed openssh", it lists our SSH as 6.6.1. I installed version 7.1p2, which is the latest available. How do I remove version 6.6.1 and make it use the version I installed now? Thanks!

      Delete
    3. So I guess you missed this part of the article:
      "After you have successfully built the OpenSSH RPMs, you can go ahead and install them on top of your current OpenSSH.

      First make sure you create a backup from your current ssh client and server configs:

      # mv /etc/ssh/ssh_config /etc/ssh/ssh_config.before
      # mv /etc/ssh/sshd_config /etc/ssh/sshd_config.before

      # rpm -Uvh openssh-7.1p1-1.x86_64.rpm openssh-server-7.1p1-1.x86_64.rpm openssh-clients-7.1p1-1.x86_64.rpm"

      Delete
  7. This worked but I had to remove the existing openssh(5.3) before installing the 7.1 rpms as there were conflicts :
    error: Failed dependencies:
    openssh = 5.3p1-112.el6_7 is needed by (installed) openssh-clients-5.3p1-112.el6_7.x86_64
    openssh = 5.3p1-112.el6_7 is needed by (installed) openssh-askpass-5.3p1-112.el6_7.x86_64
    openssh = 5.3p1-112.el6_7 is needed by (installed) openssh-server-5.3p1-112.el6_7.x86_64

    ReplyDelete
  8. I want to enable tcp-wrappers. For that what i have to do?

    ReplyDelete
    Replies
    1. May I also know how to enable tcp-wrappers?

      Help Arvin.
      I have followed your instructions on how to update openssh from v5 to 7.2P

      No output display when checking for tcp wrapper:
      $ ldd /usr/sbin/sshd | grep libwrap
      $

      Delete
  9. Thank you for documenting this procedure. It made PCI scan remediation trivial.

    I had an additional error installing version 7.2p2-1:

    "Processing files: openssh-clients-7.2p2-1.x86_64
    error: File not found:
    /root/rpmbuild/BUILDROOT/openssh-7.2p2-1.x86_64/usr/bin/slogin
    error: File not found by glob:
    /root/rpmbuild/BUILDROOT/openssh-7.2p2-1.x86_64/usr/share/man/man1/slogin.1*"

    I got past that error by commenting the following two lines from the openssh.spec file:

    %attr(-,root,root) %{_bindir}/slogin
    %attr(-,root,root) %{_mandir}/man1/slogin.1*

    ReplyDelete
  10. You totally save us Arvin Ebrahimpour
    Thank you so much!

    ReplyDelete
  11. Hi Arvin,
    I've followed all the steps, and successfully build and install rpm, but when try to connect from my other machine, i got this error: no matching host key type found. Their offer:
    I tried to verbose the connection: ssh -vv user@host,
    and got this:

    ...
    debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
    debug2: host key algorithms:
    ...
    It seems that sshd does not have host key algorithms.
    please help

    ReplyDelete
    Replies
    1. Hey Vigong,
      The new sshd server 7.0 and above has disabled many ciphers and key algorithms which are weak.
      That's the reason when you try to connect to newly installed sshd daemon, the connection get refused as the ssh client try to connect to server with keys that no longer supported due to security vulnerabilities.

      Although you still have an option to get around it by using following command where "ssh-dss" is the key which fails to negotiate and it might be different in your case, so substitute it accordingly:

      ssh -oHostKeyAlgorithms=+ssh-dss user@host

      Delete
  12. This document is fantastic. I have just used it to upgrade to openssh7.5.

    I experienced an issue that I think is worth sharing:
    When I tried to log into the server afterwards I got the error "no hostkey alg". After some digging, the following commands did the trick:
    chmod 600 ssh_host_ecdsa_key
    chmod 600 ssh_host_.........._key

    .......= use the keys you have here.

    Other than that, a really useful and incredible document. Thanks very much.

    ReplyDelete