RHEL6 my update has been unexpected aborted

Today I faced with serious issue. During update execution my server has been unexpectedly rebooted. I found root cause. It was bad disk queue timing for Qdisk cluster component. I forgot to disable cluster services during update. This is recommended action. I always remember about it but at this moment I hurry up and forgot about it. This is shame but it happens ;(

Server did not start properly and I had to use rescue CD to fix kernel. I wasted a lot of time. It was enough to start system from previous kernel - grub menu and fix yum transactions.

Let's to fix cluster node. We do not have to much time. I know all services are up and running on standby node but we have no redundancy. 

After system will bring up with previous kernel you can heck how many packages are missing or duplicted.

Before you start please create list of duplicated packages. It may be important for further moves. You can't start without it!


First step: Cancel all not completed yum transaction:
# yum-complete-transaction --cleanup-only


Second step: Cleanup system from duplicate packages:
# package-cleanup --cleandupes

Be aware at the end of checking section you can find package list which will be removed. Please save the list as separate file. It will be useful if new packages not have installed yet.

After reboot I observed that some packages are missing :( It really hurts but we must to handle it.

# mkdir /root/tmp/
# cat  packages-list-erased-`hostname`-`date +%F`

  Erasing    : ipa-client-3.0.0-37.el6.x86_64                                                                                             1/252 
  Erasing    : sssd-1.9.2-129.el6.x86_64                                                                                                    2/252
  Erasing    : libvirt-client-0.10.2-29.el6.x86_64                                                                                       3/252
  .
  ..
  ...
  Erasing    : glibc-2.12-1.132.el6.x86_64                                                                                                250/252
  Erasing    : tzdata-2013g-1.el6.noarch                                                                                                  251/252
  Erasing    : libgcc-4.4.7-4.el6.x86_64                                                                                                    252/252
  

# cat packages-list-erased-`hostname`-`date +%F` | awk '{ print $3 }' | sed 's/^[0-9]://' | \ awk -F- '{ print $1 "-" $2}' | sed 's/-[0-9].*$//' > /root/tmp/packages-list-missed- \ `hostname`-`date +%F`

# for package in `cat /root/tmp//root/tmp/packages-list-missed-`hostname`-`date \ +%F``;do yum -y install $package;done

Of course you must reboot server to take the effect and apply all changes. After reboot I have been surprised very positive. My server was up and running with all services. I checked cluster services and were also fine, only one small remark: Please check if rgmanager is up and running after reboot. I found that did not started during booting phase. 

# chkconfig --add rgmanager

# chkconfig --level=3 rgmanager on
# chkconfig --level=5 rgmanager on

# service rgmanager start  
# clustat
 
Cluster Status for RH6cluster01 @ Sat Apr 25 20:59:20 2015
Member Status: Quorate

 Member Name                                       ID   Status
 ------ ----                                                    ---- ------
 clrh6n01                                                 1 Online, rgmanager
 clrh6n02                                                 2 Online, Local, rgmanager
 /dev/VolGroupQdisk/lv_qdisk                 0 Online, Quorum Disk

 Service Name                                         Owner (Last)                                          State        
 ------- ----                                                    ----- ------                                                    -----        
 service:appRG                                        clrh6n02                                                  started      
 service:dbRG                                          clrh6n01                                                  started

Lessons and learn: do not perform update without yum lvm fs snapshoot plugin. This functionality is desirable during system update and the maintenance tasks. You can save a lot of time/money in case of update failure. You should always perform full backup system and have a shot in the locker proper back out procedure.

How to create chrooted SFTP server in EL5

I will try to show you how to install and setup sftp chroot in RHEL5. You can adopt this solution for higher RedHat Linux versions. The IT space isn't perfect. I will present different approach to provide SFTP server without active shell access. I chose EL5 for a reason. In EL6 and EL7 you have possibility to configure sftp-chroot inside ssh. In EL5 no possible to do that. Nothing more nothing less.

Let's present some details about sftp jail:

• sftp will be running on separate port higher that 2048 my port is: 9987
• there will be no dedicate IP for sftp, server will be listening on *:9987
• chroot sftp will be stored in /sftp/chroot/1 (users, data, etc.)
• there will be using jailkit tool. This is important to build proper structure dirs and binaries
• separate RedHat start/stop/status script for sftp service 
• 2 x sftp users assigned into jail (testuser1,testuser2)

Jailkit installation is out of scope this article. More details how to install jailkit you can find on below webpage:

http://olivier.sessink.nl/jailkit/index.html#download

I recommend to build RPM. You will have always possibility to uninstall jailkit software if you don't like it.

Now I will instruct you how to build proper directory structure:

for user in testuser1 testuser2;do useradd $user;echo "$user:test1234"| chpasswd;done

jk_init -v -j /sftp/chroot/1 sftp scp

Create directory /sftp/chroot/1/lib
Creating symlink /sftp/chroot/1/lib/libnss_dns.so.2 to libnss_dns-2.5.so
Copying /lib/libnss_dns-2.5.so to /sftp/chroot/1/lib/libnss_dns-2.5.so
Creating symlink /sftp/chroot/1/lib/libresolv.so.2 to libresolv-2.5.so

.

..

...

Creating device /sftp/chroot/1/dev/urandom
Creating device /sftp/chroot/1/dev/null
Create directory /sftp/chroot/1/usr/bin
Copying /usr/bin/scp to /sftp/chroot/1/usr/bin/scp
 

jk_init -v -j /sftp/chroot/1 jk_lsh/sftp/chroot/1/lib/libnsl.so.1 already exists, will not touch it
/sftp/chroot/1/lib64/libnsl.so.1 already exists, will not touch it
.

..

...
Copying /etc/jailkit/jk_lsh.ini to /sftp/chroot/1/etc/jailkit/jk_lsh.ini
writing user root to /sftp/chroot/1/etc/passwd
writing group root to /sftp/chroot/1/etc/group

jk_jailuser -m -j /sftp/chroot/1 testuser1

(no output it's ok)

jk_jailuser -m -j /sftp/chroot/1 testuser2

(no output it's ok)

Attention: We must create mentioned users before starting with jail dir creation!
 

Now time for verification:

cd /sftp/chroot/1pwd
/sftp/chroot/1
ls
dev  etc  home    lib  lib64  usr
cd etc
lsgroup  host.conf  hosts  jailkit  ld.so.cache  ld.so.conf  localtime  nsswitch.conf  passwd  protocols    resolv.conf  services cat passwd
cat passwdroot:x:0:0:root:/root:/bin/bash
testuser1:x:500:501::/home/testuser1:/usr/sbin/jk_lsh
testuser2:x:501:502::/home/testuser2:/usr/sbin/jk_lsh

It looks very good. We have users we have proper directory structure - sounds good.

We have been configured sftp-chroot service and make some tests. Let's play now.

+Create file "/etc/sysconfig/sftp-chroot" with following input:

OPTIONS="-f /etc/ssh/sftp-chroot_config"

+Copy /etc/ssh/sshd_config into /etc/ssh/sftp-chroot_config and change some parameters:

cp /etc/ssh/sshd_config /etc/ssh/sftp-chroot_config 
cat /etc/ssh/sftp-chroot_config

+Change port listening from 22 to 9987:

from:
#Port 22
to:
Port 9987

+Disable direct sftp access for root!

from:

PermitRootLogin yes 
to:
PermitRootLogin no

+Change pidfile for separate sftp proccess:

Uncomment PidFile and modify values:

from:
#PidFile /var/run/sshd.pid 
to:
PidFile /var/run/sftpchroot.pid

+Create program symbolic link for separate sftp service:

cd /usr/sbin

ln -s sshd sftp-chroot

Prepare start/stop/status script ()
cd /etc/init.d
cp sshd sftp-chroot 

You can find configuration files + script on the external link: SFTP-CHROOT

+PAM configuration:
cd /etc/pam.d
cp sshd sftp-chroot
 
You must take care about jk_lsh.ini config! This is most important for jail functionality:

cat /sftp/chroot/1/etc/jailkit/jk_lsh.ini
[testuser1]
paths= /usr/bin, /usr/lib/
executables= /usr/bin/scp, /usr/libexec/openssh/sftp-server

[testuser2]
paths= /usr/bin, /usr/lib/
executables= /usr/bin/scp, /usr/libexec/openssh/sftp-server

We have to check if our sftp service is running well. Let's to perform some sanity checks:

Service is not running:
service sftp-chroot statussftp-chroot is stopped

we must start it:
service sftp-chroot startStarting sftp-chroot:

checking status:
service sftp-chroot statussftp-chroot (pid  20238) is running...

netstat -na | grep [9]987 
tcp        0      0 0.0.0.0:9987                0.0.0.0:*                   LISTEN     
tcp        0      0 :::9987                         :::*                            LISTEN


ps -ef | grep [s]ftp-chroot
root     20238     1  0 12:20 ?        00:00:00 /usr/sbin/sftp-chroot -f /etc/ssh/sftp-chroot_config

All looks very good. Service is up and running processes wait for operation. We must be sure if we can connect to the sftp server also.

sftp -o "Port 9987" testuser1@
testuser1 password:
Connected to node01.
sftp> pwd
Remote working directory: /home/testuser1
sftp> ls
sftp> cd ..
sftp> ls
testuser1  testuser2 
sftp> cd ..
sftp> ls
dev    etc    home   lib    lib64  usr   
sftp> cd ..
sftp> ls
dev    etc    home   lib    lib64  usr 


Well we have our chrooted sftp.

To sum up you should pay special attention for location files and scripts in above chroot constellation. This is little bit tricky. You can tune your jail. How to do it you can find on jailkit home page. I hope that this article will be very useful for newbie Linux Administrators.

Migration EL6 to EL7

I would like to share with all of you my experience in migration RedHat Enterprise Linux 6 to 7. For a quite long time this feature was not available in RedHat products. Finally they reached this milestone. Now we have opportunity to switch between major releases versions. It's a little pity to now take advantage. At this moment RedHat is supporting only 64 bit intel platform. Let's focus on practice and technical points of this topic.

Before you start please perform full backup of your system. You shouldn't start any actions without backup. You must think about backout plan and estimate recovery time.

The first step should consist of apply all available patches. This is not mandatory but official RedHat recommendation. Firstly you must be sure that you have active RH subscription and software entitlements:

subscription-manager list

You should see your system on the drop list in Customer Portal (https://rhn.redhat.com/rhn/systems/SystemEntitlements.do) if you know what I mean ;-)

Let's go with system update and get rid of knowing bugs.

yum -y update

shutdown -rf now

After previous steps you should have fresh RHEL 6.6

We need to take care about migration tools which will be necessary to run pre-migration, migration and post-migration phases. Let's to do it:

rhn-channel --add --channel rhel-x86_64-server-extra-6

rhn-channel --add --channel rhel-x86_64-server-optional-6

Above command will inform yum to use additional repo. Migration software are located in separate RHN repository.

Now is good moment for installation:

yum -y install preupgrade-assistant preupgrade-assistant-ui preupgrade-assistant-contents

Next step should be installation and configuration HTTP server which will be useful to display pre-upgrade status.

Do the following commands:

yum install -y httpd

cd /etc/httpd/conf.d

ls
99-preup-httpd.conf  99-preup-httpd.conf.public  README  welcome.conf  wsgi.conf

cp 99-preup-httpd.conf 99-preup-httpd.conf.priv

cp 99-preup-httpd.conf.public 99-preup-httpd.conf
cp: overwrite `99-preup-httpd.conf'? y

service httpd start

Attention: please remember about selinux booleans and rules. I'm not using selinux on lab environment but you should at last for production systems. It concern iptables also. I disabled iptables and ip6tables to make life easier.

Hint: EL6 to EL7 migration doesn't support separate partition application space exacly /usr partition. Please pay special attention for the preupg report.

Before you start typing you should disable authentication for xml-rpc web service. How to do it? You can find the answer below:

Choose your prefer web browser and put url: "http://:8099

After that you should see RH random web page. Please find blue button "Disable Authentication" and click once. Now you have no authentication so you should send your report to the web service without issues.

Time for pre-upgrade output:

preupg -u http://:8099/submit/
Preupg tool doesn't do the actual upgrade.
Please ensure you have backed up your system and/or data in the event of a failed upgrade
 that would require a full re-install of the system from installation media.
Do you want to continue? y/n <<< "Press y"
y
Gathering logs used by preupgrade assistant:
All installed packages : 01/10 ...finished (time 00:00s)
All changed files      : 02/10 ...running

Be patient this process can take a while.

There is a link at the end of outcome. You can use it to check the result of above command. You should resolve all issues which have been reported before. You can not move further. It may cause damage your system.

This is not the end of preparation. I know this is tedious process. Without preparation your migration will be doomed. You have to be prepared for every circumstance.

Stop bullshit and let's back to work.

We will use ISO image method. We need installation media on the local file system. It can be situated everywhere. There is only 1 limit - your imagination ;-)

Before you start you must deactivate rhn yum plugin: in /etc/yum/pluginconf.d/rhnplugin.conf

from:
enabled = 1

to: 

enabled = 0 

There only one final human move left to start main migration process - running migration tool:

redhat-upgrade-tool --iso /var/isos/rhel-server-7.1-x86_64-dvd.iso

After that system will ask you for reboot. Next booting won't be normal os start. There will start main migration procedure. Migration process can take a while it depends how many packages are installed in your system. Migration procedure include some phase: replace packages, remove packages, install new packages, copy configuration files, merge configuration etc. If you want know more details I refer you to the RedHat documentation. Finally system should start automatically.

Apr 22 14:40:45 redhat upgrade[1369]: openscap = 1.0.8-1.el6_5.1 is needed by (installed) openscap-engine-sce-1.0.8-1.el6_5.1.x86_64
Apr 22 14:40:45 redhat upgrade[1369]: starting upgrade...
Apr 22 14:40:46 redhat upgrade[1369]: preparing RPM transaction, one moment...
Apr 22 14:40:46 redhat upgrade[1369]: [3/824] (6%) installing libgcc-4.8.3-9.el7...
Apr 22 14:40:47 redhat upgrade[1369]: [4/824] (6%) installing fontpackages-filesystem-1.44-8.el7...
Apr 22 14:40:47 redhat upgrade[1369]: [5/824] (6%) installing redhat-release-server-7.1-1.el7...
.
..
...
Apr 22 15:01:31 redhat upgrade[1369]: [610/610] (99%) cleaning libgcc-4.4.7-11.el6...
Apr 22 15:01:32 redhat upgrade[1369]: running %posttrans script for filesystem-3.2-18.el7
Apr 22 15:01:32 redhat upgrade[1369]: running %posttrans script for p11-kit-trust-0.20.7-3.el7
.
..
...
Apr 22 15:01:51 redhat dracut[15573]: fs-lib
Apr 22 15:01:51 redhat dracut[15576]: shutdown
Apr 22 15:01:51 redhat dracut[15579]: ========================================================================
Apr 22 15:01:51 redhat dracut[15582]: drwxr-xr-x  12 root     root            0 Apr 22 17:01 .
Apr 22 15:01:51 redhat dracut[15585]: crw-r--r--   1 root     root       5,   1 Apr 22 17:01 dev/console
Apr 22 15:01:51 redhat dracut[15588]: crw-r--r--   1 root     root       1,  11 Apr 22 17:01 dev/kmsg
.
..
...
Apr 22 15:02:47 redhat dracut[2672]: ========================================================================
Apr 22 15:02:49 redhat upgrade[1369]: running %posttrans script for hicolor-icon-theme-0.12-7.el7
Apr 22 15:02:49 redhat upgrade[1369]: upgrade finished.

In my case it was not so optimistic. I faced with one issue after migration. At first I can't access to system through ssh. One of symbolic link has been broken. I was really confused. To save the word and protect before chief shouting type the following command:

ln -s /usr/lib64/libsasl2.so /usr/lib64/libsasl2.so.2

 

Here some info about my issue:

After typing yum command I got following output:

There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   libsasl2.so.2: cannot open shared object file: No such file or directory

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.5 (default, Feb 11 2014, 07:46:25)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-13)]

If you cannot solve this problem yourself, please go to
the yum faq at:
  http://yum.baseurl.org/wiki/Faq

How we can make sure that our system has been successfully migrated? There are some methods. At the first we have to check release version:

lsb_release -irc
Distributor ID:    RedHatEnterpriseServer
Release:    7.1
Codename:    Maipo

or:
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.1 (Maipo)

If errors occurred during migration should be reported in  /var/log/upgrade.log . This is good pint to start investigation proccess 

To sum up before your starting make sure that you have valid backout plan and proper recovery procedure. Please do not the reboot machine when final migration procedure is running.