Saturday, April 25, 2015

RHEL6 my update has been unexpected aborted

Today I faced with serious issue. During update execution my server has been unexpectedly rebooted. I found root cause. It was bad disk queue timing for Qdisk cluster component. I forgot to disable cluster services during update. This is recommended action. I always remember about it but at this moment I hurry up and forgot about it. This is shame but it happens ;(

Server did not start properly and I had to use rescue CD to fix kernel. I wasted a lot of time. It was enough to start system from previous kernel - grub menu and fix yum transactions.

Let's to fix cluster node. We do not have to much time. I know all services are up and running on standby node but we have no redundancy. 

After system will bring up with previous kernel you can heck how many packages are missing or duplicted.

Before you start please create list of duplicated packages. It may be important for further moves. You can't start without it!


First step: Cancel all not completed yum transaction:
# yum-complete-transaction --cleanup-only


Second step: Cleanup system from duplicate packages:
# package-cleanup --cleandupes

Be aware at the end of checking section you can find package list which will be removed. Please save the list as separate file. It will be useful if new packages not have installed yet.

After reboot I observed that some packages are missing :( It really hurts but we must to handle it.

# mkdir /root/tmp/
# cat  packages-list-erased-`hostname`-`date +%F`

  Erasing    : ipa-client-3.0.0-37.el6.x86_64                                                                                             1/252 
  Erasing    : sssd-1.9.2-129.el6.x86_64                                                                                                    2/252
  Erasing    : libvirt-client-0.10.2-29.el6.x86_64                                                                                       3/252

  .
  ..
  ...
  Erasing    : glibc-2.12-1.132.el6.x86_64                                                                                                250/252
  Erasing    : tzdata-2013g-1.el6.noarch                                                                                                  251/252
  Erasing    : libgcc-4.4.7-4.el6.x86_64                                                                                                    252/252

  

# cat packages-list-erased-`hostname`-`date +%F` | awk '{ print $3 }' | sed 's/^[0-9]://' | \ awk -F- '{ print $1 "-" $2}' | sed 's/-[0-9].*$//' > /root/tmp/packages-list-missed- \ `hostname`-`date +%F`

# for package in `cat /root/tmp//root/tmp/packages-list-missed-`hostname`-`date \ +%F``;do yum -y install $package;done


Of course you must reboot server to take the effect and apply all changes. After reboot I have been surprised very positive. My server was up and running with all services. I checked cluster services and were also fine, only one small remark: Please check if rgmanager is up and running after reboot. I found that did not started during booting phase. 

# chkconfig --add rgmanager
# chkconfig --level=3 rgmanager on
# chkconfig --level=5 rgmanager on
# service rgmanager start  
# clustat
 
Cluster Status for RH6cluster01 @ Sat Apr 25 20:59:20 2015
Member Status: Quorate

 Member Name                                       ID   Status
 ------ ----                                                    ---- ------
 clrh6n01                                                 1 Online, rgmanager
 clrh6n02                                                 2 Online, Local, rgmanager
 /dev/VolGroupQdisk/lv_qdisk                 0 Online, Quorum Disk

 Service Name                                         Owner (Last)                                          State        
 ------- ----                                                    ----- ------                                                    -----        
 service:appRG                                        clrh6n02                                                  started      
 service:dbRG                                          clrh6n01                                                  started
Lessons and learn: do not perform update without yum lvm fs snapshoot plugin. This functionality is desirable during system update and the maintenance tasks. You can save a lot of time/money in case of update failure. You should always perform full backup system and have a shot in the locker proper back out procedure.

Friday, April 24, 2015

How to create chrooted SFTP server in EL5

I will try to show you how to install and setup sftp chroot in RHEL5. You can adopt this solution for higher RedHat Linux versions. The IT space isn't perfect. I will present different approach to provide SFTP server without active shell access. I chose EL5 for a reason. In EL6 and EL7 you have possibility to configure sftp-chroot inside ssh. In EL5 no possible to do that. Nothing more nothing less.

Let's present some details about sftp jail:
  • sftp will be running on separate port higher that 2048 my port is: 9987
  • there will be no dedicate IP for sftp, server will be listening on *:9987
  • chroot sftp will be stored in /sftp/chroot/1 (users, data, etc.)
  • there will be using jailkit tool. This is important to build proper structure dirs and binaries
  • separate RedHat start/stop/status script for sftp service 
  • 2 x sftp users assigned into jail (testuser1,testuser2)
Jailkit installation is out of scope this article. More details how to install jailkit you can find on below webpage:


I recommend to build RPM. You will have always possibility to uninstall jailkit software if you don't like it.

Now I will instruct you how to build proper directory structure:

for user in testuser1 testuser2;do useradd $user;echo "$user:test1234"| chpasswd;done
jk_init -v -j /sftp/chroot/1 sftp scp
Create directory /sftp/chroot/1/lib
Creating symlink /sftp/chroot/1/lib/libnss_dns.so.2 to libnss_dns-2.5.so
Copying /lib/libnss_dns-2.5.so to /sftp/chroot/1/lib/libnss_dns-2.5.so
Creating symlink /sftp/chroot/1/lib/libresolv.so.2 to libresolv-2.5.so

.
..
...
Creating device /sftp/chroot/1/dev/urandom
Creating device /sftp/chroot/1/dev/null
Create directory /sftp/chroot/1/usr/bin
Copying /usr/bin/scp to /sftp/chroot/1/usr/bin/scp

 
jk_init -v -j /sftp/chroot/1 jk_lsh/sftp/chroot/1/lib/libnsl.so.1 already exists, will not touch it
/sftp/chroot/1/lib64/libnsl.so.1 already exists, will not touch it
.
..
...
Copying /etc/jailkit/jk_lsh.ini to /sftp/chroot/1/etc/jailkit/jk_lsh.ini
writing user root to /sftp/chroot/1/etc/passwd
writing group root to /sftp/chroot/1/etc/group

jk_jailuser -m -j /sftp/chroot/1 testuser1
(no output it's ok)
jk_jailuser -m -j /sftp/chroot/1 testuser2
(no output it's ok)

Attention: We must create mentioned users before starting with jail dir creation!
 
Now time for verification:

cd /sftp/chroot/1pwd
/sftp/chroot/1

ls
dev  etc  home    lib  lib64  usr
cd etc
lsgroup  host.conf  hosts  jailkit  ld.so.cache  ld.so.conf  localtime  nsswitch.conf  passwd  protocols    resolv.conf  services cat passwd
cat passwdroot:x:0:0:root:/root:/bin/bash
testuser1:x:500:501::/home/testuser1:/usr/sbin/jk_lsh
testuser2:x:501:502::/home/testuser2:/usr/sbin/jk_lsh


It looks very good. We have users we have proper directory structure - sounds good.

We have been configured sftp-chroot service and make some tests. Let's play now.

+Create file "/etc/sysconfig/sftp-chroot" with following input:

OPTIONS="-f /etc/ssh/sftp-chroot_config"

+Copy /etc/ssh/sshd_config into /etc/ssh/sftp-chroot_config and change some parameters:

cp /etc/ssh/sshd_config /etc/ssh/sftp-chroot_config 
cat /etc/ssh/sftp-chroot_config

+Change port listening from 22 to 9987:

from:
#Port 22
to:
Port 9987

+Disable direct sftp access for root!

from:

PermitRootLogin yes 
to:
PermitRootLogin no

+Change pidfile for separate sftp proccess:

Uncomment PidFile and modify values:

from:
#PidFile /var/run/sshd.pid 
to:
PidFile /var/run/sftpchroot.pid

+Create program symbolic link for separate sftp service:

cd /usr/sbin
ln -s sshd sftp-chroot

Prepare start/stop/status script ()
cd /etc/init.d
cp sshd sftp-chroot 

You can find configuration files + script on the external link: SFTP-CHROOT

+PAM configuration:
cd /etc/pam.d
cp sshd sftp-chroot
 
You must take care about jk_lsh.ini config! This is most important for jail functionality:

cat /sftp/chroot/1/etc/jailkit/jk_lsh.ini
[testuser1]
paths= /usr/bin, /usr/lib/
executables= /usr/bin/scp, /usr/libexec/openssh/sftp-server

[testuser2]
paths= /usr/bin, /usr/lib/
executables= /usr/bin/scp, /usr/libexec/openssh/sftp-server


We have to check if our sftp service is running well. Let's to perform some sanity checks:

Service is not running:
service sftp-chroot statussftp-chroot is stopped

we must start it:
service sftp-chroot startStarting sftp-chroot:

checking status:
service sftp-chroot statussftp-chroot (pid  20238) is running...

netstat -na | grep [9]987 
tcp        0      0 0.0.0.0:9987                0.0.0.0:*                   LISTEN     
tcp        0      0 :::9987                         :::*                            LISTEN



ps -ef | grep [s]ftp-chroot
root     20238     1  0 12:20 ?        00:00:00 /usr/sbin/sftp-chroot -f /etc/ssh/sftp-chroot_config

All looks very good. Service is up and running processes wait for operation. We must be sure if we can connect to the sftp server also.

sftp -o "Port 9987" testuser1@
testuser1 password:
Connected to node01.
sftp> pwd
Remote working directory: /home/testuser1
sftp> ls
sftp> cd ..
sftp> ls
testuser1  testuser2 
sftp> cd ..
sftp> ls
dev    etc    home   lib    lib64  usr   
sftp> cd ..
sftp> ls
dev    etc    home   lib    lib64  usr 



Well we have our chrooted sftp.

To sum up you should pay special attention for location files and scripts in above chroot constellation. This is little bit tricky. You can tune your jail. How to do it you can find on jailkit home page. I hope that this article will be very useful for newbie Linux Administrators.

Wednesday, April 22, 2015

Migration EL6 to EL7

I would like to share with all of you my experience in migration RedHat Enterprise Linux 6 to 7. For a quite long time this feature was not available in RedHat products. Finally they reached this milestone. Now we have opportunity to switch between major releases versions. It's a little pity to now take advantage. At this moment RedHat is supporting only 64 bit intel platform. Let's focus on practice and technical points of this topic.

Before you start please perform full backup of your system. You shouldn't start any actions without backup. You must think about backout plan and estimate recovery time.

The first step should consist of apply all available patches. This is not mandatory but official RedHat recommendation. Firstly you must be sure that you have active RH subscription and software entitlements:

subscription-manager list

You should see your system on the drop list in Customer Portal (https://rhn.redhat.com/rhn/systems/SystemEntitlements.do) if you know what I mean ;-)

Let's go with system update and get rid of knowing bugs.

yum -y update
shutdown -rf now

After previous steps you should have fresh RHEL 6.6

We need to take care about migration tools which will be necessary to run pre-migration, migration and post-migration phases. Let's to do it:

rhn-channel --add --channel rhel-x86_64-server-extra-6
rhn-channel --add --channel rhel-x86_64-server-optional-6

Above command will inform yum to use additional repo. Migration software are located in separate RHN repository.

Now is good moment for installation:

yum -y install preupgrade-assistant preupgrade-assistant-ui preupgrade-assistant-contents


Next step should be installation and configuration HTTP server which will be useful to display pre-upgrade status.

Do the following commands:

yum install -y httpd
cd /etc/httpd/conf.d
ls
99-preup-httpd.conf  99-preup-httpd.conf.public  README  welcome.conf  wsgi.conf
cp 99-preup-httpd.conf 99-preup-httpd.conf.priv
cp 99-preup-httpd.conf.public 99-preup-httpd.conf
cp: overwrite `99-preup-httpd.conf'? y
service httpd start

Attention: please remember about selinux booleans and rules. I'm not using selinux on lab environment but you should at last for production systems. It concern iptables also. I disabled iptables and ip6tables to make life easier.

Hint: EL6 to EL7 migration doesn't support separate partition application space exacly /usr partition. Please pay special attention for the preupg report.

Before you start typing you should disable authentication for xml-rpc web service. How to do it? You can find the answer below:

Choose your prefer web browser and put url: "http://:8099

After that you should see RH random web page. Please find blue button "Disable Authentication" and click once. Now you have no authentication so you should send your report to the web service without issues.

Time for pre-upgrade output:

preupg -u http://:8099/submit/
Preupg tool doesn't do the actual upgrade.
Please ensure you have backed up your system and/or data in the event of a failed upgrade
 that would require a full re-install of the system from installation media.
Do you want to continue? y/n <<<
"Press y"
y
Gathering logs used by preupgrade assistant:
All installed packages : 01/10 ...finished (time 00:00s)
All changed files      : 02/10 ...running

Be patient this process can take a while.

There is a link at the end of outcome. You can use it to check the result of above command. You should resolve all issues which have been reported before. You can not move further. It may cause damage your system.

This is not the end of preparation. I know this is tedious process. Without preparation your migration will be doomed. You have to be prepared for every circumstance.

Stop bullshit and let's back to work.

We will use ISO image method. We need installation media on the local file system. It can be situated everywhere. There is only 1 limit - your imagination ;-)

Before you start you must deactivate rhn yum plugin: in /etc/yum/pluginconf.d/rhnplugin.conf

from:
enabled = 1

to: 

enabled = 0 

There only one final human move left to start main migration process - running migration tool:

redhat-upgrade-tool --iso /var/isos/rhel-server-7.1-x86_64-dvd.iso

After that system will ask you for reboot. Next booting won't be normal os start. There will start main migration procedure. Migration process can take a while it depends how many packages are installed in your system. Migration procedure include some phase: replace packages, remove packages, install new packages, copy configuration files, merge configuration etc. If you want know more details I refer you to the RedHat documentation. Finally system should start automatically.

Apr 22 14:40:45 redhat upgrade[1369]: openscap = 1.0.8-1.el6_5.1 is needed by (installed) openscap-engine-sce-1.0.8-1.el6_5.1.x86_64
Apr 22 14:40:45 redhat upgrade[1369]: starting upgrade...
Apr 22 14:40:46 redhat upgrade[1369]: preparing RPM transaction, one moment...
Apr 22 14:40:46 redhat upgrade[1369]: [3/824] (6%) installing libgcc-4.8.3-9.el7...
Apr 22 14:40:47 redhat upgrade[1369]: [4/824] (6%) installing fontpackages-filesystem-1.44-8.el7...
Apr 22 14:40:47 redhat upgrade[1369]: [5/824] (6%) installing redhat-release-server-7.1-1.el7...
.
..
...
Apr 22 15:01:31 redhat upgrade[1369]: [610/610] (99%) cleaning libgcc-4.4.7-11.el6...
Apr 22 15:01:32 redhat upgrade[1369]: running %posttrans script for filesystem-3.2-18.el7
Apr 22 15:01:32 redhat upgrade[1369]: running %posttrans script for p11-kit-trust-0.20.7-3.el7

.
..
...
Apr 22 15:01:51 redhat dracut[15573]: fs-lib
Apr 22 15:01:51 redhat dracut[15576]: shutdown
Apr 22 15:01:51 redhat dracut[15579]: ========================================================================
Apr 22 15:01:51 redhat dracut[15582]: drwxr-xr-x  12 root     root            0 Apr 22 17:01 .
Apr 22 15:01:51 redhat dracut[15585]: crw-r--r--   1 root     root       5,   1 Apr 22 17:01 dev/console
Apr 22 15:01:51 redhat dracut[15588]: crw-r--r--   1 root     root       1,  11 Apr 22 17:01 dev/kmsg
.

..
...
Apr 22 15:02:47 redhat dracut[2672]: ========================================================================
Apr 22 15:02:49 redhat upgrade[1369]: running %posttrans script for hicolor-icon-theme-0.12-7.el7
Apr 22 15:02:49 redhat upgrade[1369]: upgrade finished.


In my case it was not so optimistic. I faced with one issue after migration. At first I can't access to system through ssh. One of symbolic link has been broken. I was really confused. To save the word and protect before chief shouting type the following command:

ln -s /usr/lib64/libsasl2.so /usr/lib64/libsasl2.so.2
 
Here some info about my issue:

After typing yum command I got following output:

There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   libsasl2.so.2: cannot open shared object file: No such file or directory

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.5 (default, Feb 11 2014, 07:46:25)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-13)]

If you cannot solve this problem yourself, please go to
the yum faq at:
  http://yum.baseurl.org/wiki/Faq


How we can make sure that our system has been successfully migrated? There are some methods. At the first we have to check release version:

lsb_release -irc
Distributor ID:    RedHatEnterpriseServer
Release:    7.1
Codename:    Maipo


or:
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.1 (Maipo)

If errors occurred during migration should be reported in  /var/log/upgrade.log . This is good pint to start investigation proccess 

To sum up before your starting make sure that you have valid backout plan and proper recovery procedure. Please do not the reboot machine when final migration procedure is running.

Thursday, September 18, 2014

Running NFS server behind firewall.

All time appear new oportunities which required flexible and a large commitment. This article show you how to run NFS server behind firewall. In my scenario firewall is on the same server as NFS service.

Check if firewall is running:

service iptables status

You should see firewall status and listing of rules if any exist ;-)
Now time for add new rules into firewall configuration:

from commandline typ:
iptables -I INPUT -m state --state NEW,ESTABLISHED -p tcp -m multiport --dport 111,892,2049,32803 -j ACCEPT  

iptables -I INPUT -m state --state NEW,ESTABLISHED -p udp -m multiport --dport 111,892,2049,32769 -j ACCEPT

The last move is save rules which are now in memory into configuration file. This step must be perform if you want to save rules permanently. Please remember about it always in case of using iptables command in CMD.

service iptables save

There should also see "ensurance" that files /etc/sysconfig/iptables-config has been overwritten.Now you can check if your nfs server is available over network. You can use mount -e command.



Sunday, July 20, 2014

YUM fs-snapshoot

This functionality is very usefull. It can be used in case of if you want get back to previous os rpm stage. Let's imagine situation: you must update RedHat Enterprise server in your Company. Update has been successfully applied but application team is not satisfied from new OS packages. They foud issue and it turned out that there is a problem with some libraries. They are not compatible with currently installed application. In this case all indicates that in a short time you will must to recover the system from backup. I hope that you did that before starting OS update! If not you can be in trouble.

Now comes the useful package that will greatly facilitate our lives! Let's start:
  • installation - easy and simple one command:
    yum install yum-plugin-fs-snapshot

    Attention before you will start installation yum-plugin* (features) make sure that you have been enabled following repository:
  • post-installation steps:

    [rhel-6-server-optional-rpms]
    name = Red Hat Enterprise Linux 6 Server - Optional (RPMs)
    .
    ..
    ...
    enabled = 1
    ....
    .....
    ui_repoid_vars = releasever basearch
    At first you have to activate plugin in your  yum configuration. Till now each yum command revoke will load fs-snapshoot:

    Loaded plugins: aliases, changelog, downloadonly, fs-snapshot, kabi, product-id, refresh-packagekit, security, subscription-manager, tmprepo, verify, versionlock
    Enable plugin:

    Edit following file:
    /etc/yum/pluginconf.d/fs-snapshot.conf

    [main]
    enabled = 1
    exclude = /home /app01 /app02 /data /u01/cluster

    [lvm]
    enabled = 1
    # 'lvcreate_size_args' option must specify the snapshot LV size using -L or -l
    lvcreate_size_args = -l 25%ORIGIN


    [main] - this section is using to enable base functionality
    [lvm] - enabling lvm snapshoots

    lvcreate_size_args - determines how big compared in to the size of the original lv should be a snapshoot. Value is given in "%".


    exclude - may be usefull if you have some filesystems which should be excluded from lvm snapshoot list. Default settings treat to make snapshoot of all filesystems. In some case it can be not desirable. You can specify more than one filesystem by space delimiter.

    [root@hostname pluginconf.d]# yum update
    Loaded plugins: aliases, changelog, downloadonly, fs-snapshot, kabi, product-id, refresh-
                  : packagekit, security, subscription-manager, tmprepo, verify, versionlock
    This system is receiving updates from Red Hat Subscription Management.
    Loading support for Red Hat kernel ABI
    rhel-6-server-rpms                                                        | 3.7 kB     00:00    
    rhel-6-server-rpms/primary_db                                    |  27 MB     00:19    
    Setting up Update Process
    Resolving Dependencies
    Running transaction check
    Package e2fsprogs.x86_64 0:1.41.12-18.el6 will be updated
    Package e2fsprogs.x86_64 0:1.41.12-18.el6_5.1 will be an update
    Package e2fsprogs-libs.x86_64 0:1.41.12-18.el6 will be updated
    Package e2fsprogs-libs.x86_64 0:1.41.12-18.el6_5.1 will be an update
    Package grub.x86_64 1:0.97-83.el6 will be updated
    .
    ..
    ...

    Finished Dependency Resolution

    Dependencies Resolved

    ===========================================================
     Package                     Arch      Version                       Repository             Size
    ===========================================================
     kernel                      x86_64    2.6.32-431.23.3.el6           rhel-6-server-rpms     28 M
    Updating:
    .
    ..
    ...
    Transaction Summary
    =================================================================================================
    Install       1 Package(s)
    Upgrade      29 Package(s)

    Total download size: 145 M
    Is this ok [y/N]: y  
    Downloading Packages:
    (1/30): e2fsprogs-1.41.12-18.el6_5.1.x86_64.rpm                       | 553 kB     00:00 

    .
    ..
    ...
    (29/30): perf-2.6.32-431.23.3.el6.x86_64.rpm                              | 2.9 MB     00:02    
    (30/30): sos-2.2-47.el6_5.7.noarch.rpm                                          | 230 kB     00:00    
    -------------------------------------------------------------------------------------------------
    Total                                                            1.0 MB/s | 145 MB     02:19    
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
    fs-snapshot: WARNING: creating LVM snapshot of root LV.  If a kernel is
                          being altered /boot may need to be manually restored
                          in the event that a system rollback proves necessary.

    fs-snapshot: snapshotting / (/dev/vg_rhce01/lv_root): lv_root_yum_20140729224402
    .

    ..
    ...
      perf.x86_64 0:2.6.32-431.23.3.el6                                                             
      sos.noarch 0:2.2-47.el6_5.7                                                                   

    Complete!
  • checking if snapshoot has been successfully created:

    [root@hostname ~]# lvs -a -o name,devices
    Volume group vg_test03 is exported
    LV                                                                Devices       
    lv_root                                                         /dev/sda2(0)  
    lv_root_yum_20140729224402          /dev/sdb(0)
       

    lv_swap                                                       /dev/sda2(2178)

    We should choose 1 package from lists which was updated and remember version. After reboot we will restore system from lvm snapshoot!

    [root@hostname ~]# rpm -qa |grep nspr

    nspr-4.10.6-1.el6_5.x86_64
  • reboot server after upgrade OS
    [root@hostname ~]#
    reboot
  • restore operating system from snapshoot:
    [root@rhce01 ~]# lvconvert --merge /dev/vg_rhce01/lv_root_yum_20140729224402
      Can't merge over open origin volume
      Merging of snapshot lv_root_yum_20140729224402 will start next activation.
  • final reboot
    [root@hostname ~]# reboot
  • check if system has been restored

    [root@hostname ~]# lvs -a -o name,devices
    Volume group vg_test03 is exported
    LV                                                                Devices       
    lv_root                                                         /dev/sda2(0)    

    lv_swap                                                       /dev/sda2(2178)

    Snapshoot disappeared from lvm partition lists it mean that restore has successfully performed
  • final check:

    after upgrade:
    [root@hostname ~]# rpm -qa nspr
    nspr-4.10.6-1.el6_5.x86_64


    after restore:
    [root@hostname ~]# rpm -qa nspr
    nspr-4.10.2-1.el6_5.x86_64


    voilĂ 
    it works we returned to previous package constelation!
    Attention
    remember if update included kernel update you must choose valid kernel version via grub menu or apply changes permanently into /boot/grub/grub.conf!