Archive for June, 2011

The Crontab issue - "All crontab jobs were replaced with blank".

Today, I got the phonecall from a colleague, he said a very serious problem happened on server - "All the crontab jobs were disappeared".

Then I asked him whether we have a backup, he said maybe not. It's very emergency!

So I logged in the server, and checked all related logs. At last, I found the cause of the problem and recovered the crontab jobs.

 

The cause:

If we typed the command 'crontab' in SSH then it was aborted by some reason ( terrible networking, application exception …… ), all crontab jobs will be replaced with blank by operating system.

It sounds crazy , but when I did the test on my VMworkstation, it really happened!

Someone faced this situation on Server today.

 

How did I recover:

Fortunately, I found a backup in a directory.

 

Reflection:

First, we should backup the crontab on all servers ASAP.

Second, we should backup the crontab every time before we edit it.

 

Some operation notes:

----------------------------------------------------------------------------

[root@server~]# last
……
user2 pts/2        111.112.113.114   Sat Jun 25 09:02 - 09:18  (00:15)
user1 pts/1        111.112.113.115      Sat Jun 25 09:00 - 09:10  (00:10)
user2 pts/0        111.112.113.114   Sat Jun 25 08:57 - 09:18  (00:20)
……

[root@server cron]# ll
……
-rw------- 1 root  root   0 Jun 25 09:18 root
……

[root@server ~]# less /var/log/secure
……
Jun 25 09:18:04 server crontab[3609]: (root) REPLACE (root)
Jun 25 09:19:01 server crond[16791]: (root) RELOAD (cron/root)
……

[root@server ~]# less /home/user2/.bash_history
……
exit
su
su
exit
exit
……

[root@server ~]# less /root/.bash_history
……
crontab
crontab -l
……

[root@server user2 ]# ll
……
-rw-r--r-- 1 root     root         10628 Jun 15 13:59 crontab.bak
……

----------------------------------------------------------------------------

5 Comments

The SSH issue [command-line: line 0]

Hi Eman,

The scp command is not work on server, can you please help us to fix that?

Here is the error message:

[root@server directory]# scp user@serverdomain:/directory/file .
command-line: line 0: Bad configuration option: PermitLocalCommand

Thanks.

-------------------------------------------------------------------------------------------
Hi Buddy,

I fixed this issue, the steps are as follows:

After I checked all the profiles and binary files of SSH, I found the issue is not about the profiles, is the binary files /usr/bin/ssh and /usr/sbin/sshd.

Maybe someone changed them, and gave them some special rights, such as can't be removed and deleted.
So I got rid of all the special rights of them, reinstalled all the SSH programs. Then it was OK.

After that, I checked some logs, tried to find some information if we were hacked by someone.

But I'm sorry I couldn't find any useful information. Now I think we'd better to change all the passwords of existing users, especially the users who can login this server.

[root@server ssh]# lsattr /usr/bin/ssh
-u--ia------- /usr/bin/ssh
[root@server ssh]# lsattr /usr/sbin/sshd
-u--ia------- /usr/sbin/sshd

[root@server ssh]# chattr -uia /usr/bin/ssh
[root@server ssh]# chattr -uia /usr/sbin/sshd

[root@server ssh]# lsattr /usr/bin/ssh
------------- /usr/bin/ssh
[root@server ssh]# lsattr /usr/sbin/sshd
------------- /usr/sbin/sshd

yum install openssh-server
yum install openssh-clients
yum install openssh

[root@server ssh]# rpm -qa | grep ssh
openssh-server-4.3p2-72.el5_6.3
openssh-clients-4.3p2-72.el5_6.3
openssh-4.3p2-72.el5_6.3

-------------------------------------------------------------------------------------------
Hi Eman,

Thanks a lot.

2 Comments

Fork me on GitHub