This morning, the IDC told us that one of our server has been observed making at least 4 outgoing IRC connection attempts. This is not normal behavior. Later, I found the root cause of this IRC traffic issue.

The operation notes are as below:
-------------------------------------------------------------------------------
[root@server ~]# ps aux
...
username 26650 0.0 0.0 2068 764 ? Ss Jul20 0:21 crond
...
This progress was belong to username, and the name was crond, it was very suspicious.

[root@server ~]# last | grep username
[root@server ~]#
I checked the login history, but there was no record about the user username.

[root@server home]# ll | grep username
drwx------ 3 username username 4096 Jul 20 05:07 username
The home directory of username was modified at Jul 20 05:07.

[root@server username]# ll
total 0
But there was no normal files under it.

[root@server username]# ll -a
...
drwx------ 3 username username 4096 Aug 6 06:36 .ssh
...
Then check the hidden files, the directory .ssh was modified at 6:36 today.

[root@server .ssh]# ll -a
...
drwxr-xr-x 3 username username 4096 Jul 20 06:00 www
There was a suspicious directory under it.

There were a lot of suspicious files in this directory.
[root@server .ssh]# cd www
[root@server www]# ll
total 840
-rwxr-xr-x 1 username username 255 May 5 2010 1
-rw-r--r-- 1 username username 120 Jul 20 05:08 1.1.1.1.user
-rwxr-xr-x 1 username username 321 May 23 2009 autorun
-rwxr-xr-x 1 username username 612470 Jan 29 2010 crond
-rw-r--r-- 1 username username 54 Jul 20 05:08 cron.d
-rwxr-xr-x 1 username username 447 Feb 24 2010 go
-rwxr-xr-x 1 username username 5119 Apr 13 2010 inst
-rwxr-xr-x 1 username username 21 Jul 20 05:08 mech.dir
-rwxr-xr-x 1 username username 169692 Jul 20 05:08 pico
drwxr-xr-x 2 username username 4096 Jan 29 2010 randfiles
-rw-r--r-- 1 username username 1043 Aug 6 06:00 raw.levels
-rw------- 1 username username 6 Jul 20 05:08 raw.pid
-rw-r--r-- 1 username username 867 Aug 6 06:00 raw.session
-rw-r--r-- 1 username username 1091 Jul 20 05:08 raw.set
-rwxr-xr-x 1 username username 52 Feb 24 2010 run
-rwxr-xr-x 1 username username 194 Jul 20 05:08 update
-rwxr-xr-x 1 username username 8 Feb 24 2010 vhosts

[root@server www]# lsof crond
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
crond 26650 username txt REG 8,5 612470 67207186 crond
The file crond in this directory was the one which was running.

[root@server www]# crontab -u username -l
* * * * * /home/username/.ssh/www/update >/dev/null 2>&1
And it used the crontab jobs to update.

[root@server www]# grep 208.83.20.130 -rl .
./raw.session
./inst
./raw.set
The IP address 208.83.20.130 was included in some files.

There were a lot of domains and addresses in those files.
[root@server www]# cat raw.session
…
server Budapest.Hu.Eu.Undernet.org 7000
server Budapest.Hu.Eu.Undernet.org 6667
server Tampa.FL.US.Undernet.org 6667
server lidingo.se.eu.undernet.org 6667
…
server 194.109.20.90 6662
server 69.16.172.34 6660
server 69.16.172.34 6662
server 208.83.20.130 6667
server us.undernet.org 6667
…

There was no doubt about it, the user username was hacked.

Let’s kill the fake progress and clean them.
[root@server .ssh]# kill -9 26650
[root@server .ssh]# cat /dev/null > /var/spool/cron/username
[root@server .ssh]# rm -rf www