I have a RH7.1 machine that will not allow me to login from the console (keyboard attched to the actual machine). You type the user name at the login prompt, hit enter and the login prompt comes back blank (no password prompt). No user name works. Not even root. Has anyone ever heard of this? The machine has been operational for only 2 months. Thanks.

Strange things can happen if /var overfills. This can especially happen if the machine is also a firewall that is logging denied packets.

If you can't get in at all (check ssh, telnet etc), and ctrl-alt-delete doesn't initiate a graceful shutdown,then a hard reboot might be the only solution. Use a boot floppy and check your partitions.

The other possibility is that you've been 0wn3d by some 5cr1pt k1dd13, and your /bin/login has been trojaned. I hope not.

Has anyone ever heard of this?

Yes, sort of. 'Twas a case of a full root partition. X/GUI login bombed, but could still telnet to it, so was able to delete a bunch of hidden "trashcan" files and fix the problem. Can you (telnet/ssh)?

[edit: Dang! Aced out with a better answer! Oh, well, post count ++]

I can SSH in. I have rebooted prior to posting this thread. Still unable to login from the console.

If I've been 0wn3d it seems harmless as there is no network activity and the machine is over 99% idle. So, I am pretty sure this is not the case. I am going to end up re-doing the server this weekend (oh, did I mention it is a live web/email server). Maybe I will check out RH7.2. Thanks.

You could download a copy of chkrootkit from http://www.chkrootkit.org. It will check your system for signs of well-known rootkits...

Marc

Oh [censored]

ROOTDIR is `/'

Checking `amd'... not found

Checking `basename'... not infected

Checking `biff'... not found

Checking `chfn'... not infected

Checking `chsh'... not infected

Checking `cron'... not infected

Checking `date'... not infected

Checking `du'... not infected

Checking `dirname'... not infected

Checking `echo'... not infected

Checking `egrep'... not infected

Checking `env'... not infected

Checking `find'... not infected

Checking `fingerd'... not infected

Checking `gpm'... not infected

Checking `grep'... not infected

Checking `hdparm'... not infected

Checking `su'... not infected

Checking `ifconfig'... not infected

Checking `inetd'... not infected

Checking `inetdconf'... not found

Checking `identd'... not infected

Checking `killall'... not infected

Checking `ldsopreload'... can't exec ./strings-static, not tested

Checking `login'... not infected

Checking `ls'... not infected

Checking `lsof'... not infected

Checking `mail'... not infected

Checking `mingetty'... not infected

Checking `netstat'... not infected

Checking `named'... not infected

Checking `passwd'... not infected

Checking `pidof'... not infected

Checking `pop2'... not found

Checking `pop3'... not found

Checking `ps'... not infected

Checking `pstree'... not infected

Checking `rpcinfo'... not infected

Checking `rlogind'... not infected

Checking `rshd'... not infected

Checking `slogin'... not infected

Checking `sendmail'... not infected

Checking `sshd'... not infected

Checking `syslogd'... not infected

Checking `tar'... not infected

Checking `tcpd'... not infected

Checking `top'... not infected

Checking `telnetd'... not infected

Checking `timed'... not found

Checking `traceroute'... not infected

Checking `write'... not infected

Checking `aliens'...

/dev/ttyop /dev/ttyoa /dev/tux/.addr /dev/tux/.proc /dev/tux/tools/mirkforce/realnames

Searching for sniffer's logs, it may take a while... nothing found

Searching for HiDrootkit's default dir... nothing found

Searching for t0rn's default files and dirs... nothing found

Searching for t0rn's v8 defaults... nothing found

Searching for Lion Worm default files and dirs... nothing found

Searching for RSHA's default files and dir... nothing found

Searching for RH-Sharpe's default files... Possible RH-Sharpe's rootkit installed

Searching for Ambient's rootkit (ark) default files and dirs... nothing found

Searching for suspicious files and dirs, it may take a while...

/usr/lib/perl5/5.6.0/i386-linux/.packlist /usr/lib/perl5/5.6.0/i386-linux/auto/CGI/.packlist /usr/lib/perl5/site_perl/5.6.0/i386-linux/auto/Digest/MD5/.packlist /usr/lib/perl5/site_perl/5.6.0/i386-linux/auto/Image/Magick/.packlist /usr/lib/perl5/site_perl/5.6.0/i386-linux/auto/DBD/Pg/.packlist



Searching for LPD Worm files and dirs... nothing found

Searching for Ramen Worm files and dirs... nothing found

Searching for Maniac files and dirs... nothing found

Searching for RK17 files and dirs... nothing found

Searching for Ducoci rootkit... nothing found

Searching for Adore Worm... nothing found

Searching for ShitC Worm... nothing found

Searching for Omega Worm... nothing found

Searching for Sadmind/IIS Worm... nothing found

Searching for MonKit... nothing found

Searching for anomalies in shell history files... nothing found

Checking `asp'... not infected

Checking `bindshell'... INFECTED (PORTS:  465)

Checking `lkm'... not tested: can't exec ./chkproc

Checking `rexedcs'... not found

Checking `sniffer'... not tested: can't exec ./ifpromisc

Checking `wted'... not tested: can't exec ./chkwtmp

Checking `z2'... not tested: can't exec ./chklastlog

Oh man that sucks. I had feared the worst after you had posted that ssh in had worked - a /var or / overfill usually kills that too.

Time to burn a backup /etc, /home, /var/www, /var/spool in their entirety and reinstall from scratch.

Inspect any old datafiles that you are going to migrate into your new install - especially anything from /etc

Looking at the script, it appears to trigger if anything is listening on port 465/tcp. /etc/services lists that as "SMTP over SSL". Are you sure it's not just detecting your mailserver?

Isn't there some tool which, given a port, will tell you which local PID is listening on it?

Peter

netstat -p

Are you sure you did compile the script first? As it seems to be missing several binaries (all these "can't exec ./balahblah, not tested")

You need to do a "make sense" in chkroot's dir (Its explained in the README).
With these binaries available it will probably give some more hints, if there is really a rootkit installed.

Plus, are you running something like portsentry? Because this might also cause this
bindshell alert...

At this point I am building another machine to switch the website and email to while I sort out what to do about this.

After recompiling chkrootkit:

ROOTDIR is `/'

Checking `amd'... not found

Checking `basename'... not infected

Checking `biff'... not found

Checking `chfn'... not infected

Checking `chsh'... not infected

Checking `cron'... not infected

Checking `date'... not infected

Checking `du'... not infected

Checking `dirname'... not infected

Checking `echo'... not infected

Checking `egrep'... not infected

Checking `env'... not infected

Checking `find'... not infected

Checking `fingerd'... not infected

Checking `gpm'... not infected

Checking `grep'... not infected

Checking `hdparm'... not infected

Checking `su'... not infected

Checking `ifconfig'... not infected

Checking `inetd'... not infected

Checking `inetdconf'... not found

Checking `identd'... not infected

Checking `killall'... not infected

Checking `ldsopreload'... not infected

Checking `login'... not infected

Checking `ls'... not infected

Checking `lsof'... not infected

Checking `mail'... not infected

Checking `mingetty'... not infected

Checking `netstat'... not infected

Checking `named'... not infected

Checking `passwd'... not infected

Checking `pidof'... not infected

Checking `pop2'... not found

Checking `pop3'... not found

Checking `ps'... not infected

Checking `pstree'... not infected

Checking `rpcinfo'... not infected

Checking `rlogind'... not infected

Checking `rshd'... not infected

Checking `slogin'... not infected

Checking `sendmail'... not infected

Checking `sshd'... not infected

Checking `syslogd'... not infected

Checking `tar'... not infected

Checking `tcpd'... not infected

Checking `top'... not infected

Checking `telnetd'... not infected

Checking `timed'... not found

Checking `traceroute'... not infected

Checking `write'... not infected

Checking `aliens'...

/dev/ttyop /dev/ttyoa /dev/tux/.addr /dev/tux/.proc /dev/tux/tools/mirkforce/realnames

Searching for sniffer's logs, it may take a while... nothing found

Searching for HiDrootkit's default dir... nothing found

Searching for t0rn's default files and dirs... nothing found

Searching for t0rn's v8 defaults... nothing found

Searching for Lion Worm default files and dirs... nothing found

Searching for RSHA's default files and dir... nothing found

Searching for RH-Sharpe's default files... Possible RH-Sharpe's rootkit installed

Searching for Ambient's rootkit (ark) default files and dirs... nothing found

Searching for suspicious files and dirs, it may take a while...

/usr/lib/perl5/5.6.0/i386-linux/.packlist /usr/lib/perl5/5.6.0/i386-linux/auto/CGI/.packlist /usr/lib/perl5/site_perl/5.6.0/i386-linux/auto/Digest/MD5/.packlist /usr/lib/perl5/site_perl/5.6.0/i386-linux/auto/Image/Magick/.packlist /usr/lib/perl5/site_perl/5.6.0/i386-linux/auto/DBD/Pg/.packlist



Searching for LPD Worm files and dirs... nothing found

Searching for Ramen Worm files and dirs... nothing found

Searching for Maniac files and dirs... nothing found

Searching for RK17 files and dirs... nothing found

Searching for Ducoci rootkit... nothing found

Searching for Adore Worm... nothing found

Searching for ShitC Worm... nothing found

Searching for Omega Worm... nothing found

Searching for Sadmind/IIS Worm... nothing found

Searching for MonKit... nothing found

Searching for anomalies in shell history files... nothing found

Checking `asp'... not infected

Checking `bindshell'... INFECTED (PORTS:  465)

Checking `lkm'... You have     6 process hidden for ps command

Warning: Possible LKM Trojan installed

Checking `rexedcs'... not found

Checking `sniffer'...

eth0 is PROMISC

Checking `wted'... nothing deleted

Checking `z2'...

nothing deleted

And 'netstat -p':

Active UNIX domain sockets (w/o servers)

Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path

unix  3      [ ]         STREAM     CONNECTED     1374   646/syslogd         /dev/log

unix  3      [ ]         STREAM     CONNECTED     1373   996/xfs

unix  3      [ ]         STREAM     CONNECTED     1337   646/syslogd         /dev/log

unix  3      [ ]         STREAM     CONNECTED     1336   960/crond

unix  3      [ ]         STREAM     CONNECTED     1269   646/syslogd         /dev/log

unix  3      [ ]         STREAM     CONNECTED     1268   921/sendmail: accep

unix  3      [ ]         STREAM     CONNECTED     1181   646/syslogd         /dev/log

unix  3      [ ]         STREAM     CONNECTED     1180   869/xinetd

unix  3      [ ]         STREAM     CONNECTED     1132   646/syslogd         /dev/log

unix  3      [ ]         STREAM     CONNECTED     1131   844/sshd

unix  3      [ ]         STREAM     CONNECTED     1062   646/syslogd         /dev/log

unix  3      [ ]         STREAM     CONNECTED     1061   810/automount

unix  3      [ ]         STREAM     CONNECTED     1018   646/syslogd         /dev/log

unix  3      [ ]         STREAM     CONNECTED     1017   761/apmd

unix  3      [ ]         STREAM     CONNECTED     880    646/syslogd         /dev/log

unix  3      [ ]         STREAM     CONNECTED     879    677/rpc.statd

unix  2      [ ]         DGRAM                    840    650/klogd

Oh, yeah, netstat by default only lists Unix-domain sockets.

You actually need netstat -a -t -u -p

-t and -u include TCP and UDP ports
-a includes listening ports (otherwise it just lists connected ports)
-p shows the pid/command line of the program on that port.

# netstat -a -t -u -p

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State

PID/Program name

tcp        0      0 *:1984                  *:*                     LISTEN

828/sendmail: accep

tcp        0      0 *:32768                 *:*                     LISTEN

677/rpc.statd

tcp        0      0 *:13666                 *:*                     LISTEN

1259/LCDd

tcp        0      0 *:686                   *:*                     LISTEN

3051/rpc.dracd

tcp        0      0 *:pop3                  *:*                     LISTEN

869/xinetd

tcp        0      0 *:sunrpc                *:*                     LISTEN

662/portmap

tcp        0      0 *:http                  *:*                     LISTEN

1138/httpd

tcp        0      0 *:smtps                 *:*                     LISTEN

831/atd

tcp        0      0 *:ftp                   *:*                     LISTEN

869/xinetd

tcp        0      0 *:ssh                   *:*                     LISTEN

844/sshd

tcp        0      0 *:smtp                  *:*                     LISTEN

921/sendmail: accep

tcp        0      0 *:https                 *:*                     LISTEN

1138/httpd

tcp        0      0 tslight.com:ssh         216.179.112.3:43716     ESTABLISHED

6468/sshd

tcp        0      0 tslight.com:pop3        216.179.112.2:65063     TIME_WAIT

-

tcp        0      0 tslight.com:pop3        216.179.112.2:65062     TIME_WAIT

-

tcp       96      0 localhost.localdo:32797 localhost.localdo:13666 ESTABLISHED

1263/lcdproc

tcp        0      0 localhost.localdo:13666 localhost.localdo:32797 ESTABLISHED

1259/LCDd

tcp        0      0 tslight.com:ssh         216.179.112.2:65070     TIME_WAIT

-

tcp        0      0 tslight.com:ssh         216.179.112.3:43756     ESTABLISHED

7601/sshd

tcp        0      0 tslight.com:pop3        216.179.112.2:65069     TIME_WAIT

-

udp        0      0 *:32768                 *:*

677/rpc.statd

udp        0      0 *:684                   *:*

3051/rpc.dracd

udp        0      0 *:853                   *:*

677/rpc.statd

udp        0      0 *:sunrpc                *:*

662/portmap

[root@tslight robricc]# resize

COLUMNS=107;

LINES=24;

export COLUMNS LINES;

[root@tslight robricc]# netstat -a -t -u -p

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

tcp        0      0 *:1984                  *:*                     LISTEN      828/sendmail: accep

tcp        0      0 *:32768                 *:*                     LISTEN      677/rpc.statd

tcp        0      0 *:13666                 *:*                     LISTEN      1259/LCDd

tcp        0      0 *:686                   *:*                     LISTEN      3051/rpc.dracd

tcp        0      0 *:pop3                  *:*                     LISTEN      869/xinetd

tcp        0      0 *:sunrpc                *:*                     LISTEN      662/portmap

tcp        0      0 *:http                  *:*                     LISTEN      1138/httpd

tcp        0      0 *:smtps                 *:*                     LISTEN      831/atd

tcp        0      0 *:ftp                   *:*                     LISTEN      869/xinetd

tcp        0      0 *:ssh                   *:*                     LISTEN      844/sshd

tcp        0      0 *:smtp                  *:*                     LISTEN      921/sendmail: accep

tcp        0      0 *:https                 *:*                     LISTEN      1138/httpd

tcp        0      0 tslight.com:ssh         216.179.112.3:43716     ESTABLISHED 6468/sshd

tcp        0      0 tslight.com:pop3        216.179.112.2:65063     TIME_WAIT   -

tcp        0      0 tslight.com:pop3        216.179.112.2:65062     TIME_WAIT   -

tcp        0    183 localhost.localdo:32797 localhost.localdo:13666 ESTABLISHED 1263/lcdproc

tcp       40      0 localhost.localdo:13666 localhost.localdo:32797 ESTABLISHED 1259/LCDd

tcp        0      0 tslight.com:ssh         216.179.112.2:65070     TIME_WAIT   -

tcp        0      0 tslight.com:ssh         216.179.112.3:43756     ESTABLISHED 7601/sshd

tcp        0      0 tslight.com:pop3        216.179.112.2:65069     TIME_WAIT   -

udp        0      0 *:32768                 *:*                                 677/rpc.statd

udp        0      0 *:684                   *:*                                 3051/rpc.dracd

udp        0      0 *:853                   *:*                                 677/rpc.statd

udp        0      0 *:sunrpc                *:*                                 662/portmap

tcp 0 0 *:686 *:* LISTEN 3051/rpc.dracd

Hmm. Don't remember this one.

And I can't think of any reason that atd should be listening on the network.

What also seems suspicious to me is that port 465 which has a shell bound to it (according to chkrootkit), doesn't show up as listening in netstat's output...

Seems like netstat (probably other essential binaries, too) has been tampered with.

Marc

edit: You could of course try telnetting to this machine's port 465 and see if you get a shell or a login prompt or something.

Hmm. Don't remember this one.

Is it this? Somebody setting up a mail relay?

(back to following along...)

Yeah, that's a good call. Especially as 'grep smtps /etc/services' gives me a blank. (Although I'm running Mandrake not RH)

The machine is a mail relay. I use DRAC for pop before smtp authentication (ie. must check a pop account before you can relay through the server).

For reference I have 'netstat -a -t -u -p' output for the server I just built. It has all the necessary software that the "infected" one had except gallery and LCDProc.

# netstat -a -t -u -p

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

tcp        0      0 *:1024                  *:*                     LISTEN      640/rpc.statd

tcp        0      0 *:931                   *:*                     LISTEN      1600/rpc.dracd

tcp        0      0 *:npmp-gui              *:*                     LISTEN      1280/rpc.dracd

tcp        0      0 *:pop3                  *:*                     LISTEN      1628/xinetd

tcp        0      0 *:imap                  *:*                     LISTEN      1628/xinetd

tcp        0      0 *:sunrpc                *:*                     LISTEN      625/portmap

tcp        0      0 *:http                  *:*                     LISTEN      889/httpd

tcp        0      0 *:ftp                   *:*                     LISTEN      1628/xinetd

tcp        0      0 *:ssh                   *:*                     LISTEN      800/sshd

tcp        0      0 *:smtp                  *:*                     LISTEN      1318/sendmail: acce

tcp        0      0 *:https                 *:*                     LISTEN      889/httpd

tcp        0      0 tslight.com:ftp         216.179.112.2:61780     ESTABLISHED 1631/ftpd: 216.179.

tcp        0      0 tslight.com:ssh         216.179.112.2:61715     ESTABLISHED 1331/sshd

udp        0      0 *:1024                  *:*                                 640/rpc.statd

udp        0      0 *:929                   *:*                                 1600/rpc.dracd

udp        0      0 *:816                   *:*                                 640/rpc.statd

udp        0      0 *:609                   *:*                                 1280/rpc.dracd

udp        0      0 *:sunrpc                *:*                                 625/portmap

The machine is a mail relay.

D'oh. If Jim would actually RTFM, he might figure out that it would be a bad choice for an exploit (or so they say).

Unless you really need it, I suggest that you get rid of the portmap and rpc.whatever services.

For reference, my webserver:

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   

tcp        0      0 peculiar.differentp:ssh ellided.ellided.c:61464 ESTABLISHED 15547/sshd          

tcp        0      0 *:www                   *:*                     LISTEN      19519/apache        

tcp        0      0 *:ssh                   *:*                     LISTEN      15739/sshd          

tcp        0      0 *:pop3                  *:*                     LISTEN      14287/tcpserver     

tcp        0      0 *:smtp                  *:*                     LISTEN      14282/tcpserver     

tcp        0      0 *:ftp                   *:*                     LISTEN      197/proftpd (accept 

tcp        0      0 *:auth                  *:*                     LISTEN      154/inetd           

tcp        0      0 peculiar.differe:domain *:*                     LISTEN      132/named           

tcp        0      0 localhost:domain        *:*                     LISTEN      132/named           

udp        0      0 *:1024                  *:*                                 132/named           

udp        0      0 peculiar.differe:domain *:*                                 132/named           

udp        0      0 localhost:domain        *:*                                 132/named

And I can't think of any reason that atd should be listening on the network.

Yes, especially on smpts. Robricc, you have definitely been own3d.

Checking `lkm'... You have 6 process hidden for ps command

That's a bit of a giveaway too.

Peter

When it rains it pours. Our CSU/DSU's alarm light is on and we have no access all day. Should be fixed tomorrow. Unreal.

At least no one can further exploit that machine.

Unless it was an inside job....

Just to inject a little humor into the whole haX0r thing, I thought of this when the message about "being 0w3nd" was posted:

http://www.detonate.net/matrix/

It seems our connection is back up. If anyone would like me to post logs somewhere so they can peruse them, I will. Just name what you would like to see. I think the more people that look at what happened would do me good. I, and others, can possibly learn a lot.

If anyone is interested, the infected machine is photo-documented in the link below. In contrast, the backup server I built is a K6-200 with 64MB of RAM and a 10GB IDE hard drive. It is still running RH 7.1 since I know all the gotcha's (pop3 daemon in the IMAP package. Yeah, that makes sense).

http://spmicro.com/gallery/tslightserver