Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 1 of 2 1 2 >
Topic Options
#313599 - 04/09/2008 15:08 A botnet is attacking my server
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Beginning on August 29 2008, I've noticed a new attack strategy on the SSH server on my Linux gateway box.

Until now, I've just left SSH open, but had firewall rules to automatically blacklist any host that tries to connect too often within a short interval. This seems to have been working acceptably until now.

But a week ago, a botnet began attacking.. slowly.. a new connection attempt every few seconds, but from a different IP address each time. Up to a limit, after which the IP addresses roll around again, outside of the blacklist rules that I had set.

Cute.

Anyway, the firewall has now been adjusted to deny drop SSH by default, and I'm moving to a different strategy for remote access there. Long overdue, I suppose.

Cheers

Top
#313600 - 04/09/2008 15:12 Re: A botnet is attacking my server [Re: mlord]
canuckInOR
carpal tunnel

Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
Just use a different port?

Top
#313601 - 04/09/2008 15:26 Re: A botnet is attacking my server [Re: canuckInOR]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Fancier than that. No further comment.

-ml

Top
#313602 - 04/09/2008 15:52 Re: A botnet is attacking my server [Re: mlord]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
Yeah. Unfortunately this is the way they've gone now. The attackers realised that lots of people have a process that will autoblock any IP that does more than a certain number of invalid attempts.

In the end I decided to just disable password entry via SSH and insist on a key only. It will block an IP after the first attempt if it tries to use a password.

Top
#313603 - 04/09/2008 16:03 Re: A botnet is attacking my server [Re: tman]
tonyc
carpal tunnel

Registered: 27/06/1999
Posts: 7058
Loc: Pittsburgh, PA
Quote:
It will block an IP after the first attempt if it tries to use a password.

How does one go about setting this up? I have iptables rules to throttle / block attempts, but I also have password authentication turned off and would like to ban any IPs that try it.
_________________________
- Tony C
my empeg stuff

Top
#313604 - 04/09/2008 16:17 Re: A botnet is attacking my server [Re: tonyc]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
Originally Posted By: tonyc
Quote:
It will block an IP after the first attempt if it tries to use a password.

How does one go about setting this up? I have iptables rules to throttle / block attempts, but I also have password authentication turned off and would like to ban any IPs that try it.

DenyHosts

Top
#313605 - 04/09/2008 16:26 Re: A botnet is attacking my server [Re: tman]
Daria
carpal tunnel

Registered: 24/01/2002
Posts: 3937
Loc: Providence, RI
Also apparently fail2ban.

Top
#313606 - 04/09/2008 16:29 Re: A botnet is attacking my server [Re: tman]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Originally Posted By: tman

Mmm.. oddly enough, that tool is only a second-level filter, after the firewall.

Looks useful, though.

Cheers

Top
#313607 - 04/09/2008 16:32 Re: A botnet is attacking my server [Re: mlord]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
Originally Posted By: mlord
Originally Posted By: tman

Mmm.. oddly enough, that tool is only a second-level filter, after the firewall.

Yeah. It is after the firewall. It will add entries to hosts.deny

dbrashear's suggestion of fail2ban will add an extra firewall rule though.

Top
#313608 - 04/09/2008 16:49 Re: A botnet is attacking my server [Re: tman]
DWallach
carpal tunnel

Registered: 30/04/2000
Posts: 3810
Alternately, if you want to have fun, you could always shunt them to some kind of honeypot. Or Zork.

Top
#313623 - 04/09/2008 19:04 Re: A botnet is attacking my server [Re: DWallach]
sn00p
addict

Registered: 24/07/2002
Posts: 618
Loc: South London
Originally Posted By: DWallach
Alternately, if you want to have fun, you could always shunt them to some kind of honeypot. Or Zork.


My vote would be for:

Code:
Greetings professor falken.

Would you like to play a game?



Top
#313638 - 05/09/2008 00:06 Re: A botnet is attacking my server [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Well, the attack stopped at 16:36 local time -- I guess the botnet recognizes when there's no response on port 22 (ssh).

Curiously though, at 17:27, the firewall rejected a small flurry of twenty or so simultaneous ICMP TYPE=8 packets (from a botnet), and then some ACK-FIN attacks on the SMTP server.

Since then, things have been mostly quiet, with just the normal single-host attempts on port 22 (ssh).

Such fun!

Top
#313640 - 05/09/2008 00:12 Re: A botnet is attacking my server [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Originally Posted By: mlord
Since then, things have been mostly quiet, with just the normal single-host attempts on port 22 (ssh).

Oh, and more or less continuous incoming spam attempts all evening on the SMTP server -- the spamhaus blocklist seems to have rejected just about all of those connections, though.

Top
#313658 - 05/09/2008 13:58 Re: A botnet is attacking my server [Re: mlord]
drakino
carpal tunnel

Registered: 08/06/1999
Posts: 7868
Looks like someone is taking advantage of their large botnet to try these attacks. Been hearing from 2 other sources of some pretty heavy attacks lately.

Top
#313937 - 10/09/2008 22:08 Re: A botnet is attacking my server [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Mmm.. botnet attacking again today, about 3-4 times a minute, from about 07:58am onward. I think I'll now just stop logging the failed SSH attempts, to keep the log file from being rotated too often.

Cheers

Top
#316878 - 02/12/2008 21:38 Re: A botnet is attacking my server [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Mmm.. this type of attack finally made it onto SlashDot and other news sites today.

Top
#316879 - 03/12/2008 01:44 Re: A botnet is attacking my server [Re: mlord]
DWallach
carpal tunnel

Registered: 30/04/2000
Posts: 3810
I checked my logs, and sure enough, they're full of these sorts of probes against my ssh daemon. For now, I set it to disable password authentication, which should defeat any of these password-guessing attacks.

Top
#316888 - 03/12/2008 13:01 Re: A botnet is attacking my server [Re: DWallach]
LittleBlueThing
addict

Registered: 11/01/2002
Posts: 612
Loc: Reading, UK
great source of user ids though wink
_________________________
LittleBlueThing Running twin 30's

Top
#316891 - 03/12/2008 13:03 Re: A botnet is attacking my server [Re: LittleBlueThing]
LittleBlueThing
addict

Registered: 11/01/2002
Posts: 612
Loc: Reading, UK
I've never got round to port knocking my sshd - I guess it might be time...
_________________________
LittleBlueThing Running twin 30's

Top
#316894 - 03/12/2008 13:55 Re: A botnet is attacking my server [Re: LittleBlueThing]
DWallach
carpal tunnel

Registered: 30/04/2000
Posts: 3810
Amusement: after I set sshd to disallow password authentication, I got this:

Quote:
Dec 3 09:50:07 bunsen-honeydew sshd[54753]: Invalid user admin from 219.237.242.171
Dec 3 09:50:09 bunsen-honeydew sshd[54755]: Invalid user test from 219.237.242.171
Dec 3 09:50:11 bunsen-honeydew sshd[54757]: Invalid user guest from 219.237.242.171
Dec 3 09:50:13 bunsen-honeydew sshd[54759]: Invalid user webmaster from 219.237.242.171
Dec 3 09:50:17 bunsen-honeydew sshd[54763]: Invalid user oracle from 219.237.242.171
Dec 3 09:50:19 bunsen-honeydew sshd[54765]: Invalid user library from 219.237.242.171
Dec 3 09:50:22 bunsen-honeydew sshd[54767]: Invalid user info from 219.237.242.171
Dec 3 09:50:24 bunsen-honeydew sshd[54769]: Invalid user shell from 219.237.242.171
Dec 3 09:50:26 bunsen-honeydew sshd[54771]: Invalid user linux from 219.237.242.171
Dec 3 09:50:28 bunsen-honeydew sshd[54773]: Invalid user unix from 219.237.242.171
Dec 3 09:50:30 bunsen-honeydew sshd[54775]: Invalid user webadmin from 219.237.242.171
Dec 3 09:50:32 bunsen-honeydew sshd[54782]: Invalid user ftp from 219.237.242.171
Dec 3 09:50:34 bunsen-honeydew sshd[54784]: Invalid user test from 219.237.242.171
Dec 3 09:50:38 bunsen-honeydew sshd[54788]: Invalid user admin from 219.237.242.171
Dec 3 09:50:40 bunsen-honeydew sshd[54790]: Invalid user guest from 219.237.242.171
Dec 3 09:50:42 bunsen-honeydew sshd[54792]: Invalid user master from 219.237.242.171
Dec 3 09:50:44 bunsen-honeydew sshd[54794]: Invalid user apache from 219.237.242.171
Dec 3 09:51:00 bunsen-honeydew sshd[54810]: Invalid user admin from 219.237.242.171
Dec 3 09:51:02 bunsen-honeydew sshd[54812]: Invalid user admin from 219.237.242.171
Dec 3 09:51:04 bunsen-honeydew sshd[54814]: Invalid user admin from 219.237.242.171
Dec 3 09:51:06 bunsen-honeydew sshd[54816]: Invalid user admin from 219.237.242.171
Dec 3 09:51:12 bunsen-honeydew sshd[54822]: Invalid user test from 219.237.242.171
Dec 3 09:51:14 bunsen-honeydew sshd[54824]: Invalid user test from 219.237.242.171
Dec 3 09:51:16 bunsen-honeydew sshd[54826]: Invalid user webmaster from 219.237.242.171
Dec 3 09:51:18 bunsen-honeydew sshd[54828]: Invalid user turbo from 219.237.242.171
Dec 3 09:51:20 bunsen-honeydew sshd[54830]: Invalid user cvs from 219.237.242.171
Dec 3 09:51:22 bunsen-honeydew sshd[54832]: Invalid user ram from 219.237.242.171
Dec 3 09:51:24 bunsen-honeydew sshd[54834]: Invalid user eric from 219.237.242.171
Dec 3 09:51:26 bunsen-honeydew sshd[54836]: Invalid user wu from 219.237.242.171
Dec 3 09:51:28 bunsen-honeydew sshd[54838]: Invalid user jesica from 219.237.242.171
Dec 3 09:51:30 bunsen-honeydew sshd[54840]: Invalid user jessica from 219.237.242.171
Dec 3 09:51:32 bunsen-honeydew sshd[54842]: Invalid user hsiao from 219.237.242.171
Dec 3 09:51:34 bunsen-honeydew sshd[54844]: Invalid user chen from 219.237.242.171
Dec 3 09:51:36 bunsen-honeydew sshd[54846]: Invalid user sam from 219.237.242.171
Dec 3 09:51:38 bunsen-honeydew sshd[54848]: Invalid user chang from 219.237.242.171
Dec 3 09:51:40 bunsen-honeydew sshd[54850]: Invalid user alan from 219.237.242.171
Dec 3 09:51:42 bunsen-honeydew sshd[54852]: Invalid user allan from 219.237.242.171
Dec 3 09:51:44 bunsen-honeydew sshd[54854]: Invalid user web from 219.237.242.171
Dec 3 09:51:46 bunsen-honeydew sshd[54856]: Invalid user eva from 219.237.242.171
Dec 3 09:51:48 bunsen-honeydew sshd[54858]: Invalid user adam from 219.237.242.171
Dec 3 09:51:50 bunsen-honeydew sshd[54860]: Invalid user postgres from 219.237.242.171
Dec 3 09:51:52 bunsen-honeydew sshd[54862]: Invalid user postgres from 219.237.242.171
Dec 3 09:51:54 bunsen-honeydew sshd[54864]: Invalid user sam from 219.237.242.171
Dec 3 09:51:56 bunsen-honeydew sshd[54866]: Invalid user student from 219.237.242.171
Dec 3 09:51:58 bunsen-honeydew sshd[54868]: Invalid user student from 219.237.242.171
Dec 3 09:52:00 bunsen-honeydew sshd[54870]: Invalid user student from 219.237.242.171
Dec 3 09:52:02 bunsen-honeydew sshd[54872]: Invalid user eric from 219.237.242.171
Dec 3 09:52:04 bunsen-honeydew sshd[54874]: Invalid user fax from 219.237.242.171
Dec 3 09:52:06 bunsen-honeydew sshd[54876]: Invalid user test from 219.237.242.171
Dec 3 09:52:08 bunsen-honeydew sshd[54878]: Invalid user test from 219.237.242.171
Dec 3 09:52:10 bunsen-honeydew sshd[54881]: Invalid user test from 219.237.242.171
Dec 3 09:52:12 bunsen-honeydew sshd[54883]: Invalid user test from 219.237.242.171
Dec 3 09:52:14 bunsen-honeydew sshd[54885]: Invalid user test from 219.237.242.171
Dec 3 09:52:16 bunsen-honeydew sshd[54887]: Invalid user test from 219.237.242.171
Dec 3 09:52:18 bunsen-honeydew sshd[54889]: Invalid user test from 219.237.242.171
Dec 3 09:52:18 bunsen-honeydew sshd[54891]: Invalid user isanne from 201.218.231.142
Dec 3 09:52:18 bunsen-honeydew com.apple.SecurityServer[48]: Failed to authorize right system.login.tty by client /usr/sbin/sshd for authorization created by /usr/sbin/sshd.
Dec 3 09:52:18 bunsen-honeydew sshd[54891]: error: PAM: Authentication failure for illegal user isanne from 201.218.231.142
Dec 3 09:52:18 bunsen-honeydew sshd[54891]: Failed keyboard-interactive/pam for invalid user isanne from 201.218.231.142 port 2356 ssh2
Dec 3 09:52:20 bunsen-honeydew sshd[54894]: Invalid user test from 219.237.242.171
Dec 3 09:52:22 bunsen-honeydew sshd[54896]: Invalid user test from 219.237.242.171
Dec 3 09:52:24 bunsen-honeydew sshd[54898]: Invalid user test from 219.237.242.171
Dec 3 09:52:26 bunsen-honeydew sshd[54900]: Invalid user info from 219.237.242.171
Dec 3 09:52:28 bunsen-honeydew sshd[54902]: Invalid user lsmith from 219.237.242.171
Dec 3 09:52:30 bunsen-honeydew sshd[54904]: Invalid user lsmith from 219.237.242.171
Dec 3 09:52:32 bunsen-honeydew sshd[54906]: Invalid user dennison from 219.237.242.171
Dec 3 09:52:34 bunsen-honeydew sshd[54908]: Invalid user dennison from 219.237.242.171
Dec 3 09:52:36 bunsen-honeydew sshd[54910]: Invalid user chris from 219.237.242.171
Dec 3 09:52:38 bunsen-honeydew sshd[54912]: Invalid user chriss from 219.237.242.171
Dec 3 09:52:40 bunsen-honeydew sshd[54914]: Invalid user user from 219.237.242.171
Dec 3 09:52:42 bunsen-honeydew sshd[54916]: Invalid user username from 219.237.242.171
Dec 3 09:52:44 bunsen-honeydew sshd[54918]: Invalid user username from 219.237.242.171
Dec 3 09:52:46 bunsen-honeydew sshd[54920]: Invalid user user from 219.237.242.171
Dec 3 09:52:50 bunsen-honeydew sshd[54924]: Invalid user admin from 219.237.242.171


Suddenly, the request rate jumped radically, from one every few minutes, to one every few seconds, and all from the same IP address. Well, they're nothing if not persistent. Can anybody diagnose the "Failed keyboard-interactive/pam for invalid user" line? After I set PasswordAuthentication to "no", I thought those sorts of things wouldn't happen any more.

Top
#316897 - 03/12/2008 14:25 Re: A botnet is attacking my server [Re: LittleBlueThing]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Originally Posted By: LittleBlueThing
I've never got round to port knocking my sshd - I guess it might be time...

Code:
#!/bin/bash
#################################################################################
#
# Disable external SSH by default; require a door-knocker on tcp/$KNOCK to gain access.
#
# The current time-limited ssh_knock list can be viewed at:  /proc/net/ipt_recent/ssh_knock
#
ipt=/sbin/iptables
KNOCK=1234      ## The door-knocker port: pick a nice randomish port number here
SSHLOG="-m limit --limit 30/minute --limit-burst 20 -j LOG --log-prefix"

## ssh_add:  move somebody temporarily to the ssh_knock list.
$ipt -N ssh_add
$ipt -A ssh_add -m recent --name ssh_knock --set $SSHLOG "ssh_add: "  ## add to list
$ipt -A ssh_add -j DROP  ## pretend we ignored it

## ssh_del:  remove an IP from the ssh_knock list.
$ipt -N ssh_del
$ipt -A ssh_del -m recent --name ssh_knock --rsource --remove $SSHLOG "ssh_del: "  ## remove from list
$ipt -A ssh_del -j RETURN  ## return to caller

## ssh_accept:  accept an incoming SSH connection.
$ipt -N ssh_accept
$ipt -A ssh_accept $SSHLOG "CONNECT(ssh): "
$ipt -A ssh_accept -m recent --name ssh_knock --rcheck -j ssh_del   ## remove from list
$ipt -A ssh_accept -j ACCEPT  ## allow this one connection attempt

## ssh_filter:  restrict ssh access to only those hosts on the ssh_knock list, one attempt, 15 seconds max:
$ipt -N ssh_filter
$ipt -A ssh_filter -m recent --name ssh_knock --rcheck --seconds 15 ! --hitcount 1 -j ssh_accept
$ipt -A ssh_filter -m recent --name ssh_knock --rcheck                             -j ssh_del
$ipt -A ssh_filter -j DROP  ## ignore it

$ipt -I INPUT -p tcp --syn --dport $KNOCK -j ssh_add
$ipt -I INPUT -p tcp --syn --dport 22     -j ssh_filter


Edited by mlord (04/12/2008 02:54)
Edit Reason: Fixed $ipt

Top
#316918 - 03/12/2008 17:33 Re: A botnet is attacking my server [Re: mlord]
Mojo
Unregistered


What's the problem if you have a secure password? It's probably more likely that the datacenter or house where your server is located will flood or catch on fire before a bot guesses your user and secure password.

They're just trolling for suckers with dictionary passwords. Or am I missing something?

Top
#316919 - 03/12/2008 17:50 Re: A botnet is attacking my server [Re: ]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
They are just probing but better to not give them the ability to do so at all. Port knocking makes sure that even if there is a vulnerability in SSH then you're safe.

Top
#316920 - 03/12/2008 17:54 Re: A botnet is attacking my server [Re: ]
LittleBlueThing
addict

Registered: 11/01/2002
Posts: 612
Loc: Reading, UK
You are right of course.

But they go away if they can't see you - on a low bandwidth connection this stuff can impact us.

Plus it's fun smile

Thanks for the script Mark - I wasn't aware of ipt_recent.
I'm working the elements into my firewall - it has been a while since I hacked at it and it is good to remember how it all works...
_________________________
LittleBlueThing Running twin 30's

Top
#316929 - 03/12/2008 18:49 Re: A botnet is attacking my server [Re: LittleBlueThing]
LittleBlueThing
addict

Registered: 11/01/2002
Posts: 612
Loc: Reading, UK
oh... I think

ipt=/sbin/iptables

should be

function ipt()
{
/sbin/iptables "$@"
}

or $ipt

but thanks for the impetus - I now have a knock-locked cert-only sshd for next time we go skiing and feel the need to vpn back to the webcam aimed at the cats...


Edited by LittleBlueThing (03/12/2008 18:50)
_________________________
LittleBlueThing Running twin 30's

Top
#316931 - 03/12/2008 19:20 Re: A botnet is attacking my server [Re: DWallach]
DWallach
carpal tunnel

Registered: 30/04/2000
Posts: 3810
Turns out, to disable password-based access, you need to set:

UsePAM no
PasswordAuthentication no

And, while you're at it, you might as well disable reverse DNS verification, since it's already pretty obvious that all of those hits are attempted security attacks.

UseDNS no

Top
#316941 - 03/12/2008 23:32 Re: A botnet is attacking my server [Re: tman]
Mojo
Unregistered


Originally Posted By: tman
They are just probing but better to not give them the ability to do so at all. Port knocking makes sure that even if there is a vulnerability in SSH then you're safe.


Well OK, I suppose it's useful if you're worried about a new SSH vulnerability being found.

Top
#316943 - 04/12/2008 00:07 Re: A botnet is attacking my server [Re: ]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
Originally Posted By: Mojo
Originally Posted By: tman
They are just probing but better to not give them the ability to do so at all. Port knocking makes sure that even if there is a vulnerability in SSH then you're safe.


Well OK, I suppose it's useful if you're worried about a new SSH vulnerability being found.

Exploits have been found. If nothing else, it'll save you from trawling through page after page of invalid attempts in the logs.

Top
#316945 - 04/12/2008 00:53 Re: A botnet is attacking my server [Re: ]
jimhogan
carpal tunnel

Registered: 06/10/1999
Posts: 2591
Loc: Seattle, WA, U.S.A.
Originally Posted By: Mojo
What's the problem if you have a secure password? It's probably more likely that the datacenter or house where your server is located will flood or catch on fire before a bot guesses your user and secure password.

They're just trolling for suckers with dictionary passwords. Or am I missing something?

If your log files fill up with this kind of scripted probe chaff, you'll never see the one line in that log file that you *should* pay attention to.

This is oversimplification, but allowing log files to be flooded with this kind of stuff makes life more difficult.
_________________________
Jim


'Tis the exceptional fellow who lies awake at night thinking of his successes.

Top
#316949 - 04/12/2008 02:53 Re: A botnet is attacking my server [Re: LittleBlueThing]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14491
Loc: Canada
Originally Posted By: LittleBlueThing
oh... I think

ipt=/sbin/iptables

should be

function ipt()
{
/sbin/iptables "$@"
}

or $ipt


Duh.. yeah!

I clipped it (incompletely) out of a larger version from my rc.firewall:
Code:
function ipt(){
        /sbin/iptables "$@"
        rc=$?
        if [ $rc != 0 ]; then
                echo "FAILED: /sbin/iptables $@"
        fi
}

Top
Page 1 of 2 1 2 >