Hi,

I'm CTO of a company located in Nevada. We spread our satellites (linux-clusters) in many countries. To have a secure connection to the master system we'd like to set up secure VPNs (e.g. ipsec). First I'm happy of any hint for this (we are using debian, an apt-get solution would be great). Second but not less important I've heard that encrypted connections aren't allowed in un the USA. But what else shall I do? I don't believe that all the copmpanies there don't secure their interconnections...

Rolf (from Germany)

I have been using OpenVPN, which is available on apt. Very happy with it, it wasn't too hard for my simple mind to set up, and has been working very reliably. There is a lot of help about for it as well, which is always important. Recommend it.

Some internet providers block VPN traffic for some (usually residential) customers to force higher priced access packages to the customers who need VPN. There is no 'country wide' standard or block on VPN in the US. My company uses VPN for our remote telemarketing staff.

-Zeke

It is nearly trivial to configure a static IPSEC setup, and only somewhat more complex for dynamic "dial-in" style support. But most of the existing Linux FAQs on this are either out-of-date (pre-2.6 kernel days) or overly complex in their explanations.

But this one is really well written. Just walk through the examples given, trying them between any two machines, and you'll be an expert in no time.

Cheers

One caveat about Linux "native IPSEC" (the 2.6 kernel stuff), is that it is broken when the IPSEC endpoint is also a NAT gateway. Eg. single machine as firewall, NAT, and IPSEC gateway -- No issues with ESP, but "transport mode AH" doesn't work at all.

There are patches for this, though, which fix the problem. But who uses "transport mode AH" over the internet, anyway??

Cheers

GREAT!

I knew that I could count on you. Exactly what I hoped to hear. So now we can set up the VPNs (Cuba and Syria isn't our main destination, so there should be no problem) and I know how I can set this up due to the readme...

Thank you!

Now something about our pizzaboxes:
We want wo cover the world with small clusters which take the SIP calls. But we need a secure connection especially for the database cluster (MySQL replication) for this Asterisk cluster. And for my final degree I made VoIPonCD, an Asterisk CD you can throw into a fresh PC and this will install Debian and Asterisk and all the needed stuff. Now I want to change this a bit so we can use this for dedicated (rented) servers. I don't want to fly around the world and put some heavy servers into the racks, better just renting them and put the VoIP on it, this will happen in just 10 minutes to set up a machine in a cluster.

I thought to use Empegs for this cluster part but I definitely will not pull my loved empeg out ot my car :-). But I'm sure my Empeg would be able to do everything or nearly everything :-).

I know here are very wise guys so I'd be pleased to hear a statement again. The idea was to implement the ipsec right into the software so on a dedicated rented server I just have to scp my main script and start it at the commandline, including the setup of the ipsec tunnel.

Rolf

I'll second the vote for OpenVPN, because it's simple, and useful compared to the insanity of setting up ipsec. It also has nice features like forwarding ethernet frames if you really need that kind of transparency.

Good!

Oh, I suppose your pizza boxes will likely want/need to be using IPSec "tunnel mode". That is also described in more or less plain language at the same link I gave earlier, a couple of Next clicks further on. But don't skip pages as you read on to that point..

cheers

Now time has come to decide whether I shall use IPsec or OpenVPN. Seems like OpenVPN is much easier to handle than IPsec, but IPsec is a standard and even my router can handle this.

Can somebody give me the right direction?

Rolf

Personally, I don't see any real advantage to OpenVPN over IPsec. "Easier to configure" and "works across NAT" seem to be their list of advantages for OpenVPN, but I don't see the first as a huge issue, and you can get IPsec to work over NAT, too.

Ok, I have fixed IPs everywhere so no issues with NAT. If I could imagine how much work it might be to set up an IPsec VPN I could decide, but never did this. It's for Debian servers, especially for MySQL-Connections (replication) across the internet.

Rolf

>trivial to setup IPSEC with fixed IP addresses

Just wrote this down to a post it an put it at my monitor. Keeping your words in mind it should work. I'll try it today, yes today is the great day. Keep fingers crossed it will work as expected.

Rolf

Ok, decision is done. IPSec is the loser...

>But this one is really well written. Just walk through the examples given, trying them between any two machines, and you'll be an expert in no time.

Well, maybe you forgot that I got the first prize for the worst install of an Empeg in 2003. Also some hours of reading didn't make me feel better in IPSec (an I can tell about me that I'm not really the worst in networking). Anyway, it didn't make me happy.

Then, after maybe 4 hours of reading and brainstorming I decided to try out OpenVPN and this got me smiling just as soon as you can open the vi editor... It took me 0.32 seconds to understand OpenVPN for a bad quick and dirty but working solution.

I use debian so I don't want to comile all the stuff again:

apt-get install openvpn
openvpn --genkey --secret static.key
vi server.conf

Put this in:

dev tun
ifconfig 10.8.0.1 10.8.0.2
secret static.key

Save and copy the staic.key to the client machine.
Then, on the client machine:

apt-get install openvpn
vi client.conf

Put in:

remote server.rowi.net
dev tun
ifconfig 10.8.0.2 10.8.0.1
secret static.key

Save and:

openvpn client.conf

That's it! If the kernel has a TUN device (which almost should be), everything works and you can ping and pong from 10.8.0.1 to .2.

Just wanted to let you know. Comments welcome. Maybe I'll switch to IPSec but OpenVPN worked smarter.

Rolf

Way too complicated!

Here's how to do it with IPsec: Just run this script once after boot, on each machine:

#!/bin/sh
action="$1"
IP1=10.8.0.1
IP2=10.8.0.2
PASSKEY="1234567890123456" ## replace with something more secure
SK="/usr/sbin/setkey -c"

echo -n "Clearing ipsec configuration.. "
$SK <<-EOF
flush;
spdflush;
EOF
echo

[ "$action" = "stop" ] && exit

echo "Enabling ipsec.. "
$SK <<-EOF
add $IP1 $IP2 ah 15700 -A hmac-md5 "$PASSKEY";
add $IP2 $IP1 ah 15701 -A hmac-md5 "$PASSKEY";
spdadd $IP1 $IP2 any -P out ipsec ah/transport//require;
spdadd $IP2 $IP1 any -P out ipsec ah/transport//require;
spdadd $IP1 $IP2 any -P in ipsec ah/transport//require;
spdadd $IP2 $IP1 any -P in ipsec ah/transport//require;
EOF

Now the two machines can communcate securely, using just their regular exposed IP addresses. Appropriate firewall rules are still required to block the rest of the universe.

An improved variation on this, would be to replace [EDIT:] "ah/transport" with "esp/transport" on the above lines.. I didn't have an example of that handy here.

Ok, that's indeed impressive. Nevertheless it looks like I have to cook a new kernel due to this message:

racoon - IKE keying daemon will not be started as /proc/net/pfkey is not
available or a suitable 2.6 (or 2.4 with IPSEC backport)
kernel with af_key.[k]o module is not installed.

I use a standard Debian kernel:
Linux sip3 2.6.14.2 #5 Mon Nov 21 16:18:18 CET 2005 i686 GNU/Linux

AFAIK the IPSec _can_ be compiled into the kernel but it may be that it isn't. I want to use rented machines for this and it will be hard work to give all those machines a new kernel. Hoped it would match all 2.6 kernel but this perhaps is not true.

Nevertheless it is very very interesting here so please don't give up! I'm filling my brain with all of your postings.

Rolf

Just nuke all of that Racoon nonsense -- Racoon is only needed if you are NOT using static IP addresses everywhere. EDIT: or the warnings can be gotten rid of by doing modprobe af_key beforehand.

cheers

You can test for IPsec (for ipv4) by doing: modprobe ah4 ; modprobe esp4
If no messages appear (in shell window), then IPSec is present in the kernel.

Cheers

To elaborate on Racoon, it is a daemon to manage/validate certificates and the like for dynamic IPsec connections from random IP addresses. This is normally needed, and makes things much more complicated.

But for simple, static IP addresses, Racoon is neither needed nor used, and the whole deal gets very simple as a result.

Cheers

Here is another example, this time extending the earlier script to use "esp" instead of "ah". This allows the packets to pass through NAT and non-compliant routers and such. If that's not a concern, then an even better setup would be to combine both AH and ESP.

#!/bin/sh
action="$1"
IP1=10.0.0.53
IP2=10.0.0.14
ENCAP="esp"
if [ "$ENCAP" = "ah" ]; then
PASSKEY="1234567890123456"
CRYPT="-A hmac-md5"
else ## "esp"
PASSKEY="123456789012345678901234"
CRYPT="-E 3des-cbc"
fi
SK="/usr/sbin/setkey -c"

echo "Clearing ipsec configuration.. "
$SK <<-EOF
flush;
spdflush;
EOF

[ "$action" = "stop" ] && exit

echo "Enabling ipsec.. "
$SK <<-EOF
add 10.0.0.11 10.0.0.216 esp 15701 -E 3des-cbc "123456789012123456789012";
add $IP1 $IP2 $ENCAP 15700 $CRYPT "$PASSKEY";
add $IP2 $IP1 $ENCAP 15701 $CRYPT "$PASSKEY";
spdadd $IP1 $IP2 any -P out ipsec $ENCAP/transport//require;
spdadd $IP2 $IP1 any -P out ipsec $ENCAP/transport//require;
spdadd $IP1 $IP2 any -P in ipsec $ENCAP/transport//require;
spdadd $IP2 $IP1 any -P in ipsec $ENCAP/transport//require;
EOF

Ok, I gave it a try...

18:45:10.545802 IP ding.rowi.net > dong.rowi.net: ESP(spi=0x00003d54,seq=0x46), length 88
0x0000: 4500 006c 0007 4000 3832 b251 5558 060b [Email][email protected]..[/Email]
0x0010: 5410 e094 0000 3d54 0000 0046 6688 9675 T.....=T...Ff..u
0x0020: 4004 3205 b8c0 5a09 48c4 cd3a a791 c201 @.2...Z.H..:....
0x0030: 3f3b 63cf bfab abfd 2580 e29b e134 90a2 ?;c.....%....4..
0x0040: 3644 9b08 d5ad 21e6 aebc 570b 1721 1787 6D....!...W..!..
0x0050: 5da3 ].


Hmm, looks like there is something working. Does the modprobe work on 2.6 kernels, too? It just worked so I have one 2.4 and one 2.6 running, without any error message.

Pings to each other reaches the other host (snipplet above) but ping obviously won't be decrypted. Am I right to switch the IP addresses IP1 and IP2 in the scripts for the other host or do I have to start the scripts exactly as it is on one host at the other (without any changes)?
[EDIT: Now I edited the script at ONE server, copied it to the other server and - it seems to work. But now neither tcpdump nor iptraf see packets between these two machines, a good sign?]

What if I need one Server and 100 clients? Is this a strict P2P configuration due to the 2 IP-addresses?

Rolf

P.S.: Ok, maybe IPSec works well, too. Sounds good at this time.

Seems to work between 2 machines, but as I started the script again at the server with the configuration for the second client and started the same script at the second client they couldn't communicate between each other at all.

Due to the circumstance that this really was our live server and the client was a live server, too, I didn't know to help me but to reboot both machines. Ok, I'm still CTO of this company, but if I do this a second time I may need a new place to work...

Obviously it's not good to start the same script more than once at the server. What did I wrong? And how do I switch everything back without rebooting the whole machine?

Wow, what a day.

Rolf

Well, obviously this part (below) should only be done once on the server, as it wipes out the ipsec config each time it is run..


echo "Clearing ipsec configuration.. "
$SK <<-EOF
flush;
spdflush;
EOF

You could rearrange your script around it, to do something like this:


if [ "$action" = "stop" ]; then
echo "Clearing ipsec configuration.. "
$SK <<-EOF
flush;
spdflush;
EOF
exit
fi

And this part will also need to be adapted for more than two machines:

add $IP1 $IP2 $ENCAP 15700 $CRYPT "$PASSKEY";
add $IP2 $IP1 $ENCAP 15701 $CRYPT "$PASSKEY";

Those numbers (15700, 15701) are the "Security Parameter Index" values, and should probably be unique for each IP/IP combination/order. And I think they still have to match between the server's entries and the remote entries. Any script you use would have to take that into account as well.

Cheers

I'll try this out asap. Thank you very much, Mark.
While reading I had something in mind: Why couldn't we start a "EmpegBBS Company"? With all the knowledge of all the Empeg-people on the BBS we could even be better than any other company in the world :-).

Rolf

Heh.. But now that we know you have about 101 machines, perhaps Racoon could simplify life here. I don't know much about that, though.

Cheers