Subject: Avoiding kernel internal routing among vserver clients
From: Thomas Weber <l_vserver@mail2news.4t2.com>
Date: Tue, 31 Jul 2007 02:56:10 +0200

Hello all,

this seems to be a tricky configuration I haven't found information
about (it's also difficult to make up a good Subject for this).

Anyway, the Problem:
H1 H2 are Hosts,
VS1 VS2 VSx are vservers on these hosts.

The Setup is like this:

NetA       VS2            VS4             VS6 
                  eth0           eth0
-GW--------------- H1 ----------- H2 -----------
                  eth1           eth1 
NetB       VS1            VS3             VS5


The GW is the regular connection between NetA and NetB.
Vservers are hosted on either H1 or H2 (Hx if you like).

I'd like to have all traffic from vservers in NetA to NetB pass the GW. 
The Problem is that Hx routes the traffic between it's vservers
internally (if VS1 and VS2 are hosted on H1, traffic between VS1/2 never
hits the GW).

Right now I've partially solved the problem by using the netfilter ROUTE
target (
http://www.netfilter.org/documentation/HOWTO//netfilter-extensions-HOWTO-4.html#ss4.5)
by 'putting' packets from the lo Interface onto the ethx Interface where
I want them. So far so good. On the receiving side i 'get' them from the
ethx interface and 'put; them back onto the lo interface again, so that
the kernel can answer them.

So this is my test setup:
host: 
eth0 192.168.1.52/24, gw .254
eth1 192.168.2.52/24, gw .254
vs test1:
eth0 192.168.1.152/24
vs test2:
eth1 192.168.2.152/24

~# iptables -t mangle -L -v -n
Chain PREROUTING (policy ACCEPT 18206 packets, 1551K bytes)
 pkts bytes target     prot opt in     out     source               destination    
    
 5991  503K ROUTE      0    --  eth0   *       192.168.2.152        192.168.1.0/24 
    ROUTE oif:lo 
[...]
Chain POSTROUTING (policy ACCEPT 10881 packets, 1452K bytes)
 pkts bytes target     prot opt in     out     source               destination    
    
 6759  568K ROUTE      0    --  *      lo      192.168.2.152        192.168.1.0/24 
    ROUTE oif:eth1 

this is only to test pinging test1 from test2 right now.
tcpdump on the GW shows me that the echo requests passing the firewall.
I can also see these echo request approaching the host on eth0.
tcpdump -i eth0 -n icmp:
02:47:14.752500 IP 192.168.2.152 > 192.168.1.152: ICMP echo request, id 28948, seq 251,
length 64

BUT tcpdump -i lo -n icmp gives me:
02:43:30.752294 IP 192.168.2.152 > 192.168.1.152: ICMP echo request, id 28948, seq 27,
length 64
02:43:30.752358 IP 192.168.1.152 > 192.168.1.52: ICMP echo reply, id 28948, seq 27,
length 64
                                           ^^^^^
which is the wrong address to reply to. Is this because of the vserver
kernel patches or 'normal' Linux behaviour?

Does any of you have an idea how to solve the initial Problem? If there
is a better way then the iptables ROUTE target i'd be more than happy to
try this.

The Versions:
debian etch
util-vserver 0.30.212-1
self compiled linux-2.6.21.6-vs2.2.0.3 + netfilter ROUTE

tia,
  Tom