Subject: Re: [vserver] Avoiding kernel internal routing among vserver clients
From: Thomas Weber <l_vserver@mail2news.4t2.com>
Date: Mon, 06 Aug 2007 21:06:18 +0200

Anyone?

If noone's interested in fixing this and networking will remain this
way, i'll probably have to switch to openvz or something else. I'd like
to avoid this because I'm happily running vserver for quite some time on
a couple of boxes.

  Tom

Am Dienstag, den 31.07.2007, 02:56 +0200 schrieb Thomas Weber:
> Hello all,
> 
> this seems to be a tricky configuration I haven't found information
> about (it's also difficult to make up a good Subject for this).
> 
> Anyway, the Problem:
> H1 H2 are Hosts,
> VS1 VS2 VSx are vservers on these hosts.
> 
> The Setup is like this:
> 
> NetA       VS2            VS4             VS6 
>                   eth0           eth0
> -GW--------------- H1 ----------- H2 -----------
>                   eth1           eth1 
> NetB       VS1            VS3             VS5
> 
> 
> The GW is the regular connection between NetA and NetB.
> Vservers are hosted on either H1 or H2 (Hx if you like).
> 
> I'd like to have all traffic from vservers in NetA to NetB pass the GW. 
> The Problem is that Hx routes the traffic between it's vservers
> internally (if VS1 and VS2 are hosted on H1, traffic between VS1/2 never
> hits the GW).
> 
> Right now I've partially solved the problem by using the netfilter ROUTE
> target (
> http://www.netfilter.org/documentation/HOWTO//netfilter-extensions-HOWTO-4.html#ss4.5)
> by 'putting' packets from the lo Interface onto the ethx Interface where
> I want them. So far so good. On the receiving side i 'get' them from the
> ethx interface and 'put; them back onto the lo interface again, so that
> the kernel can answer them.
> 
> So this is my test setup:
> host: 
> eth0 192.168.1.52/24, gw .254
> eth1 192.168.2.52/24, gw .254
> vs test1:
> eth0 192.168.1.152/24
> vs test2:
> eth1 192.168.2.152/24
> 
> ~# iptables -t mangle -L -v -n
> Chain PREROUTING (policy ACCEPT 18206 packets, 1551K bytes)
>  pkts bytes target     prot opt in     out     source               destination  
      
>  5991  503K ROUTE      0    --  eth0   *       192.168.2.152        192.168.1.0/24
     ROUTE oif:lo 
> [...]
> Chain POSTROUTING (policy ACCEPT 10881 packets, 1452K bytes)
>  pkts bytes target     prot opt in     out     source               destination  
      
>  6759  568K ROUTE      0    --  *      lo      192.168.2.152        192.168.1.0/24
     ROUTE oif:eth1 
> 
> this is only to test pinging test1 from test2 right now.
> tcpdump on the GW shows me that the echo requests passing the firewall.
> I can also see these echo request approaching the host on eth0.
> tcpdump -i eth0 -n icmp:
> 02:47:14.752500 IP 192.168.2.152 > 192.168.1.152: ICMP echo request, id 28948, seq
251, length 64
> 
> BUT tcpdump -i lo -n icmp gives me:
> 02:43:30.752294 IP 192.168.2.152 > 192.168.1.152: ICMP echo request, id 28948, seq
27, length 64
> 02:43:30.752358 IP 192.168.1.152 > 192.168.1.52: ICMP echo reply, id 28948, seq 27,
length 64
>                                            ^^^^^
> which is the wrong address to reply to. Is this because of the vserver
> kernel patches or 'normal' Linux behaviour?
> 
> Does any of you have an idea how to solve the initial Problem? If there
> is a better way then the iptables ROUTE target i'd be more than happy to
> try this.
> 
> The Versions:
> debian etch
> util-vserver 0.30.212-1
> self compiled linux-2.6.21.6-vs2.2.0.3 + netfilter ROUTE
> 
> tia,
>   Tom