Subject: Re: [vserver] vservers and policy routing - stopping access to other vserver's local IPs
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Fri, 7 Nov 2008 23:40:55 +0100

On Thu, Nov 06, 2008 at 08:18:15PM +0000, Phil Stoneman wrote:
> Hi guys,
> 
> We use vserver2 quite extensively at work, and generally love it for
> all sorts of reasons. We're now in a situation where we'd like to have
> different vservers on the same host presented on different vlans. This
> is quite easy to do; we just make the vserver's interfaces 'nodev',
> and put something like this in /etc/network/interfaces:
> 
> auto eth1.108
>         iface eth1.108 inet static
>         address 192.168.8.100
>         netmask 255.255.255.0
>         vlan_raw_device eth1
> 
> So far so good. This doesn't let the vservers get out to the Internet
> via the correct default gateway, as the system only has one routing
> table. So, we use policy routing via iproute2:
> 
> auto eth1.108
>         iface eth1.108 inet static
>         address 192.168.8.100
>         netmask 255.255.255.0
>         vlan_raw_device eth1
>         post-up ip route add 192.168.8.0/24 dev eth1.108 table guesttable
>         post-up ip route add 0/0 src 192.168.8.100 via 192.168.8.1
> table guesttable
>         post-up route del -net 192.168.8.0 netmask 255.255.255.0 dev eth1.108
>         post-up ip rule add from 192.168.8.0/24 table guesttable prio 10
> 
> This works fine. The vservers can get out via the correct gateway, and
> it's all smooth. The IP rules look like this:
> 
> 0:      from all lookup 255
> 10:     from 192.168.8.0/24 lookup guesttable
> 11:     from 192.168.9.0/24 lookup guest2table
> 32766:  from all lookup main
> 32767:  from all lookup default
> 
> So as you can see, anything from the guest is sent via the alternate
> routing table. The trouble is that all packets are subject to the
> first rule and passed through table 255 for local addresses first.
> 
> In a nutshell, any of the guest vservers can contact any other guest
> vserver directly, even if we have them on a different vlan.
> 
> Does anyone know of a way around this with vservers?

you need a lot of evil trickery to convince
Linux to send out packets for local addresses,
and it's usually not worth the efford ...

best,
Herbert

> Thanks
> 
> Phil