Subject: vservers and policy routing - stopping access to other vserver's local IPs
From: "Phil Stoneman" <pstoneman@gmail.com>
Date: Thu, 6 Nov 2008 20:18:15 +0000

Hi guys,

We use vserver2 quite extensively at work, and generally love it for
all sorts of reasons. We're now in a situation where we'd like to have
different vservers on the same host presented on different vlans. This
is quite easy to do; we just make the vserver's interfaces 'nodev',
and put something like this in /etc/network/interfaces:

auto eth1.108
        iface eth1.108 inet static
        address 192.168.8.100
        netmask 255.255.255.0
        vlan_raw_device eth1

So far so good. This doesn't let the vservers get out to the Internet
via the correct default gateway, as the system only has one routing
table. So, we use policy routing via iproute2:

auto eth1.108
        iface eth1.108 inet static
        address 192.168.8.100
        netmask 255.255.255.0
        vlan_raw_device eth1
        post-up ip route add 192.168.8.0/24 dev eth1.108 table guesttable
        post-up ip route add 0/0 src 192.168.8.100 via 192.168.8.1
table guesttable
        post-up route del -net 192.168.8.0 netmask 255.255.255.0 dev eth1.108
        post-up ip rule add from 192.168.8.0/24 table guesttable prio 10

This works fine. The vservers can get out via the correct gateway, and
it's all smooth. The IP rules look like this:

0:      from all lookup 255
10:     from 192.168.8.0/24 lookup guesttable
11:     from 192.168.9.0/24 lookup guest2table
32766:  from all lookup main
32767:  from all lookup default

So as you can see, anything from the guest is sent via the alternate
routing table. The trouble is that all packets are subject to the
first rule and passed through table 255 for local addresses first.

In a nutshell, any of the guest vservers can contact any other guest
vserver directly, even if we have them on a different vlan.

Does anyone know of a way around this with vservers?

Thanks

Phil