Subject: Re: multiple routing tables and IPv6 with vservers
From: Tim Spriggs <tims@uahirise.org>
Date: Thu, 24 Feb 2011 20:43:15 -0700

 Thu, 24 Feb 2011 20:43:15 -0700
On Thursday, February 24, 2011, Jeff Jansen <jeff.jansen@kkoncepts.net> wrote:
> On Friday 25,February,2011 08:39 AM, Tim Spriggs wrote:
>> This script was written for my environment so it assumes the use of
>> "vlan" interfaces of the form:
>>
>> vlan2/vlan3/...vlanN
>>
>> instead of the other forms.
>
> If someone would not mind helping me with my ignorance :-) what's the
> advantages of using vlans in your vserver networking?  I've always used
> "novlandev" (which is the default now, I think) in networking and simply
> created the addresses as secondaries on the interface.
>
> We use tagged vlans on our switches, so I understand the concept.  But
> I'm having trouble seeing what I gain from putting every vserver on its
> own vlan.  I think I'm missing something conceptually.
>
> For folks who are using vlans in their vserver networking, I'd
> appreciate knowing why you choose to go that route and how you feel it
> improves your setup as opposed to simply creating secondary addresses on
> the interfaces.
>
> TIA
>
> Jeff Jansen
>

I am new to multiple routing tables so please step in and correct me
if anyone sees something wrong.

We have several networks with different security levels and we want to
put server on machines that are on multiple networks. For the sake of
simplicity I will explain what I am solving in a 2-vlan scenario while
the machines we have setup are still a bit more complex than this.

On vlan1 we are running some network (let's just say 192.168.1.0/24)
and on vlan2 we are running another network (similarly
192.168.2.0/24). The way we have our network topology configured,
hosts on vlan2 can contact hosts on vlan1 and hosts on vlan1 are
effectively DMZ hosts that can provide services to the world. So,
traffic can be initiated from vlan2->vlan1 but not from vlan1->vlan2.

Let's imagine we have some large computer that will host many services
on both vlans but each service itself needs to be on only one network.
Again for simplicity, imagine vserver1 on vlan1 (192.168.1.10) and
vserver2 on vlan2 (192.168.2.10). With the normal ip stack and routing
setup vserver1 will be able to send packets onto vlan2 because as the
packet is formed and sent into the network stack, the Linux kernel
will say, "hey, I have a route directly to that network from this
other interface. Let's just use it." This creates several problems:

1) traffic goes where you don't want it.
2) when hosts in vlan2 attempt to connect to vserver1, the server will
respond with it's usual ACK but the ACK will go out of the wrong
interface... as will all other traffic even if this is OK on your
network
2.1) This is not ok on a network where the boundary between vlan1 and
vlan2 is a stateful firewall because the fw never sees the ACK
2.2) This is bad anyway because it's not what you should expect

Now for the part I haven't tested yet. Communication from
vserver1->vserver2 (dmz->internal) should not work without contacting
the firewall first. While this is not optimal in terms of bandwidth,
it is needed for our security model.

Enter multiple routing tables and ip rules (thanks again for the
pointer Bertl.) With ip rules you can specify that traffic originating
from vlan1 (192.168.1.0/24) should use a routing table specially
tailored for delivering packets correctly out of the vlan1
interface... or without the "knowledge" of the vlan2 network. This is
where I get lazy in my script; I assume that vlanN corresponds to the
routing table id=N. Now we can state with confidence that traffic
coming from vserver1 (which *is* a secondary IP on the vlan1
interface) will only "know" about the vlan1 network and the vlan1
default gateway.

--

Next problem: IPv6.

We have deployed IPv6 on our networks and as such we need this
configuration to work with IPv6 if it is going to work at all. The
problem and significant difference between IPv4 and IPv6 in our
scenario is that we use SLAAC to configure the default gateway for all
of our hosts. This means we can not hard-code the default gateway and
then just add that to our secondary routing tables. The second bump is
that when our hosts are coming up and the interfaces are being
plumbed, they may not know what the default router on the network is
(or even their own network prefix for that matter) so the script needs
to wait for that information to become available and then create the
appropriate ip rule and ip route entries.

My solution was to fork and literally wait (via polling) for
information to appear on the device in question. Once it does then the
fork continues and propagates the routes into the secondary routing
tables.

I hope this clears things up a little but if not I can try to make
them more muddy by explaining more ;)

Cheers,
-Tim