Subject: Re: [vserver] Re: multiple routing tables and IPv6 with vservers
From: Bruno Prémont <bonbons@linux-vserver.org>
Date: Mon, 28 Feb 2011 22:00:40 +0100

 Mon, 28 Feb 2011 22:00:40 +0100
On Mon, 28 February 2011 Tim Spriggs <tims@uahirise.org> wrote:
> On Sat, Feb 26, 2011 at 8:12 AM, Bruno Prémont
> <bonbons@linux-vserver.org> wrote:
> > On Thu, 24 February 2011 Tim Spriggs <tims@uahirise.org> wrote:
> >
> >> On Thursday, February 24, 2011, Jeff Jansen <jeff.jansen@kkoncepts.net> wrote:
> >> > On Friday 25,February,2011 08:39 AM, Tim Spriggs wrote:
> >> >> This script was written for my environment so it assumes the use of
> >> >> "vlan" interfaces of the form:
> >> >>
> >> >> vlan2/vlan3/...vlanN
> >> >>
> >> >> instead of the other forms.
> >> >
> >> > If someone would not mind helping me with my ignorance :-) what's the
> >> > advantages of using vlans in your vserver networking?  I've always used
> >> > "novlandev" (which is the default now, I think) in networking and simply
> >> > created the addresses as secondaries on the interface.
> >> >
> >> > We use tagged vlans on our switches, so I understand the concept.  But
> >> > I'm having trouble seeing what I gain from putting every vserver on its
> >> > own vlan.  I think I'm missing something conceptually.
> >> >
> >> > For folks who are using vlans in their vserver networking, I'd
> >> > appreciate knowing why you choose to go that route and how you feel it
> >> > improves your setup as opposed to simply creating secondary addresses on
> >> > the interfaces.
> >> >
> >> > TIA
> >> >
> >> > Jeff Jansen
> >> >
> >>
> >> I am new to multiple routing tables so please step in and correct me
> >> if anyone sees something wrong.
> >>
> >> We have several networks with different security levels and we want to
> >> put server on machines that are on multiple networks. For the sake of
> >> simplicity I will explain what I am solving in a 2-vlan scenario while
> >> the machines we have setup are still a bit more complex than this.
> >>
> >> On vlan1 we are running some network (let's just say 192.168.1.0/24)
> >> and on vlan2 we are running another network (similarly
> >> 192.168.2.0/24). The way we have our network topology configured,
> >> hosts on vlan2 can contact hosts on vlan1 and hosts on vlan1 are
> >> effectively DMZ hosts that can provide services to the world. So,
> >> traffic can be initiated from vlan2->vlan1 but not from vlan1->vlan2.
> >>
> >> Let's imagine we have some large computer that will host many services
> >> on both vlans but each service itself needs to be on only one network.
> >> Again for simplicity, imagine vserver1 on vlan1 (192.168.1.10) and
> >> vserver2 on vlan2 (192.168.2.10). With the normal ip stack and routing
> >> setup vserver1 will be able to send packets onto vlan2 because as the
> >> packet is formed and sent into the network stack, the Linux kernel
> >> will say, "hey, I have a route directly to that network from this
> >> other interface. Let's just use it." This creates several problems:
> >>
> >> 1) traffic goes where you don't want it.
> >> 2) when hosts in vlan2 attempt to connect to vserver1, the server will
> >> respond with it's usual ACK but the ACK will go out of the wrong
> >> interface... as will all other traffic even if this is OK on your
> >> network
> >> 2.1) This is not ok on a network where the boundary between vlan1 and
> >> vlan2 is a stateful firewall because the fw never sees the ACK
> >> 2.2) This is bad anyway because it's not what you should expect
> >>
> >> Now for the part I haven't tested yet. Communication from
> >> vserver1->vserver2 (dmz->internal) should not work without contacting
> >> the firewall first. While this is not optimal in terms of bandwidth,
> >> it is needed for our security model.
> >>
> >> Enter multiple routing tables and ip rules (thanks again for the
> >> pointer Bertl.) With ip rules you can specify that traffic originating
> >> from vlan1 (192.168.1.0/24) should use a routing table specially
> >> tailored for delivering packets correctly out of the vlan1
> >> interface... or without the "knowledge" of the vlan2 network. This is
> >> where I get lazy in my script; I assume that vlanN corresponds to the
> >> routing table id=N. Now we can state with confidence that traffic
> >> coming from vserver1 (which *is* a secondary IP on the vlan1
> >> interface) will only "know" about the vlan1 network and the vlan1
> >> default gateway.
> >>
> >> --
> >>
> >> Next problem: IPv6.
> >>
> >> We have deployed IPv6 on our networks and as such we need this
> >> configuration to work with IPv6 if it is going to work at all. The
> >> problem and significant difference between IPv4 and IPv6 in our
> >> scenario is that we use SLAAC to configure the default gateway for all
> >> of our hosts. This means we can not hard-code the default gateway and
> >> then just add that to our secondary routing tables. The second bump is
> >> that when our hosts are coming up and the interfaces are being
> >> plumbed, they may not know what the default router on the network is
> >> (or even their own network prefix for that matter) so the script needs
> >> to wait for that information to become available and then create the
> >> appropriate ip rule and ip route entries.
> >>
> >> My solution was to fork and literally wait (via polling) for
> >> information to appear on the device in question. Once it does then the
> >> fork continues and propagates the routes into the secondary routing
> >> tables.
> >>
> >> I hope this clears things up a little but if not I can try to make
> >> them more muddy by explaining more ;)
> >
> > Beware that with IPv6 you might need to trigger rerouting of your packets
> > in order to make sure they go out on the right interface.
> > To do this, e.g. set a marker with iptables for traffic from your guest
> > (this way netfilter will cause kernel to recalculate routing for your
> > packets). The reason is that for IPv6, in opposition to IPv4 kernel first
> > tries to determine routing and only in a second time selects source
> > address, thus it may select a route that looks better for your target but
> > for which the source address is not available to your guest. Then when
> > determining your source address it will for an address from your guest but
> > maybe route through the wrong interface (under IPv6 IP addresses are
> > system wide, e.g. only very loosely bound to interfaces).
> >
> > Because of your security consideration it looks like you might be better
> > of attempting to combine linux-vserver and mainline net namespaces. (no
> > idea if it is possible or how hard it is [to implement]) At least it would
> > provide the cleanest subnet separation.
> >
> > Bruno
> 
> Hi Bruno,
> 
>   Thanks for this pointer. It may be necessary to implement...
> 
>   As it turns out, even with the config mentioned above, vserver1 can
> connect to vserver2 if it is on the same machine. I'm not sure if
> there is a way to turn the short-circuiting of IPs off in this
> configuration but it would be nice if it exists. :)

Maybe you could do something with iptables + marks + routing tables selected
by marks, tough I have never tried.
On the other hand, going the net-ns way would definitely be much more clean,
easy to understand, separation and less error-prone!

Bruno

> -Tim