Mon, 28 Feb 2011 22:00:40 +0100 On Mon, 28 February 2011 Tim Spriggs <tims@uahirise.org> wrote: > On Sat, Feb 26, 2011 at 8:12 AM, Bruno Prémont > <bonbons@linux-vserver.org> wrote: > > On Thu, 24 February 2011 Tim Spriggs <tims@uahirise.org> wrote: > > > >> On Thursday, February 24, 2011, Jeff Jansen <jeff.jansen@kkoncepts.net> wrote: > >> > On Friday 25,February,2011 08:39 AM, Tim Spriggs wrote: > >> >> This script was written for my environment so it assumes the use of > >> >> "vlan" interfaces of the form: > >> >> > >> >> vlan2/vlan3/...vlanN > >> >> > >> >> instead of the other forms. > >> > > >> > If someone would not mind helping me with my ignorance :-) what's the > >> > advantages of using vlans in your vserver networking? I've always used > >> > "novlandev" (which is the default now, I think) in networking and simply > >> > created the addresses as secondaries on the interface. > >> > > >> > We use tagged vlans on our switches, so I understand the concept. But > >> > I'm having trouble seeing what I gain from putting every vserver on its > >> > own vlan. I think I'm missing something conceptually. > >> > > >> > For folks who are using vlans in their vserver networking, I'd > >> > appreciate knowing why you choose to go that route and how you feel it > >> > improves your setup as opposed to simply creating secondary addresses on > >> > the interfaces. > >> > > >> > TIA > >> > > >> > Jeff Jansen > >> > > >> > >> I am new to multiple routing tables so please step in and correct me > >> if anyone sees something wrong. > >> > >> We have several networks with different security levels and we want to > >> put server on machines that are on multiple networks. For the sake of > >> simplicity I will explain what I am solving in a 2-vlan scenario while > >> the machines we have setup are still a bit more complex than this. > >> > >> On vlan1 we are running some network (let's just say 192.168.1.0/24) > >> and on vlan2 we are running another network (similarly > >> 192.168.2.0/24). The way we have our network topology configured, > >> hosts on vlan2 can contact hosts on vlan1 and hosts on vlan1 are > >> effectively DMZ hosts that can provide services to the world. So, > >> traffic can be initiated from vlan2->vlan1 but not from vlan1->vlan2. > >> > >> Let's imagine we have some large computer that will host many services > >> on both vlans but each service itself needs to be on only one network. > >> Again for simplicity, imagine vserver1 on vlan1 (192.168.1.10) and > >> vserver2 on vlan2 (192.168.2.10). With the normal ip stack and routing > >> setup vserver1 will be able to send packets onto vlan2 because as the > >> packet is formed and sent into the network stack, the Linux kernel > >> will say, "hey, I have a route directly to that network from this > >> other interface. Let's just use it." This creates several problems: > >> > >> 1) traffic goes where you don't want it. > >> 2) when hosts in vlan2 attempt to connect to vserver1, the server will > >> respond with it's usual ACK but the ACK will go out of the wrong > >> interface... as will all other traffic even if this is OK on your > >> network > >> 2.1) This is not ok on a network where the boundary between vlan1 and > >> vlan2 is a stateful firewall because the fw never sees the ACK > >> 2.2) This is bad anyway because it's not what you should expect > >> > >> Now for the part I haven't tested yet. Communication from > >> vserver1->vserver2 (dmz->internal) should not work without contacting > >> the firewall first. While this is not optimal in terms of bandwidth, > >> it is needed for our security model. > >> > >> Enter multiple routing tables and ip rules (thanks again for the > >> pointer Bertl.) With ip rules you can specify that traffic originating > >> from vlan1 (192.168.1.0/24) should use a routing table specially > >> tailored for delivering packets correctly out of the vlan1 > >> interface... or without the "knowledge" of the vlan2 network. This is > >> where I get lazy in my script; I assume that vlanN corresponds to the > >> routing table id=N. Now we can state with confidence that traffic > >> coming from vserver1 (which *is* a secondary IP on the vlan1 > >> interface) will only "know" about the vlan1 network and the vlan1 > >> default gateway. > >> > >> -- > >> > >> Next problem: IPv6. > >> > >> We have deployed IPv6 on our networks and as such we need this > >> configuration to work with IPv6 if it is going to work at all. The > >> problem and significant difference between IPv4 and IPv6 in our > >> scenario is that we use SLAAC to configure the default gateway for all > >> of our hosts. This means we can not hard-code the default gateway and > >> then just add that to our secondary routing tables. The second bump is > >> that when our hosts are coming up and the interfaces are being > >> plumbed, they may not know what the default router on the network is > >> (or even their own network prefix for that matter) so the script needs > >> to wait for that information to become available and then create the > >> appropriate ip rule and ip route entries. > >> > >> My solution was to fork and literally wait (via polling) for > >> information to appear on the device in question. Once it does then the > >> fork continues and propagates the routes into the secondary routing > >> tables. > >> > >> I hope this clears things up a little but if not I can try to make > >> them more muddy by explaining more ;) > > > > Beware that with IPv6 you might need to trigger rerouting of your packets > > in order to make sure they go out on the right interface. > > To do this, e.g. set a marker with iptables for traffic from your guest > > (this way netfilter will cause kernel to recalculate routing for your > > packets). The reason is that for IPv6, in opposition to IPv4 kernel first > > tries to determine routing and only in a second time selects source > > address, thus it may select a route that looks better for your target but > > for which the source address is not available to your guest. Then when > > determining your source address it will for an address from your guest but > > maybe route through the wrong interface (under IPv6 IP addresses are > > system wide, e.g. only very loosely bound to interfaces). > > > > Because of your security consideration it looks like you might be better > > of attempting to combine linux-vserver and mainline net namespaces. (no > > idea if it is possible or how hard it is [to implement]) At least it would > > provide the cleanest subnet separation. > > > > Bruno > > Hi Bruno, > > Thanks for this pointer. It may be necessary to implement... > > As it turns out, even with the config mentioned above, vserver1 can > connect to vserver2 if it is on the same machine. I'm not sure if > there is a way to turn the short-circuiting of IPs off in this > configuration but it would be nice if it exists. :) Maybe you could do something with iptables + marks + routing tables selected by marks, tough I have never tried. On the other hand, going the net-ns way would definitely be much more clean, easy to understand, separation and less error-prone! Bruno > -Tim