Subject: Re: [vserver] odd network problem
From: Chuck <chuck@sbbsnet.net>
Date: Sat, 22 Nov 2008 10:51:59 -0500

On Saturday 22 November 2008, Herbert Poetzl wrote:
> On Sun, Nov 16, 2008 at 08:15:40PM -0500, Chuck wrote:
> > On Thursday 13 November 2008, Herbert Poetzl wrote:
> > > On Thu, Nov 13, 2008 at 10:03:42AM -0500, Chuck wrote:
> > > > i stopped and restarted our openfire server and all was well with that
> > > > action. however, 2 web servers, one running an older centos and the
> > > > other running gentoo64 with openrc/baselayout2 both lost any network
> > > > communication. ips showed present on the host but were unusable even
> > > 		~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ that is unusual
> > > 
> > > > from the host. the ip blocks of the various guests and the host use 3
> > > > networks on the same nic.
> > > 
> > > > i restarted each of those affected servers. when they stopped each    
> > > > gave this error:                                                      
> > > 
> > > > RTNETLINK answers: Cannot assign requested address
> > > > RTNETLINK answers: Cannot assign requested address
> > > > 
> > > > it appears it gave one line for every ip assigned to the guest.
> > > 
> > > that happens if the IP is already there (maybe with
> > > the wrong netmask or so)
> > > 
> > > > restarting those 2 vservers cured the problem. these are the only two
> > > > affected out of the others living on that host.
> > > 
> > > this will have removed and re-added the IPs properly
> > > 
> > > > i dont think this has anything to do with it but the centos guest was
> > > > moved to this server by simply tarring its /vservers and /etc/vservers
> > > > entries. the openfire and gentoo guests were created and configured on
> > > > this host. no ip addresses clash.
> > > 
> > > > the host is running gentoo64 baselayout1 with
> > > 
> > > > kernel 2.6.22-vs2.3.0.34-gentoo
> > > > util-vserver 0.30.215
> > > > iproute2 version 2.6.22.20070710
> > > 
> > > > i am planning on updating the kernel to 2.6.26-vs2.3.0.35.6-gentoo and 
> > > > baselayout2/openrc but it is on my 'when i get to it' priority list.
> > > 
> > > > anyone seen this behavior of stopping a guest and it affecting
> > > > networking on other guests before? this is new behavior to me.
> > > 
> > > well, it is kind of expected, if you are using several
> > > IPs in the same network (i.e. with a netmask) and without
> > > secondary propagation, that when you remove the primary,
> > > all secondaries are gone too (that is a mainline 'feature')
> > > 
> > > but I'm not sure that matches what you are seeing, because
> > > as I said, the secondaries are gone, so they are not supposed
> > > to show up on the host or anywhere (in this case)
> > > 
> > 
> > hmm that behavior appears to be different than earlier versions then.
> 
> no, you are just misinterpreting what I said, please reread
> and/or read some information on how Linux Networking handles
> addresses within the same subnet (with emphasis on primary
> vs secondary addresses)
> 
> > previously, if i had 5 guests all using the same /24 network but
> > of course different ips, shutting one guest down vanished only its
> > ips and the others remained functional. 
> 
> the very same works now as well, given that your config
> is correct and/or you enabled 'promote secondaries'
> 
> > now it appears to destroy all networking within that global network. 
> > ie: guest1 may have xxx.xxx.34.23 thru 26 assigned.. guest2 may 
> > be using xxx.xxx.34.56 thru 60. both /24 , each guest may have 
> > other networks assigned as well depending on their functions. 
> > the networks for the most part are not subnetted in any way. 
> > that would create a nightmare.
> 
> nope, works fine and more importantly, as expected if done
> properly ... no nightmare, no issues ...
> 
> > i need this to be able to just shut down the ip addresses involved 
> > with the downed guest leaving everything else intact.. 
> 
> then get your setup right
> 
> > is there some setting maybe in the kernel or a capability or 
> > something that can do this? 
> 
> as I mentioned several times, you probably want to enable
> 'promote secondaries' for your setup, and make sure that
> the config holds valid addresses and prefix/netmasks
> 

ok i think i understand what happens then... this time around in the new 
hosts, i did not assign a primary ip in each network, only one in one network  
for the host. this then gave a guest a primary ip in a network the host was 
not using before starting the guest... if that guest were stopped then it 
would shut down everything in its network...  so... i assigned a primary ip 
to the host for each network that will be used, and also set up 
net.ipv4.conf.eth0.promote_secondaries=1 in sysctl.conf for backup safety.

i think this combination will cure it. i will know my next scheduled 
maintenance which is tomorrow.


> > maybe with all these upgrades we have done recently i missed 
> > some setting?
> > 
> > we have 3 /24 blocks assigned to web hosting and they are 
> > distributed throughout 10 guests according to various 
> > customers/assignment rules/website purpose etc that we have. 
> > so one /24 network may be shared in all 10 guests living on 
> > the same host. this same network may be shared with other 
> > guests on other hosts as well.
> 
> prefectly fine ...
> 
> > so if i read this right, i must now be very careful not to put 
> > any 2 guests sharing the same network on the same host? 
> 
> without 'promote secondaries' or a dummy base address, yes
> 
> > ugh... literally impossible with our setup.
> 
> (doctor, doctor, it hurts when I poke my eye)
> then don't do that :)
> 
> HTH,
> Herbert
> 
> > > best,
> > > Herbert
> > > 
> > > > -- 
> > > > 
> > > > Chuck
> > > 
> > 
> > 
> > 
> > -- 
> > 
> > Chuck
> 



-- 

Chuck