Subject: Re: [vserver] odd network problem
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Sat, 22 Nov 2008 15:53:10 +0100

On Sun, Nov 16, 2008 at 08:15:40PM -0500, Chuck wrote:
> On Thursday 13 November 2008, Herbert Poetzl wrote:
> > On Thu, Nov 13, 2008 at 10:03:42AM -0500, Chuck wrote:
> > > i stopped and restarted our openfire server and all was well with that
> > > action. however, 2 web servers, one running an older centos and the
> > > other running gentoo64 with openrc/baselayout2 both lost any network
> > > communication. ips showed present on the host but were unusable even
> > 		~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ that is unusual
> > 
> > > from the host. the ip blocks of the various guests and the host use 3
> > > networks on the same nic.
> > 
> > > i restarted each of those affected servers. when they stopped each    
> > > gave this error:                                                      
> > 
> > > RTNETLINK answers: Cannot assign requested address
> > > RTNETLINK answers: Cannot assign requested address
> > > 
> > > it appears it gave one line for every ip assigned to the guest.
> > 
> > that happens if the IP is already there (maybe with
> > the wrong netmask or so)
> > 
> > > restarting those 2 vservers cured the problem. these are the only two
> > > affected out of the others living on that host.
> > 
> > this will have removed and re-added the IPs properly
> > 
> > > i dont think this has anything to do with it but the centos guest was
> > > moved to this server by simply tarring its /vservers and /etc/vservers
> > > entries. the openfire and gentoo guests were created and configured on
> > > this host. no ip addresses clash.
> > 
> > > the host is running gentoo64 baselayout1 with
> > 
> > > kernel 2.6.22-vs2.3.0.34-gentoo
> > > util-vserver 0.30.215
> > > iproute2 version 2.6.22.20070710
> > 
> > > i am planning on updating the kernel to 2.6.26-vs2.3.0.35.6-gentoo and 
> > > baselayout2/openrc but it is on my 'when i get to it' priority list.
> > 
> > > anyone seen this behavior of stopping a guest and it affecting
> > > networking on other guests before? this is new behavior to me.
> > 
> > well, it is kind of expected, if you are using several
> > IPs in the same network (i.e. with a netmask) and without
> > secondary propagation, that when you remove the primary,
> > all secondaries are gone too (that is a mainline 'feature')
> > 
> > but I'm not sure that matches what you are seeing, because
> > as I said, the secondaries are gone, so they are not supposed
> > to show up on the host or anywhere (in this case)
> > 
> 
> hmm that behavior appears to be different than earlier versions then.

no, you are just misinterpreting what I said, please reread
and/or read some information on how Linux Networking handles
addresses within the same subnet (with emphasis on primary
vs secondary addresses)

> previously, if i had 5 guests all using the same /24 network but
> of course different ips, shutting one guest down vanished only its
> ips and the others remained functional. 

the very same works now as well, given that your config
is correct and/or you enabled 'promote secondaries'

> now it appears to destroy all networking within that global network. 
> ie: guest1 may have xxx.xxx.34.23 thru 26 assigned.. guest2 may 
> be using xxx.xxx.34.56 thru 60. both /24 , each guest may have 
> other networks assigned as well depending on their functions. 
> the networks for the most part are not subnetted in any way. 
> that would create a nightmare.

nope, works fine and more importantly, as expected if done
properly ... no nightmare, no issues ...

> i need this to be able to just shut down the ip addresses involved 
> with the downed guest leaving everything else intact.. 

then get your setup right

> is there some setting maybe in the kernel or a capability or 
> something that can do this? 

as I mentioned several times, you probably want to enable
'promote secondaries' for your setup, and make sure that
the config holds valid addresses and prefix/netmasks

> maybe with all these upgrades we have done recently i missed 
> some setting?
> 
> we have 3 /24 blocks assigned to web hosting and they are 
> distributed throughout 10 guests according to various 
> customers/assignment rules/website purpose etc that we have. 
> so one /24 network may be shared in all 10 guests living on 
> the same host. this same network may be shared with other 
> guests on other hosts as well.

prefectly fine ...

> so if i read this right, i must now be very careful not to put 
> any 2 guests sharing the same network on the same host? 

without 'promote secondaries' or a dummy base address, yes

> ugh... literally impossible with our setup.

(doctor, doctor, it hurts when I poke my eye)
then don't do that :)

HTH,
Herbert

> > best,
> > Herbert
> > 
> > > -- 
> > > 
> > > Chuck
> > 
> 
> 
> 
> -- 
> 
> Chuck