Subject: Re: [vserver] transient network drops
From: Eugen Leitl <eugen@leitl.org>
Date: Sat, 1 Sep 2007 13:52:25 +0200

On Sat, Sep 01, 2007 at 06:35:16AM -0500, Michael S. Zick wrote:

> When brtables was an add-on, it also added on spanning
> tree routing -
> Now that brtables is built-in, I would suppose that
> the spanning tree algorithm is also built in now.

The host never saw any problem. The host looks like this:

nitrogen:~# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:E0:81:58:58:B3  
          inet addr:85.10.225.7  Bcast:85.10.225.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:81ff:fe58:58b3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6171202 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6583481 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:838920870 (800.0 MiB)  TX bytes:4942355116 (4.6 GiB)
          Interrupt:16 

eth1      Link encap:Ethernet  HWaddr 00:E0:81:58:58:B2  
          inet addr:10.0.0.7  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:81ff:fe58:58b2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:2517054 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2149975 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:191384699 (182.5 MiB)  TX bytes:4881121526 (4.5 GiB)
          Interrupt:19 Base address:0x4000 
...

the guest looks like this:

v64:/# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:E0:81:58:58:B3  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6171793 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6584117 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:838971065 (800.1 MiB)  TX bytes:4943030673 (4.6 GiB)
          Interrupt:16 

The switch told me 

  85.10.225.64    00e081-5858b3     dynamic 7
and then
  85.10.225.64    00e081-5858b3     dynamic 49

which told me (the second Level 2 switch was connected
to port 49 of the Level 3 switch) that a frame from eth0 
somehow managed to emerge from eth1.
 
> I.E: The kernel is also a level 2 bridge with S.T.
> and the vserver isolation is at level 3 (ip).
> 
> I think your mention of pulling the loop cable clearing
> the problem was a key clue in what is happening.

What was happening was probably that on very rare occasions
a MAC from eth0 emerged on eth1. I know second-hand that
some X2100 M2 have been observed to do that in the wild.
 
> Why the kernel-bridge would randomly send a traffic
> packet on a redundant link ???? no idea, unless it
> had a vlan to establish routing for.

No vlans that I know of. There was an OpenVPN interface
I just disabled, for good measure.
 
> Can you snoop the bridge configuration packets on
> that cable loop?

Yes.

> Or maybe put a brtable rule in that logs what the
> kernel bridge is doing?
> 
> The kernel bridge code has been working problem
> free for many generations of Linux - but bit rot 
> happens.

I'm rather inclined to believe that a MAC jumps between
interfaces on particularly braindead hardware.

-- 
Eugen* Leitl <a href="leitlhttp://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE