Subject: Re: [vserver] Help me shed some light on a kernel bug
From: Corey Wright <undefined@pobox.com>
Date: Mon, 2 Feb 2009 23:37:30 -0600

On Mon, 02 Feb 2009 23:44:03 -0500
Cedric Veilleux <cveilleux@neopeak.com> wrote:

> I had a server running a vserver patched kernel crash repeatedly (3-4
> times) in a 48 hours period.
> 
> The only error logged was the one pasted below. There was in fact
> sometimes one of these logged, some times many similar ones before a
> crash.
> 
> So, the log output might be misleading as it did not lead immediately to
> a system crash, but everytime it happened, the system would freeze
> eventually.
> 
> What is very special is that this system had 440 days of uptime before
> the first crash. It was running on a Debian Backports kernel labeled
> 2.6.22-6~bpo40+1, with vserver-info showing:
> 
>                    Kernel: 2.6.22-3-vserver-amd64
>                    VS-API: 0x00020200
>              util-vserver: 0.30.216-pre2772; Nov  8 2008, 21:49:08
> 
> 
> So, this might very well not be a vserver related bug at all, but I am
> posting here since people more knowledgeable than me can point me in the
> right direction.
> 
> Also, I would suspect a failing system, maybe some failing system
> memory? But the wording of the error message "Kernel BUG at ..." leads
> me to believe this is a software error and not hardware related...?
> 
> Finally, I since upgraded to 2.6.26-bpo.1-vserver-amd64 and now it has
> been running fine for 4 days with no errors.
> 
> 
> Let me know what you think.

google thinks the problem is failing memory. ;-)

http://www.google.com/search?q="kernel+BUG+at+mm%2Frmap.c"

first hit: http://bugs.centos.org/view.php?id=1108

conclusions:
- "It was bad memory."
- "I replaced the memory, and found that this did indeed solve the problem."

so, survey says, "bad memory." :-D

corey
-- 
undefined@pobox.com

> Jan 28 19:05:19 mtl1 kernel: ------------[ cut here ]------------
> Jan 28 19:05:19 mtl1 kernel: kernel BUG at mm/rmap.c:629!
> Jan 28 19:05:19 mtl1 kernel: invalid opcode: 0000 [1] SMP 
> Jan 28 19:05:19 mtl1 kernel: CPU 3 
> Jan 28 19:05:19 mtl1 kernel: Modules linked in: generic piix ide_core
> ata_generic ata_piix libata ehci_hcd scsi_mod tg3 uhci_hcd thermal
> processor fan Jan 28 19:05:19 mtl1 kernel: Pid: 706:#0, comm: udevd Not
> tainted 2.6.22-3-vserver-amd64 #1 Jan 28 19:05:19 mtl1 kernel: RIP: 0010:
> [<ffffffff8027d383>]  [<ffffffff8027d383>] page_remove_rmap+0xed/0x109
> Jan 28 19:05:19 mtl1 kernel: RSP: 0018:ffff8101191cfdb8  EFLAGS: 00010246
> Jan 28 19:05:19 mtl1 kernel: RAX: 0000000000000000 RBX: ffff81011fb674a0
> RCX: 0000000000004e51 Jan 28 19:05:19 mtl1 kernel: RDX: 00000000ffffff01
> RSI: ffff81011b4048b8 RDI: ffffffff804c3d5c Jan 28 19:05:19 mtl1 kernel:
> RBP: ffff8101196b3348 R08: 000000000000000e R09: 000000007fffffff Jan 28
> 19:05:19 mtl1 kernel: R10: 0000000000000000 R11: ffffffff8031e493 R12:
> 0000000000517000 Jan 28 19:05:19 mtl1 kernel: R13: ffff81011b4048b8 R14:
> ffff810119160f40 R15: 0000000000552000 Jan 28 19:05:19 mtl1 kernel: FS:
> 0000000000000000(0000) GS:ffff81011ba10b40(0000) knlGS:0000000000000000
> Jan 28 19:05:19 mtl1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b Jan 28 19:05:19 mtl1 kernel: CR2: 00002b8cfc7fb4c8 CR3:
> 0000000119aef000 CR4: 00000000000006e0 Jan 28 19:05:19 mtl1 kernel:
> Process udevd (pid: 706[#0], threadinfo ffff8101191ce000, task
> ffff81011af18140) Jan 28 19:05:19 mtl1 kernel: Stack:  000000001ab23020
> ffff81011fb674a0 000000000afcc020 ffffffff8027516e Jan 28 19:05:19 mtl1
> kernel:  0000000000000000 ffff8101191cfea8 ffffffffffffffff
> 0000000000000000 Jan 28 19:05:19 mtl1 kernel:  ffff8101196b3348
> ffff8101191cfeb0 00000000003e9ffd 0000000000000000 Jan 28 19:05:19 mtl1
> kernel: Call Trace: Jan 28 19:05:19 mtl1 kernel:  [<ffffffff8027516e>]
> unmap_vmas+0x3f9/0x758 Jan 28 19:05:19 mtl1 kernel:  [<ffffffff8027ad26>]
> exit_mmap+0x76/0x19c Jan 28 19:05:19 mtl1 kernel:  [<ffffffff8022fe34>]
> mmput+0x2c/0x9d Jan 28 19:05:19 mtl1 kernel:  [<ffffffff802356e4>] do_exit
> +0x222/0x947 Jan 28 19:05:19 mtl1 kernel:  [<ffffffff80235e88>]
> sys_exit_group+0x0/0xe Jan 28 19:05:19 mtl1 kernel:  [<ffffffff80209d9e>]
> system_call+0x7e/0x83 Jan 28 19:05:19 mtl1 kernel: Jan 28 19:05:19 mtl1
> kernel: Jan 28 19:05:19 mtl1 kernel: Code: 0f 0b eb fe 8b 77 18 41 58 5b
> 5d 83 e6 01 f7 de 83 c6 04 e9 Jan 28 19:05:19 mtl1 kernel: RIP
> [<ffffffff8027d383>] page_remove_rmap+0xed/0x109 Jan 28 19:05:19 mtl1
> kernel:  RSP <ffff8101191cfdb8> Jan 28 19:05:19 mtl1 kernel: Fixing
> recursive fault but reboot is needed!