Subject: For a period of 2 months, the server file system has switched three times into the readonly mode
From:KLEIN Stéphane <klein.stephane@gmail.com>
Date: Thu, 29 Oct 2009 11:05:51 +0100

 Thu, 29 Oct 2009 11:05:51 +0100
Hi,

I've a vserver installed on Debian Lenny.

For a period of 2 months, the server file system has switched three
times into the readonly mode.

What do you think ?
* is it a vserver issue ?
* is it a hardware issue ?
* is it a debian issue ?

Last switch time, I've verify my disk with smartctl and fsck… my disk
seem to be clean.

This is some information about the last switch :

# uptime
10:29:53 up 15 days, 37 min,  0 users,  load average: 0.00, 0.00, 0.00

# dmesg | grep "Shorewall" -v
TTL=120 ID=21306 DF PROTO=TCP SPT=64969 DPT=8443 WINDOW=8192 RES=0x00 SYN URGP=0
[1366416.782108] EXT3-fs error (device md1): ext3 free blocks: Freeing
blocks in system zones - Block = 131072, count = 1
[1366416.782161] Aborting journal on device md1.
[1366416.782386] Remounting filesystem read-only
[1366416.782748] EXT3-fs error (device md1) in
ext3 reserve inode write: Journal has aborted
[1366416.782792] EXT3-fs error (device md1) in ext3 truncate: Journal
has aborted
[1366416.782833] EXT3-fs error (device md1) in
ext3 reserve inode write: Journal has aborted
[1366416.782875] EXT3-fs error (device md1) in ext3 orphan del:
Journal has aborted
[1366416.782915] EXT3-fs error (device md1) in
ext3 reserve inode write: Journal has aborted

# mdadm --detail /dev/md1
/dev/md1:
       Version : 00.90
 Creation Time : Wed Aug 19 15:56:46 2009
    Raid Level : raid1
    Array Size : 487749376 (465.15 GiB 499.46 GB)
 Used Dev Size : 487749376 (465.15 GiB 499.46 GB)
  Raid Devices : 2
 Total Devices : 2
Preferred Minor : 1
   Persistence : Superblock is persistent

   Update Time : Thu Oct 29 10:07:11 2009
         State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
 Spare Devices : 0

          UUID : 191320ed:72bbf45b:bfe78010:bc810f04
        Events : 0.6

   Number   Major   Minor   RaidDevice State
      0       8        2        0      active sync   /dev/sda2
      1       8       18        1      active sync   /dev/sdb2

# lspci
00:00.0 Host bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML
and 945GT Express Memory Controller Hub (rev 03)
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS,
943/940GML Express Integrated Graphics Controller (rev 03)
00:02.1 Display controller: Intel Corporation Mobile 945GM/GMS/GME,
943/940GML Express Integrated Graphics Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express
Port 1 (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1)
00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC
Interface Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE
Controller (rev 01)
00:1f.2 SATA controller: Intel Corporation 82801GR/GH (ICH7 Family)
SATA AHCI Controller (rev 01)
00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev 01)
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)

# cat /proc/cpuinfo
processor : 0
vendor id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU         T7200  @ 2.00GHz
stepping : 6
cpu MHz : 2000.186
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fdiv bug : no
hlt bug : no
f00f bug : no
coma bug : no
fpu : yes
fpu exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm
constant tsc arch perfmon pebs bts pni monitor ds cpl vmx est tm2
ssse3 cx16 xtpr lahf lm
bogomips : 4004.73
clflush size : 64
power management:

processor : 1
vendor id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU         T7200  @ 2.00GHz
stepping : 6
cpu MHz : 2000.186
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fdiv bug : no
hlt bug : no
f00f bug : no
coma bug : no
fpu : yes
fpu exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm
constant tsc arch perfmon pebs bts pni monitor ds cpl vmx est tm2
ssse3 cx16 xtpr lahf lm
bogomips : 4000.55
clflush size : 64
power management:

# dpkg -l | grep "vserver"
ii  linux-image-2.6.26-1-vserver-686   2.6.26-13lenny2          Linux
2.6.26 image on PPro/Celeron/PII/PIII/P4, Linux-VServer
ii  util-vserver                       0.30.216~r2772-6
user-space tools for Linux-VServer virtual private servers
ii  vserver-debiantools                0.6.3                    Tools
to manage debian virtual servers

# vserver-info
Versions:
                  Kernel: 2.6.26-1-vserver-686
                  VS-API: 0x00020303
            util-vserver: 0.30.216-pre2772; Dec 12 2008, 23:24:33

Features:
      CC: gcc, gcc (Debian 4.3.2-1) 4.3.2
                     CXX: g++, g++ (Debian 4.3.2-1) 4.3.2
                CPPFLAGS: ''
                  CFLAGS: '-Wall -g  -O2 -std=c99 -Wall -pedantic -W
-funit-at-a-time'
                CXXFLAGS: '-g -O2 -ansi -Wall -pedantic -W
-fmessage-length=0 -funit-at-a-time'
              build/host: i486-pc-linux-gnu/i486-pc-linux-gnu
            Use dietlibc: yes
      Build C++ programs: yes
      Build C99 programs: yes
          Available APIs: v13,net,v21,v22,v23,netv2
           ext2fs Source: e2fsprogs
   syscall(2) invocation: alternative
     vserver(2) syscall#: 273/glibc
              crypto api: beecrypt
  use library versioning: yes

Paths:
                  prefix: /usr
       sysconf-Directory: /etc
           cfg-Directory: /etc/vservers
        initrd-Directory: $(sysconfdir)/init.d
      pkgstate-Directory: /var/run/vservers
         vserver-Rootdir: /var/lib/vservers


Assumed 'SYSINFO' as no other option given; try '--help' for more information.

#fsck.ext3 -v /dev/md1
e2fsck 1.40.8 (13-Mar-2008)
/ was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 122881 has zero dtime.  Fix<y>? yes

Deleted inode 3194881 has zero dtime.  Fix<y>? yes

Deleted inode 10838017 has zero dtime.  Fix<y>?

Inode 10838018 is in use, but has dtime set.  Fix<y>? yes

Inode 10838018, i blocks is 11013, should be 0.  Fix<y>? yes

Deleted inode 10838081 has zero dtime.  Fix<y>? yes

Deleted inode 10838113 has zero dtime.  Fix<y>? yes

Inode 10838114 is in use, but has dtime set.  Fix<y>? yes

Inode 10838114, i blocks is 3707185669, should be 0.  Fix<y>? yes

Deleted inode 10838177 has zero dtime.  Fix<y>? yes

Deleted inode 10838209 has zero dtime.  Fix<y>? yes

Deleted inode 10838433 has zero dtime.  Fix<y>? yes

Deleted inode 10838465 has zero dtime.  Fix<y>? yes

[snap]

Do you need other information ?

Thanks for your help.
Stephane