Tue, 8 Jun 2010 11:04:07 +0200 Thanks Herbert, I really appreciate your support. It reminds me that there is time to support the project financially. I really hope we can fix this, and make the "management" happy - as they seem to be starting pulling rank to get debian and vmware instead. 2010/6/7 Herbert Poetzl <herbert@13thfloor.at>: > On Mon, Jun 07, 2010 at 12:02:16PM +0200, Tor Rune Skoglund wrote: >> I've ran into a strange (possibly) inode problem within >> one vserver. Althought df -i on the host shows that the >> filesystem on which the vserser lives has IUse of only 1%, >> we get "cannot create" errors on new files. > > cannot create is not necessarily an indication to running > out of inodes ... could you please strace -fF one of those > and provide the trace via some pastebin? > >> Having seen that there is a configuration option >> for the maximum number of inodes dedicated to a vserver, >> I have the following questions: > >> - What is the default limit when this option is not set? > > when the dlimit is unset, the number of inodes is unlimited > >> - What mechanism is used when checking and enforcing vserver >> inode limits? > > the kernel takes care of that > >> - The total number of inodes on the filesystem on which the >> vserver lives is 324350 at the moment. >> Could someone comment if is so high that special actions >> need to be taken? > > nope, I have several millions of inode in use, and except > for a certain memory overhead (for the caches) there is no > problem to be expected .... > >> - Could any other limit be causing the error? E.g. number of >> files limit or whatever? > > could be, strace will probably shed some light on that At the bottom, there an extract of the important parts of the strace result. We have also done some more testing of the problem. I would like to add this to describe the setup: * We have a vserver as a fileserver, running samba 3.4.6. The host and fileserver are both gentoo-based. * The client, which in this case is a debian vserver on the same host, mounts the fileserver share. Unzip/move operations with large number of files fail as described. * Another vserver client, but Gentoo based, on the same host experiences the same problems. * The client, when mounting another share, on another computer, on which there is no vservers and special stuff, gets the same problem with that share. I.e., it seems to be related to the vserver client, and not the fileserver vserver. * However, on another plain host, Debian based (without vserver patches) the unzip operation works fine with the fileserver share (!) >> Here are some server facts: >> * Gentoo vserver kernel, 2.6.32-vs2.3.0.36.28 >> * util-vserver 0.30.216 pre2883 >> * Filesystem ext3, default settings when formatting Best regards, Tor Rune Skoglund execve("/usr/bin/unzip", ["unzip", "../stern 2007 all.zip"], [/* 18 vars */]) = 0 brk(0) = 0x2049000 mmap(NULL, 4096, PROT READ|PROT WRITE, MAP PRIVATE|MAP ANONYMOUS, -1, 0) = 0x7f295c980000 access("/etc/ld.so.nohwcap", F OK) = -1 ENOENT (No such file or directory) mmap(NULL, 8192, PROT READ|PROT WRITE, MAP PRIVATE|MAP ANONYMOUS, -1, 0) = 0x7f295c97e000 access("/etc/ld.so.preload", R OK) = -1 ENOENT (No such file or directory) open("/usr/local/lib/tls/x86 64/libc.so.6", O RDONLY) = -1 ENOENT (No such file or directory) stat("/usr/local/lib/tls/x86 64", 0x7fffc2d2b560) = -1 ENOENT (No such file or directory) open("/usr/local/lib/tls/libc.so.6", O RDONLY) = -1 ENOENT (No such file or directory) stat("/usr/local/lib/tls", 0x7fffc2d2b560) = -1 ENOENT (No such file or directory) open("/usr/local/lib/x86 64/libc.so.6", O RDONLY) = -1 ENOENT (No such file or directory) stat("/usr/local/lib/x86 64", 0x7fffc2d2b560) = -1 ENOENT (No such file or directory) open("/usr/local/lib/libc.so.6", O RDONLY) = -1 ENOENT (No such file or directory) stat("/usr/local/lib", {st mode=S IFDIR|S ISGID|0775, st size=4096, ...}) = 0 open("/etc/ld.so.cache", O RDONLY) = 3 fstat(3, {st mode=S IFREG|0644, st size=21183, ...}) = 0 mmap(NULL, 21183, PROT READ, MAP PRIVATE, 3, 0) = 0x7f295c978000 close(3) = 0 access("/etc/ld.so.nohwcap", F OK) = -1 ENOENT (No such file or directory) open("/lib/libc.so.6", O RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\342\1\0\0\0\0\0@"..., 832) = 832 fstat(3, {st mode=S IFREG|0755, st size=1375536, ...}) = 0 mmap(NULL, 3482232, PROT READ|PROT EXEC, MAP PRIVATE|MAP DENYWRITE, 3, 0) = 0x7f295c413000 mprotect(0x7f295c55d000, 2093056, PROT NONE) = 0 mmap(0x7f295c75c000, 20480, PROT READ|PROT WRITE, MAP PRIVATE|MAP FIXED|MAP DENYWRITE, 3, 0x149000) = 0x7f295c75c000 mmap(0x7f295c761000, 17016, PROT READ|PROT WRITE, MAP PRIVATE|MAP FIXED|MAP ANONYMOUS, -1, 0) = 0x7f295c761000 close(3) = 0 mmap(NULL, 4096, PROT READ|PROT WRITE, MAP PRIVATE|MAP ANONYMOUS, -1, 0) = 0x7f295c977000 mmap(NULL, 4096, PROT READ|PROT WRITE, MAP PRIVATE|MAP ANONYMOUS, -1, 0) = 0x7f295c976000 arch prctl(ARCH SET FS, 0x7f295c9766e0) = 0 mprotect(0x7f295c75c000, 12288, PROT READ) = 0 munmap(0x7f295c978000, 21183) = 0 rt sigaction(SIGINT, {0x409670, [INT], SA RESTORER|SA RESTART, 0x7f295c444f60}, {SIG DFL}, 8) = 0 rt sigaction(SIGTERM, {0x409670, [TERM], SA RESTORER|SA RESTART, 0x7f295c444f60}, {SIG DFL}, 8) = 0 rt sigaction(SIGBUS, {0x409670, [BUS], SA RESTORER|SA RESTART, 0x7f295c444f60}, {SIG DFL}, 8) = 0 rt sigaction(SIGSEGV, {0x409670, [SEGV], SA RESTORER|SA RESTART, 0x7f295c444f60}, {SIG DFL}, 8) = 0 brk(0) = 0x2049000 brk(0x206c000) = 0x206c000 open("/etc/localtime", O RDONLY) = 3 fstat(3, {st mode=S IFREG|0644, st size=118, ...}) = 0 fstat(3, {st mode=S IFREG|0644, st size=118, ...}) = 0 mmap(NULL, 4096, PROT READ|PROT WRITE, MAP PRIVATE|MAP ANONYMOUS, -1, 0) = 0x7f295c97d000 read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0\0"..., 4096) = 118 lseek(3, -62, SEEK CUR) = 56 read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0\0"..., 4096) = 62 close(3) = 0 munmap(0x7f295c97d000, 4096) = 0 stat("../stern 2007 all.zip", {st mode=S IFREG|0755, st size=53229057, ...}) = 0 open("../stern 2007 all.zip", O RDONLY) = 3 ioctl(1, TIOCGWINSZ, {ws row=47, ws col=80, ws xpixel=0, ws ypixel=0}) = 0 write(1, "Archive: ../stern 2007 all.zip\n"..., 32) = 32 lseek(3, 53223424, SEEK SET) = 53223424 read(3, "SiemensKurssprungBF6rsePiererRFCc"..., 5633) = 5633 lseek(3, 51617792, SEEK SET) = 51617792 read(3, "\300\275\305a\344:W\331}\332\22\330\273\34Q\360$\306xq1\273@K`}@Nl;p%*"..., 8192) = 8192 umask(0) = 022 umask(022) = 0 umask(0) = 022 umask(022) = 0 ***** lots more of these.... umask(0) = 022 umask(022) = 0 umask(0) = 022 umask(022) = 0 read(3, "\23X\"6\332|\262\204\10\26\0\0\363C\0\0t\0\0\0\0\0\0\0\0\0 \0\0\0\231\250\2"..., 8192) = 8192 umask(0) = 022 umask(022) = 0 umask(0) = 022 umask(022) = 0 ***** some more of them.... umask(0) = 022 umask(022) = 0 umask(0) = 022 umask(022) = 0 lseek(3, 0, SEEK SET) = 0 read(3, "PK\3\4\n\0\0\0\0\0rc\303<\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\000200"..., 8192) = 8192 stat("2007", 0x62ed08) = -1 ENOENT (No such file or directory) mkdir("2007", 0777) = 0 ioctl(1, TIOCGWINSZ, {ws row=47, ws col=80, ws xpixel=0, ws ypixel=0}) = 0 write(1, " creating: 2007/\n"..., 19) = 19 chmod("2007/", 0755) = 0 stat("/etc/localtime", {st mode=S IFREG|0644, st size=118, ...}) = 0 stat("2007", {st mode=S IFDIR|0755, st size=0, ...}) = 0 stat("2007/01", 0x62ed08) = -1 ENOENT (No such file or directory) mkdir("2007/01", 0777) = -1 ENOENT (No such file or directory) ioctl(1, TIOCGWINSZ, {ws row=47, ws col=80, ws xpixel=0, ws ypixel=0}) = 0 write(2, "checkdir error: cannot create 20"..., 84checkdir error: cannot create 2007/01 unable to process 2007/01/. ) = 84 stat("2007", {st mode=S IFDIR|0755, st size=0, ...}) = 0 stat("2007/01", 0x62ed08) = -1 ENOENT (No such file or directory) mkdir("2007/01", 0777) = -1 ENOENT (No such file or directory) ioctl(1, TIOCGWINSZ, {ws row=47, ws col=80, ws xpixel=0, ws ypixel=0}) = 0 write(2, "checkdir error: cannot create 20"..., 87checkdir error: cannot create 2007/01 unable to process 2007/01/01/. ) = 87 stat("2007", {st mode=S IFDIR|0755, st size=0, ...}) = 0 stat("2007/01", 0x62ed08) = -1 ENOENT (No such file or directory) mkdir("2007/01", 0777) = -1 ENOENT (No such file or directory) ioctl(1, TIOCGWINSZ, {ws row=47, ws col=80, ws xpixel=0, ws ypixel=0}) = 0 write(2, "checkdir error: cannot create 20"..., 144checkdir error: cannot create 2007/01 unable to process 2007/01/01/http www stern de lifestyle leute 579484 htmlnvprpr1.raw. ) = 144 stat("2007", {st mode=S IFDIR|0755, st size=0, ...}) = 0 stat("2007/01", 0x62ed08) = -1 ENOENT (No such file or directory) mkdir("2007/01", 0777) = -1 ENOENT (No such file or directory) ioctl(1, TIOCGWINSZ, {ws row=47, ws col=80, ws xpixel=0, ws ypixel=0}) = 0 write(2, "checkdir error: cannot create 20"..., 144checkdir error: cannot create 2007/01 unable to process 2007/01/01/http www stern de lifestyle leute 579484 htmlnvprpr1.xml. ) = 144 ******* ... and then lots of these, until it in the end ends with: stat("2007", {st mode=S IFDIR|0755, st size=0, ...}) = 0 stat("2007/04", 0x62ed08) = -1 ENOENT (No such file or directory) mkdir("2007/04", 0777) = -1 ENOENT (No such file or directory) ioctl(1, TIOCGWINSZ, {ws row=47, ws col=80, ws xpixel=0, ws ypixel=0}) = 0 write(2, "checkdir error: cannot create 20"..., 175checkdir error: cannot create 2007/04 unable to process 2007/04/21/http www stern de wissenschaft ernaehrung RisikoDieseFetteHE4rte 586102 htmlnvprpr1.raw. ) = 175 lseek(3, 51609600, SEEK SET) = 51609600 read(3, "\214@\376\242\16\245\tj\270\317\335\306\245\247bG\251D\223g\22A\256\300=H\fx\va\212\322N"..., 8192) = 8192 stat("2007", {st mode=S IFDIR|0755, st size=0, ...}) = 0 stat("2007/04", 0x62ed08) = -1 ENOENT (No such file or directory) mkdir("2007/04", 0777) = -1 ENOENT (No such file or directory) ioctl(1, TIOCGWINSZ, {ws row=47, ws col=80, ws xpixel=0, ws ypixel=0}) = 0 write(2, "checkdir error: cannot create 20"..., 175checkdir error: cannot create 2007/04 unable to process 2007/04/21/http www stern de wissenschaft ernaehrung RisikoDieseFetteHE4rte 586102 htmlnvprpr1.xml. ) = 175 lseek(3, 53223424, SEEK SET) = 53223424 read(3, "SiemensKurssprungBF6rsePiererRFCc"..., 8192) = 5633 utime("2007/", [2010/06/03-12:27:36, 2010/06/03-12:27:36]) = 0 chmod("2007/", 0755) = 0 close(3) = 0 exit group(2) = ?