Subject: where are the lost processes?
From: Jason Drage <jasond@ibsglobalweb.com>
Date: Thu, 13 Mar 2008 11:02:38 +1100

Hi ,
I'm running kernel 2.6.22.18-vs2.3.0.32 (on x86_64), util-vserver: 0.30.214

I'm having trouble finding 'lost' processes attributed to guest.

Basically I have a guest with approx 9 processes (see below) yet
vserver-stat reports vastly different numbers, 286 after running for
a week or 60 for a freshly started guest.

Why are the numbers reported by vserver-stat and ps so different?
Where can I find more details on my machine to help me track down these
'lost processes'?

Also this issue only happens to some of my guests, on some of them
process usage reported by vserver-stat and ps match. Can anyone offer an
explanation?


Details...

Below is the output of vserver-stat for a troublesome guest,

A. When guest was discovered 'using too many processes'
CTX   PROC    VSZ    RSS  userTIME   sysTIME    UPTIME NAME
221    286   1.7G  38.2M   0m09s60   0m09s16   8d20h10 aptcache

B. After a restart (vserver aptcache stop; vserver aptcache start)
CTX   PROC    VSZ    RSS  userTIME   sysTIME    UPTIME NAME
221     60 717.9M    13M   0m00s88   0m00s88   0m55s06 aptcache

C. And after then running for almost a day
CTX   PROC    VSZ    RSS  userTIME   sysTIME    UPTIME NAME
221     95   1.6G  20.1M   0m01s31   0m00s91  21h44m04 aptcache

In all cases the actual proccesses reported by either "#ps aux" from
within the guest or "#vps aux" from the host were wildly different (see
below)

======================
root@aptcache:/#  ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   5112  1968 ?        Ss   Mar12   0:04 /sbin/init
syslog   11627  0.0  0.0  12284   764 ?        Ss   06:25   0:00
/sbin/syslogd -u syslog
root     15893  0.0  0.0    120    32 ?        R+   10:27   0:00
login                                                                              
   

root     15920  0.0  0.0  16916  2024 pts/1    Ss   10:27   0:00
/bin/bash -login
root     16151  0.0  0.0  13984  1072 pts/1    R+   10:46   0:00 ps aux
107      30636  0.0  0.0 1024944 6220 ?        Ssl  Mar12   0:01
/usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng
pidfile:/var/run/apt-cacher-ng/pid SocketPa
root     30644  0.0  0.0  36264  1240 ?        Ss   Mar12   0:00
/usr/sbin/sshd
root     30662  0.0  0.0  17792   832 ?        Ss   Mar12   0:00
/usr/sbin/cron
root     30671  0.0  0.0  51240  2788 ?        Ss   Mar12   0:00
/usr/sbin/apache2 -k start -DSSL
www-data 30673  0.0  0.0  50980  1936 ?        S    Mar12   0:00
/usr/sbin/apache2 -k start -DSSL
www-data 30676  0.0  0.0 274768  3424 ?        Sl   Mar12   0:00
/usr/sbin/apache2 -k start -DSSL
www-data 30679  0.0  0.0 274768  3400 ?        Sl   Mar12   0:00
/usr/sbin/apache2 -k start -DSSL
========================

OR

========================
# vps aux|grep aptcache
klog     11627   221 aptcache       0.0  0.0  12284   764 ?        Ss  
06:25   0:00 /sbin/syslogd -u syslog
root     16161     0 MAIN           0.0  0.0   5128   820 ?        S+  
10:48   0:00 grep aptcache
107      30636   221 aptcache       0.0  0.0 1024944 6220 ?        Ssl 
Mar12   0:01 /usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng
pidfile:/var/run/apt-cacher-ng/pid
SocketPath:/var/run/apt-cacher-ng/socket foreground:0
root     30644   221 aptcache       0.0  0.0  36264  1240 ?        Ss  
Mar12   0:00 /usr/sbin/sshd
root     30662   221 aptcache       0.0  0.0  17792   832 ?        Ss  
Mar12   0:00 /usr/sbin/cron
root     30671   221 aptcache       0.0  0.0  51240  2788 ?        Ss  
Mar12   0:00 /usr/sbin/apache2 -k start -DSSL
www-data 30673   221 aptcache       0.0  0.0  50980  1936 ?        S   
Mar12   0:00 /usr/sbin/apache2 -k start -DSSL
www-data 30676   221 aptcache       0.0  0.0 274768  3424 ?        Sl  
Mar12   0:00 /usr/sbin/apache2 -k start -DSSL
www-data 30679   221 aptcache       0.0  0.0 274768  3400 ?        Sl  
Mar12   0:00 /usr/sbin/apache2 -k start -DSSL
========================


~# vserver-info
Versions:
                   Kernel: 2.6.22.18-vs2.3.0.32
                   VS-API: 0x00020302
             util-vserver: 0.30.214; Nov  1 2007, 03:33:19

Features:
                       CC: gcc, gcc (GCC) 4.2.3 20071014 (prerelease)
(Debian 4.2.2-3)
                      CXX: g++, g++ (GCC) 4.2.3 20071014 (prerelease)
(Debian 4.2.2-3)
                 CPPFLAGS: ''
                   CFLAGS: '-Wall -g  -O2 -std=c99 -Wall -pedantic -W
-funit-at-a-time'
                 CXXFLAGS: '-g -O2 -ansi -Wall -pedantic -W
-fmessage-length=0 -funit-at-a-time'
               build/host: x86_64-pc-linux-gnu/x86_64-pc-linux-gnu
             Use dietlibc: yes
       Build C++ programs: yes
       Build C99 programs: yes
           Available APIs: v13,net,v21,v22,v23,netv2
            ext2fs Source: e2fsprogs
    syscall(2) invocation: alternative
      vserver(2) syscall#: 236/glibc

Paths:
                   prefix: /usr
        sysconf-Directory: /etc
            cfg-Directory: /etc/vservers
         initrd-Directory: $(sysconfdir)/init.d
       pkgstate-Directory: /var/run/vservers
          vserver-Rootdir: /var/lib/vservers


Assumed 'SYSINFO' as no other option given; try '--help' for more
information.

====
thanks,
Jason