Subject: Re: [vserver] chbind the init process of the host (/sbin/init)
From: Manfred Heubach <entwicklung@heubach-edv.de>
Date: Wed, 03 Sep 2008 22:28:51 +0200
Wed, 03 Sep 2008 22:28:51 +0200

Herbert Poetzl schrieb:
> On Wed, Sep 03, 2008 at 09:30:33AM +0200, Manfred Heubach wrote:
>   
>> Herbert Poetzl schrieb:
>>     
>>> On Tue, Sep 02, 2008 at 10:27:03PM +0200, Manfred Heubach wrote:
>>>   
>>>       
>>>> Hello Gyu, hello Herbert,
>>>>
>>>> well - o.k. as I cannot restrict the init process to certain IPs
>>>> and use different IPs for the guests at the same time - I will
>>>> think this all over. I have once written a script which checks all
>>>> init scripts and wraps them if necessary. I just have to adjust
>>>> this script to wrap them into e.g. --nid 2. netstat -l will not
>>>> show the sockets in --nid 2 chbind --nid 2 netstat -l will *only*
>>>> show those in --nid 2 So if I want to get a quick overview what's
>>>> listening I will end up with 2 commands (I can build an alias for
>>>> that).
>>>>     
>>>>         
>>> try chbind --nid 1 netstat -l
>>> (nid=1 is the spectator context, like xid=1 for processes :)
>>>
>>>   
>>>       
>> Then maybe this is a bug.
>> chbind --nid 1 netstat -l neither shows sockets without any 
>> nid nor sockets in nid 2
>>     
>
> all sockets have a nid (host nid=0, spectator=1)
>
> [root]$ nc -l -p 100 &
> [root]$ chbind --nid 2 --ip 127.0.0.1 -- nc -l -p 200 &
>
> [root]$ lsof -ni
> COMMAND PID USER   FD   TYPE DEVICE SIZE NODE NAME
> nc      259 root    3u  IPv4    102       TCP *:100 (LISTEN)
> [root]$ chbind --nid 1 -- lsof -ni
> COMMAND PID USER   FD   TYPE DEVICE SIZE NODE NAME
> nc      259 root    3u  IPv4    102       TCP *:100 (LISTEN)
> nc      264 root    3u  IPv4    107       TCP *:200 (LISTEN)
> [root]$ chbind --nid 2 -- lsof -ni
> COMMAND PID USER   FD   TYPE DEVICE SIZE NODE NAME
> nc      264 root    3u  IPv4    107       TCP *:200 (LISTEN)
>
> note: if you have CONFIG VSERVER PRIVACY=y then
> the spectator context is disabled in most cases
> including the network information
>
>   
This is how the Debian kernel has been built:

CONFIG VSERVER AUTO LBACK=y
# CONFIG VSERVER AUTO SINGLE is not set
CONFIG VSERVER COWBL=y
# CONFIG VSERVER VTIME is not set
# CONFIG VSERVER DEVICE is not set
CONFIG VSERVER PROC SECURE=y
CONFIG VSERVER HARDCPU=y
CONFIG VSERVER IDLETIME=y
CONFIG VSERVER IDLELIMIT=y
CONFIG VSERVER PRIVACY=y
CONFIG VSERVER CONTEXTS=256
CONFIG VSERVER WARN=y
# CONFIG VSERVER DEBUG is not set
CONFIG VSERVER=y
CONFIG VSERVER SECURITY=y

So CONFIG VSERVER PRIVACY=y is why I don't see anything in nid 1.

Do you see anything which will lead me into other problems with this
kernel config?

(I for myself see that I'm in desperate need for some RTFM sessions - I
lost track of the vserver project about 3 years ago and loads of things
have changed since then :-))


>> (Why does the output below show 0.30.215 when actually 0.30.216 is
>> installed - strange ...)
>>
>> ---------------------------
>> Versions:
>>                    Kernel: 2.6.26-1-vserver-686
>>                    VS-API: 0x00020303
>>              util-vserver: 0.30.215; Jun  1 2008, 21:56:10
>>     
>
> well, I'd say it is 0.30.215 at least at heart :)
>
> maybe you installed two competing versions of
> util-vserver in different locations?
>   
hm - maybe I should get some sleep - this is what it looks like today - ...


                   Kernel: 2.6.26-1-vserver-686
                   VS-API: 0x00020303
             util-vserver: 0.30.216-pre2772; Aug 27 2008, 16:39:45

Good night and thanks for helping!
Manfred

> HTH,
> Herbert
>
>   
>> Features:
>>                        CC: gcc, gcc (GCC) 4.1.2 20061115 (prerelease)
>> (Debian 4.                                              1.1-21)
>>                       CXX: g++, g++ (GCC) 4.1.2 20061115 (prerelease)
>> (Debian 4.                                              1.1-21)
>>                  CPPFLAGS: ''
>>                    CFLAGS: '-Wall -g  -O2 -std=c99 -Wall -pedantic -W
>> -funit-at-                                              a-time'
>>                  CXXFLAGS: '-g -O2 -ansi -Wall -pedantic -W
>> -fmessage-length=0 -                                             
>> funit-at-a-time'
>>                build/host: i486-pc-linux-gnu/i486-pc-linux-gnu
>>              Use dietlibc: yes
>>        Build C++ programs: yes
>>        Build C99 programs: yes
>>            Available APIs: v13,net,v21,v22,v23,netv2
>>             ext2fs Source: e2fsprogs
>>     syscall(2) invocation: alternative
>>       vserver(2) syscall#: 273/glibc
>>                crypto api: beecrypt
>>
>> Paths:
>>                    prefix: /usr
>>         sysconf-Directory: /etc
>>             cfg-Directory: /etc/vservers
>>          initrd-Directory: $(sysconfdir)/init.d
>>        pkgstate-Directory: /var/run/vservers
>>           vserver-Rootdir: /var/lib/vservers
>>
>>
>>     
>>>> Maybe it's better to forget about the wrapper and configure the services
>>>> to bind only to the right IPs. I will still have a problem with services
>>>> that don't offer such an option - but these are hopefully none of which
>>>> I need :-)
>>>>     
>>>>         
>>> a good way (for certain services) is to use something
>>> like the v* runlevel wrappers we had (and maybe still
>>> have?) somewhere ...
>>>
>>> i.e. make a separate runlevel script called v<service>
>>> which only does the chbind and then hand over the
>>> real stuff to the <service> script
>>>
>>> this way, you'll keep them on updates, while benefiting
>>> from the changes/upgrades to the wrapped scripts
>>> (which you simply disable)
>>>
>>>   
>>>       
>>>> Is someone interested in a script which checks that services on the host
>>>> aren't bound to *:port. I think about this as a sanity check before
>>>> starting a vserver - so a misconfigured host won't mess up the guests.
>>>> This won't be bullet proof. You can still stop a service in the guest,
>>>> then restart the same service on the host and you end up with the host
>>>> bound to the IP of a guest. But it seems to be a good way not to shoot
>>>> yourself in the foot in the first place ;-)
>>>>     
>>>>         
>>> I think a warning script might be interesting for some
>>> folks, but I also think it would be much better to fix
>>> most of the runlevel scripts, which report 'success'
>>> while the service actually failed (like klogd, ssh, ...)
>>>
>>> best,
>>> Herbert
>>>
>>>   
>>>       
>>>> Some comments follow below.
>>>>
>>>> Thanks
>>>> Manfred
>>>>
>>>> PÁSZTOR György schrieb:
>>>>     
>>>>         
>>>>> Hi,
>>>>>
>>>>> "Manfred Heubach" <entwicklung@heubach-edv.de> írta 2008-09-02 14:12-kor:
>>>>>   
>>>>>       
>>>>>           
>>>>>> PÁSZTOR György schrieb:
>>>>>>     
>>>>>>         
>>>>>>             
>>>>>>> Hi!
>>>>>>>
>>>>>>> "Manfred Heubach" <entwicklung@heubach-edv.de> írta 2008-09-02 09:07-kor:
>>>>>>>           
>>>>>>>               
>>>>>>>>> well, why not just 'restrict' the host to those two addresses
>>>>>>>>> and be done?
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>>> Well, i *want* to restrict the host to those two addresses - but
>>>>>>>> how?
>>>>>>>>             
>>>>>>>>                 
>>>>>>> I don't understand your attitude. Why do you want to run services
>>>>>>> on the host, if you use vserver?
>>>>>>>           
>>>>>>>               
>>>>>> I use vserver mainly for server consolidation. This is why I have
>>>>>> some more services on the host itself (syslog-ng, mta, bacula,
>>>>>> raid-console, ssh, snmp, ...).
>>>>>>     
>>>>>>         
>>>>>>             
>>>>> Yepp, so do I.
>>>>>
>>>>> But with some differences:
>>>>>
>>>>> syslog-ng -> I run it on a dedicated management server. If the host
>>>>> dies we still have its logs.
>>>>>       
>>>>>           
>>>> see below - no extra hardware :-(
>>>>
>>>>     
>>>>         
>>>>> MTA -> I don't understand why do you want MTA on the host. On
>>>>> Special vservers, which purpose is not to handle the mails I run
>>>>> nullmailer to let
>>>>>       
>>>>>           
>>>> Hey - great stuff - I didn't know about nullmailer until now - and
>>>> this after 10 years of Linux with postfix, courier and all the lot.
>>>>
>>>>     
>>>>         
>>>>> my error reports reach it's destination. I have some vservers which
>>>>> runs backround service, and don't have public IP but an internal for
>>>>> interchange data between servers have. Those nullmailers sends it's
>>>>> message to the management host, and that delivers to the sysadmins.
>>>>> For the "corporate" mailing I use a vserver, which handles our
>>>>> mails, and another vserver which runs webmail things.
>>>>>
>>>>> Bacula -> I run bacula-fd on the host. But I don't need any other
>>>>> bacula things in the guests.
>>>>>
>>>>> I don't know raid-console. What it is?
>>>>>       
>>>>>           
>>>> It's just the web management console of e.g. 3ware raid controllers
>>>> (3dm)
>>>>
>>>>     
>>>>         
>>>>> ssh -> I wrote it already: Yes, the (almost) only thing you may want
>>>>> also on the host.
>>>>>
>>>>> snmp -> I collect the traps on the mentioned management machine.
>>>>> What do you want to run on the host? I don't trust in snmp things,
>>>>> so I won't allow to run snmpd on the host, neither on the guests.
>>>>>       
>>>>>           
>>>> I use snmp for performance and health monitoring - does a great job
>>>> together with nagios or munin. Read only and restricted to management
>>>> IPs it cannot do much harm.
>>>>
>>>>     
>>>>         
>>>>> ... -> So... Basic approach: Just run sshd (and bacula-fd) on the
>>>>> host. Everything else in the guests. If you do the consolidate like
>>>>> this you will have easy job, when you have to move a vserver to
>>>>> another host, because of decrease the load of a host, etc.
>>>>>       
>>>>>           
>>>> I have customers with only one server (hardware) - so all has to go on
>>>> one machine.
>>>>     
>>>>         
>>>>> Cheers,
>>>>> gyu
>>>>>   
>>>>>       
>>>>>           



Herbert Poetzl schrieb:
On Wed, Sep 03, 2008 at 09:30:33AM +0200, Manfred Heubach wrote:
  
Herbert Poetzl schrieb:
    
On Tue, Sep 02, 2008 at 10:27:03PM +0200, Manfred Heubach wrote:
  
      
Hello Gyu, hello Herbert,

well - o.k. as I cannot restrict the init process to certain IPs
and use different IPs for the guests at the same time - I will
think this all over. I have once written a script which checks all
init scripts and wraps them if necessary. I just have to adjust
this script to wrap them into e.g. --nid 2. netstat -l will not
show the sockets in --nid 2 chbind --nid 2 netstat -l will *only*
show those in --nid 2 So if I want to get a quick overview what's
listening I will end up with 2 commands (I can build an alias for
that).
    
        
try chbind --nid 1 netstat -l
(nid=1 is the spectator context, like xid=1 for processes :)

  
      
Then maybe this is a bug.
chbind --nid 1 netstat -l neither shows sockets without any 
nid nor sockets in nid 2
    

all sockets have a nid (host nid=0, spectator=1)

[root]$ nc -l -p 100 &
[root]$ chbind --nid 2 --ip 127.0.0.1 -- nc -l -p 200 &

[root]$ lsof -ni
COMMAND PID USER   FD   TYPE DEVICE SIZE NODE NAME
nc      259 root    3u  IPv4    102       TCP *:100 (LISTEN)
[root]$ chbind --nid 1 -- lsof -ni
COMMAND PID USER   FD   TYPE DEVICE SIZE NODE NAME
nc      259 root    3u  IPv4    102       TCP *:100 (LISTEN)
nc      264 root    3u  IPv4    107       TCP *:200 (LISTEN)
[root]$ chbind --nid 2 -- lsof -ni
COMMAND PID USER   FD   TYPE DEVICE SIZE NODE NAME
nc      264 root    3u  IPv4    107       TCP *:200 (LISTEN)

note: if you have CONFIG_VSERVER_PRIVACY=y then
the spectator context is disabled in most cases
including the network information

  
This is how the Debian kernel has been built:

CONFIG_VSERVER_AUTO_LBACK=y
# CONFIG_VSERVER_AUTO_SINGLE is not set
CONFIG_VSERVER_COWBL=y
# CONFIG_VSERVER_VTIME is not set
# CONFIG_VSERVER_DEVICE is not set
CONFIG_VSERVER_PROC_SECURE=y
CONFIG_VSERVER_HARDCPU=y
CONFIG_VSERVER_IDLETIME=y
CONFIG_VSERVER_IDLELIMIT=y
CONFIG_VSERVER_PRIVACY=y
CONFIG_VSERVER_CONTEXTS=256
CONFIG_VSERVER_WARN=y
# CONFIG_VSERVER_DEBUG is not set
CONFIG_VSERVER=y
CONFIG_VSERVER_SECURITY=y

So CONFIG_VSERVER_PRIVACY=y is why I don't see anything in nid 1.

Do you see anything which will lead me into other problems with this kernel config?

(I for myself see that I'm in desperate need for some RTFM sessions - I lost track of the vserver project about 3 years ago and loads of things have changed since then :-))


(Why does the output below show 0.30.215 when actually 0.30.216 is
installed - strange ...)

---------------------------
Versions:
                   Kernel: 2.6.26-1-vserver-686
                   VS-API: 0x00020303
             util-vserver: 0.30.215; Jun  1 2008, 21:56:10
    

well, I'd say it is 0.30.215 at least at heart :)

maybe you installed two competing versions of
util-vserver in different locations?
  
hm - maybe I should get some sleep - this is what it looks like today - ...


                   Kernel: 2.6.26-1-vserver-686
                   VS-API: 0x00020303
             util-vserver: 0.30.216-pre2772; Aug 27 2008, 16:39:45

Good night and thanks for helping!
Manfred

HTH,
Herbert

  
Features:
                       CC: gcc, gcc (GCC) 4.1.2 20061115 (prerelease)
(Debian 4.                                              1.1-21)
                      CXX: g++, g++ (GCC) 4.1.2 20061115 (prerelease)
(Debian 4.                                              1.1-21)
                 CPPFLAGS: ''
                   CFLAGS: '-Wall -g  -O2 -std=c99 -Wall -pedantic -W
-funit-at-                                              a-time'
                 CXXFLAGS: '-g -O2 -ansi -Wall -pedantic -W
-fmessage-length=0 -                                             
funit-at-a-time'
               build/host: i486-pc-linux-gnu/i486-pc-linux-gnu
             Use dietlibc: yes
       Build C++ programs: yes
       Build C99 programs: yes
           Available APIs: v13,net,v21,v22,v23,netv2
            ext2fs Source: e2fsprogs
    syscall(2) invocation: alternative
      vserver(2) syscall#: 273/glibc
               crypto api: beecrypt

Paths:
                   prefix: /usr
        sysconf-Directory: /etc
            cfg-Directory: /etc/vservers
         initrd-Directory: $(sysconfdir)/init.d
       pkgstate-Directory: /var/run/vservers
          vserver-Rootdir: /var/lib/vservers


    
Maybe it's better to forget about the wrapper and configure the services
to bind only to the right IPs. I will still have a problem with services
that don't offer such an option - but these are hopefully none of which
I need :-)
    
        
a good way (for certain services) is to use something
like the v* runlevel wrappers we had (and maybe still
have?) somewhere ...

i.e. make a separate runlevel script called v<service>
which only does the chbind and then hand over the
real stuff to the <service> script

this way, you'll keep them on updates, while benefiting
from the changes/upgrades to the wrapped scripts
(which you simply disable)

  
      
Is someone interested in a script which checks that services on the host
aren't bound to *:port. I think about this as a sanity check before
starting a vserver - so a misconfigured host won't mess up the guests.
This won't be bullet proof. You can still stop a service in the guest,
then restart the same service on the host and you end up with the host
bound to the IP of a guest. But it seems to be a good way not to shoot
yourself in the foot in the first place ;-)
    
        
I think a warning script might be interesting for some
folks, but I also think it would be much better to fix
most of the runlevel scripts, which report 'success'
while the service actually failed (like klogd, ssh, ...)

best,
Herbert

  
      
Some comments follow below.

Thanks
Manfred

PÁSZTOR György schrieb:
    
        
Hi,

"Manfred Heubach" <entwicklung@heubach-edv.de> írta 2008-09-02 14:12-kor:
  
      
          
PÁSZTOR György schrieb:
    
        
            
Hi!

"Manfred Heubach" <entwicklung@heubach-edv.de> írta 2008-09-02 09:07-kor:
          
              
well, why not just 'restrict' the host to those two addresses
and be done?
              
                  
Well, i *want* to restrict the host to those two addresses - but
how?
            
                
I don't understand your attitude. Why do you want to run services
on the host, if you use vserver?
          
              
I use vserver mainly for server consolidation. This is why I have
some more services on the host itself (syslog-ng, mta, bacula,
raid-console, ssh, snmp, ...).
    
        
            
Yepp, so do I.

But with some differences:

syslog-ng -> I run it on a dedicated management server. If the host
dies we still have its logs.
      
          
see below - no extra hardware :-(

    
        
MTA -> I don't understand why do you want MTA on the host. On
Special vservers, which purpose is not to handle the mails I run
nullmailer to let
      
          
Hey - great stuff - I didn't know about nullmailer until now - and
this after 10 years of Linux with postfix, courier and all the lot.

    
        
my error reports reach it's destination. I have some vservers which
runs backround service, and don't have public IP but an internal for
interchange data between servers have. Those nullmailers sends it's
message to the management host, and that delivers to the sysadmins.
For the "corporate" mailing I use a vserver, which handles our
mails, and another vserver which runs webmail things.

Bacula -> I run bacula-fd on the host. But I don't need any other
bacula things in the guests.

I don't know raid-console. What it is?
      
          
It's just the web management console of e.g. 3ware raid controllers
(3dm)

    
        
ssh -> I wrote it already: Yes, the (almost) only thing you may want
also on the host.

snmp -> I collect the traps on the mentioned management machine.
What do you want to run on the host? I don't trust in snmp things,
so I won't allow to run snmpd on the host, neither on the guests.
      
          
I use snmp for performance and health monitoring - does a great job
together with nagios or munin. Read only and restricted to management
IPs it cannot do much harm.

    
        
... -> So... Basic approach: Just run sshd (and bacula-fd) on the
host. Everything else in the guests. If you do the consolidate like
this you will have easy job, when you have to move a vserver to
another host, because of decrease the load of a host, etc.
      
          
I have customers with only one server (hardware) - so all has to go on
one machine.
    
        
Cheers,
gyu