Subject: Re: [vserver] Routing problem in Debian
From: Olivier BATARD <obatard@gmail.com>
Date: Tue, 19 Jan 2010 22:06:52 +0100
Tue, 19 Jan 2010 22:06:52 +0100
Hi,

Thanks a lot for your explanations,

Works like a charm after severe battle with hp with (next time I'll buy real switch)

Thanks again
Le 15 janv. 2010 à 10:59, Laurent Spagnol a écrit :

> Olivier BATARD a écrit :
>> 
>> Hi,
>> 
>> 
>> Thanks again but I have a problem :
>> 
>> 
>> I've made two table :
>> 
>> 
>> - rt1 : One route 192.168.2.0/ 24 dev eth1
>> - rt2 : One route 192.168.3.0/34 dev eth0
>>   
> I suppose you mean /24 ? ;)
> 
>> The rules has been made too :
>> 
>> 
>> 0:      from all lookup local
>> 32764:  from 192.168.3.0/24 lookup rt2
>> 32765:  from 192.168.2.0/24 lookup rt1
>> 
>> 
>> The main table is empty.
>> 
>> Works greats, guests are isolated but when I want to add default route with :
>> 
>> ip route add default via 192.168.2.1 table rt1 or   ip route add 192.168.3.0/24 via
192.168.2.1 table rt1 
>> 
>> I've the error :
>> 
>> RTNETLINK answers: No such process
>> 
>> Thanks for your help
>>   
> An example is better for a good understanding:
> 
> vsh-13:~# ip rule add from 10.7.0.0/16 table 7 => OK
> vsh-13:~# ip route add 10.7.0.0/16 dev eth0.7 table 7
> Cannot find device "eth0.7" => You must create VLAN before
> vsh-13:~# vconfig add eth0 7 => OK
> Added VLAN with VID == 7 to IF -:eth0:-
> vsh-13:~# ip route add 10.7.0.0/16 dev eth0.7 table 7
> RTNETLINK answers: Network is down => You must enable interface.VLAN before
> vsh-13:~# ifconfig eth0.7 up => OK
> vsh-13:~# ip route add 10.7.0.0/16 dev eth0.7 table 7 => OK
> vsh-13:~# ip route add default via 10.7.0.1 table 7
> RTNETLINK answers: No such process => You must have an active IP on this network before
> vsh-13:~# ifconfig eth0.7:test1 10.7.0.53 up => OK
> vsh-13:~# ip route add default via 10.7.0.1 table 7 => OK
> 
> vsh-13:~# ip rule
> 0:    from all lookup local 
> 32765:    from 10.7.0.0/16 lookup 7 
> 32766:    from all lookup main 
> 32767:    from all lookup default 
> 
> vsh-13:~# ip route show table 7
> 10.7.0.0/16 dev eth0.7  scope link 
> default via 10.7.0.1 dev eth0.7 
> 
> So the short answer is "some operations have some prerequisites":
> ip rule add from ... table ... => no constraints
> ip route add ... dev ... table ... => interface.VLAN MUST be active
> ip route add default via ... table ... => The network MUST have at less one active
IP => you must have at less on Vserver running with one active IP on this network)
> 
> In the real life, i use the "start-stop-scripts" of Util-Vserver (/etc/vservers/.../scripts/...)
> 
> * /etc/vservers/.../scripts/initialize.d/
> Before doing anything:
> - initialisation  of  VARS (ip, vlan, device, network, prefix, gateway) for all IPs
of the Vserver
> NOTE: "gateway" is not handled with Util-Vserver, i write this address in /etc/vservers/.../interfaces/.../gateway
> - check of IPs (fping) -> abortion of start sequence if one of them is already active
elsewhere
> 
> * /etc/vservers/.../scripts/prepre-start.d/10 vlan-up
> - loading of dot1q Kernel module if required
> - VLAN(s) is (are) added to the network device if required
> - interface.VLAN(s) is (are) enabled if required (MTU should be changed at this stage)
> 
> * /etc/vservers/.../scripts/pre-start.d/10 route-up
> => At this stage, all the interfaces of Vserver are available
> - installation of routing tables if required
> 
> * /etc/vservers/.../scripts/post-start.d/99 start-end
> => At this stage, the Vserver is fully functional
> - "arping" to gateway and network to force update of ARP tables
> 
> NOTE:
> You must be extremely carefull: "/etc/init.d/util-vserver" (Debian way) and "/etc/init.d/vservers-default"
(genuine) will start several Vserver at once, so the risk is to create duplicate routing
tables.
> That why i use Kernel locks (dotlockfile) in my script to avoid this problem. (probably
better to use /usr/lib/util-vserver/lockfile ?)
> 
> The handling of routing with Util-Vserver should be a great idea ;)
> 
> Regards,
> 
> L.S.
> 
> 
>> Le 12 janv. 2010 à 23:56, Laurent Spagnol a écrit :
>> 
>>   
>>> Hi,
>>> 
>>> Message de Olivier BATARD <obatard@gmail.com>:
>>> 
>>>     
>>>> Hello,
>>>> 
>>>> We're trying to build a vserver infrastructure on Debian Lenny with
>>>> following parameter :
>>>> 
>>>> OS :
>>>> 
>>>> - One host with the kernel 2.6.26 / Debian lenny
>>>>       
>>> The usage of the Lenny Kernel is not a good idea.
>>> 
>>> with the debian tools
>>>     
>>>> - 3 guests, each one runs on a logical volume in reiserfs
>>>> 
>>>> Network :
>>>> 
>>>> - 2 Vlan, one for the the lan, one for the dmz
>>>> - some guests's ip are one the dmz, some on the lan
>>>> - on the hosts we have an /etc/interface with two interfaces with bonding
>>>> (balance-rr mode)
>>>> - Vlan have been set up in the network configuration, each guest interfaces
>>>> as bond0.vlan as device.
>>>> 
>>>> 
>>>> Here's my problem,
>>>> 
>>>> On a dmz guest, when I run a ip route show, I see the routes of the host, so
>>>> a route to the lan ...
>>>>       
>>> Linux-Vserver doesn't virtualize the network layer, so all the routes are seen by
all the guests.
>>> 
>>> which is not good for security ...
>>> I think it's not really a problem. You can use iptables on the host side for isolating
your networks.
>>> 
>>>     
>>>> What is the best solution to isolate the guest and controlling routing ? . I
>>>> googled it and I found some answers by using ip route with alternate
>>>> table ...
>>>>       
>>> Sure. I use 802.1Q VLANs and alternate routing tables.
>>> 
>>> Is it the best and easiest solution now ?
>>> In my opinion, it is the most simplest and cleanest way to do that.
>>> 
>>> How can I set up that
>>>     
>>>> correctly in Debian ?
>>>>       
>>> The solution is not specific to Debian.
>>> 
>>> An example:
>>> vconfig add eth0 7
>>> ip rule add from 10.7.0.0/16 table 7
>>> ip route add 10.7.0.0/16 dev eth0.7 table 7
>>> ip route add default via 10.7.0.1   table 7
>>> echo eth0.7      > /etc/vservers/vs 7/interfaces/0/dev
>>> echo 10.0.7.0.2  > /etc/vservers/vs 7/interfaces/0/ip
>>> 
>>> Google => "presentation-vserver.odp" for the full example.
>>> 
>>> But the author has forgotten two details:
>>> - a routing table can be enabled only when at less one IP is available on the matching
network
>>> - you must force "automatic promotion of secondary addresses to primary"
>>> echo "net.ipv4.conf.all.promote secondaries=1">>/etc/sysctl.conf
>>> 
>>> In fact, i wrote a script that handle routes during the start sequence of the Vservers.
>>> 
>>> An example is available here:
>>> http://dokuwicri.univ-reims.fr/files/vs-tools2/vs-tools/scripts/
>>> 
>>> You can get some ideas here for firewalling:
>>> http://dokuwicri.univ-reims.fr/wiki/doku.php?id=public:projets:vs-tools:configuration
du firewall
>>> My old ugly scripts are available here:
>>> https://listes.univ-reims.fr/sympa/d read/vs-tools/Sources/
>>> (http://dokuwicri.univ-reims.fr/wiki/doku.php?id=public:projets:vs-tools:start)
>>> 
>>> Regards,
>>> 
>>> L.S.
>>> 
>>> 
>>> 
>>> 
>>>     
>>>> Thanks by advance ...
>>>> 
>>>>       
>>> 
>>> -- 
>>> Laurent Spagnol
>>> Administrateur système Linux
>>> 
>>> CRI - Université de Reims
>>> 
>>> Campus du Moulin de la Housse
>>> BP 1039 - 51687 Reims cedex 2
>>> 
>>> Tel: 03.26.91.88.32
>>> Fax: 03.26.91.31.87
>>> 
>>> 
>>> 
>>>     
>> 
>> 
>> 
>>   
> 
> 
> -- 
> Laurent Spagnol
> Administrateur Linux
> 
> Université de Reims
> Centre de Ressources Informatiques
> 
> Campus du Moulin de la Housse
> BP 1039 - 51687 Reims cedex 2
> 
> Tel: 03.26.91.88.32
> Fax: 03.26.91.31.87



Hi,

Thanks a lot for your explanations,

Works like a charm after severe battle with hp with (next time I'll buy real switch)

Thanks again
Le 15 janv. 2010 à 10:59, Laurent Spagnol a écrit :

Olivier BATARD a écrit :
Hi,


Thanks again but I have a problem :


I've made two table :


- rt1 : One route 192.168.2.0/ 24 dev eth1
- rt2 : One route 192.168.3.0/34 dev eth0
  
I suppose you mean /24 ? ;)

The rules has been made too :


0:      from all lookup local
32764:  from 192.168.3.0/24 lookup rt2
32765:  from 192.168.2.0/24 lookup rt1


The main table is empty.

Works greats, guests are isolated but when I want to add default route with :

ip route add default via 192.168.2.1 table rt1 or   ip route add 192.168.3.0/24 via 192.168.2.1 table rt1 

I've the error :

RTNETLINK answers: No such process

Thanks for your help
  
An example is better for a good understanding:

vsh-13:~# ip rule add from 10.7.0.0/16 table 7 => OK
vsh-13:~# ip route add 10.7.0.0/16 dev eth0.7 table 7
Cannot find device "eth0.7" => You must create VLAN before
vsh-13:~# vconfig add eth0 7 => OK
Added VLAN with VID == 7 to IF -:eth0:-
vsh-13:~# ip route add 10.7.0.0/16 dev eth0.7 table 7
RTNETLINK answers: Network is down => You must enable interface.VLAN before
vsh-13:~# ifconfig eth0.7 up => OK
vsh-13:~# ip route add 10.7.0.0/16 dev eth0.7 table 7 => OK
vsh-13:~# ip route add default via 10.7.0.1 table 7
RTNETLINK answers: No such process => You must have an active IP on this network before
vsh-13:~# ifconfig eth0.7:test1 10.7.0.53 up => OK
vsh-13:~# ip route add default via 10.7.0.1 table 7 => OK

vsh-13:~# ip rule
0:    from all lookup local
32765:    from 10.7.0.0/16 lookup 7
32766:    from all lookup main
32767:    from all lookup default

vsh-13:~# ip route show table 7
10.7.0.0/16 dev eth0.7  scope link
default via 10.7.0.1 dev eth0.7

So the short answer is "some operations have some prerequisites":
ip rule add from ... table ... => no constraints
ip route add ... dev ... table ... => interface.VLAN MUST be active
ip route add default via ... table ... => The network MUST have at less one active IP => you must have at less on Vserver running with one active IP on this network)

In the real life, i use the "start-stop-scripts" of Util-Vserver (/etc/vservers/.../scripts/...)

* /etc/vservers/.../scripts/initialize.d/
Before doing anything:
- initialisation  of  VARS (ip, vlan, device, network, prefix, gateway) for all IPs of the Vserver
NOTE: "gateway" is not handled with Util-Vserver, i write this address in /etc/vservers/.../interfaces/.../gateway
- check of IPs (fping) -> abortion of start sequence if one of them is already active elsewhere

* /etc/vservers/.../scripts/prepre-start.d/10 vlan-up
- loading of dot1q Kernel module if required
- VLAN(s) is (are) added to the network device if required
- interface.VLAN(s) is (are) enabled if required (MTU should be changed at this stage)

* /etc/vservers/.../scripts/pre-start.d/10 route-up
=> At this stage, all the interfaces of Vserver are available
- installation of routing tables if required

* /etc/vservers/.../scripts/post-start.d/99 start-end
=> At this stage, the Vserver is fully functional
- "arping" to gateway and network to force update of ARP tables

NOTE:
You must be extremely carefull: "/etc/init.d/util-vserver" (Debian way) and "/etc/init.d/vservers-default" (genuine) will start several Vserver at once, so the risk is to create duplicate routing tables.
That why i use Kernel locks (dotlockfile) in my script to avoid this problem. (probably better to use /usr/lib/util-vserver/lockfile ?)

The handling of routing with Util-Vserver should be a great idea ;)

Regards,

L.S.


Le 12 janv. 2010 à 23:56, Laurent Spagnol a écrit :

  
Hi,

Message de Olivier BATARD <obatard@gmail.com>:

    
Hello,

We're trying to build a vserver infrastructure on Debian Lenny with
following parameter :

OS :

- One host with the kernel 2.6.26 / Debian lenny
      
The usage of the Lenny Kernel is not a good idea.

with the debian tools
    
- 3 guests, each one runs on a logical volume in reiserfs

Network :

- 2 Vlan, one for the the lan, one for the dmz
- some guests's ip are one the dmz, some on the lan
- on the hosts we have an /etc/interface with two interfaces with bonding
(balance-rr mode)
- Vlan have been set up in the network configuration, each guest interfaces
as bond0.vlan as device.


Here's my problem,

On a dmz guest, when I run a ip route show, I see the routes of the host, so
a route to the lan ...
      
Linux-Vserver doesn't virtualize the network layer, so all the routes are seen by all the guests.

which is not good for security ...
I think it's not really a problem. You can use iptables on the host side for isolating your networks.

    
What is the best solution to isolate the guest and controlling routing ? . I
googled it and I found some answers by using ip route with alternate
table ...
      
Sure. I use 802.1Q VLANs and alternate routing tables.

Is it the best and easiest solution now ?
In my opinion, it is the most simplest and cleanest way to do that.

How can I set up that
    
correctly in Debian ?
      
The solution is not specific to Debian.

An example:
vconfig add eth0 7
ip rule add from 10.7.0.0/16 table 7
ip route add 10.7.0.0/16 dev eth0.7 table 7
ip route add default via 10.7.0.1   table 7
echo eth0.7      > /etc/vservers/vs 7/interfaces/0/dev
echo 10.0.7.0.2  > /etc/vservers/vs 7/interfaces/0/ip

Google => "presentation-vserver.odp" for the full example.

But the author has forgotten two details:
- a routing table can be enabled only when at less one IP is available on the matching network
- you must force "automatic promotion of secondary addresses to primary"
echo "net.ipv4.conf.all.promote secondaries=1">>/etc/sysctl.conf

In fact, i wrote a script that handle routes during the start sequence of the Vservers.

An example is available here:
http://dokuwicri.univ-reims.fr/files/vs-tools2/vs-tools/scripts/

You can get some ideas here for firewalling:
http://dokuwicri.univ-reims.fr/wiki/doku.php?id=public:projets:vs-tools:configuration du firewall
My old ugly scripts are available here:
https://listes.univ-reims.fr/sympa/d read/vs-tools/Sources/
(http://dokuwicri.univ-reims.fr/wiki/doku.php?id=public:projets:vs-tools:start)

Regards,

L.S.




    
Thanks by advance ...

      
-- 
Laurent Spagnol
Administrateur système Linux

CRI - Université de Reims

Campus du Moulin de la Housse
BP 1039 - 51687 Reims cedex 2

Tel: 03.26.91.88.32
Fax: 03.26.91.31.87



    



  


-- 
Laurent Spagnol
Administrateur Linux

Université de Reims
Centre de Ressources Informatiques

Campus du Moulin de la Housse
BP 1039 - 51687 Reims cedex 2

Tel: 03.26.91.88.32
Fax: 03.26.91.31.87