Subject: Re: [vserver] "Stack smash" error running vserver under hardened AMD64?
From: Ed Wildgoose <lists@wildgooses.com>
Date: Fri, 21 Mar 2008 23:39:19 +0000

vitalyb wrote:
> Ed Wildgoose wrote:
>> Hi, can someone please give me some tips on vserver under hardened 
>> amd64?
>>
>> I have a working 32 bit server running the latest 2.2 vserver+grsec 
>> patch without any problems.  I have taken the kernel .config and 
>> switched it over to 64bit and recompiled on a new machine, then 
>> rsynced over the guests to this new machine.  When I try and start 
>> them under the 64bit machine (they are 32 bit images obviously) I get 
>> the error
>>
>> # vserver www2 start
>> stack smashed in main (value 0x00000000000kpp00)
>
> Had to track this down recently. I suppose it's Hardened Gentoo?
> This error happens because of dietlibc. "diet" wrapper do not
> passes "-fno-stack-protector" option to gcc (which has ssp
> enabled by default) and strips this parameter if you
> try to add it to CFLAGS manually.
>

Yep - hardened gentoo.

Some further investigation reveals that the ebuild already sets 
-fno-stack-protector and -fno-stack-protector-all

The trick seems to be to change my CFLAGS and disable optimisation (ie 
-O2).  With CFLAGS including -O2 I get the stack smashed error above - 
removing -O2 and keeping all other CFLAGS (-march=athlon64 -pipe) it 
compiles and superficially at least seems to run ok...

So I'm guessing that gcc-3.4.6 optimisations are doing something which 
triggers the problem rather than it being the code itself?  Perhaps some 
variable aliasing?

So just for reference (and benefit of google).  You can compile up 
util-vserver fine (not fully tested mind) if you compile dietlibc 
without any -O flag (presumably defaults to -O1 ?)

Any other thoughts appreciated?

Ed W