Subject: Re: [vserver] all memory consumed [resolved]
From: Herbert Poetzl <herbert@13thfloor.at>
Date: Fri, 23 Oct 2009 21:06:26 +0200

On Fri, Oct 23, 2009 at 10:35:12AM -0400, Chuck wrote:
> On Friday 23 October 2009, Michael S. Zick wrote:
> > On Fri October 23 2009, Chuck wrote:
> > > On Friday 23 October 2009, Edward Capriolo wrote:
> > > > On Fri, Oct 23, 2009 at 6:58 AM, Chuck <chuck@sbbsnet.net> wrote:
> > > > > On Friday 23 October 2009, Konrad Korzeniowski wrote:
> > > > >> Hi,
> > > > >>
> > > > >> I don't know syslog-ng,  but there is rsyslog solution:
> > > > >> http://www.rsyslog.com/doc-rsyslog_reliable_forwarding.html
> > > > >>
> > > > >
> > > > > thanks. i will look into it.
> > > > >
> > > > >> rgrds.,
> > > > >> KK
> > > > >>
> > > > >> [2009-10-23 04:16] Chuck:
> > > > >> > I  doubt this is a vserver problem, but I am placing this here just 
> in
> > > > > case
> > > > >> > there may be something.
> > > > >> >
> > > > >> > we run a remote logging server that every host and guest logs to in
> > > > > addition
> > > > >> > to local logs. when all is well this works just fine.
> > > > >> >
> > > > >> > if the remote log server is taken down, suddenly most of the hosts 
> > > begin
> > > > >> > chewing up all available memory and swap until they crash. 
> disabling
> > > > > remote
> > > > >> > logging on everything cured this but obviously is not what we want.
> > > > >> >
> > > > >> > i am also posting this in the gentoo forum in case it is a 
> syslog-ng or
> > > > > other
> > > > >> > issue.
> > > > >> >
> > > > >> > we use 2 lines in every syslog-ng config.. obviously the ip is 
> changed 
> > > for
> > > > >> > public viewing:
> > > > >> >
> > > > >> > destination remote { udp("1.3.0.1"); };
> > > > >> > log { source(src); destination(remote); };
> > > > >> >
> > > > >> > on all machines
> > > > >> > kernel is
> > > > >> >
> > > > >> > 2.6.26-vs2.3.0.35.6-gentoo
> > > > >> >
> > > > >> > util-vserver is 0.30.215
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Chuck
> > > > >
> > > > 
> > > > Chuck,
> > > > 
> > > > Strange, since UDP packets have no guarantees for transmission or
> > > > delivery one would imagine they would be sent and dropped. Maybe
> > > > syslog-ng has a delivery queue that that holds messages until it can
> > > > make a connection to the remote server and since your server is down
> > > > the messages build up.
> > > > 
> > > 
> > > after reading about the rsyslog that suddenly occurred to me too... i am 
> going 
> > > to look into the options available for the config to see if there is a 
> queue 
> > > and if so maybe it can be disabled. i always thought udp was 'blind' and 
> that 
> > > syslog-ng would just blindly send out and if it reached the destination 
> fine 
> > > if not it would not know the difference and continue on.. evidently not..
> > >
> > 
> > UDP is 'blind' in the sense that it does not provide guarantied delivery. 
> > 
> > What may be happening here is a failure *before* the packet is sent -
> > I.E: Your routing is returning "Network unreachable" - so the packet 
> > has nowhere to go.
> > 
> > Perhaps you could provide an alternate route to a packet "/dev/null" 
> > (I.E: the "dummy" interface) for when the for-real logger is down.
> > I think the stock network bridging module can be set-up to do this for you.
> > 
> 
> i think i found the answer in syslog changelog:
> 
> 2.1.4
>    Tue, 17 Mar 2009 13:35:00 +0100
> 
>    Bugfixes:
>    * Fixed a possible DoS condition triggered by a destination port
>           unreaachable ICMP packet received from a UDP destination.
>      syslog-ng started eating all available memory and CPU until it
>           crashed if this happened.
>    * Fixed the rate at which files regular were read using the file()
>      source.
>    * Report connection breaks as a write error instead of reporting
>           POLLERR as the write error path reports more sensible information
>      in the logs.

just out of curiosity, the one you used was is a
distribution provided packaged version, and if yes
what distribution, if you do not mind telling?

TIA,
Herbert

> > Mike
> > > > Edward
> > > > 
> > > 
> > > 
> > > 
> > 
> > 
> > 
> 
> 
> 
> -- 
> 
> Chuck