Subject: Re: [vserver] all memory consumed
From: "Michael S. Zick" <mszick@morethan.org>
Date: Fri, 23 Oct 2009 09:24:04 -0500

 Fri, 23 Oct 2009 09:24:04 -0500
On Fri October 23 2009, Chuck wrote:
> On Friday 23 October 2009, Edward Capriolo wrote:
> > On Fri, Oct 23, 2009 at 6:58 AM, Chuck <chuck@sbbsnet.net> wrote:
> > > On Friday 23 October 2009, Konrad Korzeniowski wrote:
> > >> Hi,
> > >>
> > >> I don't know syslog-ng,  but there is rsyslog solution:
> > >> http://www.rsyslog.com/doc-rsyslog reliable forwarding.html
> > >>
> > >
> > > thanks. i will look into it.
> > >
> > >> rgrds.,
> > >> KK
> > >>
> > >> [2009-10-23 04:16] Chuck:
> > >> > I  doubt this is a vserver problem, but I am placing this here just in
> > > case
> > >> > there may be something.
> > >> >
> > >> > we run a remote logging server that every host and guest logs to in
> > > addition
> > >> > to local logs. when all is well this works just fine.
> > >> >
> > >> > if the remote log server is taken down, suddenly most of the hosts 
> begin
> > >> > chewing up all available memory and swap until they crash. disabling
> > > remote
> > >> > logging on everything cured this but obviously is not what we want.
> > >> >
> > >> > i am also posting this in the gentoo forum in case it is a syslog-ng or
> > > other
> > >> > issue.
> > >> >
> > >> > we use 2 lines in every syslog-ng config.. obviously the ip is changed 
> for
> > >> > public viewing:
> > >> >
> > >> > destination remote { udp("1.3.0.1"); };
> > >> > log { source(src); destination(remote); };
> > >> >
> > >> > on all machines
> > >> > kernel is
> > >> >
> > >> > 2.6.26-vs2.3.0.35.6-gentoo
> > >> >
> > >> > util-vserver is 0.30.215
> > >> >
> > >> >
> > >> >
> > >>
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Chuck
> > >
> > 
> > Chuck,
> > 
> > Strange, since UDP packets have no guarantees for transmission or
> > delivery one would imagine they would be sent and dropped. Maybe
> > syslog-ng has a delivery queue that that holds messages until it can
> > make a connection to the remote server and since your server is down
> > the messages build up.
> > 
> 
> after reading about the rsyslog that suddenly occurred to me too... i am going 
> to look into the options available for the config to see if there is a queue 
> and if so maybe it can be disabled. i always thought udp was 'blind' and that 
> syslog-ng would just blindly send out and if it reached the destination fine 
> if not it would not know the difference and continue on.. evidently not..
>

UDP is 'blind' in the sense that it does not provide guarantied delivery. 

What may be happening here is a failure *before* the packet is sent -
I.E: Your routing is returning "Network unreachable" - so the packet 
has nowhere to go.

Perhaps you could provide an alternate route to a packet "/dev/null" 
(I.E: the "dummy" interface) for when the for-real logger is down.
I think the stock network bridging module can be set-up to do this for you.

Mike
> > Edward
> > 
> 
> 
>