Subject: Re: [vserver] all memory consumed [resolved]
From: Chuck <chuck@sbbsnet.net>
Date: Fri, 23 Oct 2009 10:35:12 -0400

 Fri, 23 Oct 2009 10:35:12 -0400
On Friday 23 October 2009, Michael S. Zick wrote:
> On Fri October 23 2009, Chuck wrote:
> > On Friday 23 October 2009, Edward Capriolo wrote:
> > > On Fri, Oct 23, 2009 at 6:58 AM, Chuck <chuck@sbbsnet.net> wrote:
> > > > On Friday 23 October 2009, Konrad Korzeniowski wrote:
> > > >> Hi,
> > > >>
> > > >> I don't know syslog-ng,  but there is rsyslog solution:
> > > >> http://www.rsyslog.com/doc-rsyslog reliable forwarding.html
> > > >>
> > > >
> > > > thanks. i will look into it.
> > > >
> > > >> rgrds.,
> > > >> KK
> > > >>
> > > >> [2009-10-23 04:16] Chuck:
> > > >> > I  doubt this is a vserver problem, but I am placing this here just 
in
> > > > case
> > > >> > there may be something.
> > > >> >
> > > >> > we run a remote logging server that every host and guest logs to in
> > > > addition
> > > >> > to local logs. when all is well this works just fine.
> > > >> >
> > > >> > if the remote log server is taken down, suddenly most of the hosts 
> > begin
> > > >> > chewing up all available memory and swap until they crash. 
disabling
> > > > remote
> > > >> > logging on everything cured this but obviously is not what we want.
> > > >> >
> > > >> > i am also posting this in the gentoo forum in case it is a 
syslog-ng or
> > > > other
> > > >> > issue.
> > > >> >
> > > >> > we use 2 lines in every syslog-ng config.. obviously the ip is 
changed 
> > for
> > > >> > public viewing:
> > > >> >
> > > >> > destination remote { udp("1.3.0.1"); };
> > > >> > log { source(src); destination(remote); };
> > > >> >
> > > >> > on all machines
> > > >> > kernel is
> > > >> >
> > > >> > 2.6.26-vs2.3.0.35.6-gentoo
> > > >> >
> > > >> > util-vserver is 0.30.215
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Chuck
> > > >
> > > 
> > > Chuck,
> > > 
> > > Strange, since UDP packets have no guarantees for transmission or
> > > delivery one would imagine they would be sent and dropped. Maybe
> > > syslog-ng has a delivery queue that that holds messages until it can
> > > make a connection to the remote server and since your server is down
> > > the messages build up.
> > > 
> > 
> > after reading about the rsyslog that suddenly occurred to me too... i am 
going 
> > to look into the options available for the config to see if there is a 
queue 
> > and if so maybe it can be disabled. i always thought udp was 'blind' and 
that 
> > syslog-ng would just blindly send out and if it reached the destination 
fine 
> > if not it would not know the difference and continue on.. evidently not..
> >
> 
> UDP is 'blind' in the sense that it does not provide guarantied delivery. 
> 
> What may be happening here is a failure *before* the packet is sent -
> I.E: Your routing is returning "Network unreachable" - so the packet 
> has nowhere to go.
> 
> Perhaps you could provide an alternate route to a packet "/dev/null" 
> (I.E: the "dummy" interface) for when the for-real logger is down.
> I think the stock network bridging module can be set-up to do this for you.
> 

i think i found the answer in syslog changelog:

2.1.4
   Tue, 17 Mar 2009 13:35:00 +0100

   Bugfixes:
   * Fixed a possible DoS condition triggered by a destination port
          unreaachable ICMP packet received from a UDP destination.
     syslog-ng started eating all available memory and CPU until it
          crashed if this happened.
   * Fixed the rate at which files regular were read using the file()
     source.
   * Report connection breaks as a write error instead of reporting
          POLLERR as the write error path reports more sensible information
     in the logs.



> Mike
> > > Edward
> > > 
> > 
> > 
> > 
> 
> 
> 



-- 

Chuck