Subject: Re: [vserver] all memory consumed [resolved]
From: Chuck <chuck@sbbsnet.net>
Date: Fri, 23 Oct 2009 15:29:46 -0400

 Fri, 23 Oct 2009 15:29:46 -0400
On Friday 23 October 2009, Herbert Poetzl wrote:
> On Fri, Oct 23, 2009 at 10:35:12AM -0400, Chuck wrote:
> > On Friday 23 October 2009, Michael S. Zick wrote:
> > > On Fri October 23 2009, Chuck wrote:
> > > > On Friday 23 October 2009, Edward Capriolo wrote:
> > > > > On Fri, Oct 23, 2009 at 6:58 AM, Chuck <chuck@sbbsnet.net> wrote:
> > > > > > On Friday 23 October 2009, Konrad Korzeniowski wrote:
> > > > > >> Hi,
> > > > > >>
> > > > > >> I don't know syslog-ng,  but there is rsyslog solution:
> > > > > >> http://www.rsyslog.com/doc-rsyslog reliable forwarding.html
> > > > > >>
> > > > > >
> > > > > > thanks. i will look into it.
> > > > > >
> > > > > >> rgrds.,
> > > > > >> KK
> > > > > >>
> > > > > >> [2009-10-23 04:16] Chuck:
> > > > > >> > I  doubt this is a vserver problem, but I am placing this here 
just 
> > in
> > > > > > case
> > > > > >> > there may be something.
> > > > > >> >
> > > > > >> > we run a remote logging server that every host and guest logs 
to in
> > > > > > addition
> > > > > >> > to local logs. when all is well this works just fine.
> > > > > >> >
> > > > > >> > if the remote log server is taken down, suddenly most of the 
hosts 
> > > > begin
> > > > > >> > chewing up all available memory and swap until they crash. 
> > disabling
> > > > > > remote
> > > > > >> > logging on everything cured this but obviously is not what we 
want.
> > > > > >> >
> > > > > >> > i am also posting this in the gentoo forum in case it is a 
> > syslog-ng or
> > > > > > other
> > > > > >> > issue.
> > > > > >> >
> > > > > >> > we use 2 lines in every syslog-ng config.. obviously the ip is 
> > changed 
> > > > for
> > > > > >> > public viewing:
> > > > > >> >
> > > > > >> > destination remote { udp("1.3.0.1"); };
> > > > > >> > log { source(src); destination(remote); };
> > > > > >> >
> > > > > >> > on all machines
> > > > > >> > kernel is
> > > > > >> >
> > > > > >> > 2.6.26-vs2.3.0.35.6-gentoo
> > > > > >> >
> > > > > >> > util-vserver is 0.30.215
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Chuck
> > > > > >
> > > > > 
> > > > > Chuck,
> > > > > 
> > > > > Strange, since UDP packets have no guarantees for transmission or
> > > > > delivery one would imagine they would be sent and dropped. Maybe
> > > > > syslog-ng has a delivery queue that that holds messages until it can
> > > > > make a connection to the remote server and since your server is down
> > > > > the messages build up.
> > > > > 
> > > > 
> > > > after reading about the rsyslog that suddenly occurred to me too... i 
am 
> > going 
> > > > to look into the options available for the config to see if there is a 
> > queue 
> > > > and if so maybe it can be disabled. i always thought udp was 'blind' 
and 
> > that 
> > > > syslog-ng would just blindly send out and if it reached the 
destination 
> > fine 
> > > > if not it would not know the difference and continue on.. evidently 
not..
> > > >
> > > 
> > > UDP is 'blind' in the sense that it does not provide guarantied 
delivery. 
> > > 
> > > What may be happening here is a failure *before* the packet is sent -
> > > I.E: Your routing is returning "Network unreachable" - so the packet 
> > > has nowhere to go.
> > > 
> > > Perhaps you could provide an alternate route to a packet "/dev/null" 
> > > (I.E: the "dummy" interface) for when the for-real logger is down.
> > > I think the stock network bridging module can be set-up to do this for 
you.
> > > 
> > 
> > i think i found the answer in syslog changelog:
> > 
> > 2.1.4
> >    Tue, 17 Mar 2009 13:35:00 +0100
> > 
> >    Bugfixes:
> >    * Fixed a possible DoS condition triggered by a destination port
> >           unreaachable ICMP packet received from a UDP destination.
> >      syslog-ng started eating all available memory and CPU until it
> >           crashed if this happened.
> >    * Fixed the rate at which files regular were read using the file()
> >      source.
> >    * Report connection breaks as a write error instead of reporting
> >           POLLERR as the write error path reports more sensible 
information
> >      in the logs.
> 
> just out of curiosity, the one you used was is a
> distribution provided packaged version, and if yes
> what distribution, if you do not mind telling?
> 

all versions of syslog-ng were installed via gentoo portage. some older 
vservers had v2.0.6 and others had 2.0.9 or 2.1.3. from what i can gather 
this problem was with 2.1.3 and possibly earlier versions although i did not 
research it... i updated every syslog-ng in our entire network to 2.1.4. 

after the logging server hardware upgrade is done and the server is 
reinstalled and stable i will enable remote logging again and we will 
deliberately test this on a host by host basis to be sure it is all cured.




> TIA,
> Herbert
> 
> > > Mike
> > > > > Edward
> > > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> > > 
> > 
> > 
> > 
> > -- 
> > 
> > Chuck
> 



-- 

Chuck