Subject: Re: [vserver] throttler script
From:Sandino Araico Sánchez <sandino@sandino.net>
Date: Mon, 21 Jan 2013 18:47:11 -0600
Mon, 21 Jan 2013 18:47:11 -0600
On 18/01/13 02:50, Fiedler Roman wrote:
> Von: Sandino Araico Sánchez [mailto:sandino@sandino.net] 
>
>>> On 17/01/13 08:25, Phil Parris wrote: 
>>> Im running into similar issues.  Please provide the script.  One thing i thought
about was to change cron source and add in a random sleep.
>> I wrote a couple of throttler scripts two years ago for a host with 50 vservers and
load >250. 
>>
>> The ionice-updatedb looks for known I/O consumming processes in the vserver, renices
them and puts the most agressive ones in the throttler.
>>
>> The throttler script pauses the process for some seconds (depending on the system
load) and lets it run for one second. Two methods are avilable SIGSTOP and freezer but
I stopped using the freezer on kernels 2.6.32 because some processes used to stay forever
in status DN and were impossible to kill until next host server reboot. I have been
using the SIGSTOP method since then.
> Did you stop only single processes or whole process group/all processes in guest?
If you used stopping of single processes: did you observe any problems with parents
getting confused when their child got a signal or did not respond to parent/child communication
in time?
One throttler for each process. The ionice-updatedb script runs one
throttler for each io-abusing process so in the host server you could
observe many throttler processes throttling one guest process each.

When I use the freezer the parent process does not notice about it's
frozen child, but it has happened that some of the frozen children stay
blocked forever in status DN until next host reboot and I have not
checked in recent 3.x versions if this forever-blocking has been fixed.

When I use SIGSTOP I have seen some parents notice and detach the
stopped child process. Some examples of this detaching can be observed
running emerge or revdep-rebuild in a terminal or some bash scripts
running a list of rsync commands.
>
>

-- 
Sandino Araico Sánchez 
http://sandino.net



On 18/01/13 02:50, Fiedler Roman wrote:
Von: Sandino Araico Sánchez [mailto:sandino@sandino.net] 

On 17/01/13 08:25, Phil Parris wrote: 
Im running into similar issues.  Please provide the script.  One thing i thought about was to change cron source and add in a random sleep.
I wrote a couple of throttler scripts two years ago for a host with 50 vservers and load >250. 

The ionice-updatedb looks for known I/O consumming processes in the vserver, renices them and puts the most agressive ones in the throttler.

The throttler script pauses the process for some seconds (depending on the system load) and lets it run for one second. Two methods are avilable SIGSTOP and freezer but I stopped using the freezer on kernels 2.6.32 because some processes used to stay forever in status DN and were impossible to kill until next host server reboot. I have been using the SIGSTOP method since then.
Did you stop only single processes or whole process group/all processes in guest? If you used stopping of single processes: did you observe any problems with parents getting confused when their child got a signal or did not respond to parent/child communication in time?
One throttler for each process. The ionice-updatedb script runs one throttler for each io-abusing process so in the host server you could observe many throttler processes throttling one guest process each.

When I use the freezer the parent process does not notice about it's frozen child, but it has happened that some of the frozen children stay blocked forever in status DN until next host reboot and I have not checked in recent 3.x versions if this forever-blocking has been fixed.

When I use SIGSTOP I have seen some parents notice and detach the stopped child process. Some examples of this detaching can be observed running emerge or revdep-rebuild in a terminal or some bash scripts running a list of rsync commands.



-- 
Sandino Araico Sánchez 
http://sandino.net