Subject: Re: [vserver] Guest hourly/daily/... cron job parallel execution
From: "Bendtsen, Jon" <Jon.Bendtsen@laerdal.dk>
Date: Thu, 17 Jan 2013 10:35:43 +0000

 Thu, 17 Jan 2013 10:35:43 +0000
On 17/01/2013, at 11.05, Fiedler Roman <Roman.Fiedler@ait.ac.at> wrote:

>> -----Ursprüngliche Nachricht-----
>> Von: Bendtsen, Jon [mailto:Jon.Bendtsen@laerdal.dk]
>> 
>> On 17/01/2013, at 10.07, Fiedler Roman <Roman.Fiedler@ait.ac.at>
>> wrote:
>> 
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Bendtsen, Jon [mailto:Jon.Bendtsen@laerdal.dk]
>>>> 
>>>> On 17/01/2013, at 09.43, Fiedler Roman <Roman.Fiedler@ait.ac.at>
>> wrote:

[cuuuuuut]


>> Or maybe just pick some random time, sort of what old ethernet did when
>> there is a collision because 2 or more computers tried to transmit data at the
>> same time.
> 
> Yes. Or use the context-ID modulo something.

good idea

[cuuuuuut]

>> I see your point. If so maybe we should split up daily/weekly/hourly cronjobs
>> even further, such that all the same programs which are hard linked together
>> can use the same cache and ram for instructions. But running locate to update
>> the database might not be a good choice to all run at the same time.
> 
> I would expect, that caching of the application code itself is not the main performance
boost. It is more about getting rid of the bottle-necks. As you noted, too many locate-updates
in parallel will kill disk performance. But also loosing the usual disk-cache benefit
might be problematic. If e.g. just a few databases run a job, the relevant db-content
from disk is likely the end up completely in OS RAM cache. All disk-reads from db will
return immediately. When too many DBs execute in parallel, disk content of one process
will be put in cache, eliminating pages soon need again by another job again. 

This is not really any different from running many other kinds of virtual machines in
parallel. Your storage system needs to perform.




>> Maybe we need kind of scheduler in the kernel that notices which processes
>> in the different guests are hard linked and then prioritizing running those?
>> 
>> Maybe we need some new kind of scheduler system that is made with
>> virtualization in mind, such that it adjusts to when there is a low load and then
>> tries to run the maintenance scripts, meaning that sometimes scripts are run
>> with only 20 hours between them, other times it might be 36 hours.
> 
> In my opinion, such an intelligent/learning scheduler would allow significance increase
in execution performance, but looking of the simplicity of current cron, I think, that
such a program is years away, if even written ever.

probably. But I can imagine other ways to do it. I already use the at system for some
tasks. Like my backup scripts. If the load is too high or the system is other wise not
ready, I have my backup scripts call 'at now + 1 hour' to run the script itself 1 hour
later. If it is still not ready, I call +2 hours, +4 and finally +8 hours.

But it all hinges on the guest knowing the over all system load of the virtualization
host, just like a process in a normal stand alone system can see the system load. Maybe
we need a new field in the load data? Or a new syscall to give those data?



[cuuuuut]

>> I am not starting cron daemon with nice. I put nice in side the crontab file, like
>> these examples:
>> 
>> 37 0	* * * 	root	nice -n 15
>> /usr/local/sbin/AD integration/find disabled users from AD in groups.sh
>> 0,15,30,45 *	* * *	root nice -n 5
>> /usr/local/sbin/AD integration/merge AD groups with unix.sh
> 
> Ah I see.
> 
> My current solution is:
> * Check for each guest, if cron scheduler is installed inside
> * Check if guest cron would run hourly/daily by himself, if yes let him do it. A misconfigured/malicious
guest can always run any process at any time, this has to be addressed via other means
> * If guest cron is installed,  cron.daily/hourly ... directories exist but guest scheduler
does NOT run those jobs from etc/crontab (cooperative guest) by itself, then start those
via vserver-exec
> * Make sure to run only a given number of those guest processes in parallel
> 
> So all you need to do is to install cron in guest but remove run-scripts directives
for hourly/daily from /etc/crontab to opt in.

Yes, that is a method too, but I would not really like that outsiders, like hosting
providers ran tasks inside my guest, at least not without my beforehand knowledge. I
guess it works if one administrates both the host and the guests.



JonB