Subject: Re: [vserver] Kernel Thread's accounted to a User Context
From: Eric Keller <ekeller@Princeton.EDU>
Date: Sat, 15 Dec 2007 10:35:30 -0500

As an update... I added counters to vx_schedule() (which gets called by 
schedule()) and vx_need_resched() (which gets called by scheduler_tick()).

What I found is that a process does indeed run many ticks from when 
vx_need_resched() gets called and runs out of tokens and sets 
TIF_NEED_RESCHED to when schedule gets called to pick a new task.  Since 
tokens don't go below zero, and since I guess enough time has passed 
from when the task was flagged to when vx_schedule() does get called, 
the process gets some more tokens and never gets put on hold.

I added my counters to /proc/virtual/<context>/sched, and here's a 
stripped down

[#] more /proc/virtual/777/sched
FillRate:              2,1
Interval:             40,8
TokensMin:             4
TokensMax:           200
PrioBias:              0
cpu 0: 364(usr) 1074(sys) 561(hold) 5226(tokentime) 0(idle) R- 0 4 200 
2/40 1/8 0 0
     hold 0 zero_tok(k) 732 zero_tok(u) 0 vx_sched(k) 64 vx_sched(u) 343
     consume(k) 323 consume(u) 368 slice_zero 791

[#] more /proc/virtual/777/sched
FillRate:              2,1
Interval:             40,8
TokensMin:             4
TokensMax:           200
PrioBias:              0
cpu 0: 364(usr) 1353(sys) 561(hold) 5260(tokentime) 0(idle) R- 0 4 200 
2/40 1/8 0 0
     hold 0 zero_tok(k) 977 zero_tok(u) 0 vx_sched(k) 81 vx_sched(u) 343
     consume(k) 357 consume(u) 368 slice_zero 1053

The first line (where it says cpu 0) is what vserver currently prints 
out (with me adding text in parentheses). 
The second line:
* hold: how many times the code vx_hold task gets called
* zero_tok: how many times vx_need_resched (and therefore 
scheduler_tick) gets called while the token count is 0 (the k in 
parentheses means a kernel thread, u means user thread)
* vx_sched: how many times vx_schedule gets called (again with k for 
kernel and u for user)
The third line:
* consume: how many times _vx_consume_token gets called (again with k 
for kernel and u for user)
* zero_slice: how many times vx_need_resched returns true (i.e. (slice==0)).

So, from this one can see that 279 sys ticks, 245 zero token ticks, 
vx_schedule was called 17 times, 34 tokens were consumed.  Note that the 
245 zero token ticks + 34 tokens consumed = 279 sys ticks.

When I run just a user process, there are no zero_tok ticks and it does 
indeed go on hold.

Any idea on how to make it so that the task immediately holds (is it bad 
to call schedule() from scheduler_tick())?

Thanks,
Eric