2009-09-16 08:54:45 +02:00
|
|
|
/*
|
|
|
|
|
* Only give sleepers 50% of their service deficit. This allows
|
|
|
|
|
* them to run sooner, but does not allow tons of sleepers to
|
|
|
|
|
* rip the spread apart.
|
|
|
|
|
*/
|
2011-07-06 14:20:14 +02:00
|
|
|
SCHED_FEAT(GENTLE_FAIR_SLEEPERS, true)
|
2009-09-11 12:31:23 +02:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Place new tasks ahead so that they do not starve already running
|
|
|
|
|
* tasks
|
|
|
|
|
*/
|
2011-07-06 14:20:14 +02:00
|
|
|
SCHED_FEAT(START_DEBIT, true)
|
2009-09-11 12:31:23 +02:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Prefer to schedule the task we woke last (assuming it failed
|
|
|
|
|
* wakeup-preemption), since its likely going to consume data we
|
|
|
|
|
* touched, increases cache locality.
|
|
|
|
|
*/
|
2011-07-06 14:20:14 +02:00
|
|
|
SCHED_FEAT(NEXT_BUDDY, false)
|
2009-09-11 12:31:23 +02:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Prefer to schedule the task that ran last (when we did
|
|
|
|
|
* wake-preempt) as that likely will touch the same data, increases
|
|
|
|
|
* cache locality.
|
|
|
|
|
*/
|
2011-07-06 14:20:14 +02:00
|
|
|
SCHED_FEAT(LAST_BUDDY, true)
|
2009-09-11 12:31:23 +02:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Consider buddies to be cache hot, decreases the likelyness of a
|
|
|
|
|
* cache buddy being migrated away, increases cache locality.
|
|
|
|
|
*/
|
2011-07-06 14:20:14 +02:00
|
|
|
SCHED_FEAT(CACHE_HOT_BUDDY, true)
|
2009-09-11 12:31:23 +02:00
|
|
|
|
2012-10-14 14:28:50 +02:00
|
|
|
/*
|
|
|
|
|
* Allow wakeup-time preemption of the current task:
|
|
|
|
|
*/
|
|
|
|
|
SCHED_FEAT(WAKEUP_PREEMPTION, true)
|
|
|
|
|
|
2011-07-06 14:20:14 +02:00
|
|
|
SCHED_FEAT(HRTICK, false)
|
|
|
|
|
SCHED_FEAT(DOUBLE_TICK, false)
|
|
|
|
|
SCHED_FEAT(LB_BIAS, true)
|
2009-09-11 12:31:23 +02:00
|
|
|
|
2010-10-04 17:03:22 -07:00
|
|
|
/*
|
2014-05-27 13:50:41 -04:00
|
|
|
* Decrement CPU capacity based on time not spent running tasks
|
2010-10-04 17:03:22 -07:00
|
|
|
*/
|
2014-05-27 13:50:41 -04:00
|
|
|
SCHED_FEAT(NONTASK_CAPACITY, true)
|
2011-04-05 17:23:58 +02:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Queue remote wakeups on the target CPU and process them
|
|
|
|
|
* using the scheduler IPI. Reduces rq->lock contention/bounces.
|
|
|
|
|
*/
|
2011-07-06 14:20:14 +02:00
|
|
|
SCHED_FEAT(TTWU_QUEUE, true)
|
2011-07-15 10:35:52 +02:00
|
|
|
|
2020-04-04 13:06:34 +07:00
|
|
|
/*
|
|
|
|
|
* When doing wakeups, attempt to limit superfluous scans of the LLC domain.
|
|
|
|
|
*/
|
|
|
|
|
SCHED_FEAT(SIS_AVG_CPU, false)
|
|
|
|
|
|
sched/rt: Use IPI to trigger RT task push migration instead of pulling
When debugging the latencies on a 40 core box, where we hit 300 to
500 microsecond latencies, I found there was a huge contention on the
runqueue locks.
Investigating it further, running ftrace, I found that it was due to
the pulling of RT tasks.
The test that was run was the following:
cyclictest --numa -p95 -m -d0 -i100
This created a thread on each CPU, that would set its wakeup in iterations
of 100 microseconds. The -d0 means that all the threads had the same
interval (100us). Each thread sleeps for 100us and wakes up and measures
its latencies.
cyclictest is maintained at:
git://git.kernel.org/pub/scm/linux/kernel/git/clrkwllms/rt-tests.git
What happened was another RT task would be scheduled on one of the CPUs
that was running our test, when the other CPU tests went to sleep and
scheduled idle. This caused the "pull" operation to execute on all
these CPUs. Each one of these saw the RT task that was overloaded on
the CPU of the test that was still running, and each one tried
to grab that task in a thundering herd way.
To grab the task, each thread would do a double rq lock grab, grabbing
its own lock as well as the rq of the overloaded CPU. As the sched
domains on this box was rather flat for its size, I saw up to 12 CPUs
block on this lock at once. This caused a ripple affect with the
rq locks especially since the taking was done via a double rq lock, which
means that several of the CPUs had their own rq locks held while trying
to take this rq lock. As these locks were blocked, any wakeups or load
balanceing on these CPUs would also block on these locks, and the wait
time escalated.
I've tried various methods to lessen the load, but things like an
atomic counter to only let one CPU grab the task wont work, because
the task may have a limited affinity, and we may pick the wrong
CPU to take that lock and do the pull, to only find out that the
CPU we picked isn't in the task's affinity.
Instead of doing the PULL, I now have the CPUs that want the pull to
send over an IPI to the overloaded CPU, and let that CPU pick what
CPU to push the task to. No more need to grab the rq lock, and the
push/pull algorithm still works fine.
With this patch, the latency dropped to just 150us over a 20 hour run.
Without the patch, the huge latencies would trigger in seconds.
I've created a new sched feature called RT_PUSH_IPI, which is enabled
by default.
When RT_PUSH_IPI is not enabled, the old method of grabbing the rq locks
and having the pulling CPU do the work is implemented. When RT_PUSH_IPI
is enabled, the IPI is sent to the overloaded CPU to do a push.
To enabled or disable this at run time:
# mount -t debugfs nodev /sys/kernel/debug
# echo RT_PUSH_IPI > /sys/kernel/debug/sched_features
or
# echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched_features
Update: This original patch would send an IPI to all CPUs in the RT overload
list. But that could theoretically cause the reverse issue. That is, there
could be lots of overloaded RT queues and one CPU lowers its priority. It would
then send an IPI to all the overloaded RT queues and they could then all try
to grab the rq lock of the CPU lowering its priority, and then we have the
same problem.
The latest design sends out only one IPI to the first overloaded CPU. It tries to
push any tasks that it can, and then looks for the next overloaded CPU that can
push to the source CPU. The IPIs stop when all overloaded CPUs that have pushable
tasks that have priorities greater than the source CPU are covered. In case the
source CPU lowers its priority again, a flag is set to tell the IPI traversal to
restart with the first RT overloaded CPU after the source CPU.
Parts-suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Joern Engel <joern@purestorage.com>
Cc: Clark Williams <williams@redhat.com>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/20150318144946.2f3cc982@gandalf.local.home
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-03-18 14:49:46 -04:00
|
|
|
#ifdef HAVE_RT_PUSH_IPI
|
|
|
|
|
/*
|
|
|
|
|
* In order to avoid a thundering herd attack of CPUs that are
|
|
|
|
|
* lowering their priorities at the same time, and there being
|
|
|
|
|
* a single CPU that has an RT task that can migrate and is waiting
|
|
|
|
|
* to run, where the other CPUs will try to take that CPUs
|
|
|
|
|
* rq lock and possibly create a large contention, sending an
|
|
|
|
|
* IPI to that CPU and let that CPU push the RT task to where
|
|
|
|
|
* it should go may be a better scenario.
|
|
|
|
|
*/
|
|
|
|
|
SCHED_FEAT(RT_PUSH_IPI, true)
|
|
|
|
|
#endif
|
|
|
|
|
|
2011-07-06 14:20:14 +02:00
|
|
|
SCHED_FEAT(FORCE_SD_OVERLAP, false)
|
|
|
|
|
SCHED_FEAT(RT_RUNTIME_SHARE, true)
|
2012-04-17 13:38:40 +02:00
|
|
|
SCHED_FEAT(LB_MIN, false)
|
2015-09-11 16:10:59 +02:00
|
|
|
SCHED_FEAT(ATTACH_AGE_LOAD, true)
|
|
|
|
|
|
BACKPORT: sched/fair: Add util_est on top of PELT
The util_avg signal computed by PELT is too variable for some use-cases.
For example, a big task waking up after a long sleep period will have its
utilization almost completely decayed. This introduces some latency before
schedutil will be able to pick the best frequency to run a task.
The same issue can affect task placement. Indeed, since the task
utilization is already decayed at wakeup, when the task is enqueued in a
CPU, this can result in a CPU running a big task as being temporarily
represented as being almost empty. This leads to a race condition where
other tasks can be potentially allocated on a CPU which just started to run
a big task which slept for a relatively long period.
Moreover, the PELT utilization of a task can be updated every [ms], thus
making it a continuously changing value for certain longer running
tasks. This means that the instantaneous PELT utilization of a RUNNING
task is not really meaningful to properly support scheduler decisions.
For all these reasons, a more stable signal can do a better job of
representing the expected/estimated utilization of a task/cfs_rq.
Such a signal can be easily created on top of PELT by still using it as
an estimator which produces values to be aggregated on meaningful
events.
This patch adds a simple implementation of util_est, a new signal built on
top of PELT's util_avg where:
util_est(task) = max(task::util_avg, f(task::util_avg@dequeue))
This allows to remember how big a task has been reported by PELT in its
previous activations via f(task::util_avg@dequeue), which is the new
_task_util_est(struct task_struct*) function added by this patch.
If a task should change its behavior and it runs longer in a new
activation, after a certain time its util_est will just track the
original PELT signal (i.e. task::util_avg).
The estimated utilization of cfs_rq is defined only for root ones.
That's because the only sensible consumer of this signal are the
scheduler and schedutil when looking for the overall CPU utilization
due to FAIR tasks.
For this reason, the estimated utilization of a root cfs_rq is simply
defined as:
util_est(cfs_rq) = max(cfs_rq::util_avg, cfs_rq::util_est::enqueued)
where:
cfs_rq::util_est::enqueued = sum(_task_util_est(task))
for each RUNNABLE task on that root cfs_rq
It's worth noting that the estimated utilization is tracked only for
objects of interests, specifically:
- Tasks: to better support tasks placement decisions
- root cfs_rqs: to better support both tasks placement decisions as
well as frequencies selection
Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Joel Fernandes <joelaf@google.com>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Morten Rasmussen <morten.rasmussen@arm.com>
Cc: Paul Turner <pjt@google.com>
Cc: Rafael J . Wysocki <rafael.j.wysocki@intel.com>
Cc: Steve Muckle <smuckle@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Todd Kjos <tkjos@android.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Viresh Kumar <viresh.kumar@linaro.org>
Link: http://lkml.kernel.org/r/20180309095245.11071-2-patrick.bellasi@arm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-09 09:52:42 +00:00
|
|
|
/*
|
|
|
|
|
* UtilEstimation. Use estimated CPU utilization.
|
|
|
|
|
*/
|
2018-03-09 09:52:45 +00:00
|
|
|
SCHED_FEAT(UTIL_EST, true)
|
BACKPORT: sched/fair: Add util_est on top of PELT
The util_avg signal computed by PELT is too variable for some use-cases.
For example, a big task waking up after a long sleep period will have its
utilization almost completely decayed. This introduces some latency before
schedutil will be able to pick the best frequency to run a task.
The same issue can affect task placement. Indeed, since the task
utilization is already decayed at wakeup, when the task is enqueued in a
CPU, this can result in a CPU running a big task as being temporarily
represented as being almost empty. This leads to a race condition where
other tasks can be potentially allocated on a CPU which just started to run
a big task which slept for a relatively long period.
Moreover, the PELT utilization of a task can be updated every [ms], thus
making it a continuously changing value for certain longer running
tasks. This means that the instantaneous PELT utilization of a RUNNING
task is not really meaningful to properly support scheduler decisions.
For all these reasons, a more stable signal can do a better job of
representing the expected/estimated utilization of a task/cfs_rq.
Such a signal can be easily created on top of PELT by still using it as
an estimator which produces values to be aggregated on meaningful
events.
This patch adds a simple implementation of util_est, a new signal built on
top of PELT's util_avg where:
util_est(task) = max(task::util_avg, f(task::util_avg@dequeue))
This allows to remember how big a task has been reported by PELT in its
previous activations via f(task::util_avg@dequeue), which is the new
_task_util_est(struct task_struct*) function added by this patch.
If a task should change its behavior and it runs longer in a new
activation, after a certain time its util_est will just track the
original PELT signal (i.e. task::util_avg).
The estimated utilization of cfs_rq is defined only for root ones.
That's because the only sensible consumer of this signal are the
scheduler and schedutil when looking for the overall CPU utilization
due to FAIR tasks.
For this reason, the estimated utilization of a root cfs_rq is simply
defined as:
util_est(cfs_rq) = max(cfs_rq::util_avg, cfs_rq::util_est::enqueued)
where:
cfs_rq::util_est::enqueued = sum(_task_util_est(task))
for each RUNNABLE task on that root cfs_rq
It's worth noting that the estimated utilization is tracked only for
objects of interests, specifically:
- Tasks: to better support tasks placement decisions
- root cfs_rqs: to better support both tasks placement decisions as
well as frequencies selection
Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Joel Fernandes <joelaf@google.com>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Morten Rasmussen <morten.rasmussen@arm.com>
Cc: Paul Turner <pjt@google.com>
Cc: Rafael J . Wysocki <rafael.j.wysocki@intel.com>
Cc: Steve Muckle <smuckle@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Todd Kjos <tkjos@android.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Viresh Kumar <viresh.kumar@linaro.org>
Link: http://lkml.kernel.org/r/20180309095245.11071-2-patrick.bellasi@arm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-09 09:52:42 +00:00
|
|
|
|
2015-01-13 13:45:51 +00:00
|
|
|
/*
|
|
|
|
|
* Energy aware scheduling. Use platform energy model to guide scheduling
|
|
|
|
|
* decisions optimizing for energy efficiency.
|
|
|
|
|
*/
|
2016-09-20 18:42:22 -07:00
|
|
|
#ifdef CONFIG_DEFAULT_USE_ENERGY_AWARE
|
|
|
|
|
SCHED_FEAT(ENERGY_AWARE, true)
|
|
|
|
|
#else
|
2015-01-13 13:45:51 +00:00
|
|
|
SCHED_FEAT(ENERGY_AWARE, false)
|
2016-09-20 18:42:22 -07:00
|
|
|
#endif
|
2017-12-07 19:43:46 +00:00
|
|
|
|
|
|
|
|
/*
|
2020-04-04 13:06:34 +07:00
|
|
|
* Minimum capacity capping. Keep track of minimum capacity factor when
|
|
|
|
|
* minimum frequency available to a policy is modified.
|
|
|
|
|
* If enabled, this can be used to inform the scheduler about capacity
|
|
|
|
|
* restrictions.
|
2017-12-07 19:43:46 +00:00
|
|
|
*/
|
2020-04-04 13:06:34 +07:00
|
|
|
SCHED_FEAT(MIN_CAPACITY_CAPPING, false)
|
2017-12-11 14:56:12 +00:00
|
|
|
|
2020-04-04 13:06:34 +07:00
|
|
|
/*
|
|
|
|
|
* Enforce the priority of candidates selected by find_best_target()
|
|
|
|
|
* ON: If the target CPU saves any energy, use that.
|
|
|
|
|
* OFF: Use whichever of target or backup saves most.
|
|
|
|
|
*/
|
|
|
|
|
SCHED_FEAT(FBT_STRICT_ORDER, true)
|
2018-01-16 19:01:05 +09:00
|
|
|
|
2018-03-22 14:43:33 +09:00
|
|
|
#ifdef CONFIG_SCHED_EMS
|
|
|
|
|
SCHED_FEAT(EXYNOS_MS, true)
|
2018-01-16 19:01:05 +09:00
|
|
|
#else
|
2018-03-22 14:43:33 +09:00
|
|
|
SCHED_FEAT(EXYNOS_MS, false)
|
2018-01-16 19:01:05 +09:00
|
|
|
#endif
|
|
|
|
|
|
