I have faced a sequence where the Idle Load Balance was sometime not
triggered for a while on my platform, in the following scenario:
CPU 0 and CPU 1 are running tasks and CPU 2 is idle
CPU 1 kicks the Idle Load Balance
CPU 1 selects CPU 2 as the new Idle Load Balancer
CPU 2 sets NOHZ_BALANCE_KICK for CPU 2
CPU 2 sends a reschedule IPI to CPU 2
While CPU 3 wakes up, CPU 0 or CPU 1 migrates a waking up task A on CPU 2
CPU 2 finally wakes up, runs task A and discards the Idle Load Balance
task A quickly goes back to sleep (before a tick occurs on CPU 2)
CPU 2 goes back to idle with NOHZ_BALANCE_KICK set
Whenever CPU 2 will be selected as the ILB, no reschedule IPI will be sent
because NOHZ_BALANCE_KICK is already set and no Idle Load Balance will be
performed.
We must wait for the sched softirq to be raised on CPU 2 thanks to another
part the kernel to come back to clear NOHZ_BALANCE_KICK.
The proposed solution clears NOHZ_BALANCE_KICK in schedule_ipi if
we can't raise the sched_softirq for the Idle Load Balance.
Change since V1:
- move the clear of NOHZ_BALANCE_KICK in got_nohz_idle_kick if the ILB
can't run on this CPU (as suggested by Peter)
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1370419991-13870-1-git-send-email-vincent.guittot@linaro.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Git-commit: 873b4c65b519fd769940eb281f77848227d4e5c1
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[smuckle@codeaurora.org: minor merge resolution for 3.4 in scheduler_ipi()]
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Change-Id: I3548612057cccc2ecc29429c129c44183083831f
When doing performance analysis it can be useful to see exactly
what is going on with the load balancer - when it runs and why
exactly it may not be redistributing load.
This additional tracepoint will show the idle context of the
load balance operation (idle, not idle, newly idle), various
values from the load balancing operation, the final result,
and the new balance interval.
Change-Id: I9e5c97ae3878bea44e60d189ff3cec2275f2c75e
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Add a cold_boot parameter which supplements the
boot_reason sysctl entry with information about
whether the system was booted from cold or warm state.
/proc/sys/kernel/cold_boot entry is updated with 1 or 0 when
system was booted from cold or warm boot state respecitively.
CRs-Fixed: 461256
Change-Id: I2bc5d80c8f26eb9e9dbb4b34960d991a51a224e4
Signed-off-by: David Keitel <dkeitel@codeaurora.org>
There are some new processors whose TSC clocksource won't stop during
suspend. Currently, after system resumes, kernel will use persistent
clock or RTC to compensate the sleep time, but with these nonstop
clocksources, we could skip the special compensation from external
sources, and just use current clocksource for time recounting.
This can solve some time drift bugs caused by some not-so-accurate or
error-prone RTC devices.
The current way to count suspended time is first try to use the persistent
clock, and then try the RTC if persistent clock can't be used. This
patch will change the trying order to:
suspend-nonstop clocksource -> persistent clock -> RTC
When counting the sleep time with nonstop clocksource, use an accurate way
suggested by Jason Gunthorpe to cover very large delta cycles.
CRs-fixed: 536881
Change-Id: I59c21b1ce5ae81cebc8cf3e817d9b3f951302a3d
Signed-off-by: Feng Tang <feng.tang@intel.com>
[jstultz: Small optimization, avoiding re-reading the clocksource]
Signed-off-by: John Stultz <john.stultz@linaro.org>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: e445cf1c4257cc0238d72e4129eb4739f46fd3de)
[sboyd: Took tk and clock local variables]
Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
The leap second rework unearthed another issue of inconsistent data.
On timekeeping_resume() the timekeeper data is updated, but nothing
calls timekeeping_update(), so now the update code in the timer
interrupt sees stale values.
This has been the case before those changes, but then the timer
interrupt was using stale data as well so this went unnoticed for quite
some time.
Add the missing update call, so all the data is consistent everywhere.
CRs-fixed: 536881
Change-Id: I5f7d5219749a3f6330d074ef8866533ebda3bfca
Reported-by: Andreas Schwab <schwab@linux-m68k.org>
Reported-and-tested-by: "Rafael J. Wysocki" <rjw@sisk.pl>
Reported-and-tested-by: Martin Steigerwald <Martin@lichtvoll.de>
Cc: LKML <linux-kernel@vger.kernel.org>
Cc: Linux PM list <linux-pm@vger.kernel.org>
Cc: John Stultz <johnstul@us.ibm.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>,
Cc: Prarit Bhargava <prarit@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: John Stultz <johnstul@us.ibm.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: 3e997130bd2e8c6f5aaa49d6e3161d4d29b43ab0
Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
The change that long-ago added the interrupts-disabled warning in
smp_call_function_single() cited the following deadlock as the reason:
CPU A CPU B
Disable interrupts
smp_call_function()
Take call_lock
Send IPIs
Wait for all cpus to acknowledge IPI
CPU A has not responded, spin waiting
for cpu A to respond, holding call_lock
smp_call_function()
Spin waiting for call_lock
Deadlock Deadlock
Such a deadlock is not possible however, in the event that
smp_call_function_single() is called for the currently-executing CPU.
While not a common usecase, relaxing the warning may simplify some
driver implementations.
The 'acpuclock-krait' driver in the MSM ARM sub-architecture provides
one example, where smp_call_function_single() is sometimes called from
a cpuidle path with interrupts disabled, but only ever from the target
CPU. Other callers of that same function may cross-call, but only do so
when interrupts are enabled.
Change-Id: I647b3b46fe6aa4dfb06f7750396986b80b92d700
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
watchdog driver creates per-cpu kernel threads that are bound to
separate cpus. Use of smp_processor_id() by such threads in
__touch_watchdog() is perfectly fine, as they are generally pinned to
the cpu on which they are running.
During cpu offline event, however, affinity for the watchdog thread
bound to dying cpu is broken before it is killed. There is a small
time window between affinity broken and thread dying in which thread
can run on a cpu with its affinity not set to that cpu exclusively.
This will trigger a warning from use of smp_processor_id(), which is
harmless in this case as it will provide the required heartbeat on cpu
where it is running and moreover that thread will be shortly killed.
Mute the harmless warning by use of raw_smp_processor_id() in
__touch_watchdog(). This seems less intrusive fix than killing threads
in CPU_DOWN_PREPARE event handler.
Change-Id: I7fa22ff529aeea0ec3d5610cfec87aea92cf95a0
CRs-Fixed: 517188
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
The WARN_ON_ONCE() calls at the beginning of try_to_wake_up_local()
were recently converted from BUG_ON() calls. If these hit it indicates
something is wrong and that may contribute to other system instability.
To eliminate the risk of an instance of one of these errors going
un-noticed because there was an earlier instance that occured long ago,
change to WARN_ON(). If there ever is a flood of these there are bigger
problems.
Change-Id: I392832e2b6ec24b3569b001b1af9ecd4ed6828e7
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
try_to_wake_up_local() should only be invoked to wake up another
task in the same runqueue and BUG_ON()s are used to enforce the
rule. Missing try_to_wake_up_local() can stall workqueue
execution but such stalls are likely to be finite either by
another work item being queued or the one blocked getting
unblocked. There's no reason to trigger BUG while holding rq
lock crashing the whole system.
Convert BUG_ON()s in try_to_wake_up_local() to WARN_ON_ONCE()s.
Change-Id: I75fdaaf4dcaefcf3893e0404d98c8aa91f89934d
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Steven Rostedt <rostedt@goodmis.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20130318192234.GD3042@htj.dyndns.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: 383efcd00053ec40023010ce5034bd702e7ab373
CRs-Fixed: 519942
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
migrate_tasks() uses _pick_next_task_rt() to get tasks from the
real-time runqueues to be migrated. When rt_rq is throttled
_pick_next_task_rt() won't return anything, in which case
migrate_tasks() can't move all threads over and gets stuck in an
infinite loop.
Instead unthrottle rt runqueues before migrating tasks.
Additionally: move unthrottle_offline_cfs_rqs() to rq_offline_fair()
Change-Id: If8a4a399f1a14b7f4789c1b205dcfadbde555214
Signed-off-by: Peter Boonstoppel <pboonstoppel@nvidia.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Paul Turner <pjt@google.com>
Link: http://lkml.kernel.org/r/5FBF8E85CA34454794F0F7ECBA79798F379D3648B7@HQMAIL04.nvidia.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Git-commit: a4c96ae319b8047f62dedbe1eac79e321c185749
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
This reverts commit 749c4210f338496abf2e16ef51308ff756b34f7b.
This patch is a hack intended to solve a particular instance of a more
generic problem. The cause of the spindump timeout has been fixed with
the commit f37a3dc536e4ecd72961a2d9c77f5587c5c55d8f. Furthermore this
change also introduces a race condition since the printk is actually
intended to be protected with the spinlock.
Change-Id: I8ca057f47760c7bb5b826cceeac92ae5e124e904
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
cpu3 stuck in printk more time in spin lock low_water_lock cause cpu0
get spin lock fail and system crashed.
CRs-Fixed: 521570
Change-Id: I75356a4b4171ae2888ce6cce792f569b5ca8cdcf
Signed-off-by: Tingting Yang <tingting@codeaurora.org>
pm_qos_work_fn() calls pm_qos_update_request() which does
cancel_delayed_work_sync() on the same pm_qos_work_fn()
causing deadlock. Fix this by updating the target with
default timeout value directly instead of calling
pm_qos_update_request().
CRs-Fixed: 526216
Change-Id: I7de2fb1c89f87b0ebf7427116b8920aec50d5b2b
Signed-off-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
Data corruptions in the kernel often end up in system crashes that
are easier to debug closer to the time of detection. Specifically,
if we do not panic immediately after lock or list corruptions have been
detected, the problem context is lost in the ensuing system mayhem.
Add support for allowing system crash immediately after such corruptions
are detected. The CONFIG option controls the enabling/disabling of the
feature.
Change-Id: I9b2eb62da506a13007acff63e85e9515145909ff
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
When legacy wakelock code was removed in commit
f85607a715, some of the code
for moving calls to sys_sync() from suspend paths into a
workqueue item had not been properly removed. Specifically,
one of the call sites to suspend_sys_sync_wait() has been
mistakenly replaced with a call to sys_sync(), which is not
necessary because the corresponding instance of
suspend_sys_sync_queue() was already replaced with
sys_sync(). Clean up the remnants of the legacy wakelock
code by removing the extraneous call to sys_sync() and
restoring some of the surrounding printk statements that
had been moved to suspend_sys_sync_queue() and subsequently
lost.
CRs-Fixed: 498669
Change-Id: Ifb2ede7808560f456c824d3d6359a4541c51b73f
Signed-off-by: Stepan Moskovchenko <stepanm@codeaurora.org>
Add ftrace event trace_sched_cpu_hotplug to track cpu
hot-add and hot-remove events.
This is useful in a variety of power, performance and
debug analysis scenarios.
Change-Id: I5d202c7a229ffacc3aafb7cf9afee0b0ee7b0931
Signed-off-by: Arun Bharadwaj <abharadw@codeaurora.org>
Add a new tracepoint trace_sched_enq_deq_task to track
per-cpu rt and non-rt cpu_load during task enqueue
and dequeue.
This is useful to visualize and compare the load on
different cpus and also to understand how balanced
the load is at any point of time.
Note: We only print cpu_load[0] because we only care about
the most recent load history for tracking load balancer
effectiveness.
Change-Id: I46f0bb84e81652099ed5edf8c2686c70c8b8330c
Signed-off-by: Arun Bharadwaj <abharadw@codeaurora.org>
Library functions should not grab locks when the callsites can do it,
even if the lock nests like the rcu read-side lock does.
Push the rcu_read_lock() from css_is_ancestor() to its single user,
mem_cgroup_same_or_subtree() in preparation for another user that may
already hold the rcu read-side lock.
Change-Id: I0ec7ec0059ed588d8f85bc9be8fdc42ce0ca7f5d
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Konstantin Khlebnikov <khlebnikov@openvz.org>
Acked-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Acked-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Li Zefan <lizf@cn.fujitsu.com>
Cc: Li Zefan <lizf@cn.fujitsu.com>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Git-commit: 91c63734f6908425903aed69c04035592f18d398
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
The swap token code no longer fits in with the current VM model. It
does not play well with cgroups or the better NUMA placement code in
development, since we have only one swap token globally.
It also has the potential to mess with scalability of the system, by
increasing the number of non-reclaimable pages on the active and
inactive anon LRU lists.
Last but not least, the swap token code has been broken for a year
without complaints, as reported by Konstantin Khlebnikov. This suggests
we no longer have much use for it.
The days of sub-1G memory systems with heavy use of swap are over. If
we ever need thrashing reducing code in the future, we will have to
implement something that does scale.
Change-Id: I6d287cfc3c3206ca24da2de0c1392e5fdfcfabe8
Signed-off-by: Rik van Riel <riel@redhat.com>
Cc: Konstantin Khlebnikov <khlebnikov@openvz.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: Hugh Dickins <hughd@google.com>
Acked-by: Bob Picco <bpicco@meloft.net>
Acked-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Git-commit: e709ffd6169ccd259eb5874e853303e91e94e829
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Trinity discovered that we fail to check all 64 bits of
attr.config passed by user space, resulting to out-of-bounds
access of the perf_swevent_enabled array in
sw_perf_event_destroy().
Introduced in commit b0a873ebb ("perf: Register PMU
implementations").
Signed-off-by: Tommi Rantala <tt.rantala@gmail.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: davej@redhat.com
Cc: Paul Mackerras <paulus@samba.org>
Cc: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Patch-mainline: linux-kernel @ 04/13/13, 19:49
Signed-off-by: Deepak Katragadda <dkatraga@codeaurora.com>
We could re-read rq->rt_avg after we validated it was smaller than
total, invalidating the check and resulting in an unintended negative.
Change-Id: I8543974aad539107768e9e513ca3a8c4cb79b2ff
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: David Rientjes <rientjes@google.com>
Link: http://lkml.kernel.org/r/1337688268.9698.29.camel@twins
Signed-off-by: Ingo Molnar <mingo@kernel.org>
CRs-Fixed: 497236
Git-commit: b654f7de41b0e3903ee2b51d3b8db77fe52ce728
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Commit 55ddeb0f (sched: Reset rq->next_interval before going idle) reset
a cpu's rq->next_balance when pulled_task = 0, which will be true when
the cpu failed to pull any task, causing it go idle. However that patch
relied on next_balance being calculated as a result of traversing cpu's
sched domain hierarchy.
A cpu that is the only online cpu will however not be attached to any
sched domain hierarchy. When such a cpu calls into idle_balance(), we
will end up initializing next_balance to be 1sec away! Such a CPU will
defer load balance check for another 1sec, even though we may bring up
more cpus in the meantime requiring it to check for load imbalance more
frequently. This could then lead to increased scheduling latency for
some tasks.
This patch results in a cpu's next_balance being re-calculated when its
attaching to a new sched domain hierarchy. This should let cpus call
load balance checks at the right time we expect them to!
Change-Id: I855cff8da5ca28d278596c3bb0163b839d4704bc
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
When the new signal handlers are set up, the location of sa_restorer is
not cleared, leaking a parent process's address space location to
children. This allows for a potential bypass of the parent's ASLR by
examining the sa_restorer value returned when calling sigaction().
Based on what should be considered "secret" about addresses, it only
matters across the exec not the fork (since the VMAs haven't changed
until the exec). But since exec sets SIG_DFL and keeps sa_restorer,
this is where it should be fixed.
Given the few uses of sa_restorer, a "set" function was not written
since this would be the only use. Instead, we use
__ARCH_HAS_SA_RESTORER, as already done in other places.
Example of the leak before applying this patch:
$ cat /proc/$$/maps
...
7fb9f3083000-7fb9f3238000 r-xp 00000000 fd:01 404469 .../libc-2.15.so
...
$ ./leak
...
7f278bc74000-7f278be29000 r-xp 00000000 fd:01 404469 .../libc-2.15.so
...
1 0 (nil) 0x7fb9f30b94a0
2 4000000 (nil) 0x7f278bcaa4a0
3 4000000 (nil) 0x7f278bcaa4a0
4 0 (nil) 0x7fb9f30b94a0
...
[akpm@linux-foundation.org: use SA_RESTORER for backportability]
Signed-off-by: Kees Cook <keescook@chromium.org>
Reported-by: Emese Revfy <re.emese@gmail.com>
Cc: Emese Revfy <re.emese@gmail.com>
Cc: PaX Team <pageexec@freemail.hu>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
Cc: Serge Hallyn <serge.hallyn@canonical.com>
Cc: Julien Tinnes <jln@google.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Git-commit:2ca39528c01a933f6689cd6505ce65bd6d68a530
Git-Repo:https://github.com/torvalds/linux
Git-link:https://github.com/torvalds/linux/commit/2ca39528c01a933f6689cd6505ce65bd6d68a530
Change-Id: I7ae396ee90c2339c86f7b773d15f5571f8fd54c5
Signed-off-by: Goberu Gopal <ggober@codeaurora.org>
Signed-off-by: Shashank Babu Chinta Venkata <sbchin@codeaurora.org>
Commit 88a7e37d26 (sched: provide per cpu-cgroup option to
notify on migrations) added a notifier call when a task is moved
to a different CPU. Unfortunately the two call sites in the RT
sched class where this occurs happens with a runqueue lock held.
This can result in a deadlock if the notifier call attempts to do
something like wake up a task.
Fortunately the benefit of 88a7e37d26 comes mainly from notifying
on migration of non-RT tasks, so we can simply ignore the movements
of RT tasks.
CRs-Fixed: 491370
Change-Id: I8849d826bf1eeaf85a6f6ad872acb475247c5926
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Commit 7db16c8c (sched: Fix SCHED_HRTICK bug leading to late preemption
of tasks) introduced a bug in sched_slice() calculation by using wrong
cfs_rq for tasks. rq->cfs was incorrectly used as task's cfs_rq, rather
than the correct one to which they belonged.
Fix the bug by using correct cfs_rq for tasks.
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
WQ_HIGHPRI was implemented by queueing highpri work items at the head
of the global worklist. Other than queueing at the head, they weren't
handled differently; unfortunately, this could lead to execution
latency of a few seconds on heavily loaded systems.
Now that workqueue code has been updated to deal with multiple
worker_pools per global_cwq, this patch reimplements WQ_HIGHPRI using
a separate worker_pool. NR_WORKER_POOLS is bumped to two and
gcwq->pools[0] is used for normal pri work items and ->pools[1] for
highpri. Highpri workers get -20 nice level and has 'H' suffix in
their names. Note that this change increases the number of kworkers
per cpu.
POOL_HIGHPRI_PENDING, pool_determine_ins_pos() and highpri chain
wakeup code in process_one_work() are no longer used and removed.
This allows proper prioritization of highpri work items and removes
high execution latency of highpri work items.
v2: nr_running indexing bug in get_pool_nr_running() fixed.
v3: Refreshed for the get_pool_nr_running() update in the previous
patch.
Change-Id: Id843c0a425f51f84083786fbf413d999d35771b7
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Josh Hunt <joshhunt00@gmail.com>
LKML-Reference: <CAKA=qzaHqwZ8eqpLNFjxnO2fX-tgAOjmpvxgBFjv6dJeQaOW1w@mail.gmail.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Fengguang Wu <fengguang.wu@intel.com>
Git-commit: 3270476a6c0ce322354df8679652f060d66526dc
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
Introduce NR_WORKER_POOLS and for_each_worker_pool() and convert code
paths which need to manipulate all pools in a gcwq to use them.
NR_WORKER_POOLS is currently one and for_each_worker_pool() iterates
over only @gcwq->pool.
Note that nr_running is per-pool property and converted to an array
with NR_WORKER_POOLS elements and renamed to pool_nr_running. Note
that get_pool_nr_running() currently assumes 0 index. The next patch
will make use of non-zero index.
The changes in this patch are mechanical and don't caues any
functional difference. This is to prepare for multiple pools per
gcwq.
v2: nr_running indexing bug in get_pool_nr_running() fixed.
v3: Pointer to array is stupid. Don't use it in get_pool_nr_running()
as suggested by Linus.
Change-Id: I46e9488601d764d25e4a6c707de129ab68f7064c
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Fengguang Wu <fengguang.wu@intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Git-commit: 4ce62e9e30cacc26885cab133ad1de358dd79f21
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
GCWQ_MANAGE_WORKERS, GCWQ_MANAGING_WORKERS and GCWQ_HIGHPRI_PENDING
are per-pool properties. Add worker_pool->flags and make the above
three flags per-pool flags.
The changes in this patch are mechanical and don't caues any
functional difference. This is to prepare for multiple pools per
gcwq.
Change-Id: I1824fd1c509d8ac6b0619536621a22b15b316256
Signed-off-by: Tejun Heo <tj@kernel.org>
Git-commit: 11ebea50dbc1ade5994b2c838a096078d4c02399
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
Modify all functions which deal with per-pool properties to pass
around @pool instead of @gcwq or @cpu.
The changes in this patch are mechanical and don't caues any
functional difference. This is to prepare for multiple pools per
gcwq.
Change-Id: I4be6727e1cce6f9aa2a0057b96bdc725c84f1ea8
Signed-off-by: Tejun Heo <tj@kernel.org>
Git-commit: 63d95a9150ee3bbd4117fcd609dee40313b454d9
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
Move worklist and all worker management fields from global_cwq into
the new struct worker_pool. worker_pool points back to the containing
gcwq. worker and cpu_workqueue_struct are updated to point to
worker_pool instead of gcwq too.
This change is mechanical and doesn't introduce any functional
difference other than rearranging of fields and an added level of
indirection in some places. This is to prepare for multiple pools per
gcwq.
v2: Comment typo fixes as suggested by Namhyung.
Change-Id: Iefae84798c2af580f425b92ed79117935d99f21f
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Git-commit: bd7bdd43dcb81bb08240b9401b36a104f77dc135
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
Unbound wqs aren't concurrency-managed and try to execute work items
as soon as possible. This is currently achieved by implicitly setting
%WQ_HIGHPRI on all unbound workqueues; however, WQ_HIGHPRI
implementation is about to be restructured and this usage won't be
valid anymore.
Add an explicit chain-wakeup path for unbound workqueues in
process_one_work() instead of piggy backing on %WQ_HIGHPRI.
Change-Id: Iecd17a9935ee28f856d8b726bb4c296762922bed
Signed-off-by: Tejun Heo <tj@kernel.org>
Git-commit: 974271c485a4d8bb801decc616748f90aafb07ec
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
worker_enter_idle() has WARN_ON_ONCE() which triggers if nr_running
isn't zero when every worker is idle. This can trigger spuriously
while a cpu is going down due to the way trustee sets %WORKER_ROGUE
and zaps nr_running.
It first sets %WORKER_ROGUE on all workers without updating
nr_running, releases gcwq->lock, schedules, regrabs gcwq->lock and
then zaps nr_running. If the last running worker enters idle
inbetween, it would see stale nr_running which hasn't been zapped yet
and trigger the WARN_ON_ONCE().
Fix it by performing the sanity check iff the trustee is idle.
Change-Id: I54465b67d76bfaff3138963153256cde48add673
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: stable@vger.kernel.org
Git-commit: 544ecf310f0e7f51fa057ac2a295fc1b3b35a9d3
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
An inactive timer's base can refer to a offline cpu's base.
In the current code, cpu_base's lock is blindly reinitialized
each time a CPU is brought up. If a CPU is brought online
during the period that another thread is trying to modify an
inactive timer on that CPU with holding its timer base lock,
then the lock will be reinitialized under its feet. This leads
to following SPIN_BUG().
<0> BUG: spinlock already unlocked on CPU#3, kworker/u:3/1466
<0> lock: 0xe3ebe000, .magic: dead4ead, .owner: kworker/u:3/1466,
.owner_cpu: 1
<4> [<c0013dc4>] (unwind_backtrace+0x0/0x11c) from [<c026e794>]
(do_raw_spin_unlock+0x40/0xcc)
<4> [<c026e794>] (do_raw_spin_unlock+0x40/0xcc) from [<c076c160>]
(_raw_spin_unlock+0x8/0x30)
<4> [<c076c160>] (_raw_spin_unlock+0x8/0x30) from [<c009b858>]
(mod_timer+0x294/0x310)
<4> [<c009b858>] (mod_timer+0x294/0x310) from [<c00a5e04>]
(queue_delayed_work_on+0x104/0x120)
<4> [<c00a5e04>] (queue_delayed_work_on+0x104/0x120) from [<c04eae00>]
(sdhci_msm_bus_voting+0x88/0x9c)
<4> [<c04eae00>] (sdhci_msm_bus_voting+0x88/0x9c) from [<c04d8780>]
(sdhci_disable+0x40/0x48)
<4> [<c04d8780>] (sdhci_disable+0x40/0x48) from [<c04bf300>]
(mmc_release_host+0x4c/0xb0)
<4> [<c04bf300>] (mmc_release_host+0x4c/0xb0) from [<c04c7aac>]
(mmc_sd_detect+0x90/0xfc)
<4> [<c04c7aac>] (mmc_sd_detect+0x90/0xfc) from [<c04c2504>]
(mmc_rescan+0x7c/0x2c4)
<4> [<c04c2504>] (mmc_rescan+0x7c/0x2c4) from [<c00a6a7c>]
(process_one_work+0x27c/0x484)
<4> [<c00a6a7c>] (process_one_work+0x27c/0x484) from [<c00a6e94>]
(worker_thread+0x210/0x3b0)
<4> [<c00a6e94>] (worker_thread+0x210/0x3b0) from [<c00aad9c>]
(kthread+0x80/0x8c)
<4> [<c00aad9c>] (kthread+0x80/0x8c) from [<c000ea80>]
(kernel_thread_exit+0x0/0x8)
As an example, this particular crash occurred when CPU #3 is executing
mod_timer() on an inactive timer whose base is refered to offlined CPU #2.
The code locked the timer_base corresponding to CPU #2. Before it could
proceed, CPU #2 came online and reinitialized the spinlock corresponding
to its base. Thus now CPU #3 held a lock which was reinitialized. When
CPU #3 finally ended up unlocking the old cpu_base corresponding to CPU #2,
we hit the above SPIN_BUG().
CPU #0 CPU #3 CPU #2
------ ------- -------
..... ...... <Offline>
mod_timer()
lock_timer_base
spin_lock_irqsave(&base->lock)
cpu_up(2) ..... ......
init_timers_cpu()
..... spin_unlock_irqrestore(&base->lock) ......
<spin_bug>
Allocation of per_cpu timer vector bases is done only once under
"tvec_base_done[]" check. In the current code, spinlock_initialization
of base->lock isn't under this check. When a CPU is up each time the base
lock is reinitialized. Move base spinlock initialization under the check.
CRs-Fixed: 471127
Change-Id: I73b48440fffb227a60af9180e318c851048530dd
Signed-off-by: Tirupathi Reddy <tirupath@codeaurora.org>
This reverts commit f31b9b9f30.
Based on upstream mailing list discussions, this patch is not
necessary since switch_hrtimer_base() is called with interrupts
disabled.
Change-Id: I0244a744516e1c289af25043ee8b866be54813e3
Signed-off-by: Michael Bohan <mbohan@codeaurora.org>
This reverts commit 878f1bd9dd.
Based on upstream mailing list discussions, this patch is not
necessary since switch_hrtimer_base() is called with interrupts
disabled.
Change-Id: I53c64b1a4cea3a310ef53d1729a00baee6e4534a
Signed-off-by: Michael Bohan <mbohan@codeaurora.org>
Daniel Lezcano
Linux Build Service Account
Vincent Guittot
Steve Muckle
David Keitel
Feng Tang
Thomas Gleixner
Matt Wagantall
Srivatsa Vaddagiri
Tejun Heo
Peter Boonstoppel
Syed Rameez Mustafa
Tingting Yang
Sujit Reddy Thumma
Stepan Moskovchenko
Arun Bharadwaj
Laura Abbott
Johannes Weiner
Rik van Riel
Tommi Rantala
Peter Zijlstra
Kees Cook
Tirupathi Reddy
Michael Bohan