2.7 Load Balancing
2.7.2 Work Stealing
Work stealing was named after the fact that concurrent data structures, most notably concurrent deques, allow “thieves” to “steal” tasks from “victims” without interfering with the victims’ execution7. In systems with distributed address spaces, work stealing requires cooperation between victims and thieves: victims send tasks in reaction to steal requests they receive. Because of this explicit message exchange, some authors prefer the term work requesting [178]. Work sharing and work stealing are instances of sender-initiated and receiver-initiated load balancing [81, 44]. Tasks are transferred
7This is a bit of a simplification because thief and victim may have to synchronize.
2.7.2 Work Stealing 27 from sender to receiver as a result of either the sender’s actions (work sharing) or the receiver’s actions (work stealing).
The Cilk multithreaded runtime system The idea of work stealing originated from research on parallelism in functional programming languages in the early 1980’s [59, 106, 107]. Much of the groundwork that influenced the design and implementation of schedulers was laid in the Cilk project at MIT [21]. Cilk established a provably efficient, randomized work-stealing scheduler for fully-strict computations, in which child tasks are required to synchronize with their parents (well-structured fork/join computations) [95, 49].
Tasks and child tasks A multithreaded program can be viewed as a DAG of compu-tations (vertices) linked by dependencies (edges) [48]. Spawn edges create child tasks, which represent potential parallelism, and join edges introduce synchronization. In a fully-strict DAG, every task Γ must synchronize with its parent, the task that spawned Γ. In a strict DAG, every task Γ must synchronize with one of its ancestors, the par-ent of Γ or, recursively, an ancestor of the parpar-ent of Γ. A DAG in which every task ends with a join edge is said to be terminally strict [32]. Figure 2.3 shows examples of fully-strict and strict multithreaded computations that are also terminally strict.
Terminal strictness is often implied when waiting for completion is the only means of synchronization between tasks and child tasks.
Let T1 be the execution time of a fully-strict computation DAG on one processor, TP be the execution time on P processors, and T∞ be the execution time on an infi-nite number of processors. The latter is also known as the critical path length: the theoretically shortest execution time resulting from the longest chain of sequential de-pendencies. Cilk’s work-stealing scheduler executes a computation onP processors in expected time T1/P +O(T∞), assuming the two bounds TP ≥ T1/P and TP ≥ T∞ are met. The result is near-optimal linear speedup if T1/P T∞, or equivalently, if T1/T∞ P, that is, the potential parallelism far exceeds the number of processors, which highlights the importance of breaking down a program into many independent tasks. T1 is known as the work of the computation, and the ratio T1/TS describes the work overhead relative to the serial elision with execution timeTS8.
The work-first principle Cilk’s work-first principle states that the work overhead should be minimized, since it has a big impact on performance, whereas overheads on
8The serial elision of a Cilk program is obtained by removing all Cilk keywords [95].
(a)Fully-strict computation DAG (b)Strict computation DAG
Figure 2.3: The DAG model for multithreading. Tasks, drawn as rectangles around sequen-tial computations (vertices) and continuations (horizontal edges), are connected by spawn and join edges. A spawn edge creates a new task, which may execute in parallel with other tasks. A join edge ends a task after synchronizing with the parent (a) or an ancestor (b). The work is the time it takes to execute all computations in a DAG, and thespan, orcritical-path length, is the time it takes to execute the longest path of dependencies. The ratio of work to span gives the maximum possible speedup for any number of processors. It is an indication of how much potential parallelism a DAG contains. Strictness does not require synchronization between parent and child tasks, allowing computations at different levels of the spawn tree to execute in parallel without unnecessary dependency constraints. There are two such op-portunities in this example, with the result that the strict computation DAG contains more potential parallelism than its fully-strict counterpart. In (a), the ratio of work to span is 14/12 = 1.16, whereas in (b), it is 14/10 = 1.4.
the critical path T∞ are much less important as long as sufficient parallelism exists, and steals are rare. In other words, optimizations should target the common case, the execution path where no work is stolen, even if that means adding overheads to the critical path. Specifically, as much of the scheduling cost as possible should be shifted to idle workers, since idle workers have no other work to do. Similar time, space, and communication bounds have been proved for the more general class of strict multithreaded computations, in which child tasks are required to synchronize with their ancestors but may outlive their parents [85, 32].
Work-first scheduling Cilk’s task creation strategy is the result of strict adherence to the work-first principle. As in lazy task creation [172], a task is turned into a function call, while the continuation following the spawn operation is pushed onto the deque for idle workers to steal [95]. Upon returning from the task, the worker that pushed the continuation checks if the continuation has been stolen, and if so, becomes a thief itself. Otherwise, the worker picks up and resumes the continuation.
2.7.2 Work Stealing 29 Help-first scheduling The alternative to Cilk’s task creation strategy is to queue the task and execute the continuation. Because this strategy neither assumes compiler support nor a continuation-passing programming style, it is the strategy of choice for many tasking libraries, including our own. Guo et al. call the “steal child” approach help-first, to suggest that help is needed to run a task, and to distinguish it from Cilk’s
“steal parent” approach, which is a consequence of the work-first principle [101]. Help-first is less space efficient than work-Help-first. In theory, help-Help-first may require unbounded space and may overflow heap memory, whereas work-first provably requires at mostS1P space, a multiple of the space required by the serial executionS1. A simple example may help to visualize the difference between help-first and work-first. Consider the following loop, in which a single thread spawnsN tasks before running them in sequence at the
syncstatement, all under the assumption that no thieves are present [208]:
for (i = 0; i < N; i++) spawn f(i);
sync;
With help-first, the program requires space proportional to N, because N tasks are created and enqueued before the sync statement is reached, at which point tasks are scheduled for execution. With work-first, the program runs in constant space, deferring only the continuation of the loop in each iteration. Suppose the loop spawns one million tasks, each taking up 192 bytes9. Running the program on one processor will allocate 192 MB of memory to store the tasks, which will make the loop noticeably slower than its sequential version. Using our runtime system, for example, we measure a work overhead of 2.42, compared to 1.37 for Cilk Plus, when functionfdoes nothing else but return (see Figure 2.4 (a)). Half of the work overhead can be attributed to allocating memory. By allocating memory ahead of time, the work overhead drops to 1.7. The remaining overhead compared to sequential execution is caused by task creation and deque operations, which cannot be eliminated without serializing tasks.
For comparison, we show how using a concurrent deque, similar to Cilk’s imple-mentation [95], can affect the work overhead. Private deques have the advantage of efficient operations, but require that victims steal on behalf of thieves, which is at odds with the work-first principle as formulated by Cilk.
Figure 2.4 (b) indicates that work overheads tend to be small for non-empty tasks.
Given tasks of one microsecond, which we count as fine-grained parallelism, work-first has no measurable overhead on average, while help-first adds 6% to the sequential execution time.
9The actual size of a task in our implementation.
Work overhead T1/TS
Private deque Private deque Concurrent deque Concurrent deque
(a)Task length of 0µs
Private deque Private deque Concurrent deque Concurrent deque
Private deque Private deque Concurrent deque Concurrent deque
(b) Task length of 1µs
Figure 2.4: Work overhead T1/TS for running a sequential loop that spawns one million tasks using work-first and help-first task creation strategies. Operating a concurrent deque is more expensive than operating a private deque; hence the greater work overhead. (ICC 14.0.1, -O2, Intel Core i7-4770)
Advantages of help-first are less stack pressure and better performance when steals are frequent. (Remember that one of the assumptions underlying the work-first princi-ple is that steals are rare.) A more practical advantage of help-first is that it tends to be easier to reason about than work-first. Imagine a sequence of statements that looks like this:
S1;
spawn S2;
S3;
Under work-first, S1 and S3 are executed by different threads if a thief steals the continuation following S2 while the worker is still busy with S2. For this reason, we may observe that a procedure is called by a threadT1, but returns on a different thread T2 [210]. Under help-first, S2may be executed by a different thread, butS1and S3are guaranteed to run on the same thread (intuitive function call/return).
Guo et al. have developed an adaptive work-stealing scheduler for Habanero Java that switches between help-first and work-first depending on the stealing rate and recursion depth [102]. If steals are rare, the scheduler operates under the work-first policy in order to guarantee bounded use of space. Otherwise, if steals are frequent, or if stacks exceed a certain depth, the scheduler prefers to create tasks according to the help-first policy. Scheduling decisions are reevaluated periodically.
A simpler way to try to combine the benefits of work-first and help-first is to bound the number of tasks per deque [170]. Bounded deques make it impossible to enqueue
2.7.3 Data Structures for Work Stealing 31