Additional measurements were performed to evaluate the impact of these operations in actual MPI operations. MPI operations can run from a few microseconds to multiple sec-onds, depending on the type of operation, the number of processes and the size of the buffers.
In Fig. 13.4, results for theMPI SENDandMPI BCASToperations are presented. These two operations were selected since they have the lowest latencies among the set of point-to-point and collective operations, respectively. The figure presents the latency for the MPI SENDoperation at the top and the MPI BCASToperation at the bottom. Results for Phase 1 (left) and Phase 2 (right) nodes are presented side by side for comparison. Results for 16 and 1024 processes are presented with buffer sizes from 16 bytes up to a megabyte.
The size of the CFG was set to 32 for these tests. Most applications and benchmarks that have been evaluated generate less CFG entries by the time they terminate.
13.3 MPI Performance Impact of the CFG Detection Overhead
Buffer size in bytes (Sandy Bridge) No CFG detection 16 Processes
Buffer size in bytes (Sandy Bridge)
16 256 4096 65536 1000
Figure 13.4:MPI SEND(top) andMPI BCAST(bottom) performance examples with detec-tion enabled and disabled on a 32 entry CFG loop. Results for SuperMUC Phase 1 (Sandy Bridge, left) and Phase 2 (Haswell, right) are presented.
As can be seen in the plots, the performance ofMPI SENDis only impacted significantly for message sizes of up to 4096 bytes, but only at lower process counts. For the case of 1024 processes, the overhead of the CFG detection algorithm is insignificant even for very small messages of 16 bytes. Additionally, the overhead of detection is not measurable on verification mode. This means that its overhead will only be observed when the detection algorithm has not encountered a loop, or when it exits a loop and resumes its detection.
A smaller performance impact can be observed for theMPI BCASToperation. As men-tioned before, the latency of this operation is the lowest among MPI collectives; therefore, the impact of CFG detection can be expected to be almost negligible when collectives are being used. Although the detection overhead is lower in terms of absolute latency, the percentage impact is higher in the case of Phase 2 nodes.
13 Elastic Resource Manager Performance
14 Case Studies with Distributed Memory Applications
Two computational kernels are evaluated in detail in this chapter: a matrix-matrix multi-plication kernel and a Gaussian elimination kernel. The matrix-matrix kernel is based on the Cannon algorithm, while the Gaussian elimination kernel is a naive row-block imple-mentation.
These have been selected due to their simplicity: these have well understood scalability and efficiency properties and execute fast enough. Because of this, a full sweep of pos-sible resource combinations with them can be done in a timely manner. More complex applications are already developed or under development, such as Computational Fluid Dynamics (CFD) simulations with AMR [70].
14.1 Cannon Matrix-Matrix Multiplication
Figure 14.1: Cannon matrix-matrix multiplication trace for 16 processes. MPI time in red and application time in blue.
In this section, a matrix-matrix distributed multiplication kernel based on the Can-non [54] algorithm is analyzed. The response of the CFG detection and scheduling al-gorithms of the infrastructure to its performance and scalability properties are discussed.
14.1.1 Basic and EPOP Implementations
The original implementation was a small single C source file with the MPI based Cannon algorithm. The new implementation uses MPI topologies to simplify the communication
14 Case Studies with Distributed Memory Applications
Number of MPI processes (Sandy Bridge) 4096x4096
Number of MPI processes (Sandy Bridge) 4096x4096
Efficiency (elements per second per process)
Number of MPI processes (Sandy Bridge) 4096x4096
Efficiency (elements per second per process)
Number of MPI processes (Haswell)
Number of MPI processes (Sandy Bridge) 4096x4096
Figure 14.2: Compute, MPI, efficiency and MTCT ratio (top to bottom) of a Cannon Matrix-Matrix multiplication kernel. Results for SuperMUC Phase 1 (Sandy Bridge, left) and Phase 2 (Haswell, right) are presented.
14.1 Cannon Matrix-Matrix Multiplication
with neighbor processes during computation. This is particularly helpful with the Cannon algorithm given its block wise exchanges in the main kernel. The kernel remains the same in both EPOP and basic versions of the code. This kernel is presented in Listing 14.1.
f o r( c a n n o n b l o c k c y c l e = 0 ; c a n n o n b l o c k c y c l e < s q r t s i z e ; c a n n o n b l o c k c y c l e ++){
The adaptation window was inserted in the main kernel loop. No proper adaptation code was implemented. Instead, the root process of the application redistributes the matrix data on each adaptation. A better solution will be to add an MPI based collaborative repartitioning scheme where all processes participate.
Initialize
For testing, long running applications are needed to ob-serve the behavior of the scheduler. Because of this, an ad-ditional loop was added that effectively repeats the num-ber of matrix-matrix multiplications that are performed by the application. The source matrices are not modified, therefore no changes were necessary to ensure correctness.
Although it is a very simple application, it suffers from the difficulties described in the EPOP chapter. An EPOP version of the application was also developed. Figure 14.3 illustrates its design based on EPOP blocks. It is a single EP application, with its required initialization block and a single rigid phase used for finalization. Because EPOP operates at a very coarse level, the performance of the ap-plication in both versions is indistinguishable. Because of this, the performance data presented in this evaluation are relevant to both implementations.
14.1.2 Pattern Detection
The Cannon application was also used to verify the correctness of the pattern detection functionality presented in the scheduling chapter. Figure 14.4 illustrates what occurs when the system detects the CFG of the non-EPOP implementation. At the beginning, each ap-plication process starts the detection process. Fortunately, the apap-plication is simple enough that the detected CFG of a full execution can be illustrated. The CFGs of the root process
14 Case Studies with Distributed Memory Applications
Rank 0 Other Ranks Rank 0 Other Ranks
Collapse
Figure 14.4: Cannon CFG detection process illustrated.
and other processes are illustrated on the left side of the figure. These differ in that root has more loops than the rest of the processes. There is a loop where matrix dimensions are broadcasted, and another loop where the operand matrix sub-blocks are distributed. There is an additional loop where the final results are gathered. For each of these three loops, the rest of the processes have a matching receive (one for each of the first two loops at root) and a matching send (for the final gather loop at root). The CFG collapse and reduction operations for this application are illustrated from left to right, respectively. The collapse operation simplifies the loops at each process. The reduction operation detects the loops that are present at all processes and produces the distributed loop metadata.
14.1.3 Performance Analysis
A trace with an allocation of 16 processes showing the MPI and application times for this application is presented in Fig. 14.1. As can be seen, the proportion of MPI to compute time is low. Figure 14.2 shows a detailed sweep of the performance and efficiency properties of this application based on the number of processes. It helps to remember that, in the presented infrastructure, the number of processes of an application is ensured to match the number of CPU cores that are allocated to it. In the figure, the different times for the iteration of the detected loop in the CFG are presented. From top to bottom: total time, MPI time, efficiency and MPI to compute time ratio (the MTCT metric described in the scheduling chapter).
As can be seen in the bottom plots, the number of processes for MTCT metric values below 0.1 correlate well with the number of processes where the efficiency metric of the application is near the maximum possible for each input size. The heuristic described in the scheduling chapter halves the number of process in all cases where the average or trend MTCT values are above 0.1. The quality of the decisions can be verified for this applica-tion, since its performance and efficiency has been evaluated before for a wide range of input matrices and process counts. In this case, the algorithm makes resource adaptation decisions that do not lower the application’s parallel efficiency significantly.