GASNet - Global Address Space Networking

GASNet is a highly portable communication network layer designed by Dan Bonachea for the PGAS. It is based on the Active Messaging Interface [71] and described in [6]. It has been designed as a layer between the hardware layer and the actual API, i.e., a low-level language.

This section will give an overview of the functionalities of GASNet but without detailed infor-mation on single routines not directly associated with communication. All statements made in this chapter that excess the scope of the specification, especially when referencing any readme files, refer to the GASNet release 1.18.2 and included documents obtainable online [31].

1 0

time

AMRequest(...,handler0,...)

handler0(...) AMReply(...,replyHandler0,...)

replyHandler0(...)

Figure 3.1.: The active messaging communication scheme. The dashed reply and its invoked reply handler are not mandatory but must be sent by handler0.

The GASNet programming interface is divided into two APIs: the Core API, which is the minimum interface that every implementation of GASNet has to deliver, and the Extended API, which is more user-friendly and may be implemented solely in terms of the Core API.

During the configuration of GASNet, the machine on which it will be installed is checked by a script and according to the result of the tests, one or more of twelve conduits will be installed.

These conduits ensure the portability of GASNet to different platforms while still reaching highest possible performance. To achieve maximum performance in terms of runtime saving and resource friendliness the GASNet routines are implemented in a conduit specific manner.

GASNet routines can thus be seen as a wrapper around given hardware specific languages such as IB Verbs. As it would lead to too much detail to explain every single conduit, a list can be found in App.A. For further reference, please see the readme files included in the different conduits [31].

The Core API consists of the two-phased initialization with the two routines gasnet_init and gasnet_attach, the exiting routinegasnet_exit, environment queries and several active messaging routines. There are also atomicity control routines, which enable a race condition free memory access.

The initialization routines bootstrap the job to the computation nodes, allocate a global mem-ory segment on each node and establish the communication network between the nodes. The global memory segments will then be accessible by all compute nodes as shared memory. The established communication network is of an alltoall character, such that every node can after-wards communicate with any other node which has been bootstrapped.

The communication routines of the Core API consist of different active messaging routines.

An active messaging communication typically consists of a request message sent by node pi, invoking a handler on node pj, and an optional reply message sent by node pj, again invoking a handler on node p_i. The initiated handler on the other node then processes the sent data, giving this type of communication the name Active Messages. This communication scheme is depicted in Fig. 3.1.

To steer the workflow of a GASNet application, a polling routine and a blocking routine also belong to the Core API. Both query, whether a certain user-defined condition has been fulfilled, the latter blocking the CPU until the condition is fulfilled.

The Extended API of GASNet defines more user-friendly routines than the Core API does, e.g., there are no longer different communication routines for different message sizes. Instead, the message size is taken as an input argument to the communication routines of the Extended API. It includes memory-to-memory transferring routines, which are available in a blocking and a non-blocking manner. For the non-blocking routines there are also corresponding syn-chronization routines. Additionally there are register-memory operations, also in non-blocking and blocking versions, which enable a better use and control of memory resources. There is also a split-phase barrier to synchronize the threads in their workflow. As already implied through the last sentence, the Extended API also supports threading.

The blocking put and get functions will write the data into the remote memory segment im-mediately, meaning before the function returns. The non-blocking routines on the other hand will return immediately but the application programmer has to ensure that the data has been written before using it again by calling a synchronization routine. Every non-blocking routine comes in two versions: an explicit handle version and an implicit handle version. These handles - passed as arguments - become important when synchronizing.

In case of the explicit handle version, the programmer has the possibility to only synchronize those routines, which were started with a certain handle. This gives the programmer more flexibility and makes it possible to minimize synchronization points by only synchronizing when the according data is actually needed. Another possibility for the programmer to avoid synchronizing at undesired points is available by choosing to wait only for puts, gets or actually waiting for all memory-transfer routines to have finished.

If the implicit handle versions are used, any implicit synchronization call will wait until all data is written. If not all implicit calls shall be synchronized, but only a subset, the GASNet specification supplies the so called access regions. These access regions return a handle, which can then be used for synchronizing all implicit handle communication calls made in this region.

For the synchronization of the non-blocking calls, the programmer may choose between the gasnet_wait_... calls and the gasnet_try_... calls. The latter only checks if all calls in question are done and returns immediately. If all calls are have completed successfully, gasnet_try_... will behave like gasnet_wait_..., which would block until all routines in question have returned.

These synchronization routines should not be confused with the split-phase barrier also included in the Extended API. Every node calling gasnet_barrier_notifynotifies the system, that it has reached a certain point in the application. It is possible to hand a token to the routine as an argument, which affects the second phase of the barrier. This second phase may be called in a blocking manner as gasnet_barrier_wait, which will wait until all nodes have notified

the system, or in a non-blocking manner throughgasnet_barrier_trywhich only checks if all nodes have notified the system and may be called several times after one another. This barrier has nothing to do with the synchronization of data and thus it may not be assumed, that all write and/or put operations called before the barrier have completed.

GASNet and IB Verbs have been two low-level interfaces for distributed memory systems with the possibility of creating a PGAS through their RDMA routines. Both may be used as an underlying layer for an implementation of higher level languages, like GASPI. The API described in the next section, is a shared memory API, which is often used in combination with distributed memory languages like GASPI or MPI on hybrid systems.