Linearizability

Linearizability requires all the operations to be timestamped and uses these timestamps to create a global sequence, all the nodes agree on. It assumes the nodes in the network to have loosely synchronized clocks and the timestamps create a unique and unambiguous sequence of operations. However, if the differences between the local clocks on different nodes become too big, the management of the operation sequence may become unfair, e.g., the requests from nodes with faster running clocks are favored and those from nodes with slower clocks are never executed.

Like the atomic model, the linearizability also requires, that the operation start is de-fined as the moment of its initialization by the requester node, but linearizability relaxes the time synchronization requirements of the atomic consistency. Time synchronization, even if feasible, because some inaccuracy is allowed, may induce a global operation that reduces the scalability. But, the relaxed temporal requirements makes this consistency model more feasible, compared to the atomic consistency.

The model can be realized in a distributed way, using replicas or in a centralized way, without replicas. These possible solutions can also be differentiated based on the operation that is invalidated in case of conflicts, i.e., either the write or read operation.

In a distributed implementation of the linearizability consistency model, every node manages its own replica and accesses to it. All the replicas are writable and each write request is timestamped and broadcast to all other replica holders. In Figure 5.10, node B issues two write requests and nodesA and C read the shared data item. And since the delivery of the update requests, that follow the write access, is delayed due to trans-mission, it is necessary to provide a mechanism for conflict solving. Invalidating a write request, once it is sent, would be an expensive task, regarding communication costs.

Thus, it sounds more feasible to invalidate the read requests, since these are issued and performed locally, i.e., only the local replicas are read. But, in order to enable the inval-idation mechanism, it is necessary to provide a transaction-like mechanism that allows roll-back of a read request. This is a complex task and requires at least for all the nodes to store a history of own read requests for each shared data item. And each time a write request is received, its timestamp is compared with those of the read requests, stored in the history for the particular data item. All the read requests that are newer than the incoming write, are invalidated.

The distributed implementation requires storage for the read requests and the support for invalidation of these requests. In a resource constrained system like a wireless sensor network, both these requirements can introduce memory consistency, as well as scalability

Figure 5.10: An example flow for the linearizability model with replicas (distributed)

problems. If the number of read requests is high and the node runs out of memory, then no new requests can be stored in the read request history. A possible solution for the problem would be to reject the new read requests immediately, in such a case. On the other hand, the invalidation of a read request can become an overkill for the application complexity, since it has to provide the possibility for request invalidation.

The invalidation of read accesses is a complex task, but, in many cases, it can be substituted by an easier solution, i.e., by the delaying of the delivery of the read request processing results. However, the latter assumes, that the application can work with the additional delay. Figure 5.10shows four possible combinations of timing relations between a read and a write accesses with the application of delayed read. Both, read1

and read2 were issued after write1, but in case of read1, the delay was not sufficient and the invalid response was propagated to the application, before the update for write1

arrived. Similar, both read₃ and read₄ were issued before write₂. In this case it is not necessary to change the result of the read operation, even if an update request for the read data item arrives during the delay period, as it is the case for read3.

If the application accepts delayed responses for read requests, then it is possible to postpone the delivery of the read value until it is clear, that the read request will not be invalidated anymore. This can be realized by introducing an invalidation timeout for the read request history, i.e., once the read request is issued, it is timestamped and stored in the history together with the current response, but this response can still be updated and is delivered after the validation timeout elapses (see Figure 5.10). This solution opens possibilities for memory consistency violation. After the read request is validated and the response is delivered to the application, there is no way to accept a write request older than this already handled read request. This would cause the operation sequence visible on the local node, to be different from the global one and thus, to be invalid. An example of such situation a is depicted by read₁ in Figure5.10. The optimum value for the validation timeout is strongly dependent on the accuracy of the time synchronization mechanism, but even more on the size of the network and the possible transmission delay.

The implementation of the above mentioned distributed version of the linearizability model with replicas, would require major changes in the tinyDSM middleware. The

concept of data item manager had to be redesigned, so that all the nodes posses a writable copy. This settings could be enabled by a new policy parameter–writable replicas(see Table F.1). In such a setting, there is no single manager of a shared data item and the identity of the node included in the update request is used only to distinguish the requests and order them according to the identity, if they were issued at the same moment and thus, have the same timestamp (or version). The write requests are handled immediately, the local copy is changed and the update request is broadcast to the other nodes.

Additionally, the read mechanism had to be adapted as well, i.e., all the local read requests are stored in a read request buffer and are handled after the validation period expires. The later change could be parameterized by an additional policy parameter–

read delay(see TableF.3) and could be realized by introducing a delay in the notifica-tion on the positively handled read request in the NOT3state (see Figure4.18).

The middleware has to be also modified, so that all the requests are timestamped in the issuing moment with the local time of the requesting node. This could be, for instance, realized by introducing a new policy parameter–timestamp issuing. Enabling this switch parameter for a given variable, turns on the timestamping of requests at the issuing time (see Table F.4).

The definition of variables supporting the linearizability model in the distributed way with replication, is given in Listing5.1.

#define l i n e a r i z a b i l i t y f i f o p r o c e s s i n g \\

r e p l i c a t i o n r a n g e : 0 \\

r e p l i c a t i o n d e n s i t y : 1 0 0 \\

t i m e s t a m p i s s u i n g \\

w r i t a b l e r e p l i c a s \\

d i s t r i b u t e d g l o b a l l i n e a r i z a b i l i t y u i n t 3 2 t A;

d i s t r i b u t e d g l o b a l l i n e a r i z a b i l i t y u i n t 3 2 t B ; d i s t r i b u t e d g l o b a l l i n e a r i z a b i l i t y u i n t 3 2 t C ;

Listing 5.1: The definition of variables supporting the linearizability model with replica-tion

In a centralized implementation that tries to assure the linearizability consistency requirements, a single node has to store and manage the shared data item and the data cannot be replicated (defined using the noreplication policy parameter). Each access request is transmitted to this manager node and thus, operates on the most recent version of the data item (see Figure5.11). This solution limits the parallelism of the system, but it also simplifies solving the write-write and write-read conflicts as described in Section 3.9. Two distant nodes (AandC in Figure5.11) may issue write requests in about the same moment and the messages containing these requests may arrive at the manager nodeB in incorrect order, regarding their timestamps. In this situation, the write-write conflict can be solved, i.e., the sequence and the value stored in the data item can be corrected, if necessary.

But, if a read operation would be performed between two write operations that require correction, then the value returned by this operation may be invalid, due to the write-read conflict and violates the linearizability model (see Figure 5.12). Thus, in order to ensure the rules of the model it is either necessary to invalidate all the read requests that

Figure 5.11: An example flow for the linearizability model without replicas (centralized)

Figure 5.12: A violation of the linearizability model (centralized)

Figure 5.13: An example flow for the delayed read in the linearizability model without replicas (centralized)

Figure 5.14: An example of refused write request in the linearizability model without replicas (centralized)

happened on the ’dirty’ data, or to refuse accepting a write request that is older than the most recently performed read operation. As already mentioned, invalidating read requests is either not practical in wireless sensor networks or substituted by a delayed read (see Figure 5.13). Refusing the write requests is much less complex and resource consuming (see Figure5.14). Additionally, a realization of the linearizability model that allows large delivery delays and provides small response times at the same time, can combine both, delayed read and rejection of write requests. This can be even realized in an adaptive way, e.g., by adjusting the delay for the read requests according to the delivery delay of the incoming write requests.

Except of the need to have a time synchronization mechanism, only minor changes to

the tinyDSM middleware are required, in order to implement the centralized version of the linearizability model. The requests have to be timestamped at the moment of issuing and the timestamps are transmitted to the owner node to order the requests (enabled by the newtimestamp issuingpolicy parameter). Additionally, for each shared data item, it is required to store the timestamp of the most recent read operation. This timestamp is compared with the timestamps of the incoming write requests and requests older than this stored timestamp are rejected. In the tinyDSM middleware, the write operations are by default acknowledged by the manager of the data item, so that the node that issues the write request knows the result of the operation. If the request was issued from within the replication range and the independent forwarding is used, then the corresponding update request is regarded as a positive acknowledgment. Thus, if the requests for the given variable have a defined timeout period, then in case of a refused write operation the owner can skip the sending of the negative acknowledgment message. The unanswered request will timeout on the node that issued the request and the application logic will be notified, that the request failed, so it can be retried.

It is also necessary, that all the correlated variables are owned by a single node, in order to assure the consistency model requirements. This could be realized by introducing another policy parameter–group number, which defines groups of shared variables (see Table F.2). For such a group, the ownership related operations are performed, as if the variables in the group were a single variable, i.e., the owner is set for all of them and the ownership of the whole group is transferred, in case of ownership migration. Actually, it is possible to store a single owner address for all the variables in a single group, if the individual representation of thelocal variables and metadata storage is used.

For multiple variables it becomes even more visible, that the single owner of the all correlated variables becomes a bottleneck in the system. In order to increase the robustness, the shared variables can be replicated, but they can be read only in special cases, e.g., only by the owner, but the ownership can migrate. Such update-only replicas can be also used in case of system recovery, e.g., if the current owner node disappeared.

The definition of variables supporting the linearizability model in the centralized way without replication, is given in Listing 5.2.

#define l i n e a r i z a b i l i t y f i f o p r o c e s s i n g \\

n o r e p l i c a t i o n \\

t i m e s t a m p i s s u i n g \\

group number : 1 \\

d i s t r i b u t e d g l o b a l l i n e a r i z a b i l i t y u i n t 3 2 t A;

d i s t r i b u t e d g l o b a l l i n e a r i z a b i l i t y u i n t 3 2 t B ; d i s t r i b u t e d g l o b a l l i n e a r i z a b i l i t y u i n t 3 2 t C ;

Listing 5.2: The definition of variables supporting the linearizability model without repli-cation

The linearizability consistency model was not evaluated practically, due to its strong dependence on the available time synchronization mechanisms, which is out of scope of the presented work. In general, the feasibility of accurate time synchronization mecha-nisms in WSN is questionable, mainly due to the scalability issues. The linearizability is the memory consistency model that is the closest one to the atomic consistency, from

Figure 5.15: An example flow for sequential consistency model without replication

those models discussed in this work. And, if such a strong consistency model is required by the application and an energy efficient, scalable and accurate time synchronization mechanism is available, the linearizability is the best choice. The strong dependence on time synchronization, as well as the definition of the operation start as the request issuing time, can be avoided by using the sequential memory model, described in the following section.