For a given storage space unit s at a certain working node, the optimizer aims to employ it for lower queries execution time, by either building more indexes, join cache or by building more replications. The choice of building more indexes needs further decisions about the certain vertices to index and the type of such indexes. In the same context, building more replications needs further decision about the certain vertices to replicate and the type of index to maintain them. Chapter 4 considered optimizing the indexes and join cache by formulating their benefits, costs, and access
Communication Framework
rates. Then transferred the formulas into sets of operational rules. The same is done in Chapter 5 which concludes by providing a single set of replication operational rules. The storage optimizer puts the three optimization modules together and unifies the optimization process. The abstract storage optimizer components are shown in Figure 5.1.
6.2.1 Universal Adaption
The universal adaption was already presented in Section 3.2 as the ability of the system to make an optimized decision to employ any of its resources with the best-expected option out of a set of multiple options. The basic of the adaption algorithm is stated in Section A.2.1 and projected on the storage adaption in Figure 6.3 which sketches the process architecture. The workload is collected and analysed into Heat Query graph(s). The system has predefined rules which have formulas to calculate the benefit of the vertices in the RDF graph based on the cost model. A raw data unit in the RDF graph is colored with blue in Figure 6.3. In the storage system, we have several indexes as well as the replication, and each structure is differentiated with its own unique color in Figure 6.3. The cost model of the optimization process
Workload
Figure 6.3: The process of storage space adaption
assigns to the blue raw data unit, the best-known structure, and labels it with the selected structure colour to produce a coloured data unit (the dark green in Figure 6.3 is an example indicates that the data unit was assigned to the SPO index). The coloured data unit is stored in the storage container and structured as its selected structure. The storage container is the physical representation of the storage where assigned data units are stored and the system pays its storage cost. This also means that the RDF graph is not presented physically, but it conceptually represents the storage container. This representation is modeled with the dashed line that connects the storage container with the RDF graph. This relation also means that, at some point, a coloured data unit in the storage container is going to be treated as raw data unit, such that it can be evaluated again by the cost model and possibly assigned to another place.
The Basic Storage Adaption Algorithm
We consider now transforming the basic adaption algorithm (given in Appendix A, Section A.2.1) into a more applicable storage adaption algorithm.
The basic work of the initial algorithm is to calculate for each vertex in the data graph the benefit of assigning it to any of the storage options (which are the memory
and hard disk indexes). The considered data graph is the local part of the data within a working node and its neighbours. The above benefit is further factorized by the access rate deduced from the workload. The next step is to sort the vertexes up on their benefits and select the most beneficial triples to be assigned to the best-fit index.
The methodology used to analyze the workload to find the vertices’ access rate is given in the previous chapter, Section 3.4 where we derived Formula 3.7 for vertex access rate.
The running time of the above algorithm is expensive since it needs to monitor each vertex, besides the storage cost required to keep track of vertexes’ benefits. To over-come these costs, we could reduce the accuracy of the benefits values and aggregate them into limited number of levels(e.g.five levels). This would result typically in a pyramid-shape of values. The lower big base contains the less important vertices, while the top has a small amount of more important vertices at its higher levels. The optimizer may now keep track of only the most important vertices by maintaining a priority queue sorted by the vertices’ importance. Such a priority queue may be updated at some point in time when the system has detected obvious changes in the workload. A second priority queue is kept by the optimizer in order to keep track of the vertices that have been already assigned in memory. However, the vertices in this case are the less important vertices out of what exists in memory in terms of their benefit. As a result, the optimizer checks the two priority queues and performs a swap whenever a vertex at the top of the first queue is more important than the vertex at the second priority queue’s top. This algorithm is already followed by a previous version of our system [3]. However, the method still requires a considerable amount of storage space besides the difficulty to maintain the queues of vertices.
In the next subsection, we explain a more optimal algorithm to handle the storage
space adaption that we have followed in for this version of UniAdapt.
Algorithm 1:Basic space adaption algorithm
input : RDF graphG={V, E}, the partition numberi, indexes setX, and the set of heat
12 benef it←baseBenef it·access;
13 U←U∪ {(v, χ, benef it)};
14 end
15 end
16 updateQueue(assignedV erticesQueue, U);
17 updateQueue(proposedV erticesQueue, U);
18 whilesize(proposedV erticesQueue)>0do
19 {(vp, indexp, benef itp)} ←pop(proposedV erticesQueue);
20 {(va, indexa, benef ita)} ←pop(assignedV erticesQueue);
21 if benef itp>(benef ita+BIAS)then
The algorithm runs at each working node i with the RDF graph G, the set of indexes χ, and the set of heat queries H as inputs. The first line applies the heat query to the RDF graph Gto get the sub-graphGh which has the set of verticesVh with an access rate of more than zero.
The first loop iterates over all the vertices inVh, then for each indexχ, it retrieves the access and benefit of each vertexv. However, these values are calculated differently depending on whether v is a local or remote vertex. In the case of local, the access value is set in Line 6 using the heat query access formula given by Formula 3.7.
The base benefit is the ηidx(v, χ) which is already given by Formula 4.3. In case of that v is a remote vertex, the access returned by the heat query is multiplied by the pprem which is the general access rate to the border replication looking from working nodei. We findpprem by recalling Formula 5.6. In Line 10, we set the base benefit of the remote vertex to ∆ which is the difference in access time between a local and remote vertex which basically depends on the network access time. The benefit is then calculated according to Formula 3.1 by multiplying the access by the
base benefit. A triple element of vertex, index, and benefit is created and assigned toU at the end of the loop vertices loop’s iteration (Line 13).
The next section of the algorithm deals with the assignment and swapping between indexes in the storage element. First, the priority queues are updated withU. As was earlier mentioned, we have two priority queues. The first isassignedV erticesQueue which holds the vertices that have already been assigned in the previous runs of the algorithm, ordered ascendingly by the vertices’ benefit. The second queue is proposedV erticesQueuewhich contains the vertices that are proposed for assignment and ordered descendingly by the benefit. Once the benefit values in each of the queues are updated, the loop of Line 18 may begin to perform the swap operation between the top of each queue. The loop halts when the top of the proposed queue is no longer more beneficial than the top of the assigned queue, or when the proposed queue is empty. The swap procedure also updates the queues accordingly to keep them consistent in terms of storing the assigned and proposed elements.
6.2.2 Better Algorithm: Rules-based Space Adaption Algorithm The main issue with Algorithm 1 is that it has to scale on the level of all the vertices. To avoid this problem, we make use of the operational rules derived for the indexes, join cache, and replications. Each operational rule includes an access rule, a destination index, and a benefit function. The included access rule has a set of sources and an access function. That are all the needed to calculate the benefits of each vertex represented by the sources set. However, we don’t have to project the benefits on the vertices. Instead, we project the benefit function on the sources, and compare the sources instead of comparing the vertices. Since that the number of the rules are limited and so that their sources, we can perform dramatical performance optimization to the adaption algorithm given by Algorithm 1, and use Algorithm 2 instead.
The algorithm works whenever enough workload change has been detected. The new workloadQbesides the data set graphG, the index types setXand two sets of rules are the inputs to the algorithm. The first set of rules is called the proposed rulesRp, which are the rules that have their sources representing the data that are ready to be assigned to the working nodes’ main memory. In the same context, the assigned rules are representing the data that were already assigned to memory.
The first loop updates both of the two rules sets with the new workloadQusing the procedureupdateRulesAccess(r, Q). This requires updating the inline heat queries, statistics, and recalculating the benefit formulas. In Line 3, each proposed rule is
going to have its sources sorted descendingly by their benefits, such that we have the most beneficial source element in the head of the rule sources. In the contrast, each assigned rule will have its sources ascendingly sorted by their benefits in order to bring the least beneficial source element to the head of the rule sources. The sources sorting process is carried out using Property 4 of the rules’ properties given in Section 3.4. The second loop takes one rulerp from the proposed rules set, and another rule ra from the assigned rules set. rp is the rule that has the best source, while ra has the worst source in terms of the benefits. These assignments of rp and ra are taken place in Line 6 and Line 7 respectively. This operation requires scanning the rules set and comparing the rules by their benefits. Each rule$ has its set of sources s.
However, since $ is sorted, only the source head s¯from each rule is considered in the scan process.
The algorithm halts normally when the benefit of the worst assigned rule is greater or equal than the benefit of the best proposed-rule. However, there are other practical reasons to halt the process not mentioned here, for instance, in case of no more proposed rule is available.
At Line 11, we evaluate the head source of rule rp to produce the vertices set Vˆp. The same is done in Line 12 to evaluate the head source of rule ra, and produceVˆa. The final step is to swap Vˆp with Vˆa using the procedure swapAssignment(rp, ra).
The procedure needs access to the full rules in order to get the destination index.
Moreover, the procedure marks in rp the source that has been assigned, and in ra the source that has been unassigned. This allows the algorithm to work again by keeping track of what has been assigned and what is still proposed.
In order for the algorithm to work as expected, any two rules in Rp need to be aggregated whenever there is an intersection between their set of vertices. This is because the sources of the rules represent the vertices, and we need to have one benefit value per vertex in order to correctly compare them. This is already performed during the build of the operational rules in the previous chapter, and We further discuss this point in Section 6.3.
Running Time
To analyse the running time of the algorithm, we look at its two loops. The first loop has a size ofO(|R|). In each iteration, we sort one rule by its sources sets. This costs O(|s|log|s|) for each rule, which makes the total loop cost within O(|R| · |s|log|s|).
In the second loop, we also iterate over the rules sets, and preform constant work on the head source of each rule. However, the costs of the evaluation parts are
dynamic and depend on the complexity of the source patterns and the availability of the indexes. The cost of the swap procedure is linear with the number of vertices to be swapped by the algorithm, which also depends on the detected changes in the workload which are translated into changes in rules benefits.
Compared to Algorithm 1, we evaluate the vertices in Algorithm 2 at runtime, so that we substantially reduce the high cost of maintaining the benefits on the vertices levels by maintaining the benefits on the rules’ sources level. Moreover, the running time of Algorithm 2 is delimited by|R| · |s|which is small number compared to the data graph vertices.
Algorithm 2:Rules-based space adaption algorithm
input :RDF graphG={V, E}, and two sets of the system operational rules: proposed rulesRp
and assigned rulesRa