Query Planning and Optimization
6.3 O ntario : Federated Query Processing over Semantic Data Lakes
Different subqueries, i.e., star-shaped subqueries, behave differently depending on the data source capabilities. Ontariois able to differentiate the following star-shaped groups:
1. CI: In this category, the star-shaped group do not have a constant object (instantiation) or a filter clause on object variables in any of triple patterns.
2. CII: Star-shaped groups in this category do not have a constant object or a filter clause in any of triple patters. Further, these star-shaped groups are defined over RDF-MTs described in terms of joins of two or more relations in the Data Lake.
3. CIII: Star-shaped groups in this category are composed of triple patterns with constant objects or contains filter clauses on object variables.
4. CIV: In addition to constant objects or filter clause, the star-shaped groups in this category are defined over RDF-MTs described in terms of joins of two or more relations in the Data Lake.
As to be shown in subsection 6.4.1, existing database engines (e.g., RDF or relational engines) may exhibit diverse performance during the execution of these star-shaped groups. For example, RDF engines will have expensive executions of star-shaped group in CI and CII, while relational engines will perform
6.3 Ontario: Federated Query Processing over Semantic Data Lakes Algorithm 4Query Plan Generation:Φ- query decomposition, Q - SELECT query
1: procedureCreatePlan(Φ,Q)
2: αÐ rs
3: forS QPΦdo
4: orderTriples(SQ)
5: α.push(SQ)
6: end for
7: PÐQ.pro jspq ŹQ.projs() - list of join and projected variables
8: αÐOrderSSQs(α,P)
9: whilelenpαq>1do
10: S Qi Ðα.poppq
11: δÐ rs
12: βÐ rS Qj f or S Qj Pαif shareVarspS Qi,S Qjqs
13: βÐOrderSSQs(β,P)
14: forS QjPβdo
15: JÐjoin(S Qi,S Qj)
16: α.remove(S Qjq
17: α.push(J)
18: break
19: end for
20: if|β| “ 0then
21: δ.push(S Qiq
22: end if
23: end while
24: iflenpδq>0then
25: αÐjoin(α, δq
26: end if
27: returnα
28: end procedure
badly on subqueries in CIII and CIV if no indexes exist over the instantiated or joined attributes. Contrary, since RDF engines always create indexes over the subject, predicates, and objects of the triples in an RDF graph [101], subqueries in CIII and CIV will be sped up in RDF engines.
6.3.1 Heuristics
Once the star-shaped groups and the sources where they will be executed are identified, the Ontario optimizer uses a set of heuristics to build query plans. These heuristics are guided by the general characteristics of the star-shaped groups, the type of star-shaped group, and the type of selected data sources.
General Heuristics
Ontario optimizer first consider the general characteristics of the star-shaped groups, referred to as general heuristics, while generating the execution plan of a given query. The general heuristics includes:
percentage of constants in the star-shaped groups, ordering of triple patterns within a star-shaped group,
number of projected variables within a star-shaped group, ordering of star-shaped groups based on selected data source types, and capability of storage engines for a selected data source.
Ordering between Star-Shaped Groups. Triple patterns with the highest number of constants at any part of the triple pattern are more selective. When ordering two star-shaped groups, a star-shaped group with the highest percentage of constants have higher precedence. Especially if bound join physical operators is selected, executing the more selective star-shaped group first could significantly reduces the size of intermediate results and minimize the execution time.
Triple Pattern Ordering within Star-Shaped Groups. The order of triple patterns in star-shaped group could affect the performance of the execution engine, e.g., triple stores. A set of triple patterns in a star-shaped group are ordered as follows: spoąspąsąpoąoąp. That is, a triple pattern with constants in all parts precedes a triple pattern with constants in subject and predicate. Similarly, a triple pattern with constants at subject and predicate precedes a triple pattern with a constant at subject part only, and so on.
Number of Projected Variables in Star-Shaped Groups. Given two star-shaped groups that have the same number of constant percentages, a star-shaped group with less number of projected variables have higher precedence. Star-shaped groups that project more values could consume more memory. This heuristics makes sure that such subqueries have a matching data with the rest of star-shaped groups to reduce memory usage.
Ordering based on Data Source Type. SSQs that are executed in endpoints with efficient indexes by a database management system (DBMS) are more selective than non-indexed raw file systems. For example, TripleStoreą“RDBMSą“GraphąDocumentąHDFSąLOCAL_FILE.
Source Capabilities.A sub-query evaluated by a data source with capabilities to push down join and filter operation precedes a sub-query to be evaluated in a data source with less capabilities. Capabilities include, indexes, optimization, join operators, filters, etc. For example, MongoDB supports a poor join operation between documents.
Star-Shaped Group Based Heuristics
The second type of heuristics the Ontariooptimizer uses is based on the type of star-shaped groups in a subquery. These heuristics are defined by both the category of star-shaped group and storage engine of the selected data source. They enable the transformation of subqueries in one category into subqueries in another category.
Pushing down instantiations into a Star-Shaped Group. This rule is performed whenever a star-shaped groupS S Qiof type CI is executed over an RDF engine in a queryQ. IfS S Qiis part of a join in Q,S S Qiis selected as the inner subquery of the join; a nested loop join is chosen as the physical operator.
Additionally, if variables in triple patterns ofS S Qiare part of a filter inQ, the filter is represented as an instantiation ofS S Qi. Thus, this rule transforms subqueries in CI into subqueries in CIII.
Breaking up joins in Star-Shaped GroupsThis rule is performed whenever a star-shaped group S S Qi of type CII is executed over an RDF engine in a queryQ. In this case,S S Qiis divided into as many subqueries as joins are defined in the corresponding RDF-MT and the attributes used inS S Qi. These subqueries are connected by nested loop join operators that will be executed at Ontariolevel.
Thus, this rule enables for transforming subqueries in CII into subqueries in CIV.
Pushing up instantiations into a Star-Shaped GroupThis rule is performed whenever a star-shaped groupS S Qiof type CIII is executed over an RDB engine in a queryQand the instantiation is not over an indexed attribute. IfS S Qi is part of a join inQ, hash join (or gjoin [34]) is chosen as the physical operator. Further, the selection is represented as a filter which is performed at Ontariolevel. Thus, this rule enables for transforming subqueries in CIII and CIV into subqueries in CI and CII, respectively.
6.3 Ontario: Federated Query Processing over Semantic Data Lakes
RDF-MT Catalysts
Data Lake
Relational Graph Document Raw File Data
Catalyst Lake Catalyst
Molecule Synthesizer
Data Lake Atomic Synthesizer
Graph Synthesizer
Figure 6.5:Semantic Data Lake Basic Components: Lake Catalyst catalyzes a SPARQL query into a set of star-shaped groups and select RDF-MTs matching each sub-query. RDF-MT Catalysts catalyze star-shaped groups into subqueries that can be executed in different Data Catalysts. Finally, Data Catalysts execute a sub-query in a data source by translating a given SPARQL query to native query language of the data source. Conversely, the results returned from each these components need to be synthesized (by Atomic, Molecule, and Graph Synthesizers) and passed to the upper layer.
Combining joins into a Star-Shaped GroupThis rule is performed whenever two star-shaped groups S S Qi andS S Qj of type CI are executed over an RDB engine, and there is a join between them over an indexed attribute. S S Qi and S S Qj are merged into one star-shaped groupS S Qi,j, transforming subqueries in CI into subqueries in CII.
6.3.2 Query Plan Generation
Given a SPARQL query, Ontarioproduces a decomposition composed of star-shaped groups (SSQs).
Using RDF-MT descriptions, Ontariofinds a matching RDF-MT for each SSQs. An RDF-MT matches an SSQ if it contains the same predicates as in SSQ. The selected SSQs composed a decomposition,Φ, which represents the input for the optimizer sketched in Algorithm 4. The optimizer is guided by the heuristics in section 6.3.1 and subsection 6.3.1. The planner first performs triple ordering within each SSQs, Line 3-5. The planner then orders SSQs inΦbased on their selectivity, categories, and the data sources, Line 7. Then, iteratively, it picks anS S Qi(Line 9) and joinableS S Qjs (Line 11), i.e., that share same join variable and create join between them, Line 13-16. Joins are selected and SSQs are ordered according to the heuristics in subsection 6.3.1. Figure 6.6 illustrates the Ontarioarchitecture.
6.3.3 The OntarioArchitecture
In this section, we present the architecture of Ontario. First, we present an overview of basic components in a Semantic Data Lake architecture, then we describe the Ontarioarchitecture with respect to these components. Figure 6.5 shows an overview of the Semantic Data Lake components. The top layer catalyzes, i.e., via the Lake Catalyst, a given SPARQL query into a set of star-shaped groups and selects matching RDF Molecule Templates. The Lake Catalyst passes the sub-queries to the respective RDF-MT Catalysts that are responsible for specific RDF Molecule Templates. It dispatches the decomposed query and coordinates the global query planing and optimization. In addition, once the results are returned from lower layer, i.e., from Molecule Synthesizers, the Graph Synthesizer combines (synthesizes) the molecules and generate the final result. The second layer catalyzes, i.e., via RDF-MT Catalysts, the star-shaped groups into a set of API calls to the Data Catalysts, which then transform the raw data
Document
Graph RDB
HDFS File File
File File
Data Sources
Data Integration and Semantification via mappings
Wrapper Wrapper Wrapper
Wrapper Wrapper
Source Selection and Query Decomposition
Query Planning Query Execution Result conciliation
SPARQL Query Results
TripleStore
Multi-mode Local Flat-file
Privacy and Access Control
Figure 6.6: The OntarioArchitecture
on-the-fly and synthesize results, i.e., via Molecule Synthesizer, returned from Atomic Synthesizers.
RDF-MT Catalysts are specialized components that deal with only one specific RDF Molecule Template, and provide decomposition, planning and execution of a particular star-shaped group. The third layer provide access to a specific data source by translating queries from global querying mechanism, e.g., SPARQL, to the underlying native query mechanism of the data sources via Data Catalysts. Atomic Synthesizers, on the other hand, transform raw data to RDF on demand by applying mapping rules, e.g., defined by RML or R2RML mapping languages. They are specialized to a specific data model and system interface. Atomic Synthesizers perform transformation of the results from native data sources to RDF based on transformation rules. The bottom layer, Data Lake, provides an infrastructure to store raw data and access interface to a set of heterogeneous data sources. These data sources can be characterized with different properties, such as autonomy (sources can be autonomous), data format heterogeneity (provides different data formats), access interface heterogeneity (various query languages), semantic heterogeneity (different representation of same data points), volume (different sizes from small to large data sets), access restrictions, etc.
In Figure 6.6, the Ontarioarchitecture is presented. Given a SPARQL query, Ontariodecomposes into a set of star-shaped groups with matching relevant RDF Molecule Templates. For each sub-query the set of RDF-MT Catalysts that can evaluate each matching RDF-MT are selected. Given a set of star-shaped groups and relevant data sources for matching RDF-MTs, Ontarioquery planner creates a bushy-tree plan utilizing the heuristics described in Section 6.3.1. The query optimizer, optimizes the generated plan and selects the appropriate operators based on the source capability and source type heuristics. The corresponding RDF-MT Catalysts then execute each sub-query, which internally contacts the appropriate Data Catalysts to synthesize the (sub-graph of) RDF Molecule Templates. Finally, the OntarioLake Catalyst merges the results to synthesize the final results and return back to the user. During the catalysis of the given SPARQL query in the Lake Catalyst, RDF-MT Catalysts as well as Data Catalysts, the privacy and access rules set by each data sources must be respected. Privacy and access control constrains must be checked as well during synthesis of results by Atomic Synthesizer, Molecule Synthesizer, and