Open Domain Question Answering

Answering factual queries about entities may also be regarded from the perspective of NLP research. Natural language question answering is a well-researched topic. In general, two fields are distinguished, answering questions where the answer is found within a small text paragraph, called closed book question answering, or open-book question answering (or open domain question answering) where a possible answer is found in some larger text corpus [19]. The general idea of open domain question answering is as follows: An information retrieval component is used to search a large corpus of text to retrieve relevant text passages that contain the answer to the question. This retrieval component often involves standard information retrieval ranking techniques, such as TF-IDF or BM25. Then a standard question answering method (the reading component) is used to mark the word span within this text passage that is the answer to the question.

Novel techniques for open-domain question answering have shown that using a dense passage retrieval mechanism for retrieving relevant passages together with a standard reading component outperforms methods using standard information retrieval techniques [59]. The retrieval model is trained on a small set of questions and relevant text passages. It then is able to create dense vector representations of questions and arbitrary text passages. The similarity between a question and a passage is used to determine whether it is relevant or not. Relevant passages for some questions are then put into a standard reading component for question answering that identifies an answer span within this passage that is used as an answer to the question. In the experiments, the dense passage retrieval approach is evaluated on

several standard benchmark datasets. The reported accuracy varies between 20%

and 60%, thus far from being perfect.

Knowledge graph query answering, as presented in our work, resembles question answering, since in both tasks, some query, either in natural language or in a structured query language, needs to be answered. In contrast, however, the answer in knowledge graphs are entities with a unique identifier, whereas natural language answers may be ambiguous entity names.

5.2 Preliminaries

Our hybrid query answering system is built to work with RDF-based knowledge graphs and masked language models to answer SPARQL queries. In this preliminary section, we introduce the basic ideas of language models and how to query language models with SPARQL.

An RDF-based knowledge graph was defined as a set of triples KG⊆E×R× (E ∪L) in Section 2.1. Querying such a graph is usually performed by the query language SPARQL. We restrict ourselves to simple, entity-centric queries with only a single BGP, where the variable may either be in the subject position or the object position. We call queries with a variable in subject position a subject query and a query with a variable in object position an object query.

Language models are statistical models trained on a large text corpus to learn to predict upcoming words given a sequence of words. Large transformer-based (a new architecture for artificial neural networks) language models have recently gotten much attention in natural language processing. In this work, we focus on the masked language model BERT [25]. During BERT training, the model is fed with sentences from a large text corpus to either learn to predict a word within the sentence or the next sentence given the previous sentence. This process is called self-supervised training since no additionally labeled training data is needed. After an extensive and resource-intensive pre-training, which is performed on large amounts of text for several epochs, the language model predicts words in arbitrary sentences, given the knowledge it has gathered in the training process. As an example, the sentence Albert Einstein was born in ..., would be completed by the most probable word that fits the context of the sentence. Usually, the language model forms grammatically correct sentences. From the name Albert Einstein, it might infer that this is a German-sounding name that fits a German city. The language model might have even stored that some text mentioned that Einstein was born in Ulm. So it may correctly predict the word Ulmwhich is the correct birthplace of Einstein. Following the idea of Petroni et al. [90], this mechanism is employed to answer simple BGP SPARQL queries.

5.2 Preliminaries

Example. To illustrate the querying process, we use our running example, asking for the birthplace of Einstein again.

SELECT ?birthplace WHERE (

<Albert_Einstein> <bornIn> ?birthplace.

)

This SPARQL query may be transformed into the sentence Albert Einstein was born in [MASK]. It is used as an input for the language model. The language model then returns a list of the most probable words and their confidence values.

For each relation in the knowledge graph, we need sentences that are used to translate queries. We use a natural sentence with two placeholders: one to be substituted with the subject entity, the other one with the object entity of some relation: [S] was born in [O]. This natural language query is called template, sometimes also known as prompt.

Previous work has shown that the querying process is improved by concatenating additionalcontext sentences to the template query [89]. If relevant context sentences about the entity of interest are picked, the precision may be improved by around 30%. We have implemented this idea of Petroni et al. by using the beginning of Wikipedia abstracts for the respective entities as a context paragraph.

Example. Hence, if our query is about Einstein’s birthplace, we retrieve the first five sentences of Einstein’s Wikipedia article and append it to the template sentence separated by a separator token ([SEP]). We provide context with the first sentence of Einstein’s Wikipedia article as follows:

Albert Einstein was born in the city of [MASK].[SEP]

Albert Einstein was a German-born theoretical physicist who developed the theory of relativity, one of the two pillars of modern physics (alongside quantum mechanics).

To answer the query which was translated to a template, the language model now returns a list of words that replace the [MASK] token from the sentence. The result list returns single words together with a confidence value.

1. a - 0.95 2. the - 0.93 3. Germany - 0.86 4. Ulm - 0.79 5. Berlin - 0.70

This list usually contains several words which are grammatically correct but are not necessarily a valid answer to the query. In the case of our example, we also have the correct answerUlm in the set of results, but also several results that need to be filtered out.

Thus, to build a functioning querying system, we further filter the result list of the language model by applying multiple cleaning steps which make use of the

Figure 5.1: An overview of the query answering system KnowlyBERT.