Experimental Setup

sequence from the predefined vocabulary of the complete target text sequences.

p(ym|y<m,x)=softmax(Wsh˜_m) (5.2.11)

WhereWsis weight matrix of softmax layer andpis probability. Please note that the decoder stops producing words once it encounters EOS (end of sequence) token ormis equal to M.

as Wikidata?” To the best of our knowledge, it is a pioneering work for the task of entity linking on the Wikidata knowledge graph where it considers the inherent challenges (noisy nature, long entity labels, implicit entities). Therefore, we do not compare our approach to generic entity linking approaches which typically either do not use any background knowledge or employ the well-established knowledge graphs such as DBpedia, YAGO, Freebase. Our approach Arjun comprises all three tasks illustrated in figure 5.2. To elaborate the advantage of inducing additional context post NER step, we built a "baseline"

which is an end to end neural model. The "baseline" in our case is the attentive neural network employed in Task 1 without any background knowledge (or can be seen as an end to end EL using an attentive neural network). In fact, in task (1) (cf. Figure5.2), the baseline directly maps the text to a sequence of Wikidata entities without identifying surface form candidates. Hence, the baseline approach is the modified version of Arjun. With a given input sentence, the baseline implicitly identifies the surface forms and directly links them to Wikidata entities. Unlike Arjun, the baseline does not use any KG context for the expansion of the surface forms. We also compare Arjun with the recently released SOTA for Wikidata entity linking- OpenTapioca [185], which is an end to end EL approach. We are not aware of any other end to end EL tool/approach released for Wikidata.

Training Details

Implementation details We implemented all the models using the PyTorch framework. The local KG and the semantic search is implemented using Apache Lucene Core⁶and Elastic search [238]. The semantic search returns entity candidates with a score (higher is better). We reuse the implementation of Falcon local KG [24] for the same. After empirically observing the performance, we set the threshold score to 0.85 for selecting the potential entity candidates per surface form (i.e. the parameter is optimised on the test set). We reused pre-trained word embeddings from Glove [51] for the attention-based neural network. These embeddings have been pre-trained on Wikipedia 2014 and Gigaword 5⁷. We employ 300-dimensional Glove word vectors for the training and testing of Arjun. The models are trained and tested on two Nvidia GeForce GTX1080 Ti GPUs with 11GB size. Due to brevity, a detailed description of training details can be found in our public Github.

Dataset Preparation We experimented initially with higher text sequence lengths but resorted to 25 words due to GPU memory limitation. In total, we processed 983,257 sentences containing 3,133,778 instances of surface forms (not necessarily unique entities) which are linked to 85,628 individual Wikidata entities. From these 3,133,778 surface forms occurrences, approximately 62% do not have an exact match with a Wikidata entity label.

Results

Table5.1summarises the performance of Arjun compared to the baseline model and another NED approach. We observe nearly 8% improvement in the performance over baseline, and Arjun significantly outperforms another end to end EL tool OpentTapioca. Arjun and OpenTapioca generate entity candidates on the fly, i.e., out of Millions of Wikidata entities, the task here is to reach to top-1 entity. This contrasts with other end to end entity linking approaches such as [37], which rely on a pre-computed list of 30 entity candidates per surface form. This translates into extra complexity due to large search space for generating entity candidates in the case of Arjun. Our solution demonstrates a clear advantage of using KGs as background knowledge in conjunction with an attention neural network model. We now detail some success and failure cases of Arjun.

6https://lucene.apache.org/core/

7https://nlp.stanford.edu/projects/glove/

Success Cases of Arjun Arjun achieves 0.77 F-Score for the surface form extraction task. Arjun identifies the correct surface form for our exemplary sentence S1 (i.e. ASIC) and links it to the entity label Application Specific Integrated Circuitofwiki:Q217302. The baseline can not achieve the linking for this sentence. In the Local KG, the entity label ofwiki:Q217302is enriched with aliases that also containASIC. This allows Arjun to provide the correct linking to the Wikidata entity containing the long label. Background knowledge induced in the attentive neural network also allows us to link several long entities correctly. For example, in the sentence "The treaty of London or London convention or similar may refer to," the gold standard links the surface form London conventionwith the labelConvention on the Prevention of Marine Pollution by Dumping of Wastes and Other Matter(c.f.

wiki:Q1156234). The entity label has 14 words, and Arjun provides correct linking. OpenTapioca, on the other hand, have a high recall(it has a high number of False Positives). However, the precision is relatively quite low. The limited performance of OpenTapioca was because it finds limitation in linking non-Wikipedia entities that constitute a major portion of the dataset. This demonstrates the strength of Arjun in also linking non-standard, noisy entities which are not part of Wikipedia.

Failure Cases of Arjun Despite the successful empirical demonstration of Arjun, we have a few types of failure cases. For example in the sentence: ‘Two vessels have borne the name HMS Heureux, both of them captured from the French’ has two gold standard entities (HeureuxtoFrench ship Heureux (wiki:Q3134963) and French toFrench (wiki:Q150)). Arjun links Heureuxto L’Heureux (wiki:Q56539239). This issue is caused by the semantic search over the Local KG while searching for the potential candidates per surface form. In this case, L’Heureux is also returned as one of the potential entity candidates for the surface formHeureux. A similar problem has been observed in correctly mapping the surface formCatalantowiki:Q7026(Catalan Language) where Arjun links Catalanto Catalan (wiki:Q595266). Another form of failure case is when Arjun identifies and links other entities which are not part of the gold standard. The sentence ‘Tom Tailor is a German vertically integrated lifestyle clothing company headquartered in Hamburg’ has two gold standard entity mappings:vertically integratedtovertical integration(wiki:Q1571520andHamburgto Hamburg(wiki:Q1055). Arjun identifiesTom(wiki:Q3354498) andTailor(wiki:Q37457972) as the extra entities and can not linkvertically integrated. For brevity, a detailed analysis of the failure cases per entity type (very long label, noisy non-standard entity), performance loss due to semantic search can be found in our Github.

Limitations and Improvements for Arjun Arjun is the first step towards improving a deep learn-ing model with additional contextual knowledge for EL task. Arjun can be enhanced in various directions considering current limitations. We list some of the immediate future extensions:

1. Enhancing Neural Network Multiple layers: Arjun currently has a Bi-LSTM and a single layer LSTM for the encoder and the decoder, respectively. It has been empirically observed in sequence

Table 5.1: Performance of Arjun compared to the Baseline.

Method Precision Recall F-Score

baseline 0.664 0.662 0.663

OpenTapioca [185] 0.407 0.829 0.579

Arjun 0.714 0.712 0.713

to sequence models for machine translations that the models show significant improvements if stacked with multiple layers [239]. Therefore, with more computing resources, the neural network model used in Arjun can be enhanced with multiple layers.

2. Alternative Models: In this article, our focus is to empirically demonstrate how background knowledge can be used to improve an attentive neural network for entity linking. Several recent approaches [31,32,240] enhance the performance NER and can be used in our models for task (1) and task (3).

3. Improving NER:there is a room of improvement regarding surface form extraction where Arjun currently achieves an F-score of 0.77. The latest context-aware word embeddings [241] can be re-used in Arjun or completely replacing NER part with latest language models such as BERT [32].

4. Replacing Semantic Search: Another possibility of improvement is in the second step of our approach (i.e., inducing background knowledge). Currently, we rely on very trivial semantic search (same as [24]) over the Local KG to extract Wikidata entity candidates per surface form.

Ganea et al. [196] developed a novel method to embed entities and words in a common vector space to provide a context in an attention neural network model for entity linking. This approach could potentially replace semantic search. Classification is seen as one of the most reasonable and preferred ways to prevent out of scope entity labels [37]. On the contrary, Sakor et al. [24]

illustrated that expanding the surface forms the way we did, works pretty well for short text.

We hypothesised that it should also work for Arjun, which is not completely true if we see our empirical results. Hence, in this paper, we do not claim that every step we took was the best, but after our empirical study, we demonstrate that the candidate expansion by Sakor et al. doesn’t work well. However, it solves our purpose of inducing context in the NN, which is the main focus of the paper. It leads to an interesting discussion: what is the most efficient way to induce KG context in a NN, maybe the classification one?- one need to prove empirically, and we leave it for future work.

5. Coverage restricted to Wikidata: Effort can be made in the direction to develop a common EL approach targeting multiple knowledge graphs with standard and nonstandard entity formats.