visualizing a concept map — as a subsequent step that can be approached independently, in particular within the visualization community.16
3.3 Subtasks and Challenges
In order to create a summary concept map for a collection of documents, multiple subtasks have to be solved. We will illustrate these tasks and their challenges for computational approaches based on the example shown in Figure 3.1.
3.3.1 Concept Mention Extraction
Given the input documents𝐷, the first task is to identify all mentions of concepts in the text of these documents. We will denote the set of mentions as𝑀.
Mentions of concepts can be of various syntactic types as the annotation study by Villalon et al. (2010) (see Section 2.2.3) already showed. Examples shown in the upper part of Figure 3.1 are nouns such as hypnosis and caffeine, proper nouns such asginkgo
biloba, more complex noun phrases likethe core symptoms of ADHD but also verb phrases that describe activities such aseating a healthy, nutritious diet. In addition, concepts are often referred to usingpronouns, as in sentence (6) of the example.
A main challenge of this subtask is to find an extraction approach with a goodtrade-off between precision and recall. A perfect method would exactly identify the highlighted spans of Figure 3.1. In practice, however, most methods are not perfect. If the precision is high, i.e. all identified spans are indeed mentions of concepts, some constructions are usually missed, lowering the recall. On the other hand, methods trying to obtain a higher recall might extract too many mentions, including false positives. In light of the subsequent steps, a high recall is important, as concepts that are missed here can never make it into the summary, but a very large set of extracted mentions also makes the later selection of summary-worthy elements harder in terms of the number of options to select from. An-other challenge isgeneralizability: Extracting spans of a certain syntactic structure might yield only correct mentions on the text it was designed for, but may be too broad and cover many undesired spans on other types of text. Ideally, we want a method that is broadly applicable to many types of text.
3.3.2 Concept Mention Grouping
Having identified all concept mentions 𝑀, the next subtask is to group mentions of the same concept together. More formally, we want to find a partitioning 𝐶 of𝑀 such that
16A vast amount of past and ongoing research exists on visualizing graphs. Surveys such as Di Battista et al.
(1998), Herman et al. (2000), Katifori et al. (2007) and Liu et al. (2014) can serve as entry points into this field.
Chapter 3. Structured Summarization with Concept Maps
Concept Mention Extraction and Grouping
(1) Caffeine , which is a mild CNS stimulant , reduces ADHD symptoms . (2) Hypnosis has little to no effect on the core symptoms of ADHD .
(3) Herbal supplements such as ginkgo biloba have been used to treat the symptoms of ADHD . (4) Hypnotherapy might help with common ADHD symptoms .
(5) However, it seems that it is not a very effective means of controlling ADHD symptoms . (6) Eating a healthy, nutritious diet can clearly benefit all children .
𝑀 = {Caffeine,a mild CNS stimulant,ADHD symptoms,Hypnosis, … } 𝐶 = {{Caffeine}, {ADHD symptoms, … }, {Hypnosis,Hypnotherapy,it}, … } Relation Mention Extraction and Grouping
(1) Caffeine, which is a mild CNS stimulant, reduces ADHD symptoms.
(2) Hypnosis has little to no effect on the core symptoms of ADHD.
(3) Herbal supplements such as ginkgo biloba have been used to treat the symptoms of ADHD.
(4) Hypnotherapy might help with common ADHD symptoms.
(5) However, it seems that it is not a very effective means of controlling ADHD symptoms.
(6) Eating a healthy, nutritious diet can clearly benefit all children.
𝑂 = {(Caffeine,is,a mild CNS stimulant), (Caffeine,reduces,ADHD symptoms), … } 𝑅 = {(𝑐4, {has little to no effect on,is not a very effective means of controlling}, 𝑐1), … } Importance Estimation
𝑐1= {ADHD symptoms, … } 0.9 ({Hypnosis, … }, ?, {ADHD symptoms, … })
𝑐2= {ginkgo biloba} 0.3 {might help with} 0.3
𝑐3= {all children} 0.8 {has little to no effect on,is not a very …} 0.8
… …
Concept Map Construction
reduces is
has little to no effect on
have been
used to treat such as can clearly benefit caffeine
mild CNS stimulant
ADHD symptoms
hypno-therapy
herbal supplements
ginkgo biloba
children
eating a healthy, nutritous diet
Figure 3.1: Subtasks of CM-MDS illustrated by examples. Six sentences are used as input. Men-tions of concepts and relaMen-tions are highlighted, color indicates groupings. Lightly drawn parts of the graph are not part of the summary. Note that the input sentences were picked to illustrate specific challenges, a real input text would be more coherent and have more complex sentences.
3.3. Subtasks and Challenges
every subset of 𝐶contains mentions of a single, unique concept. The main challenge of this subtask is the variety of expressions that can be used to refer to the same concept.
While several mentions of a concept can use exactly the same words, such asADHD symptoms
in (1) and (5) of the example, the same concept can also be mentioned usingsynonymsor paraphraseslikethe symptoms of ADHDor using apronoun.
In addition to mentions that clearly refer to the same concept, one might also want to group mentions of slightly different concepts. Examples include concepts with ahyponymy relation, such asthe symptoms of ADHDandthe core symptoms of ADHD, or ameronymy rela-tion, such as hypnotherapy and hypnosis. Since the ultimate goal is to create a summary of restricted size, the differentiation between them might not be needed. Ideally, such a merging decision has to be made considering the propositions that should be expressed by the final summary map, as they define the necessaryconcept granularity. And of course, one has to avoid that this leads to propositions that are not assertedby the text. For in-stance, if the second concept mention in sentence (6) would besome children, grouping it with other mentions to a conceptchildrenwould be problematic, since the (hypothetical) sentence would not assert that eating a healthy diet is good for (all) children.
Finally, ascalabilitychallenge arises if the methods that determine whether two men-tions refer to the same concept are based on pairwise comparisons, causing a quadratic, i.e.
𝒪(𝑛2), runtime complexity. Such approaches can therefore only scale to a certain number of concept mentions and consequently only to document collections of a limited size.
3.3.3 Relation Mention Extraction
The goal of relation extraction is to identify all mentions of relations𝑂, where each ele-ment𝑜 ∈ 𝑂is a tuple(𝑚1, 𝑞, 𝑚2)with𝑞expressing the relationship between the concepts referred to by𝑚1 and𝑚2. We refer to the tuple as aproposition.17
Similar to concept extraction, challenges of this subtask are thevarietyof expressions that are used to describe relationships and finding a goodtrade-off between precision and recallwhile ensuringgeneralizability. For relation mentions, the dominant class of verb-mediated expressions, such asreducesormight help within Figure 3.1, accounts for only 47%
of mentions in Villalon et al. (2010)’s annotation study. Other examples are prepositional phrases or adjectival phrases likesuch asin sentence (3). Moreover, some relations might be expressedacross sentence boundariesand are therefore more difficult to extract. However, if one tries to go beyond sentence boundaries, an additional scalability challenge might arise, as any of the𝑛2pairs of concept mentions in scope could be the source and target of a beyond-sentence relation mention. Within a sentence,𝑛is typically so small (𝑛 ≤ 5) that this does not constitute a challenge.
17We use the term proposition both on the document level, referring to a tuple of two concept mentions and a relation mention, and on the concept map level, referring to a tuple of two concepts and a relation.
Chapter 3. Structured Summarization with Concept Maps
Furthermore, since this subtask introduces propositions, the assertednessof them by the text has to be ensured. As negative examples, consider the following two sentences:
No studies prove that chiropractic helps with symptoms of ADHD.
Some people think diet supplements improve symptoms of ADHD.
Here, because both sentences consist of several nested statements, the inner propositions
(chiropractic, helps with, symptoms of ADHD)and(diet supplements, improve, symptoms of ADHD)should not be extracted, as they are not valid without their context. Using them for CM-MDS would introduce false statements into the concept map. However, as both concept and relation labels of the concept map should be as conciseas possible, we want to drop unnecessary nested clauses such as, for instance, non-essential relative clauses. A careful understanding of sentence structure is necessary to satisfy both goals.
3.3.4 Relation Mention Grouping
Similar to concept grouping, the goal of this subtask is to partition mentions of the same relation together. In contrast to concepts, we first group all mentions 𝑂 by the pairs of concepts they were extracted for and then partition the mentions per pair. The result is a set𝑅of relations, containing tuples(𝑐1, {𝑞1, 𝑞2, … }, 𝑐2)that consist of two concepts𝑐1, 𝑐2
— both sets of concept mentions — together with a set of relation mentions describing a unique relation between the two concepts.
As an example, consider sentences (2), (4) and (5) in Figure 3.1, each of them contain-ing a mention of a relation between the conceptshypnosis(which itself has three different mentions) andADHD symptoms(having five mentions in total). During relation grouping, we want the mentions in (2) and (5) to form a relation, as they express the same idea, while mention (4) is a different relation. Figure 3.1 shows the desired result. The challenges here are similar as for concepts, stemming mainly from thevarietyof ways in which the same relation can be expressed. In contrast to concepts, the scope and thereby also the relevance of this subtask is much smaller though because the partitioning is done per pair. That makes the sets of mentions that have to be partitioned substantially smaller, such that in practice comparison-based quadratic approaches are also more feasible.
3.3.5 Concept and Relation Labeling
In𝐶and𝑅, both concepts and relations are represented as sets of mentions. To show them in a concept map, a representative label is needed. Formally, this is a function𝑙 mapping from 𝐶 ∪ 𝑅 to sequences of words. One approach is to select one of the mentions as the representative label. For the concept map in Figure 3.1,ADHD symptomshas been chosen
3.3. Subtasks and Challenges
among the different mentions of that concept. And for the relation mention group discussed above,has little to no effect onis used as the representative label.
The main goal of labeling is to haveconciselabels. There can be cases where, in order to achieve that goal, we want to go beyond the available mentions, as for example the concept labeledchildren, which has only one, slightly longer mentionall children. Note that this is a first step towards abstractive concept maps, but a shortened mention is still a sequence of words present in the source text. Similar as for concept grouping, shortening mentions can introduce propositions that are notassertedby the text. If we had a concept with the single mentionsome children, labeling it aschildrenwould make the propositions it participates in more general than sanctioned by the text.
3.3.6 Importance Estimation
Since the summary concept map, being a summary, should typically only contain a subset of all concepts 𝐶 and relations 𝑅 found in the previous steps, the importance of these elements has to be determined. The subsequent and final step can then use this information to construct a map that contains only the most important concepts and relations. One way to model this is as a function 𝜈assigning real-valued scores to elements in𝐶 ∪ 𝑅.18 Figure 3.1 shows examples for this approach, giving more importance to the concept labeled
ADHD symptomsbut less toginkgo biloba. Correspondingly, relations also receive scores.
A challenging factor of this subtask is thesizeof𝐶and𝑅. The bigger these sets are, the more options one has to chose from, which tends to make the selection harder. Often, de-ciding whether a concept is important and should therefore be part of the summary requires external knowledgein addition to what is in the source text, e.g. common sense knowl-edge. In addition, these decisions can also be user-specific. Factors such as the specific information need of a user, their personal background knowledge and subjective feelings and opinions can influence which content should be part of a useful summary.
Discussing the challenges of concept and relation extraction earlier, we pointed out that a higher recall might make this subtask more difficult. A similar, but contrary relationship between these tasks exists with regard to precision: If the extraction approach has a non-perfect precision, introducing some erroneously extracted concepts and relations, they can still be filtered out during the selection, compensating the earlier mistakes. Thus, a notion ofqualitycan be an additional criterion when scoring concepts and relations.
3.3.7 Concept Map Construction
Finally, having concepts, relations and estimates of their importance, we have to construct the summary concept maps. Due to the grouping of concept and relation mentions before,
18Note that element-wise scoring is not the only possible approach to this problem. See Section 6.3 and surveys such as Liu (2009) for alternative ways to model this subtask.
Chapter 3. Structured Summarization with Concept Maps
propositions that mention the same concept are connected, yielding a graph𝐺 = (𝐶, 𝑅).
According to Definition 5, the summary concept map has to satisfy the size and connected-ness constraints while providing as many important aspects of the text as possible. Thus, this subtask amounts to selecting a subgraph 𝐺′ = (𝐶′, 𝑅′)with 𝐶′ ⊆ 𝐶 and 𝑅′ ⊆ 𝑅 from𝐺with respect to this goal and the constraints.
Figure 3.1 illustrates some challenges that this subtask entails. First of all, the size-constrained selection problem, i.e. choosing a subset𝐶′ ⊆ 𝐶such that|𝐶′| ≤ ℒ𝐶, already poses the problem ofcombinatorial explosion. Forℒ𝐶 = 3, there are 92 possible subsets of𝐶. In general, ∑ℒ𝑘=1𝐶 (|𝐶|𝑘)subsets exist among which one would need to find the best according to some criterion, e.g. the sum of importance estimates of the selected elements.
Second, theconnectednessrequirement makes the search for the best subset difficult, as it renders some subsets invalid. For instance, subgraphs can only containmild CNS stimulant
if caffeineis also included. Rather than simply adding the highest-scoring concepts, one might need to add some with a lower importance estimate in order to be able to include other high-scoring ones. Moreover, the graph𝐺can consist of several completely discon-nected components, as shown in the example. ADHD symptomsandchildren, both concepts deemed to be important, cannot be in a connected subgraph with the others.
We can formulate the selection as a constrained optimization problem. Given the graph 𝐺 = (𝐶, 𝑅), importance estimates 𝜈 ∶ 𝐶 ∪ 𝑅 → ℝ≥0 and size restrictions ℒ𝐶, ℒ𝑅, we want to find the subgraph𝐺′ = (𝐶′, 𝑅′)that maximizes importance:
arg max
𝐺′
∑
𝑐∈𝐶′
𝜈(𝑐) + ∑
𝑟∈𝑅′
𝜈(𝑟) (3.1)
s.t. |𝐶′| ≤ ℒ𝐶 ∧ |𝑅′| ≤ ℒ𝑅 ∧ 𝐺′ is connected
Here, the objective function prefers subgraphs containing more important elements while the constraints ensure connectedness and size. As a simpler formulation, one can also just select a subset of concepts𝐶′and let𝐺′be the node-induced subgraph of𝐺, i.e. the graph with nodes𝐶′and all edges that exist between these nodes in𝐺.
arg max
𝐺′
∑
𝑐∈𝐶′
𝜈(𝑐) (3.2)
s.t. |𝐶′| ≤ ℒ𝐶 ∧ 𝐺′ is connected
In this second formulation, only the importance estimates for concepts are used in the objective function. While it does not ensure that the number of relations is below the limit, it is still of practical relevance, as we will show later.
The complexity of optimization problems similar to Equation 3.2 has been studied by several authors. An unconstrained version of the problem, i.e. without|𝐶′| ≤ ℒ𝐶, was studied by Álvarez-Miranda et al. (2013) and El-Kebir and Klau (2014) and shown to be