Towards mobile language evolution exploitation

DOI 10.1007/s11042-011-0973-0

Towards mobile language evolution exploitation

Gideon Zenz·Nina Tahmasebi·Thomas Risse

© Springer Science+Business Media, LLC 2012

Abstract Knowing about the evolution of a term can significantly help when search- ing for relevant information, especially in case of sudden evolutions (e.g. as of dramatical changes in political situations). Here, some terms get a completely new meaning or are used in new or different ways. In mobile situations it is important to be able to effectively retrieve information, since this is usually done in a hurry and interaction possibilities with mobile devices are limited. In this paper we describe a methodology using word sense discrimination to discover term evolution. We present two mobile interfaces for easy access and exploration of this evolution, as well as a user-study to show its usefulness. We conclude the paper with an outlook of further research possibilities in this new topic.

Keywords Language evolution·Mobile applications·Ambient media· Social networks

1 Introduction

In the age of ambient media, the user demands constant and fast support for her needs. Mobile and smart devices provide excellent facilities for giving immersive, location based support for activities. Given the limited interaction possibilities and screen sizes, as well as being mobile usually implies an urgent information need,

This work is partly funded by the European Commission under ARCOMEM (IST 270239) and LiWA (IST 216267).

G. Zenz (

B

N. Tahmasebi

e-mail: tahmasebi@L3S.de T. Risse

e-mail: risse@L3S.de

efficient and effective means to query are important. In this paper we want to tackle the problem of language evolution for search.

Imagine an elderly person at home and checking what happened to the family on Facebook. Especially younger people tend to use terms in a way that is alien to them, as their meaning has recently changed. As example, consider the term anthrax. Depending on the time of the observation, this term’s sense could be about the band, or about the disease. This changed dramatically and unexpectedly in 2001 when letters with anthrax were mailed in the United States, thus the general public became aware of this term in relation to terrorism, weapons of mass destructions, etc.

Our terminology evolution application, can excavate such connections automatically, leading easier and faster to intended results than using standard search on e.g.

Google or Wikipedia. The aforementioned person might now simply grab their iPad and, by a quick query, learn of the term’s evolution.

To achieve such knowledge we employ word sense discrimination as pre- processing step. Word sense discrimination algorithms are used in many applications such as information retrieval, automatic machine translation and question answering.

The aim of word sense discrimination is to divide term collections into coherent groups of terms where each group represents one word sense or meaning.

To improve usability, the results from the word sense discrimination steps are made available in the ambient scenario. We devised two mobile usable interfaces, one for the professional user and one for the ordinary mobile user. The professional interface requires deeper knowledge of languages and web applications. It is meant to be used in places like libraries or museums, where more detailed knowledge of a query term is wanted. The second user-interface aims at the everyday mobile user and features a more concise user-interface. It aims at making the results from the word sense discrimination understandable by allowing the query term, with its sense, to be presented in its textual context.

The paper is structured as follows. In the next section we give an overview on the state of the art. Afterwards in Section3we present our method to derive the perception of a term in a given corpus. Two user-interface studies on how to present the analysis results to the user will be shown in Section4. In Section5we present our corpus as well as a user-study conducted on one user-interface. Afterwards our vision for using the technology in social networks by elderly people will be presented in Section6. Finally the paper concludes and gives an outlook on future work.

2 Related work

The core method to identify language evolution is word sense discrimination. The aim of word sense discrimination is to find senses of words present in a given text corpus. Several methods based on co-occurrence analysis and clustering have been proposed by, among others, [2,10,14] as well as [12].

Schütze [12] presented the idea of context group discrimination. Each occurrence of an ambiguous word in a training set is mapped to a point in word space. The similarity between two points is measured by cosine similarity. A context vector is then considered as the centroid (or sum) or the vectors of the words occurring in the context. This set of context vectors are then clustered into a number of coherent clusters. The representation of a sense is the centroid of its cluster.

The use of dependency triples is one alternative approach for word sense discrim- ination and was first described by Lin [7]. In this paper, a word similarity measure is proposed and an automatically created thesaurus, which uses this similarity, is evaluated. This method has the disadvantage of using hard clustering which is less appropriate for word senses due to ambiguity and polysemy of words. The author reports the method to work well but no formal evaluation is performed. In [9], Pantel presents a clustering algorithm called Clustering By Committee (CBC), which outperforms popular algorithms like Buckshot, K-means and Average Link in both recall and precision. The paper proposes a method for evaluating the output of a word sense clustering algorithm to WordNet, which has since been widely used [3,5].

In addition, it has been implemented in the WordNet::Similarity package by Bruce in [10], and is thus be used for cluster quality evaluation.

Language technologies have also shown to be useful for ambient assisted living by Gams in [11]. To the best of our knowledge, more dedicated work on our topic is currently not available.

3 From text to ambient perception

In this section we will explain the details of our processing pipeline for word sense discrimination. It is a process with three major steps: natural language processing (NLP) for the extraction of relevant terms and information, co-occurrence graph creation and graph clustering.

The main task of word sense discrimination is to automatically find the senses (meanings) of words present in a collection. The output is related sets of terms that are found in the collection representing senses. This grouping of terms is derived from clustering and we therefore refer to such an automatically found sense as a cluster.

Clustering techniques can be divided into hard and soft clustering algorithms.

In hard clustering an element can only appear in one cluster, while soft clustering allows each element to appear in several. Due to the ambiguous property of words, soft clustering is most appropriate for word sense discrimination. The techniques can be further divided into two major groups, supervised and unsupervised. Because of the vast amount of data found in our collections (see Section5.1), we are using an unsupervised technique proposed by Dorow [4], called curvature clustering. The curvature clustering is the core of the processing pipeline described next.

3.1 The processing pipeline for word sense discrimination

As a self-contained presentation, we include an overview of the processing pipeline as presented by Tahmasebi in [13]. The processing pipeline depicted in Fig.1consists of four steps; text cleaning, natural language processing, creation of cooccurrence graph and clustering. These constitute the three major steps involved in word sense discrimination with the addition of text cleaning. Each step is performed for a separate subset of the collection. Each subset represents a time interval and the granularity can be chosen freely.

Fig. 1 Overview of the word sense discrimination processing pipeline showing all four steps involved beginning with text cleaning

Text pre-processing The first step towards finding word senses is to prepare the documents in the archive for the subsequent processing. For The Times Archive that we use in our experiments this means extracting the content from the provided XML documents and performing an initial cleaning of the data.

Natural language processing The next step is to extract nouns and noun phrases from the cleaned text. To this end, it is first passed to a linguistic processor that uses a part-of-speech tagger to identify nouns. In addition, terms are lemmatized if a lemma can be derived. Lemmas of identified nouns are added to a term list which is considered to be the dictionary corresponding to that particular subset. The lemmatized text is then given as input to a second linguistic processor to extract noun phrases. The noun phrases, as well as the remaining nouns for which the first part-of- speech tagger was not able to find lemmas, are placed in the dictionary.

Co-occurrence graph creation After the natural language processing step, a co- occurrence graph is created.

Using the dictionary corresponding to a particular subset, the documents in the subset are searched for lists of nouns and noun phrases. Terms from the dictionary, that are found in the text separated by an and, an or or a comma, are considered to be co-occurring. For example in the sequence . . . cars like BMW, Audi and Fiat . . . the terms BMW, Audi and Fiat are all co-occurring in the graph. Once the entire subset is processed, all co-occurrences are filtered. Only co-occurrences with a frequency above a certain threshold are kept. This procedure ensures that the level of noise is reduced and most spurious connections are removed.

Graph clustering The clustering step is the core step of word sense discrimination and takes place once the co-occurrence graph is created. The curvature clustering algorithm by Dorow [4] is used to cluster the graph. The algorithm calculates the clustering coefficient by Strogatz [15] of each node, also called curvature value, by counting the number of triangles that the node is involved in. The triangles, representing the interconnectedness of the node’s neighbors, are normalized by the total number of possible triangles. Depicted in Fig. 2 is a graph which illustrates the calculations of curvature values using different triangles. Low-value nodes are removed and the results are clusters with sets of semantically related terms.

Fig. 2 Graph to illustrate curvature value. Nodes are labeled with name: curvature value

vw : 1

fiat : 1

bmw : 1/3

audi : 2/3 porsche : 0

4 User-interface and implementation

In order to make the results of the language evolution process end-user accessible, we devised two mobile applications. Both applications allow the user to explore the evolution of a term, however one is mainly intended for tablets while the other is intended for smart-phones or small tablets. Both applications were developed for an always-on scenario where a server stores the background knowledge and transfers upon request the needed information to the application in a compressed format.

Professional user-interface The first interface that we devised is aimed at the professional user. For a given query term, it visualizes all word sense clusters found for the term over the entire collection. The clusters are shown on a timeline and allow the user to scroll back and forward in time to search for word senses. For each cluster, a cluster representative is shown on the timeline. These are chosen from the terms in the cluster that have the highest curvature value. In Fig.3, we see cluster representatives like petersburg and ekaterinoslaf f. Choosing to click on one of the cluster representatives will show the full cluster with all its member terms and their connections like the graph shown in Fig.2. This representation enables a deeper understanding of the cluster terms and their relations. This professional user- interface will display, in addition to the cluster information, the normalized term frequency for the query term. This information can be very helpful for gaining a quick overview of the evolution of a term. As example, we chose the term Petersburg in Fig.3. This Russian city has had many different names over the years with Petersburg being the predominant name over time. However, between 1914–1991 the city was

Fig. 3 Professional

user-interface showing clusters for term Petersburg

named Petrograd as well as Leningrad. If we look at the normalized frequency distribution for Petersburg, we find that the frequency drops radically from 1914 to 1915. This already is a good indication that there has been some change in the meaning of Petersburg. If we wish to further investigate what happens with the term, we can choose to take a more in-depth look into the clusters to see if there is some indication of what happened.

We envision the use of this interface in places like museums or libraries where a deeper understanding of terms is needed and the user permits more time for the search and understanding of evolution.

Ambient user-interface The second interface is aimed at the casual or possibly elderly user. While the aim of the previous interface was to provide as much detail as possible to the user, for this interface we envision users with less time and experience.

Instead of giving in-depth analysis of the evolution of a term, this interface shows text examples for the most relevant clusters. This enables the user to understand the term evolution from the context of the original documents. As we can see from Fig.4, the query term(s) are highlighted and shown in the context of one or more sentences from the original text in the archive. These sentences aid in understanding the context of the term as well as help the user decide which documents to take a closer look at.

In order to achieve this, for each query term, we retrieve corresponding clusters from the entire time period of the archive. Using these terms from the clusters, we search through a full text index to retrieve all relevant text excerpts. The cluster terms as well as the query terms are highlighted and the resulting documents are displayed in the order of their relevance.

We envision the use of this interface in a more leisure or ad-hoc manner. Here little time will be spend by the user and thus a deeper understanding of evolution is not needed. More importance is given to the ease-of-use and the speedy recognition of the meaning of a term.

This application could be particularly interesting for elderly people who are active in social networks like Facebook or Twitter. More details about this scenario will be discussed in Section6.

Fig. 4 Ambient user-interface showing example phrases for term Anthrax

5 Evaluation

In this section we will discuss the corpora used for our experiments, as well as a user- study to show the effectiveness of the devised ambient user-interface.

5.1 Corpus

To have sufficiently large background knowledge, we applied our methodology to a large dataset, the The Times Archive, London, as a sample of real world modern English. The corpus contains newspaper articles spanning from the year 1785to 1985. The digitization process was started in the year2001when the collection was digitized from microfilm and OCR technology was applied to process the images.

The resulting201years of data consist of 4,363 articles in the smallest dataset and 91,583 in the largest. The number of space separated tokens range from four million tokens in 1785–68 million tokens in 1928. In total we found 7.1 billion tokens that translate into an average number of 35 million tokens per year. In Fig.5we give an overview of these numbers on a year by year basis.

Cluster quality To evaluate the quality of the clusters we measure the correspon- dence between clusters and word senses and rely upon WordNet [8] as a reference for word senses. We follow the method by Pantel [9] and compare the top k members of a cluster to a sense S in WordNet. If the similarity is above between S and the cluster is above a given threshold, then we say that the cluster corresponds to a word sense. The quality of clusters is then given as the amount of clusters that correspond to word senses.

Applying the method described in Section 3 to The Times Archive results in between 221–106,000 unique co-occurrences between nouns and noun phrases. Each

1800 1820 1840 1860 1880 1900 1920 1940 1960 1980 0

500 1000 1500 2000 2500

Average article length in no of terms

Year

Statistics over Times Articles

1800 1820 1840 1860 1880 1900 1920 1940 1960 19800 2 4 6 8 10 x 10⁴

Article count

Average article length Article count

Fig. 5 Number of articles and average length of articles in The Times archive from 1785 to 1985

co-occurrence corresponds to one edge in the graph as gives us a measure of the size of the graph. Each graph is clustered and the resulting clusters are evaluated.

On average 69% of all clusters contain more than two WordNet terms and can thus be evaluated. On average85±2% of our clusters correspond to word senses. The remaining clusters are a mix between wrongly devised clusters as well as clusters containing terms that are not found in WordNet. The first category of clusters contain terms that make no sense when gathered together, while the second category of clusters can contain proper nouns like people names or locations and are therefore not recognized by WordNet as a word sense.

With an average of 85% correspondence to word senses, we found that the cluster quality was sufficiently high to continue using the clusters as a basis for our user- interface evaluations. A further evaluation of the cluster quality is indirectly done once we evaluation the efficiency of our user-interfaces.

5.2 User-study

In order to assess the quality and applicability of our ambient user-interface, we devised an initial user-study with 5 participants. The participants were all experts in computer science and between 20 and 30 years old.

Participants We first analyzed their general behavior in information retrieving tasks. Most of our participants were not keen on spending large amounts of time for an information search task. Instead, they very much plan a strategy for searching in advance. On the presentation layer, they strongly prefer good accessibility of doc- uments as much as attractive presentation. In general, well-known, reliable sources of information are preferred. Therefore, The Times Archive provided an interesting and trustworthy corpus for our candidates, which was found to be important in search usability studies by Ingwersen [1]. Serving snippets from a news paper corpus, as our application does, may lead to showing contradicting or incomplete information. We therefore assessed whether this might have a negative impact on the information retrieval task, but this was generally declined by our participants. Our participants prefer to solve retrieval tasks themselves instead of asking for professional help, and generally prefer the Internet as first source.

Even though Google and Wikipedia were significantly the preferred way of assessing information, traditional printed media was still well respected and in some cases also preferred. This was for reasons like better readability of printed paper, and especially because traditional printed resources are more believed to be a serious and reliable source of information. On the contrary, electronic resources are believed to be faster to retrieve and more up-to-date.

Procedure The participants were not payed for conducting the survey. We tested all participants in a lab setting, using Android smart phones as ambient devices. Data was recorded using paper surveys before and after each task. The participants were first briefly introduced to their task and the ambient scenario. We refrained from an in-depth description of the procedure in order not to influence the participants.

Throughout the study, subjective ratings were reported on a 5-point Likert-scale, with 1 meaning always/strongly agree/very good, and five meaning the opposite.

Table 1 Query terms used in

user-study Term

1. Car 2. Zephyr

3. Flight 4. Yeoman

5. Aeroplane 6. Camera

7. Zermatt 8. Iran

9. Jet 10. Anthrax

11. Train 12. Mussolini

The first part of the survey contained general questions as discussed in the previous paragraph. Thereafter, participants were given 5 min time per query of Table1. For each query, we asked the participants to rate how well they were able to assess the meaning of the query with the presented information, how suitable the amount of presented information was, and how usable this would be in an ambient situation.

We concluded each sessions with three general questions on the usability of the tool.

Results Evaluating comprehensive tasks like this is difficult, as there are no correct answers and the goal is not necessarily to minimize time used. We assessed the helpfulness and the ease of use of our application, as can be seen in Fig. 6. For each query, we asked the participants to rate the answers given regarding (1) how well the answers made the meaning of the query understandable (2) how sufficient the amount of the presented information was and (3) how well the result fits in an ambient scenario.

Although we only had a limited number of participants, leading to a high standard deviation around 1, our application showed generally good-to-average results. The spikes we see for queries like Query 2 are usually because the used The Times Corpus has OCR issues especially with older text, leading sometimes to non-understandable sentences.

Fig. 6 Evaluation results of queries in Table1rating 1–5 from very good to bad on how clear the meaning of a term became, information quantity, and ambient usability

As the general questions yielded, the information need satisfaction was rated good (2.8±1.3). The application seemed to be helpful for the information retrieval task (2.6±0.54) and comparably easy to use (2.4±1.67).

6 Visions

Social networks are becoming more and more integrated into the daily life of people.

Currently these systems are mainly used by younger generations on a daily basis.

However, they are also increasingly used by older generations to stay in contact.

A major property of social networks is their high dynamics and frequency in the communication. As a result the messages exchanged are rather short and the authors are trying the “optimize” their entries by deriving new words or acronyms. This is especially the case for micro-blogging sites like Twitter but can also be observed in other systems. Some of these derived terms become standards while other might disappear again after a short time. The written language in these environments is evolving with a higher rate than it has been observed in traditional media. This raises new communication challenges for elderly people because they are often not aware about the new terms and acronyms used in social networks. As a result they are reduced to using social networks only to communicate with people from their own generation. Due to the partial anonymity provided by social networks, these forums should be excellent vehicles to stimulate the cross generation communication.

However, the new kinds of language barriers could hinder such a development.

To overcome this barrier we envision extending our technology to analyze social network content and to allow the mobile usage of the evolution technology.

Find evolving word senses in social networks Social networks serve as vast sources for hot and new information, making it useful to excavate this information in order to detect new and relevant information. They also serve as vast knowledge resource, therefore allowing to identify new semantic links of information or to identify new meanings of words. We envision using micro-blogging sites like Twitter to help the users find new word senses and annotate the identified web resources with the found word senses. These word senses will be weighted based on the popularity of the web resources as found in Twitter. The annotations will aid in identifying the most current word sense. Most words have many different senses listed in a dictionary, however, using Twitter as a measure of popularity, we can find out which of the listed senses that are currently in use and popular. Here, we need to identify hot or bursty resources by using frequency based statistics. Using e.g. numerical discrepancy [6], identifying hot resources can be solved in linear time. Having identified such hot resources and respectively their accompanying tweets, we can also facilitate the rest of the tweet to identify the senses of words co-occurring with these resources. These resources can now also be annotated with these senses.

The automatically generated annotations can finally be used to give the reader additional information for the understanding of a certain term.

Understanding term evolution on the go Similar to the younger generation, mobility becomes more important for elderly people. Already the usage of mobile phones today allows them to be more independent from their homes. With the increasing

availability of mobile internet devices or pad computers elderly people can stay online while sitting in the park, waiting for the doctor or while staying in the hospital.

In Section4we presented a case study on how users can get additional context information. Our vision for the future is a tighter and seamless integration with existing services. Because we need to present additional information on an already small display, good ways for this presentation need to be explored in the future.

7 Conclusions

In this paper, we presented a solution for providing language evolution “on the go”.

As a basis we used The Times Archive, a large real-world corpus, allowing us to identify significant evolutions in language. We devised two applications tailored for mobile devices, one for professional, one for contemporary users, which allow for easy access to language evolution “on the go”. We conducted an initial user-study on the contemporary, ambient user-interface, which establishes the good behavior and usability of our interface.

In the future, we want to focus more on exploiting social networks to improve access to information even more, especially for current events. Furthermore, we want to continue working on an ambient user-interface based on the results of our study. In a future, larger scale ambient user-study, we especially want to attract more older people as participants, to get a more realistic impression of the special needs of elderly people in an ambient environment.

Acknowledgement We would like to thank Times Newspapers Limited for providing the archive of The Times for our research.

References

1. Borlund P, Ingwersen P (1997) The development of a method for the evaluation of interactive information retrieval systems. J Doc 53:225–250

2. Davidov D, Rappoport A (2006) Efficient unsupervised discovery of word categories using symmetric patterns and high frequency words. In: ACL ’06: proceedings of the 21st international conference on computational linguistics and the 44th annual meeting of the ACL. Sydney, Australia, pp 297–304

3. Dorow B (2007) A graph model for words and their meanings. PhD thesis, University of Stuttgart 4. Dorow B, Eckmann J-P, Sergi D (2004) Using curvature and markov clustering in graphs for

lexical acquisition and word sense discrimination. In: Workshop MEANING-2005

5. Ferret O (2004) Discovering word senses from a network of lexical cooccurrences. In: COLING

’04: proceedings of the 20th international conference on computational linguistics, vol 1326.

Geneva, Switzerland

6. Lappas T, Arai B, Platakis M, Kotsakos D, Gunopulos D (2009) On burstiness-aware search for document sequences. In: KDD, pp 477–486

7. Lin D (1998) Automatic retrieval and clustering of similar words. In: Proceedings of the 17th international conference on computational linguistics. Montreal, Quebec, pp 768–774

8. Miller GA (1995) WordNet: a Lexical database for English. Commun ACM 38:39–41

9. Pantel P, Lin D (2002) Discovering word senses from text. In: KDD ’02: proceedings of the 8th ACM SIGKDD international conference on knowledge discovery and data mining. Edmonton, Alberta, pp 613–619

10. Pedersen T, Bruce R (1997) Distinguishing word senses in untagged text. In: Proceedings of the 2nd conference on empirical methods in natural language processing. Providence, RI, pp 197–207

11. Pogorelc B, Bosni´c Z, Gams M (2011) Automatic recognition of gait-related health problems in the elderly using machine learning. Multimed Tools Appl 1–22 (2011).

doi:10.1007/s11042-011-0786-1

12. Schütze H (1998) Automatic word sense discrimination. Comput Linguist 24(1):97–123 13. Tahmasebi N, Niklas K, Theuerkauf T, Risse T (2010) Using word sense discrimination on

historic document collections. In: JCDL, pp 89–98

14. Van de Cruys T (2008) Using three way data for word sense discrimination. In: Proceedings of the 22nd international conference on computational linguistics (Coling 2008). Manchester, UK, pp 929–936. Coling Organizing Committee

15. Watts D, Strogatz S (1998) Collective dynamics of “small-world” networks. Nature 393:440–442

Gideon Zenz is a Ph.D. Student at L3S Research Center in Hanover. He holds a M.Sc. in Computer Science since 2008 from Leibniz University of Hanover. He is working on the FP7 ARCOMEM project and others since 2008. His special interests are databases, algorithms, and search technology.

Nina Tahmasebi is a Ph.D. student at L3S Research Center in Hanover. She holds a Bachelor of Science in Mathematical Statistics since 2007 from Göteborg University and a Master of Science in Engineering Mathematics since 2008 from Chalmers University of Technology. She is currently working on the FP7 ARCOMEM project and has previously been involved in the LiWA project.

Her research interest are mathematics and mining algorithms for language evolution tracking.

Thomas Risse is the deputy managing director of the L3S Research Center in Hanover. He received a PhD in Computer Science from the Darmstadt University of Technology, Germany in 2006. Before he joined the L3S Research Center in 2007 he lead a research group about intelligent information environments at Fraunhofer IPSI, Darmstadt. Currently he is the scientific and technical director of the FP7 ARCOMEM project on Web archiving using social media information. Thomas Risse’s research interests are semantic evolution, data management in distributed systems, and self- organizing systems. He serves regularly as program committee member or project reviewer. He published several papers at the relevant international conferences.