Dark side of UGC: A user-centric perspective on the impact of user-generated content

(1)

Dark side of UGC: A user-centric

perspective on the impact of

user-generated content

Ankit Kariryaa

Supervisor: Prof. Dr. Johannes Schöning

Department of Mathematics and Computer Science

University of Bremen, Germany

This dissertation is submitted for the degree of

Doctor of Philosophy

(2)

Dark side of UGC: A user-centric perspective on the impact of user-generated content Dissertation by Ankit Kairryaa submitted for the degree of Doctor of Engineering (Dr.-Ing.) to the faculty of Mathematics and Computer Science at the University of Bremen. Date of Submission: August 18, 2020

Date of Colloquium: September 28, 2020

The following people served as readers for this thesis:

Reader . . . . Professor Johannes Schöning

University of Bremen Reader . . . .

Professor Fabian Gieseke University of Münster

(3)

Declaration

I hereby declare that except where specific reference is made to the work of others, the contents of this dissertation are original and have not been submitted in whole or in part for consideration for any other degree or qualification in this, or any other university.

Ankit Kariryaa September 2020

(4)

This thesis is dedicated to the loving memory of my uncle (Chacha)

Akhilesh Yadav * 17/06/1966

(5)

Acknowledgements

I am fortunate to have found a wonderful supervisor, Johannes Schöning, to guide me in my research journey. Johannes, I would always be grateful for the support you offered me in academic as well as personal matters, often going beyond what was required of you. Your trust, ideas, and creativity helped me explore broad and significant topics in my research.

I would also like to thank the great colleagues of the HCI group and FB3 for all the intriguing discussions, laughs, and lunches, to my sailing mate Daniel Diethei for the statistics counseling and the table tennis games, to Gian-Luca Savino for making the PhD a lot more interesting with wide-ranging discussions, to Rainer Malaka for your zeal to bring a positive change and for the India trip where I saw my own country in a new light, as well as other inspiring colleagues: Benjamin Tannert, Daria Soroko, Nadine Wagener, Carolin Stellmacher, Simon Runde, Petra Tienken, Jasmin Niess and Philipp Harms.

In Winter 2018, I visited the University of Copenhagen under the guidance of Fabian Gieseke and spend one of the most productive times in my PhD. Thank you, Fabian Gieseke, Stefan Oehmcke, Vinnie Ko, and Mathias Perslev for the insightful discussions that taught me more about machine learning than many courses. That research stay also led to many incredible things and enduring collaboration with Martin Brandt, Christian Igel, Rasmus Fen-sholt, Jim Tucker, and many other wonderful colleagues from the University of Copenhagen, NASA, and the University of Toulouse.

(6)

vi I also thank my other co-authors including Brent Hecht, Issac Johnson, Hendrik Heuer, and Andreas Jungherr. Our collaboration taught me many skills, from writing to theory, and helped me grow as a researcher.

Last but not least, I would like to thank my family and friends for their unconditional support: my parents Satish Yadav and Sheela Yadav for giving me everything they could, especially the freedom to become the person I am today, my wife Tetiana Gren for being my pillar of support, grandfather Om Kanwar Yadav for teaching me how to be happy, my joyful younger brother Pulkit Kumar Kariryaa, my late uncle Akhilesh Yadav for showing me the way to be a better person and aunt Savita Yadav for being a guiding light in research in the family. Also thanks to my other uncle, aunts, and cousins for all the love and care. Thank you, Sanjeev Sharma, Manu Bamba, Gaurav Sharma, and Balwant Kumar for your friendship.

(7)

Abstract

User-generated content (UGC) has been on the rise with the emergence of Web 2.0 and it is the driving force behind prominent online platforms such as Wikipedia, OpenStreetMap, Facebook, and Twitter. UGC has led to numerous innovations in academia and industry and has transformed our world in multiple ways. While the positive impact of UGC is abundant, there is limited research on its negative impacts. In this thesis, we study the impact of the UGC from the perspective of the users, who are both its creators and consumers. This thesis has four main contributions, detailed below.

The first contribution improves our understanding of the impact of the UGC on the geo-privacy of the user. We investigate the accuracy of localness methods used for the categorization of UGC at the city, county, and state scale. Through a user study, we show that some methods can assess the location of a content producer with very high accuracy. Thus, the research establishes a standard for localness methods as well as highlights the impact of UGC on the geo-privacy of users.

The second contribution improves our understanding of the impact of national symbols in UGC on political communication. Through a study of tweets by politicians and political parties in Germany and the USA, we analyze the role of flag emoji, looking into their usage, meaning, and association with the audience engagement. The results show that flags remain an influential symbol in online communication and they are associated with significantly higher engagement for most political parties, which helps improve to our understanding of UGC in politics.

(8)

viii In the third contribution, we present a tool for limiting the impact of UGC on the online privacy of the users. UGC producers often reveal personal information about themselves and online passwords commonly contain personal information. Current password meters do not consider personal information and, therefore, their users are susceptible to guessing attacks. We present the MoiPrivacy password meter, that extends a neural network- and heuristic-based approach and considers a user’s personal information while calculating the password strength and feedback. Through a user study, we find that MoiPrivacy significantly limits the inclusion of personal information in passwords, thus limiting the negative impacts of UGC on online security.

In the fourth contribution, we present a deep learning-based approach for identifying individual trees in sub-meter satellite imagery at a very large scale. This approach was used to map the crown size of each tree >3 m2in a land area spanning 1.3 million km2in the West African Sahara, Sahel, and sub-humid zone. We detected over 1.8 billion individual trees, or 13.4 trees ha-1, with a median crown size of 12 m2along a rainfall gradient from 0 to 1000 mm. Our assessment suggests a way to monitor trees outside forests globally and to explore their role in mitigating degradation, climate change, and poverty.

While content generation is associated with some adverse impacts on the user, it also offers an opportunity for large scale UGC-based citizen science platforms. In the future, large scale citizen platforms might be crucial for tackling global challenges including cli-mate change and shrinking biodiversity and the presented approach could be crucial for bootstrapping such platforms.

(9)

Zusammenfassung

User Generated Content (UGC) ist mit dem Aufkommen des Web 2.0 auf dem Vormarsch und ist die treibende Kraft hinter prominenten Online-Plattformen wie Wikipedia, OpenStreetMap, Facebook und Twitter. UGC hat zu zahlreichen Innovationen in der akademischen Welt und der Industrie geführt und unsere Welt auf vielfältige Weise verändert. Während die positiven Auswirkungen von UGC vielfach gezeigt wurden, gibt es nur wenige Untersuchungen über seine negativen Auswirkungen. In dieser Arbeit untersuchen wir die Auswirkungen des UGC aus der Perspektive der Nutzer, die sowohl seine Schöpfer als auch seine Konsumenten sind. Diese Arbeit umfasst vier Hauptbeiträge, die im Folgenden näher erläutert werden.

Der erste Beitrag verbessert unser Verständnis der Auswirkungen der UGC auf die Geoprivatsphäre der Benutzer. Wir untersuchen die Genauigkeit der Lokalitätsmethoden, die für die Kategorisierung von UGC auf Stadt-, Bezirks- und Staatsebene verwendet werden. Mit Hilfe einer Nutzerstudie zeigen wir, dass einige Methoden den Standort eines Nutzers mit sehr hoher Genauigkeit bestimmen können. Auf diese Weise legt diese Forschungsarbeit einen Standard für Lokalisierungsmethoden fest und zeigt die Auswirkungen von UGC auf die Geoprivatsphäre der Benutzer auf.

Der zweite Beitrag verbessert unser Verständnis des Einflusses von nationalen Symbolen in UGC auf die politische Kommunikation. Anhand einer Studie über Tweets von Politikern und politischen Parteien in Deutschland und den USA analysieren wir die Rolle der Flaggen-Emoji, indem wir ihren Gebrauch, ihre Bedeutung und ihren Zusammenhang mit dem Engagement des Twitter-Publikums untersuchen. Die Ergebnisse zeigen, dass Flaggen nach

(10)

x wie vor ein einflussreiches Symbol in der Online-Kommunikation sind und dass sie für die meisten politischen Parteien mit einem signifikant höheren Engagement verbunden sind. Somit trägt diese Arbeit zu einem besseren Verständnis von UGC in der Politik bei.

Im dritten Beitrag stellen wir ein Instrument zur Begrenzung der Auswirkungen von UGC auf die Online-Privatsphäre der Benutzer vor. UGC-Produzenten geben oft persönliche Informationen über sich selbst preis und Online-Passwörter enthalten häufig persönliche Informationen. Die derzeitigen password meters berücksichtigen keine persönlichen Informa-tionen, und daher sind ihre Benutzer anfällig für gezielte Angriffe mit erratenen gestohlenen Zugangsdaten (“guessing attacks”). Wir stellen den password meter MoiPrivacy vor, der einen auf neuronalen Netzwerken und Heuristiken basierenden Ansatz erweitert und die persönlichen Informationen eines Benutzers bei der Berechnung der Passwortstärke berück-sichtigt und diese visualisiert. In einer Nutzerstudie fanden wir heraus, dass MoiPrivacy die Verwendung persönlicher Informationen in Passwörtern signifikant verringert und so die negativen Auswirkungen von UGC auf die Online-Sicherheit begrenzt.

Im vierten Beitrag stellen wir einen deep learning Ansatz zur Identifizierung einzelner Bäume in Submeter-Satellitenbildern in sehr großem Maßstab vor. Dieser Ansatz wurde verwendet, um die Kronengröße jedes Baumes >3 m2 in einer Landfläche von 1,3 Mil-lionen km2 in der westafrikanischen Sahara, der Sahelzone und der subhumiden Zone zu kartieren. Wir entdeckten über 1,8 Milliarden Einzelbäume oder 13,4 Bäume ha-1. Im Mittel betrug die Baumkronengröße 12 m2 entlang eines Niederschlagsgradienten von 0 bis 1000 mm. Unsere Untersuchung liefert eine Methode, Bäume außerhalb der Wälder global zu überwachen und ihre Rolle bei der Minderung von Degradation, Klimawandel und Armut zu untersuchen. Die Generierung von Inhalten ist zwar mit einigen negativen Auswirkungen auf den Nutzer verbunden, bietet aber auch die Möglichkeit für groß angelegte UGC-basierte Bürger-Wissenschaftsplattformen. In der Zukunft könnten groß angelegte Bürgerplattformen für die Bewältigung globaler Herausforderungen wie Klimawandel und

(11)

xi abnehmende biologische Vielfalt von entscheidender Bedeutung sein, und der vorgestellte Ansatz könnte für das Bootstrapping solcher Plattformen entscheidend sein.

(12)

Publications

The thesis builds upon four publications that which I have (co-)authored during my PhD studies. These are are full papers and I describe my contribution in them below. Publications I and III are published in peer-reviewed conference, and publication II and IV are published in peer-reviewed journal. Publications I-IV are listed in the order they appear in the contri-butions section (see section 1.1) and each publication displays its index in the bibliography, which is ordered alphabetically. Lastly, I list other publications that I have (co-)authored during my PhD studies, but which are not included in the thesis.

Included in thesis

Publication I:[132] Ankit Kariryaa, Isaac Johnson, Johannes Schöning, and Brent Hecht. 2018. Defining and Predicting the Localness of Volunteered Geographic Information using Ground Truth Data. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). Association for Computing Machinery, New York, NY, USA, Paper 265, 1–12. DOI:https:// doi.org/ 10.1145/ 3173574.3173839.

This publication extended the earlier work of the Johnson et al. [123]. I was involved in the planning of the study, and development of the user survey, which saw equal contributions from all co-authors. I developed the data collection pipeline and implemented the localness metrics, and analyzed the data. I also contributed to all parts of the manuscript.

(13)

xiii Publication II:[133] Ankit Kariryaa, Simon Rundé, Hendrik Heuer, Andreas Jungherr, and Johannes Schöning. 2020. The Role of Flag Emoji in Online Political Communication. In Social Science Computer Review (2020). DOI:https:// doi.org/ 10.1177/ 0894439320909085. I was involved in planning and conceptualization of the study along with Rundé and Schöning. I developed and implemented the data collection system, and analyzed the data with support from Rundé and Heuer. I also contributed to all parts of the manuscript.

Publication III: [134] Ankit Kariryaa and Johannes Schöning. 2020. MoiPrivacy: Design and Evaluation of Personal Password Meter. In 19th International Conference on Mobile and Ubiquitous Multimedia (MUM 2020), November 22–25, 2020, Essen, Germany. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3428361.3428397.

I was responsible for conceptualization, design, and development of MoiPrivacy. Later on for its evaluation through a user study and for writing the manuscript. Johannes Schöning supported me throughout the process.

Publication IV: [30] Martin Brandt, Compton J. Tucker, Ankit Kariryaa, Kjeld Rasmussen, Christin Abel, Jennifer Small, Jerome Chave, Laura Vang Rasmussen, Pierre Hiernaux, Abdoul Aziz Diouf, Laurent Kergoat, Ole Mertz, Christian Igel, Fabian Gieseke, Johannes Schöning, Sizhuo Li, Katherine Melocik, Jesse Meyer, Scott Sinno, Eric Romero, Erin Glennie, Amandine Montagu, Morgane Dendoncker, Rasmus Fensholt. 2020. An unexpectedly large count of non-forest trees in the western Sahara and Sahel. Nature (2020). DOI:https: // doi.org/ 10.1038/ s41586-020-2824-5.

I wrote the code for the deep learning framework for detecting the individual trees, supported by Sizhuo Li, Johannes Schöning, Fabian Gieseke, Jesse Meyer and Christian Igel. I also contributed to the analysis and the manuscript.

(14)

xiv

Not included in thesis

Publication V: [131] Ankit Kariryaa. 2020. MaskIt: Masking for efficient utilization of in-complete public datasets for training deep learning models. arXiv preprint arXiv:2006.12004. Publication VI: [157] Brianna M Lind, Martin Brandt, Ankit Kariryaa, Jennifer L Small, Katherine A Melocik, Compton J Tucker, Niall P Hanan. 2019. Very High Resolution Satellite Imagery Reveals Many Millions of Termite Mounds Across the West African Sahel. Poster in AGUFM, 2019, B23F-2611.

Publication VII: [18] Niels van Berkel, Julio Vega, Ankit Kariryaa, C. Estelle Smith and Ye Yuan. 2018. CHI 2018. Report in IEEE Pervasive Computing, Vol. 17, Issue 3.

Publication VIII: [135] Ankit Kariryaa, Tony Veale, and Johannes Schöning. 2017. Activity and mood-based routing for autonomous vehicles. In Workshop on Mobile Interaction With and In Autonomous Vehicles workshop at MobileHCI 2017.

(15)

Chapter 1 Introduction and Motivation

User-generated content (UGC) has been on the rise with the emergence of the Web 2.0 [55]. With Web 2.0, more and more online platforms encourage their users to become creative contributors instead of just passive consumers [155]. Even so not everyone contributes equally on these platforms and the majority of the people are still passive consumers [95, 102, 155], the sheer amount of users have resulted in billions of photos, posts and videos created every day [163].

This led, in a short period, to a wave of platforms that relied on UGC for knowledge repositories e.g., Wikipedia and OpenStreetMap, social networking e.g., Facebook and Twitter, and discourse e.g., Reddit and Slashdot. UGC radically transformed computing beyond expectations and had a similar effect on numerous wide-ranging fields such as economics [160, 185, 250], social science [234, 262], and marketing [105, 155]. Krumm et al. [146] define UGC as,

“Any form of content, such as images, videos, text and audio, that have been posted by users on online platforms such as social media and wikis.”

In research, prominence of UGC as a data source has grown over the years and now it is considered fundamental in many different domains including human-computer interaction

(20)

2

Fig. 1.1 Examples of user generated content. UGC refers to content posted on online platforms by the users of these platforms. It can include texts, photos, videos, and GPS coordinates and can be found on many online platforms for social media, discourse and wiki. (HCI) (e.g. [1, 88, 181]), computational social science (e.g. [74, 126, 165, 223, 239]), and public health research (e.g. [62, 83, 84, 263]). The use of Twitter, a micro-blogging website and a prominent platform for UGC, is especially popular across domains and some researchers have even termed is as the model organism for big data [254]. Among numerous other things, researchers use UGC for predicting social unrest (e.g., [45, 129]), for determining the support for governmental policies (e.g., [136, 284]), for enhancing situational awareness during natural hazards (e.g., [114, 242, 266]), for predicting stock markets (e.g., [25, 201]), for tracking the spread of infectious diseases [98, 216] and understanding long term migration of people (e.g., [64, 79, 167, 240]).

(21)

3 Beyond research, many commercial products, such as search engines e.g., Google and Bing, and voice assistants e.g., Alexa1, Siri2, and Cortana3, which are used by billions of people every day, rely on the enormous amount of knowledge generated about the world through UGC for their functioning [232, 267, 277].

There is no doubt that UGC has a net positive impact on society, nevertheless, there is little research on its negative impacts [104]. On social media and other UGC platforms, user’s often reveal personally identifiable information (PII) (for example, age, gender and sex), interests (e.g., political beliefs and cultural preferences), and locations (e.g., home location, work location and current location) [9, 90]. But even on platforms, where users do not reveal PII e.g., Wikipedia and OpenStreetMap, personal information, such as gender, education, and faith, can be determined with high accuracy from the content produced by the users [210]. Thus in most cases, content generation is associated with erosion of privacy for the users generating the content. On the value of privacy Posner wrote [194]:

“When people today decry lack of privacy, what they want, I think, is mainly something quite different from seclusion: they want more power to conceal information about themselves that others might use to their disadvantage.” [194] Location information is one of the most widely available forms of personal information. Users frequently share their location in form of check-ins and in location fields on online platforms [103], but more frequently as a by-product of everyday activities using technolo-gies that rely on GPS, IP address or mobile network [226]. Location information has many useful applications, e.g. content customization, location-based alerts, and transaction verifica-tion [190]. However, it also presents a significant challenge to the privacy of the user. The US President’s Council of Advisors on Science and Technology in their 2014 report on big data and privacy, highlighted the benefits and harms of the big data after considering the viewpoint

1_{https://alexa.amazon.com} 2_{https://www.apple.com/siri/}

(22)

4 of various shareholders including industry, individuals, government, and researchers [196]. They cited violations of locational privacy as one of the paramount issues associated with big data. Seemingly harmless location information can reveal sensitive personal facts such as health issues and sexual orientation, and continuous locational track of an individual’s private life can result in tracking and stalking [61]. As an example, the web-service Please Rob Me4, raises awareness about over-sharing of location information which can also lead to a home robbery in certain cases. Indeed these incidents happen in real life, in 2016 a California man was sentenced to 8 years in prison for targeting 33 females and burglarizing their home by tracking their movements through social media [198]. Keßler and McKenzie point to further arguments on the negative impact of the erosion of location privacy in their geoprivacy manifesto [138].

Besides location, another kind of information that is frequently available through UGC is political interests. Political discourse is a common use of UGC platforms, and political actors, media personal, and public alike use them to comment on the news of the day, post opinions or research information [126]. UGC has had a tremendous impact on political systems globally and in this case, the role of social media outshines other UGC platforms such as blogs and wikis. While the narrative around the impact of social media has almost reversed in the last decade, its crucial role in retooling politics is widely acknowledged [127]. In the early years, social media was celebrated for providing a voice to the masses around the world, and acclaimed for its role in facilitating activism such as in protests after Moldovan parliamentary election in 2009 and in Arab Spring [5, 112, 230]. However, in the more recent years, social media is also seen as a threat to democratic societies, due to its role in spreading extremism, hate speech, and fake news [4, 253] and creating echo chambers [85]. UGC and online behavior can be used to predict sensitive personal attributes such as ethnicity, political and religious views and other personal traits with a very high accuracy [88, 145, 283]. This information can then be used for individual psychological targeting to manipulate and

(23)

5 influence people [6, 144]. The issue has been in the spotlight after the Cambridge Analytica incident, where millions of people in 68 countries were micro-targeted for political gains during critical moments of political life [37, 38, 119].

The impact of erosion of privacy goes far beyond tracking and manipulation. For example, the erosion of privacy has a major impact on the online security of a user. Many of the online authentication mechanism directly or indirectly use personal information. For instance, security questions (e.g., what is your mother’s maiden name?), which originated in banking systems and are frequently used as a fallback authentication mechanism on online platforms, rely on the hardness of an information-retrieval problem and assume that the personal information in question is not in public domain [99, 219]. However, as Rabkin [200] demonstrated this assumption does not hold in the age of UGC, where personal information such as the name of the user’s parents or high school is frequently in the public domain. Similarly, passwords, which are the most common authentication method on the internet, frequently contain personal information. Former studies [33] have reported that as high as 90% of all passwords are based upon personal information. While this percentage has reduced in the last years, recently conducted studies also report that more than one-third of the passwords still contains basic and sensitive personal information, such as name, date of birth, and phone number of the user [247]. Thus, revealing personal information and interests through UGC also harms the online security of the users.

The governmental agencies, digital rights groups (such as Electronic Frontier Foundation5 and Center for Democracy and Technology6), and researchers have for a long time warned of the various threats of erosion of privacy to both the individuals as well as the society [23, 48, 73, 82, 187, 196, 281]. In the field of computing too, there have been increasing calls for studying and mitigating these issues. For example, in a recent article Hecht et al. [104] have

5_{https://www.eff.org/} 6_{https://cdt.org/}

(24)

6 emphasized on the need for understanding and mitigating the negative impacts of computing, with a particular focus on the peer-review process.

Taking the view of Hecht et al. and privacy advocates into consideration, in this thesis, we study the negative impacts of the UGC on the users with the lens of HCI. Oulasvirta and Hornbaek [184] described scientific progress in the field of HCI as improvements in the ability to solve important problems related to human use of computing. Extending Laudan’s philosophy of science [149], they defined human-computer interaction research as problem-solving, where the solution results in an improvement to problem-solving capacity. In HCI, the research problem could be either empirical, conceptual, or constructive and the solution can be defined in terms of significance, effectiveness, efficiency, transfer, and confidence.

In this thesis, we study the negative impacts of UGC from a user-centric perspective: • User’s localness [132]: In the context of location information, we define the localness

of a user and study the accuracy with which home, voting, and familiar location of a user can be predicted through UGC.

• User’s association with national symbol [133]: In relation to political interests, we study the usage and meaning of national flags, the most influential and widely used national symbols, for politicians and political parties, and analyze their association with user engagement.

• User’s online security [134]: In context of UGC and online security, we present the design and evaluation of a tool that limits the use of personal information in the password, thus improving their strength.

While content generation is associated with an adverse impact on the user, it is hard to ignore its potential and value. In the future, the greatest opportunity that UGC might offer is for large scale citizen science platforms, while advancement in deep learning and novel data sources, including very high-resolution satellite imagery, would be at the core of such

(25)

1.1 Contributions 7 platforms. One can imagine a global platform for monitoring biodiversity, where the data is populated by deep learning pipelines and further refined by citizen scientists. We believe, in the future, these platforms would be crucial for tackling global challenges including climate change, extinction of species and continuously shrinking biodiversity. This notion is also shared by several researchers in the field of remote sensing and sustainability, where the number of calls for research in this direction has been growing over the year [3, 49]. Lastly, as potential future application of UGC, we present a system for:

• Large scale identification of individual trees [30]: In the last part, we present a machine learning based approach for identifying individual tress at a very large scale. This method was used to identify more than 1.8 billion individual trees in western Sahara and Sahel. In the future, this approach and the data thus collected, may become the core of a global biodiversity monitoring platform powered by "citizen science".

1.1 Contributions

This thesis has four main contributions, detailed below.

• Firstly, we define the localness of a user and look into the accuracy of the techniques used to assess the localness from UGC with location information. Georeferenced tweets, geotagged Instagram photos, and other volunteered geographic information (VGI), which is a subset of UGC, are critical to research and practice across a wide swath of computing. For many applications of VGI, it is important to determine the “localness” of the VGI contributor e.g., the content poster, to a specific region. This is true for applications ranging from recommender systems that surface venues that are “local favorites” (e.g., [92, 156, 214]) to research that seeks to understand the local perspective on certain issues (e.g., [107, 264, 284]). However, considering the importance of the concept of localness to VGI, we know surprisingly little about the

(26)

1.1 Contributions 8 concept. First and foremost, there are no broadly accepted definitions of “local” in computing, with most projects adopting definitions that are ad-hoc and often unstated. Second, while there are techniques that have been widely used to assess the localness of VGI, they have not been validated. To address this question, we conducted a review of the literature that has engaged with the concept of localness. Examining 30 papers, we identified the meaning of “local” in field of computing and then through a user study, we verified how well different metrices used to determine localness work for various definitions of localness.

Examining the localness problem with the lens of Oulasvirta and Hornbaek [184], defining localness is a conceptual problem. Prior definitions of localness were ad-hoc, and we defined the meaning of local in computing through a literature study. Validating the accuracy of localness matrices is an empirical problem as we quantify the accuracy of different metrics for each definition based upon ground truth. Since our research improves the problem-solving capacity of the large body of research that relies on the localness assumption, the solution is assessed to be significant for the stakeholders. Some metrics were found to be faster or more accurate than the others which improve the effectiveness and efficiency of research in the domain.

• Secondly, the North Atlantic Treaty Organization (NATO) has underscored that ma-nipulation of public opinion regarding social media in political context is a pressing policy concern for its member countries [27]. In the political context, national flags are the most important and widely used symbols, and they hold significant power over its audience [221]. Political actors use flags to establish an association between them and symbols of common identity, to send subliminal cues triggering subconscious prejudices, or to signal alignment with a shared set of values or beliefs. Exposure to flags can increase feelings of attachment to the nation as an abstract concept or feeling of superiority towards other nations [137, 218]. Flags thus serve as powerful

(27)

1.1 Contributions 9 symbols of belonging and specific interpretations of a nation. However, while they are a common element of political communication, very little is known about their uses and effects in online political communication. First and foremost, who uses the flag in political circles, how does the meaning of national flag differ for various politicians and what effect does it have on their audience. To fill this gap, we examine the use of flag emoji (digital representation of the national flag in form of emoji) on Twitter by the most important political parties and Members of Parliament (MPs) in Germany and the USA. This allows us to examine whether the use of flag emoji varies between parties and MPs depending on their political leaning, depends on events of shared national importance, varies in semantic context, and has divergent effects for different parties depending on their emphasis of national identity.

Examining the use and effect of national flag emoji is an empirical problem as it elucidates an unknown phenomenon and interprets an unknown effect. Our research finds varying use, meaning and effect of flag emoji for various political parties. Po-litical parties on the far right of the poPo-litical spectrum, use flag emoji with higher frequency and usually for these parties, tweets with flag emoji are associated with higher engagement. These two findings taken together provide us empirical evidence that political parties on right and especially parties endorsing nationalistic thought might benefit more from the introduction of flag emoji in the Unicode character set. While we only found empirical evidence of association with higher engagement, and cannot conclude that it is directly caused by the flag emoji usage, it still points to important challenges for Unicode consortium and the designers of emoji. They might have unseeingly provided comparative advantage in online communication to some political parties.

• Next, we describe a tool that mitigates the impact of easily accessible public informa-tion on the online security of the producer of UGC. Usage of social media implies a risk

(28)

1.1 Contributions 10 for online security, as personal information is commonly used in passwords [41, 247]. Former studies [33] have reported that as high as 90% of all passwords are based upon personal information. While this ratio has decreased in the last years, recently conducted studies also report that more than one-third of the passwords still contains basic and sensitive personal information, such as name, date of birth and phone number of the user [247]. In targeted attacks on individuals, attackers often exploit this knowl-edge and use personal information obtained through social media and other sources in their attack. Research has shown that having access to someone’s personal information increases the success of password cracking in the 20 attempts by more than 200% and in the first 100 attempts, it is increased by more than 600% [154]. To solve this problem, we present MoiPrivacy a personal password meter. MoiPrivacy extends the current state of the art in password meter research by including personal information in the password strength estimation and textual feedback. It aims to improve the online security of users by using personal information from their social media profiles. We found that providing feedback about the use of personal information in passwords prevents users from using it in passwords compared to a baseline condition, in which we used a state of the art password meter [258]. MoiPrivacy extension led users to create secure passwords and limit the inclusion of personal information in passwords. The design and development of the MoiPrivacy password meter is a constructive prob-lem and its evaluation is an empirical probprob-lem. Prior research on password meters which focused on improving password strengthen but did not consider the problem of personal information in password. Our solution improves understanding of the construction of a password meter that can limit the use of personal information in passwords. Our evaluation shows that this approach can reduce the personal informa-tion in the password by 55%. The soluinforma-tion is relevant to the general public as well as researchers dealing with problems of online security.

(29)

1.1 Contributions 11 • Lastly, we describe a machine learning approach for detecting individual trees in high-resolution satellite imagery on a very large scale. A large proportion of dryland trees and shrubs grow isolated without canopy closure. These non-forest trees play a crucial role for biodiversity and provide ecosystem services such as carbon storage, food resources and shelter for humans and animals [14, 246]. However, most public interest is devoted to forests, while trees outside forests are not well documented [222]. Here, we mapped the crown size of each tree >3 m2in a land area spanning 1.3 million km2in the West African Sahara, Sahel, and sub-humid zone, using sub-meter satellite imagery and deep learning [150]. We detected over 1.8 billion individual trees, or 13.4 trees ha-1, with a median crown size of 12 m2 along a rainfall gradient from 0 to 1000 mm. The canopy cover increases from 0.1% (0.7 trees ha-1) in hyper-arid areas, 1.6% (9.9 trees ha-1) in arid, and 5.6% (30.1 trees ha-1) in semi-arid to 13.3% (47 trees ha-1) in sub-humid areas. Although the overall canopy cover is low, the relatively high density of isolated trees challenges prevailing narratives about dryland desertification [54, 77, 207], and even the desert shows a surprisingly high tree density. Our assessment suggests a way to monitor trees outside forests globally, and to explore their role in mitigating degradation, climate change and poverty.

The problem tackled here would lie outside the domain of HCI and it would more appropriately fit in the domains of remote sensing and geography. In this case, we apply the Laudan’s philosophy of science [149], on which the Oulasvirta and Hornbaek [184] based their definition of HCI. Prior research on trees detection with satellite imagery either focused on forest instead of individual trees or was limited in scope to a small area. We developed models that were applicable for large part of northern Africa, opening up the possibility of detecting individual trees on the global scale. Our assess-ment lays the groundwork for better understanding the contribution of human agency and climate change to the distribution of dryland trees and their role in mitigating

(30)

(31)

1.2 Structure of the Thesis 13

1.2 Structure of the Thesis

This section outlines the structure of the thesis. Each contribution in the previous section is addressed by a chapter. This structure is graphically illustrated in Figure 1.2. Chapter 2 presents the work on defining and predicting the localness of UGC. Chapter 3 presents the work on understanding the role of flag emoji in online political communication. Chapter 4 presents the work on design, development and evaluation of a personal password meter. Chapter 5 presents the work on detecting individual trees with machine learning and very-high resolution imagery. Chapter 6 outlines the general contributions of this work, and presents the conclusion of the thesis.

A note on the writing style

In this thesis I use the term “we” instead of “I”, as it is customary in my field and because the work described here was conducted in collaboration with others. However, to describe my personal contribution in the publications section, the term “I” is used.

(32)

(33)

2.1 Introduction 15

2.1 Introduction

Georeferenced tweets, geotagged Instagram photos, and other volunteered geographic infor-mation (VGI) are critical to research and practice across a wide swath of computing. For many applications of VGI, it is important to determine the “localness” of the VGI contributor (e.g., the content poster) to a specific region. This is true for applications ranging from recommender systems that surface venues that are “local favorites” (e.g., [92, 156, 214]) to research that seeks to understand the local perspective on certain issues (e.g., [107, 264, 284]). In fact, the concept of localness is so central to VGI that, when defining the term volunteered geographic information, prominent geographer Michael Goodchild wrote [89]:

“The most important value of VGI may lie in what it can tell us about local activities... that go unnoticed by the world’s media, and about life at a local level. It is in that area that VGI may offer the most interesting, lasting and compelling value. . . ”

However, considering the importance of the concept of localness to VGI, we know surprisingly little about the concept. First and foremost, there are no broadly accepted definitions of “local” in computing, with most projects adopting definitions that are ad-hoc and often unstated. Second, while there are techniques that have been widely used to assess the localness of VGI, they have not been validated (let alone validated against a concrete definition of “local”). In other words, we have little understanding of how well these techniques work, and for which conceptions of localness.

The goal of this paper is to begin the process of addressing these two important gaps in the literature. We first asked the following question: What do we mean in computing when we describe users or information as “local”? To address this question, we conducted a review of the literature that has engaged with the concept of localness. Examining 30 papers, we identified that by “local”, researchers and practitioners typically mean one of three things:

(34)

2.1 Introduction 16 where someone currently lives (which we term the “LivesIn” definition of local), where someone currently votes (“VotesIn”), and the places with which someone is very familiar (“Familiarity”).

After identifying these three definitions, we then posed our second question: “How well do existing localness assessment techniques work, and for which definitions?” We focus on four common localness assessment techniques in particular: LocationField (e.g., [103, 173, 179], nDays (e.g., [122, 153, 193], Plurality (e.g., [162, 178, 208]), and GeometricMedian (e.g., [46, 123, 128]). To enable our understanding of these techniques, we collected the first ground truth localness dataset from VGI contributors. The dataset consists of information from 132 Twitter users and 58,945 total place-tagged tweets, and it was collected via a survey deployed on Twitter.

The results of this analysis provide important methodological guidance for the many practitioners and researchers engaging with the concept of localness. In particular, our results suggest a straightforward set of best practices: If considering a population that frequently geotags (or “placetags”) content, researchers and practitioners should use either Plurality or GeometricMedian. However, if researchers are considering a population that does not frequently use geotagging (or placetagging) functionality, the LocationField approach is an excellent second option.

Our results, however, also point to some important challenges for localness assessment. While Plurality, GeometricMedian, and LocationField all perform reasonably well for “single location” definitions of localness (LivesIn and VotesIn), even at the city scale, all existing localness assessment techniques perform worse for Familiarity. Our results additionally problematize the use of the nDays technique, with the three others being better alternatives in most cases.

Finally, our work additionally highlights several exciting opportunities for future work in this research area. In particular, through our evaluation of localness assessment techniques,

(35)

2.2 Related Work 17 we were able to identify a series of opportunities for the improvement of these techniques. We close the paper with a discussion of these opportunities, as well as a number of other implications for researchers and practitioners that emerge from our results.

2.2 Related Work

2.2.1 Importance of Localness in Computing

The concept of the “local” has become important to a diverse array of research areas in computing, including in geographic HCI [19]. For instance, within the recommender systems domain, a common challenge is surfacing relevant restaurant recommendations from “locals” (e.g., [14,15,58]). Indeed, this area has become sufficiently prominent that systems that provide this service were recently featured in an article in The New York Times [45]. In information retrieval, it has been found that search preferences and information needs in map search differ significantly between locals and non-locals (e.g., [26,54]). For social computing, the concept of “local” has been used to reveal biases in Wikipedia [47], identify potential gaps in coverage of sharing economy and mobile crowdsourcing (e.g., [35,50]), among other applications (e.g., [59]). Data science also frequently engages with localness, e.g., for understanding geographically-variable opinions on policy (e.g., [57]) and in studies of public health (e.g., [7]). Further applications of the concept of local can be found in Table 2.1.

2.2.2 Investigations of Localness

This paper is most directly motivated by the work of Johnson et al. [123]. In this paper, Johnson and colleagues investigated the four localness assessment techniques that we consider here and found that they can give different results for the same Twitter user. However, as noted by Johnson et al., they did not assess how well each technique performed against

(36)

2.2 Related Work 18 ground truth data, in part because no such dataset existed. One of the key goals of this paper was to develop this dataset and to perform this assessment.

Another important source of motivation for this work is the small set of papers that has considered ground truth localness data, but for very specialized contexts. In particular, Sen et al. [227] sought to understand the “geographic provenance” of cited sources on Wikipedia and developed a model for assessing this provenance based on ground truth data about source locations. The known accuracy of the model enabled by this ground truth allowed the authors to make important claims about the degree to which local vs. foreign sources are used to describe different areas of the world. However, this model is limited to the assessment of the localness of the URLs of news articles and other Wikipedia sources (not people), and only works at a national scale.

Localness Assessment Techniques

Johnson et al. [123] identified four localness assessment techniques in the computing litera-ture: nDays, Plurality, GeometricMedian and LocationField. We evaluate each of these four techniques in our experiment, and summarize each of them briefly below:

• nDaysis a temporal range-based technique that assigns a user as local to a region if they produced content in the place at least n days apart. We found the value of n varied from 2 to 30 days( [148, 193]), while n = 10 days ( [107, 123, 153]) was the most commonly used value.

• Plurality, as its name suggests, assigns a user as local to the region (or regions in case of a tie) from which she or he produced the most content. • LocationField extracts the entry in the location field in a user’s profile

(if it exists) and turns that text location (toponym) into machine-readable coordinates using a geocoder (we use Google’s Geocoder). The method

(37)

2.2 Related Work 19 then assigns the user as local to the region output by the geocoder (or, in the case of less granular scales, the containing region).

• GeometricMedianis the most technically complex of the approaches. It assigns a user as local to the point that minimizes the distance between all the locations from which the user has posted content, and then returns the region associated with that point. It additionally has the requirements that the user have at least five posts and that half of the user’s posts be within 30km of the median point (to avoid situations in which, e.g., a user posts from Anchorage, AK and New York, NY and is assigned as local to a district in rural Canada).

2.2.3 Geographic Information and Twitter

The vast majority of prior work on Twitter, including most localness work, has used Twitter coordinate geotags to associate tweets with geographic locations. However, very recent work by Tasse et al. [248] strongly problematized the use of coordinate geotags in a research context:

“The geotags that are still present are getting stranger: job posting bots, weather and sports bots, deleted accounts, and other accounts are creating a growing fraction of all public geotagged tweets... It is not clear how much more research can be done with coordinate geotags.”

Instead, Tasse and colleagues suggest that researchers change their focus to Twitter’s placetags, which associate tweets not with specific coordinates, but rather with named places (e.g., “Nashville, TN”, “Boise River Greenbelt”, “McSorley’s Old Ale House”). We followed this recommendation in this paper, meaning that the methodological guidance afforded by our results applies in this new “placetag era”.

(38)

2.3 Definitions of Localness 20 DEFINITION EXAMPLES

LivesIn

Hecht and Gergle, 2010 [106], Abbar et al. 2015 [1], Abdullah et al., 2015 [2], Culotta, 2014 [53], Girardin et al., 2008 [87], Li et al., 2013 [153], Malik et al., 2015 [162], Mislove et al., 2011 [171], Morais and Andrade, 2014 [173], Naaman et al., 2012 [179], Reiderer et al., 2015 [208], Tasse et al., 2017 [248], Hecht and Stephens, 2014 [107] , Popescu and Grefenstette, 2010 [193], Musthag and Ganesan, 2013 [178], Hecht et al. 2011 [103], Jurgens et al., 2015 [128], Poblete et al., 2011 [192], Johnson et al., 2016 [123], Fiorio et al., 2017 [79], Kogan et al., 2015 [141], Sen et al. 2015 [227]

VotesIn Zhang and Counts, 2015 [284] Familiar

Eckle and Albuquerque 2015 [67], Kumar et al., 2017 [147], Kumar et al., 2017 [148], White and Buscher, 2012 [275], Wu et al., 2011 [280], Zielstra et al., 2014 [285], Ludford et al., 2007 [161]

Table 2.1 Localness definitions.The definitions for localness we identified in the literature and papers that used these definitions.

2.3 Definitions of Localness

While there exists a large literature in computing that engages with the concept of localness, there exists no formal definition(s) of who is a “local”. To address this problem, we conducted a survey of 30 papers in the computing literature that engaged with the concept of localness. In doing so, we leveraged the example and straightforward methods of Johnson et al. [122], which faced a similar definitional question with respect to the vehicle routing literature (i.e. we used a set of core papers, in our case those referenced by [123], iteratively employed keyword and citation network approaches to identify further papers, and collaboratively identified themes in the found literature).

We identified three general themes in how our 30 papers had implicitly and explicitly defined what “local” means: (1) where a person lives, (2) where a person votes, (3) and areas with which a person has a great deal of familiarity. We also note that these definitions can be split into two categories: (1) single location definitions, which assume that a person can be

(39)

2.3 Definitions of Localness 21 local to a single region at a time, and (2) multiple location definitions, which allow people to be local to multiple regions simultaneously. We provide the details about each individual definition, which we respectively term LivesIn, VotesIn, and Familiar, immediately below. We also show which papers utilized which definition in Table 2.1.

The LivesIn definition of localness – a “single location” definition – is relatively straight-forward. Papers that use this definition assume that a person is local only to the region in which they live. Often this region is defined at the city scale, but occasionally it is defined at the neighborhood scale, county scale, and even state and country scale as well. For example, Malik et al. [162] employed the LivesIn definition of localness in their exploration of the biases in geotagged tweets, Morais & Andrade [173] used this definition to understand the difference in annotations shared by tourists and residents, and Fiorio et al. [79] used this definition to estimate short- and long-term migration.

The VotesIn definition – another “single location” definition – is analogous to the LivesIn definition, but applies to the location in which a person votes versus that in which they live. This is an important distinction, as college students, migrants, and others often live in different constituencies than those in which they vote. Zhang and Counts [284] employed this definition of localness to predict same-sex marriage policy change in U.S. states using publicly available Twitter data.

Finally, the Familiarity definition of localness is different from the other definitions in that it does not restrict the assignment of localness to a single region for a given user. This makes Familiarity our only “multiple location” definition. Familiarity labels someone as a local to a given region if they have a sufficient amount of on-the-ground knowledge about the region, with that amount often being extensive. For instance, Zielstra et al. [285] used this definition to study the relationship between knowledge of a place and OpenStreetMap editing patterns and Kumar et al. [148] used this definition to characterize locations using Flickr photos.

(40)

2.4 Methods 22 It is important to note that the papers in our study very often did not explicitly define what they meant by “local”. In these cases, determining the definition that was employed required deeply reading the paper for the underlying assumptions being made. It is our hope that our work can highlight the need to formally declare the definition of local that one is using. Our small schema of localness definitions should make it easier to do so.

Lastly, our review of localness research focused explicitly on the computing literature given the immediate need for increased structure in this literature. However, localness and related ideas like heimat (e.g., [22, 70]), sense of place (e.g., [21, 206, 244, 251]), homeness[225], place attachment [159], place dependence [276], place identity [197, 251], dwelling identity, community identity and regional identity[51] have been studied in the humanities and social sciences for decades (e.g. in geography, sociology, economics). Additionally, further operationalizations of the term “local” appear in various legal and other contexts (e.g. in the food industry [261]). An exciting direction of future work is to engage deeply with these literatures to introduce more sophisticated systematic definitions of localness that can be adopted by the computing literature. In this study, however, our contribution lies in formalizing existing definitions in the computing literature and evaluating how well we can operationalize them with localness assessment techniques.

2.4 Methods

2.4.1 Survey Design

We designed a survey to collect ground truth information such that we could compare the accuracy of each of the four localness assessment approaches with respect to each of the three definitions of localness. Specifically, we asked participants for where they live (LivedIn), where they vote (VotesIn), and locations with which they were familiar. Given

(41)

2.4 Methods 23 recent concerns in the United States about the privacy of voter information [20], we made all VotesIninformation optional.

Most of the 30 papers in our literature view considered localness at the city or county scales, but a few used less granular scales. As such, we focused our analyses at three scales: city, U.S. county, and U.S. state. Similarly, we selected the United States as our study area as it is the region in which much of the localness literature has been conducted (e.g., [1, 2, 21, 93, 103]). As is discussed in more detail below, a compelling direction of future work involves extending our study to other countries.

To gather LivedIn information, we asked the following question: “In which city and state do you live?” Participants were then asked about VotesIn with the optional question “In which city are you registered to vote?”. For Familiarity, we allowed participants to enter up to five cities with which they were familiar. We also asked them to indicate how familiar they were with each entered location on a five-point scale ranging from 1 (“Slightly familiar”) to 5 (“Very familiar”). For each location for which they indicated they were familiar, participants were asked to list their relationship with the location (“I have visited it”, “I have lived in it”, or “Other”, with “Other” including an open text box to describe the relationship). In the below analyses, we consider any Familiarity rating of four or above to be “familiar”, otherwise we treat the corresponding location as not familiar.

The survey, which was implemented in Qualtrics, closed with two final open-ended questions: “Do you have any additional thoughts to share about the areas to which you consider yourself local?” and “Do you have any additional comments about this survey?”

All our survey procedures followed the guidance provided by the IRB and similar organizations at our various institutions. The full text of our survey is available in the Supplementary Materials included with our submission.

(42)

2.4 Methods 24

2.4.2 Survey Sample

Since we focused on Twitter users who use placetags, we created a potential participant list by gathering a set of users for whom their most-recent placetagged tweet was in the United States from the Twitter streaming API for one week during the summer of 2017. In total, we developed a potential survey population of approximately 830,000 users in this fashion.

Our next challenge was finding a way to deploy our survey to this population, and this challenge was a serious one. A well-known approach for collecting ground truth information from social media at scale is the technique outlined by Nichols and Kang in their TSATracker work [181]. At a high level, this approach involves creating a Twitter bot that pings users with a request to tweet at the bot with a desired piece of information. However, this approach was not feasible for us as our research questions necessitated that users to fill out a survey (as opposed to TSATracker, which, e.g., asked a single question about the length of airport security lines). Unfortunately, taking a similar approach to Nichols and Kang with tweets that include a link to a survey (or any link, for that matter) is considered spam by Twitter’s Terms of Service and is banned [256].

This highlights an important issue, not just for this paper, but also for work that engages with social media more generally: if a research question requires data outside of what can be gathered using the standard public behavioral trace information, how can one gather this information at scale?

To partially address this issue, we turned to a version of the TSATracker approach, but one that is formally sanctioned by Twitter: we used Twitter’s ad platform. Specifically, instead of tweeting at users in our target population, we simply uploaded our list of users to Twitter’s ad system and targeted these users via paid ads. It is interesting to note that the exact same content we would have tweeted at users using the TSATracker approach was no longer considered spam as soon as it became a paid ad. We used two ads: one with a monetary incentive (offering a chance to win one of four $25 gift cards) and one with an

(43)

2.4 Methods 25 altruistic incentive. Our study ran for one week in Summer 2017 and from the two ads, we received 22,600 impressions and 222 clicks (1.0% click-through rate (CTR)), and 29,434 impressions and 237 clicks (0.8% CTR), respectively. Overall, we received 136 complete responses and 25 partial responses. Partial responses are those in which the participant did not reach the end of the survey, but did provide us with some information. As long as these partial responses contained LivesIn information, we considered them for the final analysis where relevant.

As we will show below, the scale afforded by the Twitter advertising platform allowed us to get a broad sense of the relative performance of each localness definition. However, the ad platform is sufficiently expensive and low-throughput that gathering information for a project that requires more ground truth data – e.g., training more complex localness models for each localness definition – would not be tractable using this approach. We return to this issue in the Discussion section.

2.4.3 Supplementary Data Collection and Data Cleaning

After filtering out survey responses in which the input Twitter handle was invalid or the LivesIncity was outside the United States (or non-existent), we were left with 132 responses. The accidental input of a Twitter display name instead of a Twitter handle was a common reason for invalid responses. Since display names are non-unique, we had to filter these users out. On inspection of the raw data, we found that some people had filled in the LivesIn city also as a Familiar city, while many others did not. We assumed that people were familiar with cities where they lived and included the LivesIn city in the list of Familiar cities when it was not explicitly included.

Next, we downloaded the most-recent tweets for each of survey participants using the Twitter API, up to 3,200 tweets per user (3,200 is the maximum allowed by the API). We then deleted all tweets that did not have placetags within the United States. On examination

(44)

2.4 Methods 26 of our placetags, we found that approximately 80% of tags were at the city scale, less than 2% of the total placetags were at a scale more local than the city and the rest of the tags were at the state scale or less granular.

We used the Google Geocoding API to determine the city, county and state from the place names in each placetag. In our evaluation of the localness assessment techniques, we eliminated from consideration any tweets whose placetags were at a scale more general than the given scale of analysis. In other words, when analyzing tweets at the city scale, we eliminated any tweets that were tagged at a scale more general than a city, and did the same for county- and state-scale analyses. Some of our participants exclusively geotagged at a state or higher scale, or they had provided only state-scale information in the survey. When performing the county-scale analysis, we excluded 14 such participants and were left with 118 participants. Additionally, two participants had specified only county-level information in the survey and they were removed from city-level analysis, leaving us with 116 valid responses at the city level.

2.4.4 Localness Assessment Techniques

Johnson et al. [123] provided an open-source implementation for all the four localness assessment techniques we consider here. However, since we were dealing with placetags and not geotags as in the case of Johnson et al., we had to re-implement some aspects of the four assessment techniques. We describe these adaptations below:

nDays: For every user, we took the available placetagged tweets and aggregated them into enumeration units at each analysis scale (i.e. we grouped them into cities, counties, and states). As per the definition of nDays, a user was considered local to all of the cities, counties, and states in which they posted at least one pair of tweets more than n days apart (n = 10).

(45)

2.5 Results 27 Plurality: Similar to the case for nDays, the placetagged tweets for each user were first assigned to their corresponding cities, states, and counties. As per the definition of Plurality [123], a user was considered local to the city, county or state (or multiple regions in case of a tie) that contained the maximum number of tweets. This was done separately at each analysis scale (i.e. separately for cities, counties, and states).

LocationField: The LocationField method does not consider placetags like the other approaches; it simply involves looking at the location field in a user’s profile. As such, we followed the standard practice for this method described above (relying on Google’s Geocoding API). If the geocoder returned a region at the desired scale of analysis, we consider that the output of the method. If not, the method was considered to have returned no output.

GeometricMedian: We use the centroid of the bounding box of the place provided in the placetag as a representative point for the place. We then used the implementation of Johnson et al. to calculate the geometric median given this point representation.

2.5 Results

The results of our city-, county-, and state-level analyses can be found in Table 2.2. Overall, these tables reveal extensive variation in the precision and recall of the various localness assessment techniques for each scale-definition combination.

We split the presentation of our results into two parts. We first provide a high-level overview of the most prominent trends present in Table 2.2, organizing our discussion by localness definition type (i.e. single location definitions and multiple location definitions). We then present a discussion of the types of failures we observed for each localness assessment technique, with an eye towards how the techniques may be improved.

(46)

(47)

2.5 Results 29 at the city scale, both have precisions and recalls above 70%. At the county scale, both techniques have precisions and recalls above 80%, and we see continued improvement at the state scale. Interestingly, for state-scale single-location definitions of localness, our results suggest that Plurality exhibits near perfect performance.

The accuracy of the LocationField technique for our single location definitions is even better news for localness assessment. Table 2.2 shows that just by looking at the location field entry in a user’s profile, one can achieve precisions at or above those of Plurality and GeometricMedian. Of course, Table 2.2 also shows that LocationField has a very poor recall, indicating that it often cannot return a local region for users. However, in the case of LocationField, this is much less of a concern: while only a small minority of Twitter users geographically reference their tweets – this number has been observed at 1-3% for geotags [103, 174] – Table 2.2 shows that around 85% of users populate their location fields (and this is roughly the same percentage observed by Hecht et al. [103] as well). In other words, a recall of 47% is not a major issue if you can consider roughly 40 time the number of users in the first place; you will end up with a lot more users with local regions. That said, a number of localness projects consider only a population of users who frequently georeference their posts. In these cases, the Plurality and GeometricMedian should be preferred given their higher recalls. We discuss these dynamics in more detail below.

The accuracy of nDays is the worst of all the methods for single location definitions. With respect to recall, we see performance on par with Plurality and GeometricMedian. However, nDays’ precision is terrible at all scales. Most of this low precision can be explained by a mismatch between the output of nDays and the nature of single location definitions of localness, a point that we discuss in our failure analysis sub-section below.

(48)

2.5 Results 30 Multiple Location Definitions (Familiarity)

The most immediate pattern in the Familiarity results is that they are substantially worse across the board with respect to recall. A key issue here is that techniques that are designed to output one location – GeometricMedian, Plurality and LocationField – are not well suited to capturing the familiarity geography of users, a common need of research and applications in the localness space. However, even nDays, which by design regularly outputs multiple locations still has relative poor recall (and precision). We also performed a sensitivity analysis by setting the threshold of Familiarity to three instead of four on our survey’s five-point familiarity scale. There were no meaningful changes in any of the relative patterns in Table 2.2 (e.g. recall expectedly dropped ~3-5% across the board, but the trends remained the same).

More generally, with regard to precision, we see roughly the same trends as we saw with the single location definitions: Plurality, GeometricMedian, and LocationField have quite high precisions (even greater than 80% at the city scale) and nDays is substantially worse.

2.5.2 Failure Analysis

As noted above, a key goal of our research project was not only to gain an understanding of the accuracy of localness assessment techniques, but also to inform the design of improvements to these techniques where possible. To address this goal, we examined the users for which each technique failed at each scale and attempted to determine the cause for the failures. In this section, we outline some of the common reasons for error for each of the assessment techniques.

LocationField

Although we saw that LocationField performed surprisingly well, especially given the size of the population of users that input LocationField information, there were some clear

Dark side of UGC: A user-centric perspective on the impact of user-generated content