Satyendra Awasthi, Madhupriya Pathak, Rajesha N, Manasa G, Narayan Choudhary, L.
Ramamoorthy
6.1 I NTRODUCTION
Hindi is an Indo-Aryan language, a descendent of Sanskrit, which is spoken in the central and northern India, in the states of Bihar, Chhattisgarh, Delhi, Haryana, Himachal Pradesh, Jharkhand, Madhya Pradesh, Rajasthan, Uttarakhand and Uttar Pradesh. It is the official language of the Union of India and is also lingua franca across India. Being the most intelligible language of India, it is currently reported to be spoken as the first language by 528.35 million people in India (as per 2011 census of India) i.e. a total of 43.63% of the populace of India speaks Hindi as their primary language.
According to the constitution of India the official languages of the union of India are Hindi and English, where Hindi to be written in Devanagari. Based on the provisions mentioned in the Official Language Act, Hindi is used for official activities such as communications between the Central Government and State Government which recognizes Hindi as official language, judiciary and parliamentary proceedings.
Hindi is written in Devenagari script, a Left to Right script which is a descendent of Brahmi script. The script is also used to write several other languages of India and neighboring countries such as Nepali, Marathi, Maithili etc.
Hindi text corpus has been collected from various areas in India, mostly from Uttar Pradesh. The greater part of the corpus has been taken from Kendriya Hindi Sansthaan (Central Institute of Hindi), Delhi and Agra libraries and Bhartiya Bhasha Sansthan (Central Institute of Indian Languages), Mysore library.
LDC-IL has tried to cover the entire category in its standard list. Some categories such as novel, short stories have huge greater proportion of content share than the other domains or sub-domains such as science, technology, economics etc.
6.2 P ECULIARITIES OF H INDI TEXT
The Hindi text corpus can be broadly classified into two: literary text and non-literary text. These two explicitly show their differences in terms of the types and the source of the word usage and variety that it brings into corpus. Literary texts are mainly narratives, and more or less they contain the elements of fiction. Some instances of such texts are: novels, short stories and plays. On the other hand the non-literary texts convey information based on their primary purpose. These range from the articles in academic journals, legal documents to the texts of various scientific or technical subjects. The literary texts have emotional elements, cultural information, dialectical variations, ambiguity etc. whereas the non-literary texts show a tendency of usage of technical and scientific terms, foreign words etc.
42 Hindi Raw Text Corpus
6.3 D ATA S AMPLING N OTES
6.3.1 Principles of Data Sampling
Hindi text data sampling strictly follows the LDC-IL text corpus collection generic guidelines, which are noted in the LDC-IL generic corpus documentation.
6.3.2 Fieldworks Undertaken
Hindi text corpus is collected from various libraries in India, mostly from Uttar Pradesh. The text materials were collected by conducting fieldworks undertaken during the period from 2005 to 2008. The greater part of the text has been taken from Kendriya Hindi Sansthaan, Delhi and Agra library and Central Institute of Indian Languages, Mysore library.
Overall, the following libraries served as the source of the Hindi text corpus:
1. Allahabad Public Library, Allahabad 2. Banaras Hindu University, Varanasi
3. Central Institute of Indian Languages, Mysore 4. Kendriya Hindi Sansthaan, Agra, and
5. Kendriya Hindi Sansthaan, Delhi
Collected text materials have been published at various places in India. Such as Delhi, Uttar Pradesh, Rajsthan, Madhya Pradesh, Uttarakhand, Bihar, Himachal Pradesh, Hariyana, Jharkhand, Maharashtra, Kerala etc.
An attempt has been made to cover the entire category in its standard list. Some categories like novel, short stories have huge amount of books but some categories like physics, chemistry, economics have very less amount of books. Literary texts are easily available in Hindi but getting scientific text is very difficult. Some categories like epigraphy, finance, oceanology text are very less then Literary texts in Hindi.
Collecting text data from the field is a difficult job. Most of the libraries do not allow to take huge amount of text from their shelves at a time because it is against their rules and principles. For a particular period, they issue maximum three or four books. Even if the librarian allowed to take many books at a time, the photocopy kiosk had issues as there was a long queue.
Sometime Xerox attendants refused to photocopy randomly selected pages because of the long queue waiting and it takes up more time for them to turn the pages compared to continuous page photocopying they are accustomed to. It was another issue that the field worker/linguist had to carry a huge list of photocopy bundles with them which was many times cumbersome to travel with.
Despite all the issues as above, the linguists working on the data collection had to deal with and get going.
6.3.3 Data Inputting
All the text has been typed in Unicode using the InScript Keyboard directly onto the XML files. The data has been inputted by around 26 inputters.
Linguistic Resources for AI/NLP in Indian Languages 43
6.3.4 Validation and Normalization Workshops
As per the validation and normalization workshop conducted by LDC-IL, it was recommended by the experts that the Hindi text corpus should remain true to the text.
6.3.5 Proofreading
Hindi text data has been proofread by internal resource persons and the resource persons engaged in the Short-term goal oriented projects (Text corpus cleaning Workshops). An account of such workshops is as below:
1. August 2010
2. 31st Dec. 12 – 1st March 13 3. 1st June – 31st July, 2015 4. 23rd May – 15th July, 2016 5. 2nd August – 28th September 2018
Text manipulation has been avoided thoroughly, and only the mistakes occurred during the input process has been corrected with a reference to the hard copies of the texts. Since poetry doesn’t serve the purpose of general machine learning due to its anomalous scheme, grammar and construction which doesn’t adhere to the day to day language needs, therefore instances of poetry have been removed from the running texts.
The printed material collected for the corpus is contemporary, mainly published after 1990.
6.3.6 Data Extracted from Websites
Hindi News corpus data is extracted from News websites of "Ranchi Express" (http://ranchiexpress.com),
"Dainik Bhaskar" (https://www.bhaskar.com), "Rajasthan Patrika" (https://www.patrika.com), and "Nav Bharat Times" (https://navbharattimes.indiatimes.com). The news content was categorized based on the content of the text and archived. The period of selection of the news corpus ranges from 2005 to 2008.
6.4 T RANSLITERATIONS IN LDC-IL H INDI TEXT CORPUS
For easy reference and uniformity of metadata, some entries in the metadata file, namely ‘Title’,
‘Headline’, ‘Author’, ‘Editor’, ‘Translator’ are transliterated from Hindi to Roman letters. Numeric characters were transliterated from Hindi to roman system.
For such purpose the LDC-IL transliteration scheme for Devenagari to Roman characters is given below:
LDC-IL Transliteration Schema
Devanagari characters to Roman and Hindi Numerals to Roman
Vowels
and Vowel SignsVowel
अ आ इ ई उ ऊ ऋ ए ऐ ओ औ अं अः अँ ऑMatra
ा िा ा ा ा ा ा ा ा ा ां ाः ाँ ाKey
a A i I u Ux
E ai O auM H m' ao
44 Hindi Raw Text Corpus
These are the borrowed sounds, however they are listed in the schema since they occur frequently in the literary text.
6.5 O VERVIEW OF R EPRESENTED D OMAINS
The size of LDC-IL Hindi Text Corpus is: 10317177 Words and 52569629 characters, gathered from 1223 different titles, including the extracts from newspapers. The data can be categorized into two classes namely ‘Typed and cleaned corpus’ and ‘Crawled corpus’. The crawled data has been crawled mainly from news websites and archived using the standard processing of LDC-IL text corpus preparation.
The following table gives a summary of the typed and crawled text of the Hindi Raw Text Corpus.
Text Type Word Count Keystroke/Character Count
Typed+Cleaned 5315913 26812263
Consonant
Consonant
क् ख् ग् घ् ङ्Key
k kh g gh ng'Consonant
च् छ् ज् झ् ञ्Key
c ch j jh nj'Consonant
ट् ठ् ड् ढ् ण् ड़् ढ़्Key T Th D Dh N D' Dh'
Consonant
त् थ् द् ध् न्Key
t th d dh nConsonant
प् फ् ब् भ् म्Key
p ph b bh mConsonant
य् र् ल् व्Key
y r l vConsonant
श् ष् स् ह्Key
sh S s hBorrowed
Consonant
क़ ख़ ग़ ज़ फ़Key k'a Kh'a g'a j'a ph'a
Numerals (Devanagari to Roman mapping)
Devanagari
० १ २ ३ ४ ५ ६ ७ ८ ९Roman 0 1 2 3 4 5 6 7 8 9
Linguistic Resources for AI/NLP in Indian Languages 45
Crawled 5001264 25757366
Total
10317177 52569629
Table 6-1: Representation of the typed and crawled Hindi Text Corpus
The representation of the four major domains covered has been shown in the table below:
Domain
Domain Word
Count Percentage
Aesthetics 3822697 37.05%
Mass Media 5012327 48.58%
Science & Technology 549143 5.32%
Social Sciences 933010 9.04%
Total 10317177 100.00%
Table 6-2: Representation of the Domains in Hindi Text Corpus
As each domain has several sub-domains, the following table shows the representation of the several domains, both within the domain and across all the domains.
6.5.1 Aesthetics
The Aesthetics category of Hindi text corpus covers 22 sub-categories bearing a total of 38,22,697 words along with the overall percentage of 37.05%. The representational details are given in the table below.
# Sub Category Word
11 Literature-Children's Literature 33749 0.88% 0.33%
12 Literature-Criticism 149527 3.91% 1.45%
13 Literature-Diaries 16585 0.43% 0.16%
14 Literature-Essays 60496 1.58% 0.59%
15 Literature-Letters 20421 0.53% 0.20%
16 Literature-Novels 1646693 43.08% 15.96%
46 Hindi Raw Text Corpus
17 Literature-Plays 163518 4.28% 1.58%
18 Literature-Science Fiction 3544 0.09% 0.03%
19 Literature-Short Stories 1235074 32.31% 11.97%
20 Literature-Speeches 5634 0.15% 0.05%
21 Literature-Travelogues 16151 0.42% 0.16%
22 Mythology 28991 0.76% 0.28%
Total 3822697 100% 37.05%
Table 6-3: Aesthetics Category Representation
6.5.2 Mass Media
The Mass Media category of Hindi text corpus covers 5 sub-categories bearing a total of 50,12,327 words along with the overall percentage of 48.58%. The representational details are given in the table below.
# Sub Category Word 5,49,143 words along with the overall percentage of 5.32%. The representational details are given in the table below.
Linguistic Resources for AI/NLP in Indian Languages 47
Table 6-5: Science and Technology Category Representation
6.5.4 Social Sciences
10 Personality Development 5206 0.56% 0.05%
11 Philosophy 45591 4.89% 0.44%
12 Political Science 97128 10.41% 0.94%
13 Religion/Spiritual 260543 27.92% 2.53%
14 Sociology 29889 3.20% 0.29%
15 Sports 17395 1.86% 0.17%
Total 933010 100% 9.04%
Table 6-6: Social Sciences Category Representation
48 Hindi Raw Text Corpus
6.6 C OPYRIGHT C ONSENTS
The Hindi text corpus has been collected from various sources therefore the copyright for the same stays with the different sources. However, for this purposes consents have been sought from all the stakeholders. Most of the copyrights belong to private parties with only a minor part belonging to the government agencies, either state or the central.