Labelling data is one of the most fundamental activities in science, and has underpinned practice, particularly in medicine, for decades, as well as research in corpus linguistics since at least the development of the Brown corpus. With the shift towards Machine Learning in Artificial Intelligence (AI), the creation of datasets to be used for training and evaluating AI systems, also known in AI as corpora, has become a central activity in the field as well. Early AI datasets were created on an ad-hoc basis to tackle specific problems. As larger and more reusable datasets were created, requiring greater investment, the need for a more systematic approach to dataset creation arose to ensure increased quality. A range of statistical methods were adopted, often but not exclusively from the medical sciences, to ensure that the labels used were not subjective, or to choose among different labels provided by the coders. A wide variety of such methods is now in regular use. This book is meant to provide a survey of the most widely used among these statistical methods supporting annotation practice. As far as the authors know, this is the first book attempting to cover the two families of methods in wider use. The first family of methods is concerned with the development of labelling schemes and, in particular, ensuring that such schemes are such that sufficient agreement can be observed among the coders. The second family includes methods developed to analyze the output of coders once the scheme has been agreed upon, particularly although not exclusively to identify the most likely label for an item among those provided by the coders. The focus of this book is primarily on Natural Language Processing, the area of AI devoted to the development of models of language interpretation and production, but many if not most of the methods discussed here are also applicable to other areas of AI, or indeed, to other areas of Data Science.

Autorentext

Silviu Paun got his Ph.D. from the University of Essex in 2017 with a thesis on topic models. Since then he has been at Queen Mary University of London. His research focuses on models of annotation, probabilistic and neural, for creating resources and to more efficiently train machine learning models. His models have been deployed to create the Phrase Detectives coreference corpus, one of the largest crowdsourced NLP corpora, created using the Phrase Detectives Game-With-A-Purpose.Ron Artstein received his Ph.D. in Linguistics from Rutgers University in 2002, held positions at the Technion-Israel Institute of Technology and the University of Essex, and is presently a research scientist at the Institute for Creative Technologies, University of Southern California. His current research focuses on the collection, annotation, and management of linguistic data for human-machine interaction, analysis of corpora, and the evaluation of implemented dialogue systems; he has published work on theoretical and computational linguistics, conversational dialogue systems, and human-agent and human-robot interaction.Massimo Poesio received his Ph.D. from the University of Rochester in 1994. He is a Professor in Computational Linguistics at Queen Mary University of London and a Turing Institute Fellow. His main interests are in anaphora resolution, disagreements in language interpretation, the use of games-with-a-purpose for creating NLP resources, and semantic interpretation in dialogue.