The Multimodal Learning Analytics Handbook

This handbook is the first book ever covering the area of Multimodal Learning Analytics (MMLA). The field of MMLA is an emerging domain of Learning Analytics and plays an important role in expanding the Learning Analytics goal of understanding and improving learning in all the different environments where it occurs. The challenge for research and practice in this field is how to develop theories about the analysis of human behaviors during diverse learning processes and to create useful tools that could augment the capabilities of learners and instructors in a way that is ethical and sustainable. Behind this area, the CrossMMLA research community exchanges ideas on how we can analyze evidence from multimodal and multisystem data and how we can extract meaning from this increasingly fluid and complex data coming from different kinds of transformative learning situations and how to best feed back the results of these analyses to achieve positive transformative actions on those learning processes. This handbook also describes how MMLA uses the advances in machine learning and affordable sensor technologies to act as a virtual observer/analyst of learning activities. The book describes how this "virtual nature" allows MMLA to provide new insights into learning processes that happen across multiple contexts between stakeholders, devices and resources. Using such technologies in combination with machine learning, Learning Analytics researchers can now perform text, speech, handwriting, sketches, gesture, affective, or eye-gaze analysis, improve the accuracy of their predictions and learned models and provide automated feedback to enable learner self-reflection. However, with this increased complexity in data, new challenges also arise. Conducting the data gathering, pre-processing, analysis, annotation and sense-making, in a way that is meaningful for learning scientists and other stakeholders (e.g., students or teachers), still pose challenges in this emergent field. This handbook aims to serve as a unique resource for state of the art methods and processes.

Chapter 11 of this book is available open access under a CC BY 4.0 license at link.springer.com.

Autorentext

Michail (Michalis) Giannakos is a professor of interaction design and learning technologies at the Department of Computer Science of the Norwegian University of Science and Technology (NTNU). He is the head of the Learner-Computer Interaction lab and his research focuses on the design and study of emerging technologies in online and hybrid education settings, and on developing new ways for humans to interact with interactive learning systems. Giannakos has co-authored more than 200 manuscripts published in prestigious peer-reviewed journals and conferences (including Computers & Education, Computers in Human Behavior, IEEE Pervasive Computing, IEEE TLT, BIT, BJET, ACM TOCE, ACM IDC, ICLS/CSCL, Interact). Giannakos, together with Mike Horn, is the Editor-in-Chief of the International Journal of Child-Computer Interaction (Elsevier). He is also on the Editorial Board of IEEE Transactions in Learning Technology, IEEE Transactions on Education, Behaviour & Information Technology, and the International Journal of Information Management, and has served as a guest editor on highly recognized journals such as BJET, Computers in Human Behavior, ACM TOCE and IEEE Multimedia. He has served as an evaluator for the European Commission (EC) and the US-NSF, and he recently authored a book on Experimental Studies in Learning Technology and Child-Computer Interaction (Springer) and co-authored the Educational Data Literacy textbook (Springer). Giannakos has worked at several research projects funded by diverse sources like the EC, Microsoft Research, The Research Council of Norway (RCN), US-NSF, the German agency for international academic cooperation and Cheng Endowment. Giannakos is one of the experts in the Norwegian task force (formed by the ministry of education and research) for introducing learning analytics to Norwegian K-12 schools and universities. He is also a recipient of a Marie Curie/ERCIM fellowship, the Norwegian Young Research Talent award and he is one of the outstanding academic fellows of NTNU (2017-2022). Giannakos usually attends the ACM Interaction Design and Children (IDC) conference (he was a Chair for IDC 2018, and PC-chair for IDC 2022), the ACM conference on Innovation and Technology in Computer Science Education (ITiCSE) (he was a Chair for ITiCSE 2020), and the Learning Analytics and Knowledge (LAK) conference. Daniel Spikol is Associate Professor of Computational Thinking in the Computer Science and Science Education departments at the University of Copenhagen (UCPH). He is part of the university's Center for Digital Education, which investigates and helps support digital education transformation. Daniel holds a PhD in Computer Science for Linnæus University, Sweden (2010), an MSc in Visual Studies from Massachusetts Institute of Technology (1992), and a BFA from Rhode Island School of Design (1987). The central theme of his research investigates how to integrate computational thinking and tinkering into education. His work combines physical computing and robotics with STEM and STEAM education to provide an authentic context to engage people in science and computing. Additionally, his research investigates how people collaborate with multimodal learning analytics (inspired by social signal processing ambient computing). Daniel combines science, design, digital art, and computer science to investigate how people learn and play. He has 70 peer-reviewed scientific publications. He has worked for different companies and research institutes and founded several companies. Daniel has been active in developing SOLAR's SIG for Multimodal Learning Analytics, participated in the program committees for different conferences such as LAK, EC-TEL, ICALT, AIED, IDC, and ICQE. He has also been a principal investigator for European and national research projects. Previous to UCPH, he was the scientific manager of the Internet of Things and People Research Lab at Malmö University and part of the Computer Science and Media Technology department. Daniele Di Mitri is a research group leader at the Education Technology group at DIPF Leibniz Institute for Research and Information in Education and a lecturer at the Goethe University of Frankfurt. Daniele received his Ph.D. title at the Open University of The Netherlands (2020) in Learning Analytics. His Ph.D. project, the Multimodal Tutor, investigated the potentials of collecting and analysing multimodal data during physical interactions for automatic feedback and human behaviour analysis. Daniele's current research focuses on designing responsible Artificial Intelligence applications for education and human support. He is a "Johanna Quandt Young Academy" fellow. He was elected "AI Newcomer 2021" at the KI Camp by the German informatics society and received the "Martin Wolpers Award" in technology-enhanced learning 2018. He is a member of the editorial board of Frontiers in Artificial Intelligence journal, a member of the CrossMMLA, a special interest group of the Society of Learning Analytics Research, and he is the chair of the Learning Analytics Hackathon (LAKathon) series. Kshitij Sharma is is associate professor of learning technologies at the Department of Computer Science of the Norwegian University of Science and Technology (NTNU). He is member of the Learner-Computer Interaction lab and his research focuses on the using artificial intelligence and machine learning to study students' behaviour and design emerging technologies in online and hybrid education settings for them to learn in more effective and efficient manner. Sharma holds a Ph.D. in computer science from EPFL, Lausanne, Switzerland. Sharma has co-authored more than 100 manuscripts published in prestigious peer-reviewed journals and conferences (including B…