Foundation Models for Natural Language Processing

This open access book provides a comprehensive overview of the state of the art in research and applications of Foundation Models and is intended for readers familiar with basic Natural Language Processing (NLP) concepts.

Over the recent years, a revolutionary new paradigm has been developed for training models for NLP. These models are first pre-trained on large collections of text documents to acquire general syntactic knowledge and semantic information. Then, they are fine-tuned for specific tasks, which they can often solve with superhuman accuracy. When the models are large enough, they can be instructed by prompts to solve new tasks without any fine-tuning. Moreover, they can be applied to a wide range of different media and problem domains, ranging from image and video processing to robot control learning. Because they provide a blueprint for solving many tasks in artificial intelligence, they have been called Foundation Models. After a brief introduction to basic NLP models the main pre-trained language models BERT, GPT and sequence-to-sequence transformer are described, as well as the concepts of self-attention and context-sensitive embedding. Then, different approaches to improving these models are discussed, such as expanding the pre-training criteria, increasing the length of input texts, or including extra knowledge. An overview of the best-performing models for about twenty application areas is then presented, e.g., question answering, translation, story generation, dialog systems, generating images from text, etc. For each application area, the strengths and weaknesses of current models are discussed, and an outlook on further developments is given. In addition, links are provided to freely available program code. A concluding chapter summarizes the economic opportunities, mitigation of risks, and potential developments of AI.

Offers an overview of pre-trained language models such as BERT, GPT, and sequence-to-sequence Transformer Explains the key techniques to improve the performance of pre-trained models Presents advanced pre-trained models for a broad range of NLP tasks This book is open access, which means that you have free and unlimited access

Autorentext

Dr. Gerhard Paaß is a Lead Scientist at the Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS). With a background in Mathematics, he is a recognized expert in the field of Artificial Intelligence, particularly in the area of Natural Language Processing. Dr. Paaß has previously worked at UC Berkeley in California and the University of Technology in Brisbane. He has served as reviewer and conference chair at various international conferences, including NeurIPS, CIKM, ECML/PKDD, ICDM, and KDD, where he regularly is a member of the program committee. Dr. Paaß has received a "best paper" award on probabilistic logic, is the author of about 70 publications for international conferences and journals and he is co-author of the best-selling Open Access book "Foundation Models for Natural Language Processing" (Springer, 2023). He is currently involved in the creation of a computer center for Foundation Models. Besides experimental researchon Foundation Models, he holds lectures for Deep Learning and Natural Language Understanding at the University of Bonn and in industry. Dr. Dirk Hecker is deputy director of the Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS and managing director of the Fraunhofer Big Data and Artificial Intelligence Alliance, the largest alliance of the Fraunhofer-Gesellschaft with more than 30 member institutes. In addition, Dr. Hecker is a member of the board of directors of the Fraunhofer Academy, which is a further education institution of the Fraunhofer-Gesellschaft aimed at specialists and executives of technology-driven companies. Dr. Hecker has many years of experience in leading research and industrial projects in the field of data mining and Machine Learning. His current work focuses on Big Data Analytics, Predictive Analytics, Deep Learning and Explainable AI. He studied geo-informatics at the Universities of Cologne and Bonn and received his PhD from the University of Cologne. Dr. Hecker is the author of numerous publications in the above-mentioned fields and is active as a subject matter expert and auditor on the topic of Artificial Intelligence in a wide variety of committees.

Inhalt

1. Introduction.- 2. Pre-trained Language Models.- 3. Improving Pre-trained Language Models.- 4. Knowledge Acquired by Foundation Models.- 5. Foundation Models for Information Extraction.- 6. Foundation Models for Text Generation.- 7. Foundation Models for Speech, Images, Videos, and Control.- 8. Summary and Outlook.