Modeling data from visual and linguistic modalities together creates opportunities for better understanding of both, and supports many useful applications. Examples of dual visual-linguistic data includes images with keywords, video with narrative, and figures in documents. We consider two key task-driven themes: translating from one modality to another (e.g., inferring annotations for images) and understanding the data using all modalities, where one modality can help disambiguate information in another. The multiple modalities can either be essentially semantically redundant (e.g., keywords provided by a person looking at the image), or largely complementary (e.g., meta data such as the camera used). Redundancy and complementarity are two endpoints of a scale, and we observe that good performance on translation requires some redundancy, and that joint inference is most useful where some information is complementary. Computational methods discussed are broadly organized into ones forsimple keywords, ones going beyond keywords toward natural language, and ones considering sequential aspects of natural language. Methods for keywords are further organized based on localization of semantics, going from words about the scene taken as whole, to words that apply to specific parts of the scene, to relationships between parts. Methods going beyond keywords are organized by the linguistic roles that are learned, exploited, or generated. These include proper nouns, adjectives, spatial and comparative prepositions, and verbs. More recent developments in dealing with sequential structure include automated captioning of scenes and video, alignment of video and text, and automated answering of questions about scenes depicted in images.

Autorentext

Kenichi Kanatani received his B.E., M.S., and Ph.D. in applied mathematics from the University of Tokyo in 1972, 1974, and 1979, respectively. After serving as Professor of computer science at Gunma University, Gunma, Japan, and Okayama University, Okayama, Japan, he retired in 2013 and is now Professor Emeritus of Okayama University. He was a visiting researcher at the University of Maryland, U.S., (1985-1986, 1988-1989, 1992), the University of Copenhagen, Denmark (1988), the University of Oxford, U.K. (1991), INRIA at Rhone Alpes, France (1988), ETH, Switzerland (2013), University of Paris-Est, France (2014), and Linkoping University, Sweden (2015). He is the author of K. Kanatani, Group-Theoretical Methods in Image Understanding (Springer, 1990), K. Kanatani, Geometric Computation for Machine Vision (Oxford University Press, 1993), K. Kanatani, Statistical Optimization for Geometric Computation: Theory and Practice (Elsevier, 1996; reprinted Dover, 2005), and K. Kanatani, Understanding Geometric Algebra: Hamilton, Grassmann, and Clifford for Computer Vision and Graphics (AK Peters/CRC Press 2015). He received many awards including the best paper awards from IPSJ (1987), IEICE (2005), and PSIVT (2009). He is a Fellow of IEICE and IEEE.Yasuyuki Sugaya received his B.E., M.S., and Ph.D. in computer science from the University of Tsukuba, Ibaraki, Japan, in 1996, 1998, and 2001, respectively. After serving as Assistant Professor of computer science at Okayama University, Okayama, Japan, he is currently Associate Professor of computer science and engineering at Toyohashi University of Technology, Toyohashi, Aichi, Japan. His research interests include image processing and computer vision. He received the IEICE best paper award in 2005.Yasushi Kanazawa received his B.E. and M.S. degree in information engineering from Toyohashi University of Technology in 1985 and 1987, respectively, and his Ph.D in information and computer science from Osaka University in 1997. After engaging in research and development of image processing systems at Fuji Electric Co., Tokyo, Japan, and serving as Lecturer of Information and Computer Engineering at Gunma College of Technology, Gunma, Japan, he is currently Associate Professor of computer science and engineering at Toyohashi University of Technology, Aichi, Japan. His research interests include image processing and computer vision.

Inhalt
Acknowledgments.- Figure Credits.- Introduction.- The Semantics of Images and Associated Text.- Sources of Data for Linking Visual and Linguistic Information.- Extracting and Representing Visual Information.- Text and Speech Processing.- Modeling Images and Keywords.- Beyond Simple Nouns.- Sequential Structure.- Bibliography.- Author's Biography.