This book is one outcome of the NATO Advanced Studies Institute (ASI) Workshop, "Speechreading by Man and Machine," held at the Chateau de Bonas, Castera-Verduzan (near Auch, France) from August 28 to Septem­ ber 8, 1995 - the first interdisciplinary meeting devoted the subject of speechreading ("lipreading"). The forty-five attendees from twelve countries covered the gamut of speechreading research, from brain scans of humans processing bi-modal stimuli, to psychophysical experiments and illusions, to statistics of comprehension by the normal and deaf communities, to models of human perception, to computer vision and learning algorithms and hardware for automated speechreading machines. The first week focussed on speechreading by humans, the second week by machines, a general organization that is preserved in this volume. After the in­ evitable difficulties in clarifying language and terminology across disciplines as diverse as human neurophysiology, audiology, psychology, electrical en­ gineering, mathematics, and computer science, the participants engaged in lively discussion and debate. We think it is fair to say that there was an atmosphere of excitement and optimism for a field that is both fascinating and potentially lucrative. Of the many general results that can be taken from the workshop, two of the key ones are these: . The ways in which humans employ visual image for speech recogni­ tion are manifold and complex, and depend upon the talker-perceiver pair, severity and age of onset of any hearing loss, whether the topic of conversation is known or unknown, the level of noise, and so forth.

Inhalt

Speechreading by Humans.- Psychology of Human Speechreading.- Word Recognition in Speechreading.- Children with Hearing Loss: Speechreading Skills.- Differences in Visual Intelligibility Across Talkers.- The Use of Auditory and Visual Information in Phonetic Perception.- Bimodal Speech Perception: A Progress Report.- Auditory-Visual Speech Perception as a Direct Process: The McGurk Effect in Infants and Across Languages.- Seeing Brains Reading Speech: A Review and Speculations.- Perception of Conflicting Audio-Visual Speech: an Examination across Spanish and German.- Audio-Visual Speech Perception Without Speech Cues: A First Report.- Perception of Synthetic Visual Speech.- Homopheneity in speechreading: Effects of phonemic equivalence classes on the structure of the lexicon.- Aspects of Modality in Audio-Visual Processes.- Exploiting sensor fusion architectures and stimuli complementarity in AV speech recognition.- Does movement on the lips mean movement in the mind?.- The Dynamics of Audiovisual Behavior in Speech.- Where and When are the Heard and Seen Speech Integrated: Magnetoencephalographical (MEG) Studies.- Multiphasic Analysis of the Basic Nature of Speechreading.- How can coarticulation models account for speech sensitivity to audio-visual desynchronization?.- Working Memory and Speechreading.- Encoding of Visual Speaker Attributes and Recognition Memory for Spoken Words.- A Study of the Semantic Memory Access by Perceptual Modalities with a Semantic Priming Experiment.- Lips and Jaw Movements for Vowels and Consonants: Spatio-Temporal Characteristics and Bimodal Recognition Applications.- Which components of the face do humans and machines best speechread?.- Speechreading by Machines.- Visionary Speech: Looking Ahead to Practical Speechreading Systems.- Talking Heads and Speech Recognisers That Can See: The Computer Processing of Visual Speech Signals.- Automatic Speechreading using dynamic contours.- Active Shape Models for Visual Speech Feature Extraction.- 2D Deformable Models for Visual Speech Analysis.- Fast Matching of a Dynamic Lip Model to Color Video Sequences under Regular Illumination Conditions.- Towards a Robust Speechreading Dialog System.- Robust Face Feature Analysis for Automatic Speachreading and Character Animation.- Time Delay Neural Networks for Articulatory Estimation from Speech: Suitable Subjective Evaluation Protocols.- Relations of Audio and Visual Speech Signals in a Physical Feature Space: Implications for the Hearing-impaired.- On the Integration of Auditory and Visual Parameters in an HMM-based ASR.- Channel Separability in the Audio-Visual Integration of Speech: A Bayesian Approach.- Audiovisual Sensory Integration Using Hidden Markov Models.- Neural-fuzzy networks and phonetic feature recognition as a help for speechreading.- Rationale for Phoneme-Viseme Mapping and Feature Selection in Visual Speech Recognition.- Panel discussions.- Human Speechreading: Learning and Psychophysics.- Human Speechreading: Psychology and Cognition.- Sensory Integration by Humans and Machines.- Databases, Standards and Comparisons.- Machine Recognition and Applications.