Elevator Pitch:

Many individuals require additional support for communication and language learning, such as the use of hearing aids. However, these devices are seldom sufficient to restore complete understanding. The innovation in your project is to add an additional channel of speech information on a person’s eyeglasses. This nonobtrusive device will perform continuous real-time acoustic analysis of his or her interlocutor’s speech and transform several continuous acoustic features of the talker’s speech into continuous visual cues displayed on the eyeglasses. The proposed activity will benefit the society by providing a research and theoretical foundation for a system that would be naturally available to almost all individuals at a very low cost.

Illustration of visual cues on the iGlasses in face-to-face conversation

Illustration of the iGlasses connected to an iPhone

In Depth – The Problem:

The need for language aids is pervasive in today’s world. Millions of individuals have language and speech challenges, and these individuals require additional support for communication and language learning. For example, about 10% of the population in the United States is hearing-impaired, and this population is even larger worldwide.

Currently, however, the needs of these persons, such as limited understanding in face-to-face communication, are not being met. This proposal develops technology to supplement the common face-to-face language interaction to enhance intelligibility, understanding, and communication.

The Solution:

We propose to develop and implement iGlasses that will show on LEDs acoustic features extracted from the speech that will complement the information conveyed by the face and help many different individuals perceive speech. This device will supplement the sound of speech and speechreading of the face with an additional informative visual input. Acoustic characteristics of the speech will be transformed into readily perceivable visual characteristics.

The goal of the proposed research is to develop and test the technology required to design a device seamlessly worn by the listener, which will perform continuous real-time acoustic analysis of his or her interlocutor’s speech. The iGlasses would transform several continuous acoustic features of the talker’s speech into continuous visual features, which will be simultaneously seen in the periphery of the lens. These acoustic features provide important linguistic information not directly observed on the face and are transformed into visual cues intended to enhance intelligibility and ease of comprehension. Pilot research has demonstrated that the acoustic features can be extracted and tracked in real time and that individuals can learn to use these visual cues to enhance their speech perception. This system does not require any learning on the part of the talker and is perceptually and linguistically motivated because it is directly based on acoustic and phonetic properties of speech and gives continuous rather than only categorical information.