Speech Communication through the Skin: Design of Learning Protocols and Initial Findings

Computer Human Interaction (CHI)


Evidence for successful communication through the sense of touch is provided by the natural methods of tactual communication that have been used for many years by persons with profound auditory and visual impairments. However, there are still currently no wearable devices that can support speech communication effectively without extensive learning. The present study reports the design and testing of learning protocols with a system that translates English phonemes to haptic stimulation patterns (haptic symbols). In one pilot study and two experiments, six participants went through the learning and testing of phonemes and words that involved different vocabulary sizes and learning protocols. We found that with a distinctive set of haptic symbols, it was possible for the participants to learn phonemes and words in small chunks of time. Further, our results provided evidence of the memory consolidation theory in that recognition performance improved after a period of inactivity on the part of a participant. Our findings pave the way for future work on improving the haptic symbols and on protocols that support the learning of a tactile speech communication system in hours as opposed to the much longer periods of time that are required to learn a new language.

Related Publications

All Publications

Federated Learning for User Privacy and Data Confidentiality Workshop At ICML - July 24, 2021

Federated Learning with Buffered Asynchronous Aggregation

John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Michael Rabbat, Mani Malek, Dzmitry Huba

ISMAR - July 29, 2021

Instant Visual Odometry Initialization for Mobile AR

Alejo Concha, Michael Burri, Jesus Briales, Christian Forster, Luc Oth

ICSA - November 6, 2019

Auralization systems for simulation of augmented reality experiences in virtual environments

Peter Dodds, Sebastià V. Amengual Garí, W. Owen Brimijoin, Philip W. Robinson

UAI - July 28, 2021

A Nonmyopic Approach to Cost-Constrained Bayesian Optimization

Eric Hans Lee, David Eriksson, Valerio Perrone, Matthias Seeger

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy