Designing a Multisensory ASL Game

Research

• More than 90% of deaf children are born to hearing parents, yet only 22.9% of families with deaf children in the U.S. report regularly signing at home.
• Deaf children who are exposed to ASL during infancy are more likely to acquire vocabulary at an age-appropriate rate.

• Multisensory feedback (haptic, visual, auditory) enhances learning and reinforcement more effectively than unimodal feedback.
• Music has been found to enhance EEG power across the brain, supporting the role of auditory stimuli in learning activities.

Approach

We designed a three-level ASL learning game with the following features:


• Sensor-equipped glove to detect correct ASL gestures
• Haptic vibrations for real-time feedback
• Visual UI confirmation for correct gestures
• Auditory reinforcement (spoken letter pronunciation)
• Classical background music to test cognitive effects (experimental group)

User Testing and Iterations

We conducted user testing with 23 participants to evaluate the system's usability and measure the impact of multisensory feedback on ASL gesture learning and retention.

Participants were pre-screened and met the following eligibility criteria:

• Right-handed: The glove used in the study was designed for the right hand. Left-handed participants might have experienced slower response times, potentially skewing the results.
• Normal or corrected vision and hearing: Ensured participants could fully engage with the visual and auditory components of the system.
• No prior knowledge of sign language: Prevented any bias related to pre-existing familiarity with ASL gestures.‍
• Full mobility of the five fingers on the right hand: Ensured participants could perform the required gestures without physical limitations.

Participants rated the system 86/100 for usability (SUS score).

Results

The experimental group with classical background music demonstrated better immediate recall of ASL gestures. However, both the control and experimental groups scored similarly high on visual short-term memory tasks, indicating no clear correlation between background music and gesture retention.

This project demonstrated that a multisensory game incorporating visual, haptic, and auditory reinforcement shows promise for improving gesture retention and engagement for ASL learners.

Reflections

With additional time and funding, I would have refined both the experimental design and system usability.

• Qualitative User Research – Conducting interviews with hearing individuals who are learning ASL or who have learned ASL in the past could guide development and ensure that the system addresses users' needs.
• Sensor Accuracy – Participants frequently adjusted the glove between rounds, affecting sensor accuracy. Future iterations could use self-calibrating sensors to reduce errors.
• Haptic Feedback Consistency – Some users didn’t notice the vibrations, likely due to hand size differences. Adjusting vibration intensity or using custom-fit gloves could improve feedback.