Empowering individuals and organizations to achieve more requires a commitment to inclusive computing experiences and leveraging technological advancements to aid those with disabilities. Over the years, Microsoft Research has worked closely with people with disabilities to develop and refine innovative assistive technologies. Here’s a look at some of their key projects:
Find My Things: Personalized Object Recognition
Find My Things is a teachable AI tool that helps people with blindness or low vision locate lost personal items. It’s a personalized object recognition tool that can be trained using just a few videos of a particular item. Available within the Seeing AI mobile app, the tool was recognized for its innovative design, being a finalist in the accessible design and artificial intelligence categories of Fast Company’s Innovation by Design Awards. Find My Things was created by the Microsoft Research Teachable AI Experiences (Tai X) team and a group of citizen designers and integrated into Seeing AI earlier in the year.
Foundation Models and Navigation
As part of the Microsoft Research’s Accelerate Foundation Models Research (AFMR) initiative, a team from Waseda University is developing a system utilizing vision and language foundation models. This aims to improve outdoor navigation capabilities for people with blindness or low vision.
ASL Citizen Dataset
Microsoft released ASL Citizen, the first crowdsourced isolated sign language dataset, in collaboration with the Deaf community. The project addresses the data shortage hindering the advancement of AI systems that assist sign language users. Researchers have also explored fairness, accountability, transparency, and ethics considerations in collecting sign language data.
PeopleLens: Personalization in Social Scenarios
PeopleLens, an advanced research prototype tested by students in the United Kingdom, uses AI and spatialized audio to identify individuals in a room. This assists users in social situations and improves interactions. Development of the PeopleLens research team identified the value personalization could add to such an experience and informed the creation of Find My Things’ object recognition capabilities.
Ludic Design for Accessibility
Researchers introduced Ludic Design for Accessibility (LDA), a multistep method that emphasizes play and exploration in the design of assistive technologies. This approach has influenced multiple projects, including an audio-based app to encourage physical activity among people with vision impairments and a study. Microsoft Research India collaborated with the Srishti Manipal Institute of Art, Design and Technology to incorporate LDA into its curriculum.
ORBIT Dataset and Benchmark
Microsoft released the Open Research for Blindness and Impaired Vision (ORBIT) dataset and benchmark. This project collected videos of personal items from the blind and low-vision community to create a training dataset inclusive of objects they might use (like guide canes) and more representative of image/video quality variation.
Alt-Text Experience Research
Researchers developed and tested interfaces for writing alt text from scratch and for providing feedback on AI-generated alt text within Microsoft PowerPoint, influencing improvements to the alt-text authoring experience available in the presentation software. Prior work includes the study of people’s experiences with alt text across different types of websites and specifically when it comes to image on social media.
Teleworker Perspectives During the Pandemic
The shift to remote work during the pandemic offered teleworkers with disabilities the chance to share how their experience was affected by having their colleagues join them in working remotely. This plays a role in expanding how Microsoft understands the impact of remote and hybrid work on people with disabilities.
Microsoft Expressive Pixels
VR and AR Accessibility
Researchers in the Microsoft Research Ability Group received an honorable mention at the 2020 ACM CHI Conference for a haptic and auditory white cane designed for navigating VR environments. This project builds on previous research to make virtual and augmented reality more accessible to people with mobility issues or hearing disabilities.
SeeingVR Toolkit
SeeingVR, a set of tools designed to enhance VR experiences for those with low vision was made open source. These tools provide audio and visual augmentations, such as magnification lenses and text-to-speech.
Sign Language–Based Systems Workshop
Microsoft Research hosted a workshop to explore sign language recognition, generation, and translation. Key insights from the workshop were later published and won best paper at the International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS).
Code Jumper Technology Transfer
Hands-Free Music Project
The Microsoft Hands-Free Music project won an SXSW Interactive Innovation Award for music and audio innovation, also being a finalist in the “innovation in connecting people” category. The project includes music-making tools controlled via eye gaze and other hands-free techniques.
Eye Control in Windows 10
Eye Control was introduced to Windows 10, allowing users to control their PCs with eye movements. This feature leverages gaze-tracking technology and allows users to type, control their mouse, and activate text-to-speech. This development included collaboration between the Microsoft Research Enable team and the Windows Text Input team.
Seeing AI Mobile App
Seeing AI, a mobile app that leverages computer vision and natural language processing to help people with blindness or low vision identify objects and people in their surroundings, was released for iOS and later Android. Features include object and currency recognition, as well as enriched descriptions and question-answering powered by generative AI.
Microsoft Soundscape
Microsoft partnered with Guide Dogs and Future Cities Catapult to launch the Cities Unlocked prototype, later known as Microsoft Soundscape. Soundscape employs 3D audio cues to assist users in navigating their surroundings. It became available on iOS and was later made open source.
Eye Gaze Wheelchair
Microsoft Research’s continued dedication to assistive technology highlights its commitment to inclusive design and accessibility, with innovations that contribute to a more accessible and empowering world for all.
