Jakob Kilian
back
A close-up shot of the glove in use. The electronics are visible on the outside, and a cable extends from the right side of the image.
Unfolding Space – Glove 2
Step 1 of donning: mount battery pack at the upper arm
Step 2 of donning: plug in the power cable at the battery pack
Step 3 of donning: plug in the power cable at the glove
Step 4 of donning: mount the electronics on the glove
Closeup shot of the electronics that are mounted on the glove. The black 3D camera, a cooling fan and PCBs are visible.
A person wearing the glove and stretching his hand in the direction of an obstacle made of cardboard.
A person looking concentrated and wearing the glove in an obstacle course made of cardboard boxes.

Unfolding Space

  • A sensory substitution wearable enabling visually impaired users to haptically perceive the depth of their surroundings. Vibration motors on the back of the hand translate a live 3D camera image into spatial information — position encodes direction, intensity encodes distance. The project spans four years of iterative development across interaction design, custom electronics, embedded software, and textile construction, culminating in an empirical study with blind and sighted participants and a peer-reviewed publication.

Concept, Design, Development, Study

Jakob Kilian

Study Execution

Jakob Kilian; Kjell Wistoff

Thesis Supervisors

Prof. Dr. Lasse Scherffig; Prof. Dr. Siegfried Wahl

Funding

Zeiss Vision Science Lab; KickStart @ TH Köln
  • main
  • areas design, research, tech
  • tags paper, raspberry pi, sensory substitution, study, time-of-flight, visually impaired, wearable

Sensory Substitution describes the brain’s capacity to interpret information from one sensory modality through another – vision through touch, for instance. The field has a history of more than fifty years, grounded in findings on neuroplasticity that demonstrate the brain’s ability to adapt to changes in its sensory periphery well into adulthood.

Yet practical implementations remain rare, and those that exist are seldom used. Sophisticated in technology, they often fall short in usability and interaction design. 

In 2018, I started Unfolding Space in an attempt to address that gap. Over the span of four years, I developed successive prototype generations, each of which was tested with blind and sighted users, toward an affordable, learnable device grounded in interaction design. In 2021, I conducted a formal empirical study, and the findings were published in a peer-reviewed paper in 2022. What follows tells that story.

Previous Prototypes (2018-2020)

Person wearing a glove with visible electronics on the ouside and a big cable going out to a backpack on his back.
2018 prototype in default position
closeup of vibration motors and cables sewed on top of the textile of the glove
Vibration motors on the glove
A Person using the glove in an outside urban situation.
Early outside tests in 2018
A person using a escalator in a subway station.
Test in a subway station in 2018
The arm of a person with lots of loose cables taped to the arm itself with tape. The person also holds a laptop in the other arm. The picture looks very experimenetal
First functional test also in 2018

Talk at 37th Chaos Communication Congress

Screenshot showing a recording of a talk.

In 2023, I had the opportunity to present my project at 37c3 in Hamburg. The talk was recorded and provides a quick 30-minute overview of the project, followed by a Q&A session. The video can also be streamed here.

Background Information

Basic Functionality

A depth image from a 3D camera is haptically projected on the back of the hand by using LRA vibration motors. The location of a vibration (X and Y) depicts an object’s relative position in space, the strength of the vibration represents its distance. Also watch the demo video on the right for better understanding.

Project Background

There is a multitude of projects that have dealt with the topic of Sensory Substitution, but to date there are only very few practical implementations of the idea, which in turn are used by a negligible small number of people. While extremely sophisticated technology is used, design and user-friendliness often suffer. I therefore started researching the topic in an open and iterative rapid prototyping process. Problems of existing solutions should be included, the operation and learning should be easier and a higher acceptance for the system should be created by addressing usability and interaction design requirements.

Still from a video showing three frames at once: a real camera image, a depth image from far in black to close objects in white and a matrix of nine circles showing the intensity of each motor.

A demo from a user’s perspective comparing the reality with the depth image and the translation into vibration.

A person wearing a glove and a backpack that ar connected with cables. On the glove one can see electronic parts.
2018 prototype with technical equipment in a backpack

Latest Prototype & Empirical Study In 2021

In the summer of 2021, I was able to test the latest prototype in an empirical study with 14 blind and sighted test subjects as part of my master’s thesis. As expected, it turned out that the path to a practical aid for blind people is still long – nevertheless, the study revealed a number of aspects that the Unfolding Space Glove already meets or which are valuable for further developments. Also check section “Publication” for further details and a link to the paper.

An outstreched arm wearing the glove an a battery pack on the upper arm. The image contains text describing the position of these parts.
The entire system in use
Closeup photo of the electronics that are mounted on the glove with text labels describing their position: connection to the glove, the 3D ToF camera, a Raspberry Pi Compute Module, a cooling fan and the USB-C power connector.
The glove's electronics in closeup

Technical Details

The Prototype now consists of only three components: a 5V USB power bank, a USB-C power cable and the glove itself hosting a mini computer and the 3D camera.

Once those three are connected you are ready to go:

  • External hardware or specific premises are no longer required.
  • Runs indoor and outdoor.
  • No need specific lighting conditions.
  • A single battery charge holds around eight hours.

Total component costs are about $ 500.  For an overview, see the pictures on the left. For details, parts lists and building instructions, please go to the GitHub repo linked below.

Future of the Project

After four years of active development, I will let this project rest due to lack of time and money. Of course, questions, tips or feedback are always welcome and maybe the project will continue with or without me at another time. That would make me really happy – the foundation stone has been laid, all the data is open. 

Empirical Study and Publication

Timeline:

  • Summer 2021: Study conducted
  • Fall 2021: Data collected and statistically analyzed
  • January 2022: Master’s thesis about the project and study finished
  • February 2022: Paper published in MDPI with over 30 citations
overview image with a picture of the unfolding space glove and the 3 keywords design, implementation and evaluation
Project overview used in the MDPI paper

This paper documents the design, implementation and evaluation of the Unfolding Space Glove—an open source sensory substitution device. It transmits the relative position and distance of nearby objects as vibratory stimuli to the back of the hand and thus enables blind people to haptically explore the depth of their surrounding space, assisting with navigation tasks such as object recognition and wayfinding. The prototype requires no external hardware, is highly portable, operates in all lighting conditions, and provides continuous and immediate feedback—all while being visually unobtrusive. Both blind (n = 8) and blindfolded sighted participants (n = 6) completed structured training and obstacle courses with both the prototype and a white long cane to allow performance comparisons to be drawn between them. The subjects quickly learned how to use the glove and successfully completed all of the trials, though still being slower with it than with the cane. Qualitative interviews revealed a high level of usability and user experience. Overall, the results indicate the general processability of spatial information through sensory substitution using haptic, vibrotactile interfaces. Further research would be required to evaluate the prototype’s capabilities after extensive training and to derive a fully functional navigation aid from its features.

MDPI – Sensors | Scientific Paper

The main paper was cited about 40 times by 2026. It was published on February 26, 2022, in the special issue "Spatial Perception and Navigation in the Absence of Vision" of MDPI Sensors.

ResearchGate | All Publications

You can find me on ResearchGate. Next to the MDPI paper you can also find the theoretical thesis (only in German) I wrote as my bachelor thesis.

Impressions from the Study

three pictures: a, showing a person doing a vision test at a computer b, showing the obstacle course used in the study from a top view c, showing the grid system of the obstacle course in closeup
A) Vision test during the study B) The obstacle course from a top view C) The grid system in closeup
three images (A, B and C tagged) showing different people using the glove in the obstacle course
Three subject using the glove and the white cane during the study

Code and Content

Unfolding Space © 2018 by Jakob Kilian is licensed under CC BY 4.0

In order to inspire other designers, developers or commercial providers to work within this field and to make Sensory Substitution available to as many people as possible this project is Open Source. All contents are under CC-BY-4.0 license, the code is under MIT license. Please check the LICENSE files in the respective repositories. This website hosts general information about the project. Please find all components, files, documentation and publications related to this project at the locations linked below:

GitHub | Main & Code

This is the main repository of the project. Besides general information, software code and documentation on project setup, installation and use can be found here.

GitHub | Hardware

Building instructions for the glove and the Unfolding Space Carrier Board, PCB files, parts lists / BOMs, 3D design files and more...

GitHub | Monitor

Multi platform project in Unity 3D of the Monitoring App that allows to monitor the activity of the device via Wifi and adjust settings. Including builds and documentation.

Vimeo | Study Videos

Collection of exemplary videos from the study in 2021 showing: pocedure of the trials, examples of efficient wayfinding with the glove, comparisons between white long cane and glove and more ...

Hackaday | Instructions & Blog

Project page on hackaday.io, where you can find building instructions, I regularly post news and you can discuss about the project.

Awards

Winner of the "Kölner Design Preis / Toby E. Rodes Award" 2018

Logo of the Cologne Design Price

HackadayPrize 2019 - Honorable Mention (Best Concept)​

Logo of the HackadayPrize 2019

Golden Pin Concept Design Award 2019 - Design Mark Winner​

Logo of the Golden Pin Concept Award

Lucky Strike Junior Designer Award 2018 – Special Recognition​

Logo of the Lucky Strike Junior Designer Award 2018

Press Releases

In English:
Other Languages:

Credits

The project wouldn’t have been possible without the help of many people, to whom I would like to express a big thank you at this point:

  • Kjell Wistoff for his active support in setting up, dismantling and rebuilding the study room, organising the documents and documenting the study photographically.
  • Trainer Regina Beschta for a free introductory O&M course and the loan of the study long cane.
  • Tim Becker and Matthias Krauß from Press Every Key for their open ear when giving advice on software and hardware.
  • Köln International School of Design/KISD (TH Köln) and the responsible parties for making the premises available over this long period of time.
  • Tom Bieling, Ulrike Gollner & Gesche Joost by publishing their work on gloves as interfaces.

I would also like to thank all those who supported and guided me in developing the prototype over the past few years, as well as in implementing and evaluating the study. Specifically, and probably more than I can remember at this point:

Jannik Nitz, Connor Shafran, Johanna Warchol, Marco Reitano, Brandon Gilles, Niklas Isselburg, Eduard Paal, Dmitry Kuznetsov, Sebastian Miller, Sören Stirn, Ronnie Balcazar, …

Sponsoring

Big thanks also to these companies supporting my project:

Logo of pmd tech

pmdtechnologies

Providing a Pico Flexx 3D camera.

Logo of Munitec

Munitec GmbH

Providing multiple glove samples that made into the final prototype.