Jakob Kilian
← back

Unfolding Space

  • project
  • areas: design, research, tech
  • tags: paper, raspberry pi, sensory substitution, study, time-of-flight, visually impaired, wearable

The Unfolding Space Glove allows blind or visually impaired users to haptically sense the depth of their surrounding space and thus (hopefully) better navigate through it. The device was drafted and developed within the framework of a four-year design research project. In 2021, the prototype was tested in an empirical study with 14 sighted and blind subjects with the results being published. 

Substituting missing visual stimuli by means of another sensory modality is referred to as Sensory Substitution in scientific terminology. The code and all components of the project have been released to the public under an Open Source licence. Depending on the field and audience, however, the device can have other labels as well: 

Sensory Substitution Device

There is a tradition of more than 50 years of research on Sensory Substitution. It is related to recent findings in neuroplasticity and cognitive science that describe the brain's ability to adapt to changes in the sensory periphery even in adulthood.

Navigation Aid System

Rather than replacing the full depth of visual information, the purpose of the device is to aid blind or visually impaired people in navigating independently and effectively. Object recognition is thus deliberately not included in the functional spectrum.

Visual Prosthesis

The two aforementioned aspects also make it, at least partially, a Visual Prosthesis. Users of similar sensory substitution devices report vision-like experiences after some training - the corresponding brain activity can be observed in MRI scans.

Current Prototype (2021)

Glove-closeup.jpg
Glove-frontal_w_user-2.jpg
Glove-Donning_PowerBank.jpg
Glove-Donning_Cable-1.jpg
Glove-Donning_Cable-2.jpg
Glove-Donning_Velcro.jpg
PCB_closeup.jpg
Glove-Using_2.jpg
Glove-Using_3.jpg

Previous Prototypes (2018-2020)

front_02.jpg
closeup.jpg
action02.jpg
underground.jpg
prototype.jpg

Background Information

Basic Functionality

A depth image from a 3d camera is haptically projected on the back of the hand by using vibration motors. The location of a vibration depicts an object’s relative position in space, the strength of the vibration represents its distance. See the demo video on the right for better understanding.

Project Background

There is a multitude of projects that have dealt with the topic of Sensory Substitution, but to date there are only very few practical implementations of the idea, which in turn are used by a negligible small number of people. While extremely sophisticated technology is used, design and user-friendliness often suffer. I therefore started researching the topic in an open and iterative rapid prototyping process. Problems of existing solutions should be included, the operation and learning should be easier and a higher acceptance for the system should be created by addressing usability and interaction design requirements.

Current Prototype & Empiric Study in 2021

Over the years, several prototype versions were created, up to the current version presented here. In the summer of 2021, I was able to test it in an empirical study with 14 blind and sighted test persons as part of my master’s thesis. As expected, it turned out that the path to a practical aid for blind people is still long – nevertheless, the study revealed a number of aspects that the Unfolding Space Glove already meets or which are valuable for further developments. Also see section “Publication” for further Details and a link to the paper.

Early protoype from 2018 with technical equipment in a backpack

Technical Details

The Prototype now consists of only three components: a 5V USB power bank, a USB-C power cable and the glove itself hosting a mini computer and the 3D camera.

Once those three are connected you are ready to go:

  • External hardware or specific premises are no longer required.
  • Runs indoor and outdoor.
  • No need specific lighting conditions.
  • A single battery charge holds around eight hours.

Total component costs are about $ 500.  For an overview, see the pictures on the left. For details, parts lists and building instructions, please go to the GitHub repo linked below.

Future of the Project

After four years of active development, I will let this project rest due to lack of time and money. Of course, questions, tips or feedback are always welcome and maybe the project will continue with or without me at another time. That would make me really happy – the foundation stone has been laid, all the data is open. 

Publications

Unfolding Space (Paper)

Go to the separate post about the (well cited) Unfolding Space paper that has been published by MDPI in 2022.

Code and Content

Unfolding Space © 2018 by Jakob Kilian is licensed under CC BY 4.0

In order to inspire other designers, developers or commercial providers to work within this field and to make Sensory Substitution available to as many people as possible this project is Open Source. All contents are under CC-BY-4.0 license, the code is under MIT license. Please check the LICENSE files in the respective repositories. This website hosts general information about the project. Please find all components, files, documentation and publications related to this project at the locations linked below:

GitHub | Main & Code

This is the main repository of the project. Besides general information, software code and documentation on project setup, installation and use can be found here.

GitHub | Hardware

Building instructions for the glove and the Unfolding Space Carrier Board, PCB files, parts lists / BOMs, 3D design files and more...

GitHub | Monitor

Multi platform project in Unity 3D of the Monitoring App that allows to monitor the activity of the device via Wifi and adjust settings. Including builds and documentation.

Vimeo | Study Videos

Collection of exemplary videos from the study in 2021 showing: pocedure of the trials, examples of efficient wayfinding with the glove, comparisons between white long cane and glove and more ...

Hackaday | Instructions & Blog

Project page on hackaday.io, where you can find building instructions, I regularly post news and you can discuss about the project.

Awards

Winner of the "Kölner Design Preis / Toby E. Rodes Award" 2018

HackadayPrize 2019 - Honorable Mention (Best Concept)​

Golden Pin Concept Design Award 2019 - Design Mark Winner​

Lucky Strike Junior Designer Award 2018 – Special Recognition​

Press Releases

In English:
Other Languages:

Credits

The project wouldn’t have been possible without the help of many people, to whom I would like to express a big thank you at this point:

  •  Kjell Wistoff for his active support in setting up, dismantling and rebuilding the study room, organising the documents and documenting the study photographically.

  • Trainer Regina Beschta for a free introductory O&M course and the loan of the study long cane.

  • Tim Becker and Matthias Krauß from Press Every Key for their open ear when giving advice on software and hardware.

  • Köln International School of Design/KISD (TH Köln) and the responsible parties for making the premises available over this long period of time.

  • Tom Bieling, Ulrike Gollner & Gesche Joost 

And all those who provided guidance in the development of the prototype over the past years and now in the implementation and evaluation of the study. Specifically (and probably more that I can not remember at this point):

Jannik Nitz, Connor Shafran, Johanna Warchol, Marco Reitano, Brandon Gilles, Niklas Isselburg, Eduard Paal, Dmitry Kuznetsov, Sebastian Miller, Sören Stirn, Ronnie Balcazar

Hardware sponsoring was granted by:

pmdtechnologies

Providing a Pico Flexx 3D camera.

Munitec GmbH

Providing multiple glove samples that made into the final prototype.

order

home, type, featured, project, thing, note, design, interaction, experience, data, product, tech, hardware, software, sound, praqma, research, cognitive-science, cognitive science, design-research, design research, teaching, publication

hi there!

I’m Jakob. I am a creative technologist, designer, maker and (re-)searcher, and this website shares my work and thoughts.

Privacy policy: I don’t track, I don’t collect or retain data.

tech

I often work on interceptions of design and engineering, bringing up some hardware or software (or both) projects, code bases and notes.

design

During my »Integrated Design« studies at KISD in Cologne, Germany, I worked in various design disciplines, fostering a holistic approach to design. While I explored traditional fields such as communication, industrial, service and interface design, my focus sharpened on these broader terms:

  • interaction: (inter-)action possibilities between humans and machines /computers.
  • experience: the aesthetic and sensory reception of audio/visual installations.
  • data: making raw data accessible and processable.
  • product: physical “things”.

research

This section showcases projects from my recent academic work. This includes projects from my own studies and the results of my work as a scientific employee at KISD from 2022 to 2025.

blog

Blogs are outdated. That’s why I started writing one.
All too often, I find myself down a rabbit hole, researching topics that haven’t received enough coverage on the internet yet, or simply stumbling upon ideas that are too valuable to keep to myself.
I’m passionate about sharing, which is why I started this blog. I hope you enjoy it!

music

Praqma is a producer and DJ of electronic music, blending diverse house and techno elements, a temporary utopia to escape the world’s imperfection.

Some of my early work is available on streaming and purchase platforms. Since 2023, I have focused solely on developing new hardware and software for techno live performances. See the »Modulator« and »Clipdeck« projects below, for example. In October 2025, I uploaded the first one-hour live set from this project.