The idea for this project originated from the observation that blind individuals are unable to experience information contained in a 2D plane, such as painting or text directly. But what if 2D images on paper could somehow convey they presence when touch without requiring the user to wear any device? Or what if they could tell a story as well; or if they produced different sounds depending on where the user places his/her finger? How much richness about a 2D image could be delivered in this way? Moreover, for sighted individuals, what if the image was completely hidden, so that only non-visual information about the hidden image is available? How much detail could be conveyed through touch and audio? What other applications would such an interface have. What novel applications and user-experiences would it enable?
This project explores the possibility of creating a drawing and storytelling book for children similar to coloring books, except that the pages are completely blank, and there is no visual information at all. Hidden within each page, however, there are capacitive touch sensors in the form of different characters or objects contained in the story. When a child moves explores a blank page with their finger, they hear sounds corresponding to the different characters on the page, or they hear stories or dialogue told by the characters. In this way, a child is encouraged to explore this hidden landscape and discover it piece by piece. Moreover, the child/user is also encouraged to draw the images and characters that are on the page, and in this way to unravel what the presented scene looks like. This is somewhat similar to coloring books, but yet very different because the user has no visual information, only auditory and positional information, and they have complete freedom to choose how the characters and objects should look. This interface affords a new kind of multimodal exploratory drawing and storytelling experience. It might also serve as a valuable educational tool for both sighted and non-sighted children.
The idea of embedding capacitive touch sensors inside paper could also be extended into numerous other applications beside the aforementioned. For instance, we might 3D objects that the user is supposed to place at specific locations on a page with specific orientation in for a secret audio message to be revealed or for the story to continue. Or we could have an interactive map / globe, so that when a user touches a specific country on the map/globe, they hear the world "Hello" in the language of that country. Or we could augment a board game so that it includes visual, auditory, and interactive components while still being a board game and a computer game.
My primary objective was in testing the feasibility of building the proposed system, determining the kinds of tools and equipment necessary for it, and building a versatile platform. By making this work public and documenting the process of building it, my intent is artists and content designers to be able to develop content suitable for such an interface, without having to reinvent the interface. To test how the platform could be used, I also developed a simple use case demonstration example. The goals for this particular use-case are to stimulate creative imagination in kids and to encourage creative expression through drawing with minimal guidance.