Mixed reality interaction model to control connected objects.
It's based on exchanging digital information between people and objects to allow objects a better adaption to their environment and usage context. This approach allows to reduce mental fatigue by constant decision making and information overload.
This project has been submitted to publish so some information is now
closed for general public.
Request more info
We want to explore a simple way to control our growing ecosystem of connected objects, often plenty of functionalities placed on multiple menus and options. Moreover we want to avoid frustration for not getting an expected response while interacting.
During everyday life, people interact with multiple tools, from basic objects such as an spoon to a more complex device such an Smart TV. In the following years there's a forecast of even more connected devices. Many of them have lots of functionalities in order to provide the user a higher number of options to fit their needs. While this increase of capabilities can be beneficial to create a more rich and personalized experience, the complexity of interfaces to control them has also been increased. This leads to a higher cognitive effort for the user due to information overload and decision making.Digital Sense explores how objects can learn and adapt to users if they can identify and get information from them on each interaction. In order to do that we posed the following hypothesis:
" Objects can better adapt their responses to people if they can have memories of the shared experiences together. Similar to human relationships, on each interaction a learning can happen to shape future responses "
Our research also focuses on the idea that despite the fact that objects tend to increase on features and capabilities we mainly use a reduced set of these. This reduction will be directly related to the concept presented before.
Example of reduction of complexity
As stated on research goals an exploration on how seamlessly interact with connected objects must be done. Since we expect our interface with everyday objects the best quote to explain our approach is the following:
" Everyday interactions with objects must be done without thinking " Donald D.Norman
We refer to this quote in two aspects to reduce cognitive overload: choosing cost and task structure. Choosing cost is related to the inherent cost of choosing between a large number of options. Task structure refers to the number of steps required to complete a task and the possibilities available in each moment. Both concepts must be minimized in order to reduce user cognitive load.
On the left, everyday when we chose our dressing we make an effort. On the right, an example of a simple task structure: warm oil on a pan, break an egg, wait to be cooked and eat.
Then, Digital Sense interaction model main capabilities, sharing preferences and recognition must contemplate these two main aspects in order to adress our research goals.
In order to explore different ways to exchange digital information we created some fake prototypes which consisted on mixed reality solutions using smartphones camera.
Some fake prototypes to explore interactions
With these prototypes we identified the opportunity of using augmented reality on objects to transfer or embed information. We also saw how this information can be smoothly transferred to personal accounts.Our first insights allowed us to create our first working prototype, FaceTag which consisted on a smartphone app that identified both objects and users using image recognition to later exchange their information.
In terms of recognition, we decided to mimic human most common way to identify objects to reinforce user mental model while interaction and we maintained it to also identify objects
Our goal was to define an intuitive interaction model to transfer this information. This system had to fit two needs; simplify the number of steps and avoid mental overload reducing choice cost.
To do that we used the concept of digital identity, consisting on a digital representation of a physical element which had embedded all their own information. Once created this representation we created a method to interact which this digital entities which consisted on a three step process
We decided to create a minimal interface in order to reduce the choices given to user. Once a certain identity is detected, users only have to drag and drop its representation closer to other one to related. We based this simple but effective interaction on Gestalt law of proximity When two representations are closer they are linked. Otherwise if user separates them, the link is interrupted
To explore possible applications of this concept we brainstormed and conducted some interviews based on the first prototype. We identified some potential everyday objects which can use this system such as TV, speakers, lights or the fridge. Also we detected a pattern that show great potential of our concept:
" Sometimes while interacting with an object, users don’t know exactly what they want and even multiple responses can be valid for them "
Imagine watching TV or listening to music. In a similar context there are multiple responses that can be valid for the user. This degree of freedom lets the object the opportunity to decide. Moreover, object will be able to better decide if it can access to memories of previous experiences.
To test this concept we created a second prototype BeSpeaker which consisted on two speakers that shaped their identity from different user’s interactions. We also trained the system with three human identities with different musical preferences. Given a certain time, each speaker was able to response differently depending on their past memories. You can see the results of this study here
The main contribution of concept is the proposal of a system that allow manipulation of digital information providing an intuitive and easy way to configure and control behaviours of physical objects. This can enhance the experience of interacting with connected objects by reducing information overload and mental fatigue due to constant decision making.
Elements on Digital Sense approach
Digital Sense is composed by an Android app to recognize humans and objects and allow manipulation of digital information and multiple connected objects.
To allow identities manipulation, this app must use smartphone camera to detect humans and objects. On one hand, human recognition has been implemented via a Face Recognizer created with OpenCV. We train the system giving it a certain number of tagged faces from multiple users. On the other hand, object recognition is based on Vuforia markers that we place direcly in connected objects. Both systems work together to detect either a human or a tagged object.
Communication between connected objects and main app has been setup up using a common shared network. These objects must have a Wi-fi module to have network access.
Current implementation supports multiple user detection and connection to three connected objects: two speakers and a lamp. Each object has been preconfigured with a set of possible responses when they are activated. Within time, object's responses are shaped depending on whom interacts with them. To do that, we use a probabilistic model based on both user preferences and memory of the specific object.
In the following image you can see the full process of linking identities working.
A user and objects are detected and a representation is placed on main app. Then with a simple drag an drop, both identities can be linked
Finally, system is also prepared to allow interaction between multiple objects but further research must be done to define possible behaviors. Right now, when two objects are placed together, the selected object behavior is copied to the other one mimicking the first one.
We validated interaction model and perception of digital identities via interface specific tasks and extracted qualitative data based on Thinking Aloud method.
Users described the system as intuitive and easy to use, facilitating the interaction with everyday objects.
One user interacting with the interface
Our results demonstrate that interaction model has been well understood. We believe that using similar metaphors for interacting with both human and objects has helped to reinforce this mental model. Testers were also able to extrapolate the concept to other daily objects present on their environment.
Finally, we must consider the importance of human decision capabilities. Despite the fact that our proposal can lead into a reduction of mental fatigue due to constant decision making and decision cost, some situations may require final user decision.
Further research is needed to determine when users expect automatic responses and which ones they want to influence more on these responses . Moreover, we can also explore how to use this system to suggest possible object reactions rather than directly triggering an action.
Since augmented and mixed reality market is expected to increase with consumer ready devices, it will be a great to translate this model to a head-device since only a camera is needed to operate.
Other research line would be how to allow recognition and data transfer without a camera, using wearable devices and technologies such as RFID.
Finally it will be also an interesting research topic explore which data is relevant for objects to understand an specific user need or mood. Here we could consider daily routines, emotional status or contextual information as an important factor to shape object response.
In this research project was responsible for all process from research definition, to concept creation and final implementation and testing. I also wrote a research article about our results.