Observing use of augmented reality with The Sinjensen5000

I’ve been working a bit on augmented reality (AR) projects lately and one of the things that commonly comes up is: yeah it’s fun but so what? I know what I think of augmented reality and what all the other tech-geek-blog-twits think of it. But what about “normal” people? How do they use it and interact with it? It seemed that the only way to get around the “so what” issue was to start doing some observation and testing. Time for an analysis and creativity mashup.

An observation platform for “normal” human use of augmented reality

One of the challenges of improving deployment of augmented reality projects is not having much information on how people use it and interact with it. The common deployment of AR via a web browser doesn’t allow for much observation of use: the subjects are in their own environments playing with the technology.

I wanted to put together an environment where a random group of people would be able to interact with an augmented reality project and I would be able to watch them. My goals were fairly broad, I wasn’t looking for any sort of specific behavior, just an open-minded “What do they do with this?” approach. Something anecdotal to talk about next time someone looked at AR and said “Looks fun, but so what?”

I chose to make a performance art project as my observation platform. A large party/event/festival in Burlington, VT called the Art Hop would provide my audience.

What happens when “normal” people use augmented reality? (A small study)

While the entire project was immensely fun and entertaining, my real goal was to see what people would do with it, how they’d react, if they’d seen this sort of thing before and so on. Here are some of the things I learned, in no particular order:

• Basic technology issues came first. Many people spent a lot of brainpower discovering that the webcam was built into the screen of the Macbook Pro. When presented with the technology many of audience members chose to approach it like a puzzle, trying to figure out how it worked.
• Once they had a basic sense of how to move and control the digital mug the audience appeared to enjoy playing with it: pretending to drink with it, tipping it upside down, putting it on their heads and so on. Most laughed and joked with each other and myself while using the technology.
• Interest in the technology seemed gender neutral and non-age-specific. Kids definitely enjoyed it (though they were often very confused by my talk of Sinjensen Corp’s first mover advantage in the emergent digital dishware market). But so did college kids, parents and a few grandparents as well.
• Once the basic technology and haptics of the system were worked out, some audience members who were still approaching the technology like a puzzle would attempt to figure out how the technology worked, running it through some tests and learning things like: breaking the black border makes the mug disappear, testing to see if the blank side of the paper the glyph was printed on would also produce an image and so on.
• Audience members typically interacted with the augmented reality project for two to three minutes. Though some played with it much longer (usually children of either gender or dads who were obviously of the puzzle-solving mentality).
• Only one person, a college student, mentioned that she had heard of something like this before but was unsure where or how.
• One of the other artists exhibiting near the entrance/exit of the Thoughtfaucet studio was able to overhear conversations on the way out. She reported that many of those exiting the studio were talking about the technology and the children up into the five to seven age band were questioning their parents about whether it was real or not.

It seemed that the audience tended to break into a two groups. Puzzlers wanted to know how the thing worked. They’d start by figuring out the camera and proceed from there to test the system. These tended to ask direct questions to me about how it worked but I would stay in character and let them know that the engineers at the Sinjensen Corporation worked day and night to advance the innovative systems driving the Sinjensen5000. For the determined I gave a business card.

Then there were the Playfuls. These folks took hold of the glyph and didn’t display any active test-making behavior but instead tipped the digital mug, pretended to drink with it, pretended to bonk their friends with it and so on. This group rarely had technology questions for me. They were much more likely to ask why we only had a mug or other questions more related to the performance aspect than the technology aspect of the project.

No one who interacted with the project gave it a non-response. Though some audience members who watched the demonstration but did not have a chance to play with the glyph did drift off, it was just as often that non-players would stay and watch what others would do with the technology.

The Sinjensen5000 Digital Dishwasher: a performance-art-based research methodology for augmented reality projects.

The basic premise of the performance art piece was that in the future, all dishware will be digital due to the looming changeover from analog dishware to digital dishware. The Sinjensen Corporation (yes, I took the name from a scene in A Fish Called Wanda) is positioning itself to capitalize on the “emerging market of digital dishware products” with its Sinjensen5000 Digital Dishwasher. I played the role of a salesman demonstrating this amazing device, a prototype of their machine which would be available to consumers “very very soon.”

The Audience: Art Hop Burlington

Every year in Burlington, VT there’s a party called the Art Hop. Burlington businesses and art studios south of Main street, along the Pine Street corridor, open their doors to five or six hours of constant foot traffic. People wander through looking at art, buying some stuff, eating and drinking. It starts around 5pm and goes until 11pm or later on the main night, Friday. The Thoughtfaucet studio is located near the epicenter of the party, well positioned to get a lot of people. In addition to the Sinjensen5000, we hosted oil painter Gillian Klein, block printer (and ethnobotony mobile-technologist) Allaire Diamond and several other artists.

The audience wasn’t selected based on any sort of affiliation or interest in technology, augmented reality or anything computer related. They weren’t at a technology conference. These were people out looking at art or enjoying a great party. All ages were represented including kids, college students, middle aged and elderly folks. Probably the majority of people were in the 35-50 age band. But everyone was represented. The audience was sufficiently engaged with life to spend an evening walking around looking at art.

The Hardware and Software Behind the Sinjensen5000

The physical setup was straightforward:

• an old dishwasher that was left by a previous tenant
• a Macbook Pro (which has a built in webcam).

For the augmented reality software, I used a fairly out-of-the-box implementation of FLARTools (based on the fantastic ARToolkit) and FLARManager, using the Blender->Papervision/FLARManager workflow. The setup would display a coffee mug (modified from the Blender open source repository) when an AR glyph was shown to the webcam.

I made the glyph quickly in Adobe Illustrator and had it reproduced on cardstock at Kinkos (thanks Binney!).

The complete rig was a “magic mirror” model of augmented reality. This means that a viewer was looking at a mirror projection of themselves that included the augmented reality content. This model of AR is common as a web distribution and has the advantage of showing AR content in the same frame as the viewer themselves. I suspect this helps with the wonder/believability of the project but that would take another study to determine.

The performance: Sinjensen Corp Director of Product Marketing

I wore a suit and spoke like a fast talking salesman for most of the evening, taking on the role of the salesman for the Sinjensen Corporation demoing the Sinjensen5000. During the evening, a former sales trainer for Xerox and Ricoh commended me on the authenticity and effectiveness of my role playing.

A typical demo would have five or so audience members that had wandered in (though sometimes it was just one and several times the audience was upwards of 15 people). It would progress like so:

1. Get permission to give them a demonstration of the Sinjensen5000 Digital Dishwasher
2. Remove a glyph from the dishwasher itself
3. Demonstrate the technology while giving marketing-speak spiel
4. Hand the glyph over to an audience member
5. Observe their reactions more closely while continuing to give marketing-speak spiel

Step One: Get their permission

I would begin by asking if they’d like a demonstration of the Sinjensen5000 Digital Dishwasher. I think the entire night only three people (who were obviously painfully shy) declined. I asked anyone who made even the slightest eye contact with me.

Step Two: Introduce the glyph

Next I would pull one of the glyphs from inside the dishwasher. This would often bring two or three more people over to watch.

Step Three: Demonstrate the technology

I would place the glyph within the webcam’s vision and (most of the time) a coffee mug would appear in the augmented reality display. I would move the glyph (and the mug it projected) to show that the mug was free from “spam, worms or any other security threat that you wouldn’t want in your digital dishware–especially if you are a heavy user of social media sites such as Facebook, Twitter or LinkedIn.”

This allowed the audience to see that actions on the screen were being manipulated by movement of the glyph.

Step Four: Get the technology into the hands of the audience

I would quickly get a member of the audience to hold the glyph and interact with the augmented reality technology themselves. While they were experiencing the system and trying to figure it out, I was keeping up a steady stream of marketing speak (benefits of digital dishware vs old analog dishware, features of the digital dishware such as xbox360, Wii and Playstation compatibility etc).

Step Five: Observe

After I’d handed off the glyph I was free to put my mouth on autopilot and observe how people used the technology, what aspects of it piqued their interest and so on. While I was able to observe the audience during all of the preceding steps, with my hands free I could easily watch the audience very closely.

PS. The dishwasher is for sale dirt cheap if anyone is in the market. Works fine. Sold. 😉