Augmented Reality Can Help Children With Autism Tap Into Their Imaginations

One researcher’s system helps them improve their play skills

1 April 2015

Playing pretend as a child—whether using a paintbrush as a wand or imagining a large cardboard box to be a castle—is more than just for fun. It is also an essential developmental activity that teaches children social and emotional skills and builds their self-esteem. However, most children with autism—a neurodevelopmental condition that affects the ability to communicate and interact with others—are less engaged in imaginative play. And this can have a profound impact on them into their adult lives.

That’s why Ph.D. candidate Zhen Bai designed an augmented reality (AR) system that she hopes will nudge such children toward more imaginative play. Her system lets children see themselves on a computer screen as they would in a mirror. She then gives the children simple physical objects—foam blocks, for example—that appear on the screen as a car, train, or airplane. The system’s computer-vision program detects where and how the child moves the block and mimics the activity in the image on the screen.

Bai is a student at the Graphics & Interaction Group at Cambridge’s Computer Laboratory. Her research paper, “Using Augmented Reality to Elicit Pretend Play for Children With Autism,” was published in December in IEEE Transactions on Visualization and Computer Graphics. It is available in the IEEE Xplore Digital Library. Her paper describes her observational experiment, which confirmed that the AR system could engage children with autism in imaginary activities. And from what she learned, Bai is developing the system to help children move to more imaginative play without the assistance of the technology.


The AR system relies on a Web camera with a wide field of view, a computer and a 60-centimeter monitor, and those simple foam blocks. The blocks are marked with black and white patterns resembling the Quick Response bar codes put on tags placed on merchandise and marketing materials, for example, for customers to scan with their smartphones and get more information about the product. The computer detects the marks on the blocks in each frame captured by the Web camera, and calculates their position and orientation. The program then superimposes a 3-D model of the toy. The children then see themselves playing with the car on the computer screen and not the foam block, augmenting the child’s real-life activity. Bai designed her system using the open-source Goblin XNA library.

Bai notes that children with autism work especially well with computers. Most of the children in her study use them daily. Twelve participants, ages 4 to 7, began the experiment but two dropped out because of the severity of their condition. Bai had the children under observation for 10 minutes each. It took nine months to recruit the participants.

“The experiment with a small number of participants addressed our core research question, which is: To what extent can AR enhance pretend play for children with autism,” Bai says. “It also allowed us to examine other considerations, such as how the children would react to the mirror display. These findings are informing the next stage of our design.”

 The children were recruited from the Cambridge Autism Research Center, the Cambridge branch of the National Autistic Society in England, and local autism outreach events. The researchers used the Childhood Autism Rating Scale to determine the severity of each participant’s condition. They also evaluated their verbal skills using the British Picture Vocabulary Scale, which assesses how well children understand spoken English.

To begin, the researchers gave each of the children five minutes to get comfortable seeing themselves on the screen and play with the foam objects in front of them. They were then asked to play in two conditions: one using the AR system and another comparable play setting without the AR system.

For each play scenario, whether a car, train, or airplane, Bai prompted the children with, for example, “Show me how to play with the block as an airplane.” She wanted to inspire pretend play without giving the children specific instructions. Each session with the AR system lasted five minutes.

Then it was up to the parents to describe how the AR setting affected their children. Bai points out it is more reliable to have parents, who have seen their children at play before, rate their child’s engagement with play rather than having the researcher do it. She asked the parents to rate their activities on factors such as cooperativeness and attentiveness, and how often they smiled, and rating both AR and ordinary settings. 

“There’s always the potential for bias, meaning parents might think their children performed better than they actually did,” Bai points out. This is why Bai gave the parents rating factors to consider. They also conducted follow-up interviews with the parents to discuss the ratings.

Eight out of 10 parents reported that their children were more engaged in the AR setting. Some said their children performed activities they had not done previously at home. Bai points out that with the AR program the children engaged in imaginative play and keep make-believe games going longer than they could on their own without the benefit of the augmentation. The AR system concurred. A video analysis showed the children engaged in over 50 percent more pretend-play scenarios per minute with the AR compared to playing without it.

The researchers also surveyed participants using the Fun Toolkit, a popular survey method for measuring the engagement of young children in play. Asked which environment was more fun, 9 out of the 10 chose the AR system and indicated they would prefer to use the system when playing with friends. Among reasons given: “Can see things that is not actually there” and “It’s funny.” Virtual play items that the children said they wanted included dinosaurs, superheroes, and volcanoes. 


One concern in applying the system, Bai notes, is that the children might begin to rely too much on it and not engage in pretend play without it. She plans to gradually fade out the visual effects to see if the system can motivate the children to continue to use their imaginations in the real world without AR. She adds that it might be helpful to have the children choose the images that appear on the screen so that they can more often come up with play scenarios of their own.

Although her system is in its infancy, Bai sees it as perhaps one day being a tool for child development, especially in remote areas with few options for treating children with autism.

IEEE membership offers a wide range of benefits and opportunities for those who share a common interest in technology. If you are not already a member, consider joining IEEE and becoming part of a worldwide network of more than 400,000 students and professionals.

Learn More