a Computer-Based Instructional Environment for Autism
Dorothy Strickland, Susan
Osborne, Chan Evans
1. Phase I Specific Aims and Overview of Results
The Phase I research was intended
to determine if virtual reality is useful as a learning tool for children
with autism. While a previous pilot study by the PI had indicated initial
success in acceptance of the technology by individuals with autism (Strickland,
Marcus, Mesibov, & Hogan, 1996), there had been no prior controlled
study documenting the effectiveness of virtual reality as a teaching tool
for children with autism or the generalization of skills acquired as a
result of VR instruction to real world settings in this population.
There were three major aims in Phase I:
(a) build a prototype of a low cost, commercially feasible virtual reality
system that is usable in school settings with children with autism, (b)
demonstrate the system's instructional potential for children with autism
by establishing its efficacy in teaching these children to comprehend
the spoken names of common objects in their environment, and (c) establish
that skills learned in a virtual reality environment can generalize to
the real world.
Phase I period was from March 24, 1997
to August 15, 1997. Key personnel included Dr. Dorothy Strickland, PI,
Dr. Susan Osborne, Education Specialist, Dr. Ann Schulte, Psychologist,
Dr. Lee Marcus, Psychologist, Autism expert, Karoline Matthewson, Teaching
All aims of Phase I grant were met. A
low cost VR system (Aim One) was built from (a) a low-end, off-the-shelf
PC, (b) a commercial game headset, (c) free VR world toolkits, and (d)
instructional object identification software based on standard programming
languages and user interface packages. The efficacy of the VR system as
an instructional tool for children with autism was tested through a series
of single subject design experiments using a multiple baseline, multiple
probe design. The results of these studies showed that children with autism
rapidly accept and use a VR headset, can acquire new skills in the headset,
and these skills generalize to the real world (Aims Two and Three).
2. Virtual Reality System Development
(a) System Design. After evaluating the
strengths and limitations of existing VR components and the capabilities
of children with autism, a VR-system design was chosen that emphasized
the ability of a VR headset to limit and focus visual attention with the
simplest user interface.
Display Views .
Two display screens were needed for the tests of the VR system's instructional
potential. One displayed an object full screen size in the center of the
screen, and one involved a display of four different objects in different
corners of the screen. A black background was used in both types of displays.
The headset chosen had a horizontal field of view of 60 degrees per eye.
Since a person has a normally wider field of view, the user is visually
aware of the dark area within the headset that is on both sides of the
two screens. By displaying all objects against a black background that
matched the side color, there was the illusion of blackness in the full
range of vision with distinct objects isolated from each other and their
Safety features in the VR test design included
placing objects at visual distances of over 40 cm to reduce the effects
of headset accommodation/convergence mismatch (Mon-Williams, Wann, &
Rushton, 1993) and adjusting the headset to the children's smaller inter-ocular
distance. The further object placement distance also avoided the need
for stereo vision, which is a problem for many individuals, and increased
display speed. All virtual reality instructional sessions were limited
to 10 minutes to reduce the physical discomfort and eyestrain resulting
from prolonged headset use. These steps followed the Risks and Protections
section of the Phase I grant proposal.
Generating Objects .
The three dimensional objects originally created with the currently available
VR software toolkits were significantly more primitive looking than the
corresponding real world objects, even on the flat screen display. Because
of low screen resolution, restricted field of views, and generally poor
optics, the same objects displayed in a low cost headset appeared even
more primitive and cartoonish. For children with autism, who are cognizant
of small variations in their environment, it was felt that this would
have introduced an uncontrolled variable into the real world versus headset
match. In addition, generating the large number of three dimensional objects
to represent the real world objects would have required a significant
part of the total funding and time for Phase I. For these reasons, the
same object pictures used as the pre-test recognition measurements described
below were scanned in and used for the objects displayed in the headset.
If other views of an object were needed, scanned images of the other views
were overlaid on the screen rather than generating and manipulating true
3-D computer objects. Since unrestricted manipulation by the children
of objects was not part of the object identification test scenario, image
scanning provided a closer visual match between the objects in the headset
screen and the corresponding real world objects.
A mouse was chosen to use as object identification pointer for the children.
All the children in the test were familiar with using a mouse for selection
on a flat computer screen. Initial pre-trial testing indicated that the
children had no difficulty using the mouse to click on an object displayed
in the headset if the mouse position was displayed on the headset screens
with a white arrow similar to a normal mouse marker they had previously
Initially the design also included using
the built in head tracker in the headset to allow a child to select an
object by looking at it. However, there is always a noticeable latency
between when a user moves in a virtual setting and when this movement
is displayed in the screen using head trackers. This latency is aggravated
by fast, jerky head motion, which was exhibited some of the children in
our study who were hyperactive. Because all children could use the mouse
and the tracker introduced uncontrollable variables with no discernible
benefits, it was dropped from the user interface. The objects remained
displayed in front of the children, no matter how they moved their heads.
By not regenerating images based on head motion, the virtual environment
motion problem of temporal aliasing was also eliminated. Temporal aliasing
occurs when a smooth head motion appears as discreet object location jumps
because of screen update rates and resolutions. Steady objects against
a black background had the desired benefit of simplifying scenes and forcing
focus on the learning scenario.
Eight VR scenes were used as reinforcers. These included characters the
children knew such as Barney rotating in a circle, a walking bunny beating
a drum, a rotating carousel, Gumby stretching, Pokey enlarging and shrinking,
and visually stimulating images such as nesting circles spinning outwards.
After the child made a certain number of correct responses (determined
individually for each child at each session), a reinforcer screen was
Teacher Interface. A menu driven program allowed the tester to define
the known and unknown objects for each child and conduct the individualized
tests through pull down menu choices. The program generated a random placement
of objects on the screen to avoid any display pattern, which the children
might detect. A history of the object placements and child choice times
were saved in a file and displayable from a menu. (Appendix D)
(b) Hardware and Software . A 266 MHz PC
with 64 Megs of memory and 512K cache from Dell Computer Corporation was
used as the base of the VR system. This standard machine came with a Matrox
Millenium PCI video board with 4MB memory. Virtuality's 2000E Series Visette
Pro was chosen as the headset for the tests. Virtuality Inc., based in
England, was the oldest virtual reality commercial gaming company at the
time of the tests and its standard headset interface worked with both
the hardware and VR software without modification. A Logitech mouse with
one active button was used for object choice.
The operating system was Microsoft Windows
95, which came installed on the computer. Several VR software toolkits
were evaluated for object generation, including VREAM, SuperScape, Sense8,
MultiGen, Divisor, and Alice. Alice was chosen because it met the minimum
VR world control we needed with the scanned image design, it had been
interfaced to the chosen headset, and it was free. Alice was developed
at a university, which was kind enough to help with initial problems and
ship us the unreleased Beta copy necessary for headset display. Alice
allows creation of three dimensional environments and movement within
these environments. The free language of Python was necessary to program
the Alice commands. Visual Basic was used for the menu programming. (Appendix
3. Experimental Procedures
(a) Participants. A sample of ten children
between the ages of six and nine who attended public school classes for
children with autism were pretested. All but one child had unequivocal
diagnosis of autism based on early history, behavioral observations, test
results, and parent and teacher reports. The initial sample was culturally
diverse, as seen in Table 1 below. The mean age for the sample was 7 years
7 months (range equaled 5 years 10 months to 10 years 2 months). Adaptive
behavior (communication, daily living skills, socialization, motor skills
and maladaptive behavior) was assessed using the Vineland Adaptive Behavior
Scales (VABS) (Sparrow, Balla, & Cicchetti, 1984). The VABS is a standardized
instrument with a mean of 100 and a standard deviation of 15. The mean
score for this sample was 57, with a range from 45 to 68. That is, the
average level of daily living skills for the sample was below the first
percentile relative to children their age. Levels of autism for children
in this sample were assessed using The Childhood Autism Rating Scale (Schopler,
Reichler, & Renner, 1988). On this measure, scores range from 15 for
a normally functioning child to 60 for a child with severe autism. The
mean of this sample was 37.5 (severe) and scores ranged from 30 (very
mild) to 56.5 (severe).
Prior to training, the African American
girl, who had not yet received a formal diagnosis of autism, correctly
labeled all objects in the pretest and was dropped from the sample. One
Caucasian male who transferred to another school between the pretesting
and the actual study was also dropped from the sample. One Hispanic boy,
whose levels of activity and distraction prohibited him from completing
the pretest and mastering operation of the mouse in the virtual environment,
was also dropped from the sample. Although he appeared interested in the
helmet and would wear it for brief periods, he removed it repeatedly,
apparently looking for the same scene in the real world.
Another child exited the study after the
first three sessions. Although this Caucasian boy accepted the helmet
easily, he had very limited skills with the computer mouse and could not
use it accurately while in the VR helmet. His mother believed, and the
experimenters concurred, that this particular task did not meet the child's
educational needs at that time. In both cases where children were dropped
from the sample because of difficulty mastering the mouse and complying
with instruction, the children's performance had improved over sessions,
but it was clear they would not be able to complete the experimental trials
within the limited time frame the testing room was available to the investigators.
Table 1. Sample Characteristics
Initial Sample Final Sample
The final sample included six children who completed training and follow-up
sessions. The preponderance of male participants is consistent with research
indicating an approximate ratio of males to females of 4:1 diagnosed with
autism (Turnbull, Turnbull, Shank, & Leal, 1995).
(b) Pretest Procedures and Materials.
Objects . During meetings to inform parents and teachers about the
study and obtain parental consent for study participation, parents and
teachers were asked to list common household objects that were likely
to be known and not known by the study participants. That initial list
included kitchen tools used by students in their cooking activities at
school, but that they were unlikely to be able to identify by name (such
as a grater, saucepan, lid, dustpan, eggbeater, and spatula), and common
objects (such as a shoe and banana) that were likely to be known to children.
These objects were photographed against
a black background. Prints of the objects were then assembled into a book
such that four photographs of objects, arranged in a 2 by 2 grid, appeared
on each page.
Prior to beginning the study, the experimenters
used this booklet to pretest study participants regarding their knowledge
of object names. Each child was tested individually on two separate occasions.
On each occasion, the children were seated at a desk, shown each page
of the book of photographs and asked to "point to the ...."
. Children were asked to identify only one object in each array of four
objects on a page. Correct and incorrect responses were identified on
a record sheet.
Three or more objects that had been unknown
during the pretest were then selected as test objects for each child.
In addition, four objects that had been known to all children (e.g., banana,
shoe) were selected as VR stimuli. These "known objects" were
added as VR stimuli because during the pretesting with photographs, several
children became upset when they did not know items on several consecutive
trials. Their distress was quickly allayed if the experimenters alternated
trials of known and unknown objects. Including known objects as VR stimuli
also provided opportunities to expose participants to the reinforcement
screens presented in virtual reality, contingent on correct responses.
Teaching Participants to Use the Helmet .
For most children, the pretesting to identify known and unknown objects
took place in their classroom, prior to the close of school for the summer.
However, the pretest training with the helmet took place just prior to
the experimental trials, which were conducted during the summer.
Both pretest training and the actual experimental
sessions took place in a local public school in a setting that was designed
to mimic the physical layout and routines which the children used in their
classrooms. The layout included two play centers, set up with activities
the children enjoyed, and a work center, consisting of the computer that
operated the instructional program, a monitor, and the helmet, all arranged
on a child sized desk. The instructional routine included rotation through
the three centers several times during a visit to the training room, a
pictorial schedule posted for each child showing their sequence of activities
for their visit to the VR training room, and a hand bell, which was rung
each time the child was to rotate to a new activity. To assure that children
were not frightened, a school staff member who knew the child was employed
by the project for the summer and she was the primary instructor during
the pretraining and experimental sessions.
Only one child, who had taken part in an
earlier pilot study, had previously used any VR equipment. Each participant
was introduced individually to the virtual reality helmet during a single
30 to 45 minute session. He or she was brought into the training room
by a parent or a familiar teacher and shown the work and play centers
and his or her schedule.
After playing at one center until he or
she appeared comfortable with the new surroundings, the child was called
over to the VR work center. In introducing the helmet, a teacher, parent,
or sibling placed the helmet on his or her own head while the target child
watched. The participating child was then invited to try on the helmet
and "see the merry-go-round" or other picture displayed in the
helmet. Using this procedure, all participants tried on the helmet without
hesitation. During this acclimatization phase, the experimenter removed
the helmet after a minute or two while students were still showing great
interest in it. Throughout the study, sessions in the helmet were limited
to no more than 10 continuous minutes and no child showed evidence of
side effects common to longer headset exposures such as dizziness, eyestrain,
or discomfort (although several sessions might take place during one visit
to the training room as the child rotated through the centers). Test design
safety issues were described above. After two sessions, most participants
were able to put on the helmet and adjust it themselves, the remainder
needed assistance from the experimenters.
to Use the Mouse in the Virtual World . Each child was seated at
the computer and put on the helmet. Displayed in the helmet was one known
object placed in the middle of the visual field. A school staff member
assisting with the trials placed the child's hand on the mouse and used
a hand-over-hand technique to move the pointer around the screen and to
"click" on the object. After hearing the request "Click
on the ...", the child had the opportunity to respond. If the child
did not respond or attempt to respond by moving the mouse within a few
seconds, the teacher provided physical guidance and then praise: "Good
job! You clicked on the...", and a reward screen was shown (e.g.,
Barney). No participating child required more than 10 minutes of practice
using physical prompts to respond, although children varied in their proficiency
with the mouse.
(c) Experimental Design. The experimental
design most appropriate for the evaluation of interventions involving
small numbers of subjects and subjects who represent a diverse population
is a variation of the multiple baseline design called a multiple probe
design (Horner & Baer, 1978). For this project, the design could be
termed a "multiple baseline across objects design" because,
for each child, the focus of data collection was changes in the child's
ability to identify objects based on their spoken name. For each participant,
VR instruction was introduced sequentially on three of the unknown objects
from the pretest. Baseline data were collected prior to any instruction
and again each time a new object had been taught (hence the term "multiple
baseline"). For experimental control to be demonstrated it is expected
that children's performance will only improve on the baseline measures
for an object if instruction has been completed on that item. "Multiple
probe" refers to a type of design where baseline data (or "probes")
are collected several times per condition, but not for each session during
the baseline phase for each item. A multiple probe design is appropriate
when subjects of the treatment are unable to perform the task at all (Alberto
& Troutman, 1994) and allows one to avoid requiring subjects to repeatedly
attempt tasks for which they have not yet received instruction.
In this case, six separate experiments
were conducted in which each participating child served as his or her
own control while learning three new objects. In this design, experimental
control over the dependent variable is demonstrated when consistent changes
in behavior occur only following intervention. The lagged intervention
across objects allows for the procedures to be replicated across teaching
objectives while controlling for maturation, history, and other possible
confounding variables. The series of six experiments allows for replication
over a small number of diverse individuals.
Data (percent of correct responses) were
graphed daily and data were compared across phases of the experiment to
evaluate the effect of the intervention. Percent of overlapping data points
before and during intervention provides evidence of quantitatively different
behavior before and after intervention and is replicated across objects
to be learned in the multiple probe design.
Baseline Measurements .
Baseline measures were collected in both the real world and in the virtual
world on objects known and not known to children on the pretest.
1. In the real world baseline probes, the experimenter asked children
individually to identify the four known objects and three target items
(to be learned) from an array of ten items. For each probe, this procedure
was repeated two to three times.
2. In the VR baseline, the experimenter asked children to identify known
and target items from arrays of four randomly placed objects displayed
in the headset. For each probe, this procedure was repeated two to three
3. In both baseline conditions, the experimenter alternated requests
for known and target items to avoid frustrating the children with long
strings of requests to which they could not comply.
4. Neither positive nor negative feedback was given during baseline trials.
Description of VR
Training Screens. Target objects appeared one per screen, with
three distracter items. In each trial, six screens each showing four objects
were presented to the child. For half of the screens, the child was asked
to click on known objects, while for the other screens, the child was
asked to click on the target object. The objects were randomly placed
on each screen so that children could not use position cues to guide their
responses, and trials with known and target objects were randomly sequenced
so children could not identify the target object based on the order of
Distracter items also changed for each
screen in the trial, however, guessing produced a 25% chance of a correct
response and some students were able to eliminate objects they could identify
from the unknowns. In order to allow for some lucky guesses, the criterion
for unknown objects in the virtual environments was set at no more than
50%. In several instances, children consistently selected objects in the
virtual environment that they had not identified in photographs or from
the real objects themselves. Although it is unclear whether children really
"knew" these objects or whether they merely perseverated on
an unknown choice, these objects were discarded and VR probes continued
until there were three objects not identified by the child in either environment.
Intervention. Target objects were taught using the multiple-baseline- across-objects
design. The initial object was presented alone in the center of the visual
field in the headset. The experimenter instructed the child to "click
on the ..." and guided the child's hand if necessary. As the child
clicked on the object, the experimenter said "Yes, that's the ...
. Good job." When the child was able to click on the single object
independently for four successive trials, the object appeared on the screen
in a 2 x 2 matrix with three distracter items. The position of the target
item and the specific distracter items changed randomly with each screen.
Screens containing the target were interspersed with screens containing
known items to ensure that there would be ample opportunities for successful
responses throughout the training session and to reduce boredom from responding
consistently to the same target object.
A teaching session terminated when the
student responded correctly to six consecutive requests for the target
item spread across twelve trials. If the child did not meet criterion,
10-minute trials continued (interspersed with breaks and play periods)
until the child had responded correctly to the target item six consecutive
When the training criterion had been met,
the experimenters presented the VR baseline screens, and if the child
correctly identified the recently taught target item, tested near generalization
with a transfer probe consisting of four real objects (including the target)
displayed on a small table covered with a black cloth. (Near generalization
refers to a task that is very similar, but not identical to the tasks
used for training.) If the child was again successful in selecting the
target item, the experimenter conducted "word game," the real
world baseline and generalization probe in which the child was asked to
get the target item from an array of ten real objects. Each child's three
target items and four known items were tested during each of two trials
of the word game. If at any point, the child did not meet criterion for
the target item being taught, he or she continued training sessions until
the student again met the teaching phase criterion.
Results indicated that children with autism
(a) accepted the helmet readily, (b) were able to learn in the virtual
world, and (c) were able to transfer information learned in the virtual
world to the real world, although results of generalization probes were
mixed. All students, including the two whose limited motor skills or level
of agitation and activity precluded participation in this study, accepted
the helmet and appeared to enjoy using it. Even the younger, and also
autistic, sibling of one participant looked forward daily to trying the
helmet when he and his mother came to pick up his older brother. After
several sessions, all participating children were able to put on the helmet
independently and adjust the size. All were able to remove it when the
reinforcement figure appeared on the screen to cue that the session was
over. The only resistance to the helmet occurred when one child caught
her hair in the headset.
Although results varied by child, all
participants demonstrated that they learned from activities presented
in the helmet. Space limitations preclude presenting graphs for all participants
however, graphs for two children are presented in Figure 1. Interpretation
of the graph for one child, Taylor, is presented below. (All child names
Three household objects unknown to Taylor
on the pretest were selected for training. Following a session to teach
headset and mouse use, baseline sessions in the real world and in the
VR world (real world probes and virtual reality probes) established that
Taylor could not reliably identify the objects pail, peeler, or mug. Training
in the headset began for the object pail as described above. Note that
Taylor immediately learned to identify pail from among four choices in
the VR helmet and from among four real objects (transfer probe), but had
greater difficulty selecting the real object from among 10 real objects
(real world probe). Each failure to demonstrate at least 75% accuracy
on the real world probe was followed by another training session (depicted
As soon as Taylor met criterion for identifying
a pail in a real world probe, training on the peeler began. Again Taylor
immediately learned peeler in the VR world but had difficulty making the
transfer to the real object presented with three others. Three training
sessions were required before this child met the criterion of 75% correct
on the real world probe. At that time, training began on the last object,
mug. Taylor met criterion immediately in VR but achieved only 65% accuracy
with real objects. Because Taylor reached criterion for two of the three
objects, training halted and Taylor returned one week later for follow-up.
At follow-up, Taylor did not reach criterion for any object in the real
world probe but did for two of three objects in VR. Table 2 summarizes
data for all children who participated in training in the VR study.
Table 2. Results by Training Objectives for Children Who Engaged in
|Accurately used mouse in helmet
|Learned new objects in VR*
|Generalized new objects in real world**
|Maintained new knowledge after 1 week ***
Y = child met criteria for success N = child did not meet criteria for
* increased from 25% to at least 75% accuracy for 2 or more objects
** achieved at least 75% accuracy on 2 or more objects
*** achieved at least 75% accuracy on at least 2 objects at follow-up
From studying the profiles of the two
children who did not benefit from participation, it is apparent that VR,
at least in this application, may have limited utility with young children
with severe attention deficits and hyperactivity. In addition, children's
ability to use the mouse or other hand input device while in the headset
is necessary for some applications. Whether children without some facility
with mouse or joystick could benefit from using a tracker in the headset
has yet to be determined.
One limitation of the Phase I study is
that only identifying household objects by name was taught. Teaching function
of the objects, in the virtual or real world, at the same time as teaching
the names could enhance maintenance and generalization of new information.
This limitation will be addressed in Phase II where film clips demonstrating
object use will be incorporated in the VR training modules.
5. Phase I Summary
All aims of Phase I were successfully
achieved. These included (a) building an inexpensive VR learning system
usable by children with autism, (b) demonstrating with multiple baseline
across object control tests that VR instructional techniques can be effective
in teaching children with autism, and (c) demonstrating generalization
occurred between virtual and real environments for children with autism,
although the generalization results were mixed. Phase I established the
instructional potential for VR in helping children with autism. Phase
II will incorporate information gained from Phase I development and trials
in developing critical applications that will benefit from the unique
features of this advanced technology.