Evaluating a Computer-Based Instructional Environment for Autism

1. Phase I Specific Aims and Overview of Results
The Phase I research was intended to determine if virtual reality is useful as a learning tool for children with autism. While a previous pilot study by the PI had indicated initial success in acceptance of the technology by individuals with autism (Strickland, Marcus, Mesibov, & Hogan, 1996), there had been no prior controlled study documenting the effectiveness of virtual reality as a teaching tool for children with autism or the generalization of skills acquired as a result of VR instruction to real world settings in this population.

There were three major aims in Phase I: (a) build a prototype of a low cost, commercially feasible virtual reality system that is usable in school settings with children with autism, (b) demonstrate the system's instructional potential for children with autism by establishing its efficacy in teaching these children to comprehend the spoken names of common objects in their environment, and (c) establish that skills learned in a virtual reality environment can generalize to the real world.

Phase I period was from March 24, 1997 to August 15, 1997. Key personnel included Dr. Dorothy Strickland, PI, Dr. Susan Osborne, Education Specialist, Dr. Ann Schulte, Psychologist, Dr. Lee Marcus, Psychologist, Autism expert, Karoline Matthewson, Teaching assistant

All aims of Phase I grant were met. A low cost VR system (Aim One) was built from (a) a low-end, off-the-shelf PC, (b) a commercial game headset, (c) free VR world toolkits, and (d) instructional object identification software based on standard programming languages and user interface packages. The efficacy of the VR system as an instructional tool for children with autism was tested through a series of single subject design experiments using a multiple baseline, multiple probe design. The results of these studies showed that children with autism rapidly accept and use a VR headset, can acquire new skills in the headset, and these skills generalize to the real world (Aims Two and Three).

2. Virtual Reality System Development

(a) System Design. After evaluating the strengths and limitations of existing VR components and the capabilities of children with autism, a VR-system design was chosen that emphasized the ability of a VR headset to limit and focus visual attention with the simplest user interface.

Display Views . Two display screens were needed for the tests of the VR system's instructional potential. One displayed an object full screen size in the center of the screen, and one involved a display of four different objects in different corners of the screen. A black background was used in both types of displays. The headset chosen had a horizontal field of view of 60 degrees per eye. Since a person has a normally wider field of view, the user is visually aware of the dark area within the headset that is on both sides of the two screens. By displaying all objects against a black background that matched the side color, there was the illusion of blackness in the full range of vision with distinct objects isolated from each other and their surroundings.

Safety features in the VR test design included placing objects at visual distances of over 40 cm to reduce the effects of headset accommodation/convergence mismatch (Mon-Williams, Wann, & Rushton, 1993) and adjusting the headset to the children's smaller inter-ocular distance. The further object placement distance also avoided the need for stereo vision, which is a problem for many individuals, and increased display speed. All virtual reality instructional sessions were limited to 10 minutes to reduce the physical discomfort and eyestrain resulting from prolonged headset use. These steps followed the Risks and Protections section of the Phase I grant proposal.

Generating Objects . The three dimensional objects originally created with the currently available VR software toolkits were significantly more primitive looking than the corresponding real world objects, even on the flat screen display. Because of low screen resolution, restricted field of views, and generally poor optics, the same objects displayed in a low cost headset appeared even more primitive and cartoonish. For children with autism, who are cognizant of small variations in their environment, it was felt that this would have introduced an uncontrolled variable into the real world versus headset match. In addition, generating the large number of three dimensional objects to represent the real world objects would have required a significant part of the total funding and time for Phase I. For these reasons, the same object pictures used as the pre-test recognition measurements described below were scanned in and used for the objects displayed in the headset. If other views of an object were needed, scanned images of the other views were overlaid on the screen rather than generating and manipulating true 3-D computer objects. Since unrestricted manipulation by the children of objects was not part of the object identification test scenario, image scanning provided a closer visual match between the objects in the headset screen and the corresponding real world objects.

User Choice. A mouse was chosen to use as object identification pointer for the children. All the children in the test were familiar with using a mouse for selection on a flat computer screen. Initial pre-trial testing indicated that the children had no difficulty using the mouse to click on an object displayed in the headset if the mouse position was displayed on the headset screens with a white arrow similar to a normal mouse marker they had previously used.

Initially the design also included using the built in head tracker in the headset to allow a child to select an object by looking at it. However, there is always a noticeable latency between when a user moves in a virtual setting and when this movement is displayed in the screen using head trackers. This latency is aggravated by fast, jerky head motion, which was exhibited some of the children in our study who were hyperactive. Because all children could use the mouse and the tracker introduced uncontrollable variables with no discernible benefits, it was dropped from the user interface. The objects remained displayed in front of the children, no matter how they moved their heads. By not regenerating images based on head motion, the virtual environment motion problem of temporal aliasing was also eliminated. Temporal aliasing occurs when a smooth head motion appears as discreet object location jumps because of screen update rates and resolutions. Steady objects against a black background had the desired benefit of simplifying scenes and forcing focus on the learning scenario.

Reinforcement Screens. Eight VR scenes were used as reinforcers. These included characters the children knew such as Barney rotating in a circle, a walking bunny beating a drum, a rotating carousel, Gumby stretching, Pokey enlarging and shrinking, and visually stimulating images such as nesting circles spinning outwards. After the child made a certain number of correct responses (determined individually for each child at each session), a reinforcer screen was shown.

Teacher Interface. A menu driven program allowed the tester to define the known and unknown objects for each child and conduct the individualized tests through pull down menu choices. The program generated a random placement of objects on the screen to avoid any display pattern, which the children might detect. A history of the object placements and child choice times were saved in a file and displayable from a menu. (Appendix D)

(b) Hardware and Software . A 266 MHz PC with 64 Megs of memory and 512K cache from Dell Computer Corporation was used as the base of the VR system. This standard machine came with a Matrox Millenium PCI video board with 4MB memory. Virtuality's 2000E Series Visette Pro was chosen as the headset for the tests. Virtuality Inc., based in England, was the oldest virtual reality commercial gaming company at the time of the tests and its standard headset interface worked with both the hardware and VR software without modification. A Logitech mouse with one active button was used for object choice.

The operating system was Microsoft Windows 95, which came installed on the computer. Several VR software toolkits were evaluated for object generation, including VREAM, SuperScape, Sense8, MultiGen, Divisor, and Alice. Alice was chosen because it met the minimum VR world control we needed with the scanned image design, it had been interfaced to the chosen headset, and it was free. Alice was developed at a university, which was kind enough to help with initial problems and ship us the unreleased Beta copy necessary for headset display. Alice allows creation of three dimensional environments and movement within these environments. The free language of Python was necessary to program the Alice commands. Visual Basic was used for the menu programming. (Appendix E)

3. Experimental Procedures

(a) Participants. A sample of ten children between the ages of six and nine who attended public school classes for children with autism were pretested. All but one child had unequivocal diagnosis of autism based on early history, behavioral observations, test results, and parent and teacher reports. The initial sample was culturally diverse, as seen in Table 1 below. The mean age for the sample was 7 years 7 months (range equaled 5 years 10 months to 10 years 2 months). Adaptive behavior (communication, daily living skills, socialization, motor skills and maladaptive behavior) was assessed using the Vineland Adaptive Behavior Scales (VABS) (Sparrow, Balla, & Cicchetti, 1984). The VABS is a standardized instrument with a mean of 100 and a standard deviation of 15. The mean score for this sample was 57, with a range from 45 to 68. That is, the average level of daily living skills for the sample was below the first percentile relative to children their age. Levels of autism for children in this sample were assessed using The Childhood Autism Rating Scale (Schopler, Reichler, & Renner, 1988). On this measure, scores range from 15 for a normally functioning child to 60 for a child with severe autism. The mean of this sample was 37.5 (severe) and scores ranged from 30 (very mild) to 56.5 (severe).

Prior to training, the African American girl, who had not yet received a formal diagnosis of autism, correctly labeled all objects in the pretest and was dropped from the sample. One Caucasian male who transferred to another school between the pretesting and the actual study was also dropped from the sample. One Hispanic boy, whose levels of activity and distraction prohibited him from completing the pretest and mastering operation of the mouse in the virtual environment, was also dropped from the sample. Although he appeared interested in the helmet and would wear it for brief periods, he removed it repeatedly, apparently looking for the same scene in the real world.

Another child exited the study after the first three sessions. Although this Caucasian boy accepted the helmet easily, he had very limited skills with the computer mouse and could not use it accurately while in the VR helmet. His mother believed, and the experimenters concurred, that this particular task did not meet the child's educational needs at that time. In both cases where children were dropped from the sample because of difficulty mastering the mouse and complying with instruction, the children's performance had improved over sessions, but it was clear they would not be able to complete the experimental trials within the limited time frame the testing room was available to the investigators.

Table 1. Sample Characteristics

Initial Sample Final Sample

The final sample included six children who completed training and follow-up sessions. The preponderance of male participants is consistent with research indicating an approximate ratio of males to females of 4:1 diagnosed with autism (Turnbull, Turnbull, Shank, & Leal, 1995).

(b) Pretest Procedures and Materials.

Identifying Test Objects . During meetings to inform parents and teachers about the study and obtain parental consent for study participation, parents and teachers were asked to list common household objects that were likely to be known and not known by the study participants. That initial list included kitchen tools used by students in their cooking activities at school, but that they were unlikely to be able to identify by name (such as a grater, saucepan, lid, dustpan, eggbeater, and spatula), and common objects (such as a shoe and banana) that were likely to be known to children.

These objects were photographed against a black background. Prints of the objects were then assembled into a book such that four photographs of objects, arranged in a 2 by 2 grid, appeared on each page.

Prior to beginning the study, the experimenters used this booklet to pretest study participants regarding their knowledge of object names. Each child was tested individually on two separate occasions. On each occasion, the children were seated at a desk, shown each page of the book of photographs and asked to "point to the ...." . Children were asked to identify only one object in each array of four objects on a page. Correct and incorrect responses were identified on a record sheet.

Three or more objects that had been unknown during the pretest were then selected as test objects for each child. In addition, four objects that had been known to all children (e.g., banana, shoe) were selected as VR stimuli. These "known objects" were added as VR stimuli because during the pretesting with photographs, several children became upset when they did not know items on several consecutive trials. Their distress was quickly allayed if the experimenters alternated trials of known and unknown objects. Including known objects as VR stimuli also provided opportunities to expose participants to the reinforcement screens presented in virtual reality, contingent on correct responses.

Teaching Participants to Use the Helmet . For most children, the pretesting to identify known and unknown objects took place in their classroom, prior to the close of school for the summer. However, the pretest training with the helmet took place just prior to the experimental trials, which were conducted during the summer.

Both pretest training and the actual experimental sessions took place in a local public school in a setting that was designed to mimic the physical layout and routines which the children used in their classrooms. The layout included two play centers, set up with activities the children enjoyed, and a work center, consisting of the computer that operated the instructional program, a monitor, and the helmet, all arranged on a child sized desk. The instructional routine included rotation through the three centers several times during a visit to the training room, a pictorial schedule posted for each child showing their sequence of activities for their visit to the VR training room, and a hand bell, which was rung each time the child was to rotate to a new activity. To assure that children were not frightened, a school staff member who knew the child was employed by the project for the summer and she was the primary instructor during the pretraining and experimental sessions.

Only one child, who had taken part in an earlier pilot study, had previously used any VR equipment. Each participant was introduced individually to the virtual reality helmet during a single 30 to 45 minute session. He or she was brought into the training room by a parent or a familiar teacher and shown the work and play centers and his or her schedule.

After playing at one center until he or she appeared comfortable with the new surroundings, the child was called over to the VR work center. In introducing the helmet, a teacher, parent, or sibling placed the helmet on his or her own head while the target child watched. The participating child was then invited to try on the helmet and "see the merry-go-round" or other picture displayed in the helmet. Using this procedure, all participants tried on the helmet without hesitation. During this acclimatization phase, the experimenter removed the helmet after a minute or two while students were still showing great interest in it. Throughout the study, sessions in the helmet were limited to no more than 10 continuous minutes and no child showed evidence of side effects common to longer headset exposures such as dizziness, eyestrain, or discomfort (although several sessions might take place during one visit to the training room as the child rotated through the centers). Test design safety issues were described above. After two sessions, most participants were able to put on the helmet and adjust it themselves, the remainder needed assistance from the experimenters.

Teaching Participants to Use the Mouse in the Virtual World . Each child was seated at the computer and put on the helmet. Displayed in the helmet was one known object placed in the middle of the visual field. A school staff member assisting with the trials placed the child's hand on the mouse and used a hand-over-hand technique to move the pointer around the screen and to "click" on the object. After hearing the request "Click on the ...", the child had the opportunity to respond. If the child did not respond or attempt to respond by moving the mouse within a few seconds, the teacher provided physical guidance and then praise: "Good job! You clicked on the...", and a reward screen was shown (e.g., Barney). No participating child required more than 10 minutes of practice using physical prompts to respond, although children varied in their proficiency with the mouse.

(c) Experimental Design. The experimental design most appropriate for the evaluation of interventions involving small numbers of subjects and subjects who represent a diverse population is a variation of the multiple baseline design called a multiple probe design (Horner & Baer, 1978). For this project, the design could be termed a "multiple baseline across objects design" because, for each child, the focus of data collection was changes in the child's ability to identify objects based on their spoken name. For each participant, VR instruction was introduced sequentially on three of the unknown objects from the pretest. Baseline data were collected prior to any instruction and again each time a new object had been taught (hence the term "multiple baseline"). For experimental control to be demonstrated it is expected that children's performance will only improve on the baseline measures for an object if instruction has been completed on that item. "Multiple probe" refers to a type of design where baseline data (or "probes") are collected several times per condition, but not for each session during the baseline phase for each item. A multiple probe design is appropriate when subjects of the treatment are unable to perform the task at all (Alberto & Troutman, 1994) and allows one to avoid requiring subjects to repeatedly attempt tasks for which they have not yet received instruction.

In this case, six separate experiments were conducted in which each participating child served as his or her own control while learning three new objects. In this design, experimental control over the dependent variable is demonstrated when consistent changes in behavior occur only following intervention. The lagged intervention across objects allows for the procedures to be replicated across teaching objectives while controlling for maturation, history, and other possible confounding variables. The series of six experiments allows for replication over a small number of diverse individuals.

Data (percent of correct responses) were graphed daily and data were compared across phases of the experiment to evaluate the effect of the intervention. Percent of overlapping data points before and during intervention provides evidence of quantitatively different behavior before and after intervention and is replicated across objects to be learned in the multiple probe design.

Baseline Measurements . Baseline measures were collected in both the real world and in the virtual world on objects known and not known to children on the pretest.

1. In the real world baseline probes, the experimenter asked children individually to identify the four known objects and three target items (to be learned) from an array of ten items. For each probe, this procedure was repeated two to three times.

2. In the VR baseline, the experimenter asked children to identify known and target items from arrays of four randomly placed objects displayed in the headset. For each probe, this procedure was repeated two to three times.

3. In both baseline conditions, the experimenter alternated requests for known and target items to avoid frustrating the children with long strings of requests to which they could not comply.

4. Neither positive nor negative feedback was given during baseline trials.

Description of VR Training Screens. Target objects appeared one per screen, with three distracter items. In each trial, six screens each showing four objects were presented to the child. For half of the screens, the child was asked to click on known objects, while for the other screens, the child was asked to click on the target object. The objects were randomly placed on each screen so that children could not use position cues to guide their responses, and trials with known and target objects were randomly sequenced so children could not identify the target object based on the order of screens.

Distracter items also changed for each screen in the trial, however, guessing produced a 25% chance of a correct response and some students were able to eliminate objects they could identify from the unknowns. In order to allow for some lucky guesses, the criterion for unknown objects in the virtual environments was set at no more than 50%. In several instances, children consistently selected objects in the virtual environment that they had not identified in photographs or from the real objects themselves. Although it is unclear whether children really "knew" these objects or whether they merely perseverated on an unknown choice, these objects were discarded and VR probes continued until there were three objects not identified by the child in either environment.

Intervention. Target objects were taught using the multiple-baseline- across-objects design. The initial object was presented alone in the center of the visual field in the headset. The experimenter instructed the child to "click on the ..." and guided the child's hand if necessary. As the child clicked on the object, the experimenter said "Yes, that's the ... . Good job." When the child was able to click on the single object independently for four successive trials, the object appeared on the screen in a 2 x 2 matrix with three distracter items. The position of the target item and the specific distracter items changed randomly with each screen. Screens containing the target were interspersed with screens containing known items to ensure that there would be ample opportunities for successful responses throughout the training session and to reduce boredom from responding consistently to the same target object.

A teaching session terminated when the student responded correctly to six consecutive requests for the target item spread across twelve trials. If the child did not meet criterion, 10-minute trials continued (interspersed with breaks and play periods) until the child had responded correctly to the target item six consecutive times.

When the training criterion had been met, the experimenters presented the VR baseline screens, and if the child correctly identified the recently taught target item, tested near generalization with a transfer probe consisting of four real objects (including the target) displayed on a small table covered with a black cloth. (Near generalization refers to a task that is very similar, but not identical to the tasks used for training.) If the child was again successful in selecting the target item, the experimenter conducted "word game," the real world baseline and generalization probe in which the child was asked to get the target item from an array of ten real objects. Each child's three target items and four known items were tested during each of two trials of the word game. If at any point, the child did not meet criterion for the target item being taught, he or she continued training sessions until the student again met the teaching phase criterion.

4. Results

Results indicated that children with autism (a) accepted the helmet readily, (b) were able to learn in the virtual world, and (c) were able to transfer information learned in the virtual world to the real world, although results of generalization probes were mixed. All students, including the two whose limited motor skills or level of agitation and activity precluded participation in this study, accepted the helmet and appeared to enjoy using it. Even the younger, and also autistic, sibling of one participant looked forward daily to trying the helmet when he and his mother came to pick up his older brother. After several sessions, all participating children were able to put on the helmet independently and adjust the size. All were able to remove it when the reinforcement figure appeared on the screen to cue that the session was over. The only resistance to the helmet occurred when one child caught her hair in the headset.

Although results varied by child, all participants demonstrated that they learned from activities presented in the helmet. Space limitations preclude presenting graphs for all participants however, graphs for two children are presented in Figure 1. Interpretation of the graph for one child, Taylor, is presented below. (All child names are fictitious.)

Three household objects unknown to Taylor on the pretest were selected for training. Following a session to teach headset and mouse use, baseline sessions in the real world and in the VR world (real world probes and virtual reality probes) established that Taylor could not reliably identify the objects pail, peeler, or mug. Training in the headset began for the object pail as described above. Note that Taylor immediately learned to identify pail from among four choices in the VR helmet and from among four real objects (transfer probe), but had greater difficulty selecting the real object from among 10 real objects (real world probe). Each failure to demonstrate at least 75% accuracy on the real world probe was followed by another training session (depicted by arrows).

As soon as Taylor met criterion for identifying a pail in a real world probe, training on the peeler began. Again Taylor immediately learned peeler in the VR world but had difficulty making the transfer to the real object presented with three others. Three training sessions were required before this child met the criterion of 75% correct on the real world probe. At that time, training began on the last object, mug. Taylor met criterion immediately in VR but achieved only 65% accuracy with real objects. Because Taylor reached criterion for two of the three objects, training halted and Taylor returned one week later for follow-up. At follow-up, Taylor did not reach criterion for any object in the real world probe but did for two of three objects in VR. Table 2 summarizes data for all children who participated in training in the VR study.

Table 2. Results by Training Objectives for Children Who Engaged in VR Training

Y = child met criteria for success N = child did not meet criteria for success

* increased from 25% to at least 75% accuracy for 2 or more objects
** achieved at least 75% accuracy on 2 or more objects
*** achieved at least 75% accuracy on at least 2 objects at follow-up

From studying the profiles of the two children who did not benefit from participation, it is apparent that VR, at least in this application, may have limited utility with young children with severe attention deficits and hyperactivity. In addition, children's ability to use the mouse or other hand input device while in the headset is necessary for some applications. Whether children without some facility with mouse or joystick could benefit from using a tracker in the headset has yet to be determined.

One limitation of the Phase I study is that only identifying household objects by name was taught. Teaching function of the objects, in the virtual or real world, at the same time as teaching the names could enhance maintenance and generalization of new information. This limitation will be addressed in Phase II where film clips demonstrating object use will be incorporated in the VR training modules.

5. Phase I Summary

All aims of Phase I were successfully achieved. These included (a) building an inexpensive VR learning system usable by children with autism, (b) demonstrating with multiple baseline across object control tests that VR instructional techniques can be effective in teaching children with autism, and (c) demonstrating generalization occurred between virtual and real environments for children with autism, although the generalization results were mixed. Phase I established the instructional potential for VR in helping children with autism. Phase II will incorporate information gained from Phase I development and trials in developing critical applications that will benefit from the unique features of this advanced technology.
return to top