Clinical Scenario
On the first day of the staff being back to school, Mr. Stark, the principal, informed Ms. Bell, a special education teacher, that she was gaining a new student who had recently transferred from out of state. The new student, Kristin, was a 6-year-old girl with autism spectrum disorder (ASD) who reportedly had “significant communication deficits.” At first, Ms. Bell was very nervous, as she did not have experience working with students with such significant communication needs. However, she was intrigued about how to enhance Kristin’s learning opportunities and excited for this new experience in the classroom.
Ms. Bell thoroughly reviewed Kristin’s file and discovered that the only information in Kristin’s individualized educational program (IEP) pertaining to her present level of communication performance was that she exhibited “some receptive language skills.” It was unclear to Ms. Bell how this had been determined. Additionally, there was no information addressing Kristin’expressive communication abilities or modalities used. Ms. Bell typically conducts preference assessments on all her students to systematically identify each student’s preferred stimuli (e.g., items, activities, interactions). These preferred stimuli are then used as reinforcers for students engaging in or completing various instructional tasks or activities. However, given that the current IEP did not provide details on Kristin’s communication skills, Ms. Bell was not sure whether Kristin would be able to complete the preference assessment. Ms. Bell sought out Ms. Davis, the school’s speech-language pathologist (SLP), to gain insight into how she should proceed with Kristin. Ms. Davis explained that they first needed to establish a reliable way for Kristin to intentionally communicate. Ms. Davis typically recommends formal communication assessments, especially for students new to the district. However, scores from formal assessments may not be valid due to a variety of factors for students like Kristin with severe developmental disabilities (e.g., assessments can be overwhelming to students and can be challenging to administer). Ms. Davis considered conducting an informal communication assessment but wondered if the informal assessment would be rigorous enough to obtain a clear understanding of Kristin’s communication skills to inform the preference assessment. Due to this dilemma, Ms. Davis decided that the first step was to investigate the literature to see if there was empirical support for conducting a formal and/or informal communication skill assessment prior to a preference assessment.
Background
It is common for students with severe developmental disabilities to have difficulty staying engaged in instructional activities, particularly activities that are not interesting to those students (Alberto & Troutman, 2013; Pace, Ivancic, Edwards, Iwata, & Page, 1985). Research supports using reinforcement techniques during the instruction of students with developmental disabilities to enhance their learning opportunities. In fact, the literature indicates that systematic implementation of this reinforcement strategy increases the students’ overall outcomes (Hagopian, Kuhn, Long, & Rush, 2005; Mangum, Fredrick, Pabico, & Roane, 2012). It is vital to consider the student’s preferences when identifying potential reinforcers because the stimuli used must be highly rewarding for the student in order to have an impact on the success of the instruction (Fisher, Piazza, Bowman, & Amari, 1996). The term preference assessment refers to the process in which a student is evaluated to identify stimuli (e.g., items, activities, interactions) that are highly motivating for the individual and that can be used to enhance the learner’s engagement in various tasks or activities.
Unfortunately, many students with developmental disabilities have difficulty communicating their likes and dislikes (Best, Heller, & Bigge, 2009; Heller, Forney, Alberto, Best, & Schwarzman, 2009). As a result, it may be necessary to assess the student’s communication skills prior to a preference assessment so that teachers and other professionals working with the student have a better understanding of the student’s ability to clearly express his or her preferences. There are currently two methods for assessing students’ communication skills: formal and informal assessments. Formal assessments provide differential diagnosis information. They can help gather information on the nature and degree of the communication disorders and, consequently, provide information needed to accurately and effectively plan intervention programs (Purse & Gardner, 2013; Wiig, 2001). Scores from formal assessments are usually compared to a standard measure to determine the current skill or age-equivalent level of the student being assessed. In contrast, informal assessment strategies are based on observations in natural settings. Informal assessments are often used to supplement formal assessments or to evaluate students who may be challenging to assess (Purse & Gardner, 2013). In addition, informal assessments offer flexibility in assessing communication skills during specific tasks, objectives, or activities; they are specific to the skills under observation (Ruiz-Primo & Furtak, 2004). To date, there is little consensus on which form of communication skills assessment best precedes and supports preference assessments for students with severe developmental disabilities. Because of the value of both informal and formal assessments, Ms. Davis and Ms. Bell were unsure which assessment format would best identify Kristin’s present level of performance.
The Clinical Question
Ms. Davis used the PICO format (population, intervention, comparison, outcome; Sackett, Straus, Richardson, Rosenberg, & Haynes, 2000) to investigate the following clinical question: Should practitioners (P) conduct a formal (I) or informal communication assessment (C) prior to completing a preference assessment to accurately identify preferences for students with severe developmental disabilities (O)?
Search for Evidence
Ms. Davis searched literature on the use of preference assessments with students diagnosed with severe developmental disabilities, including students with intellectual disabilities and students with autism spectrum disorder, published in the last 30 years (1984–2014). These years were selected to ensure all relevant studies were considered, yet results were restricted to fairly recent investigations to best inform current practices. Ms. Davis evaluated articles that met the following inclusion criteria: (a) the primary focus was on the use and/or evaluation of preference assessments; (b) the studies used experimental or quasi-experimental research designs; (c) the participants included students with severe developmental disabilities; (d) the authors included information on the assessment of participants’ communication skills; (e) the authors included information on the students’ overall communication skills and response modalities; (f ) the articles were published in peer-reviewed journals; and (g) participants were between 3 and 22 years old.
Ms. Davis divided the search process into three phases to help her systematically retrieve articles. In Phase 1, she searched four databases: Cumulative Index of Nursing and Allied Health Literature (CINAHL), Educational Resources Information Clearinghouse (ERIC), ProQuest, and PsycINFO. Each database was searched using the same keywords and keyword combinations, including reinforcer assessment, preference assessment, severe disabilities, and special education (see Table 1 for a summary of search keywords). During Phase 2, Ms. Davis conducted an ancestral search by scanning the reference lists of articles that met the inclusion criteria to identify other potentially relevant articles. In Phase 3, Ms. Davis hand-searched the Journal of Applied Behavior Analysis (see Figure 1 for a description of the multifaceted search process). After the search process was completed, Ms. Davis recruited Ms. Bell to evaluate all of the articles to ensure only those that met the inclusion criteria were analyzed. Ms. Davis and Ms. Bell used a checklist with predetermined categories that consisted of the following: (a) author name(s); (b) publication year; (c) research design; (d) purpose of the study; (e) communication assessment tool used; (f ) communication skill level: receptive and expressive; (g) modality of communication; (h) modality of the reinforcer/preference (e.g., tangible, activity, social); and (i) participant information (i.e., age, gender, and disability).
Evaluating the Evidence
Using this search process, 10 studies met the inclusion criteria and were further evaluated using the method developed by Reichow, Volkmar, and Cicchetti (2008; Reichow, 2011) to determine evidence-based practices. This evaluative method includes specific procedures for evaluating group and single-subject research designs. Given that all 10 studies used single-subject designs, Ms. Davis felt it was appropriate to use the Reichow evaluative method to assess these studies. Studies were rated as strong (S), adequate (A), or weak (W) based on primary and secondary quality indicators. Primary indicators consisted of: (a) adequate descriptions of participant characteristics, (b) independent variables described with replicable precision, (c) operationally defined dependent variables, (d) stable baseline conditions with at least three measurement points, (e) visual analysis of a stable level/trend with minimal data overlap between conditions, and (f ) three separate demonstrations of experimental control. Secondary indicators consisted of: (g) >.80 inter-observer agreement, (h) >.60 kappa obtained for at least 20% of sessions, (i) blind raters, (j) >.80 procedural fidelity, (k) generalization or maintenance assessed, and (l) social validity adequately assessed. A strong rating was given to studies that clearly provided information on all of the primary quality indicators (a–f ) and at least three secondary quality indicators (g–l). An adequate rating was given to studies in which at least four primary and two secondary quality indicators were appropriately described. A weak rating was given if fewer than four primary quality indicators were clearly described and at least two secondary quality indicators were missing.
After reviewing the studies for quality indicators, Ms. Davis and Ms. Bell noted that all but one (i.e., Grindle & Remington, 2005) of the studies received a weak rating due to insufficient descriptions in the methodology and/or the researchers’ lack of experimental control. It was evident they would not be able to draw definite conclusions from these studies. However, Ms. Davis and Ms. Bell decided to organize the information into two main categories (i.e., formal and informal assessment) and extract information that could potentially help them identify the best process, procedures, and tools to evaluate Kristin.
Communication Skills Assessments
Six of the 10 studies formally assessed the participants’ communication skills prior to conducting preference assessments (see Table 2 for a summary of the studies that met the inclusion criteria under this heading). Findings for these studies reported that that participants’ preferred stimuli were identified from the preference assessments, suggesting that participants appropriately communicated their preferences during the assessment process. Furthermore, all six studies reported similar preference assessment results regardless of whether participants’ expressive and receptive communication skills (Kang et al., 2013; Keen & Pennell, 2010; Kooistra, Buchmeier, & Klatt, 2012; Mason, McGee, Farmer-Dougan, & Risley, 1989; Petursdottir, Carp, Matthies, & Esch, 2011) or only receptive communication skills (Grindle & Remington, 2005) were assessed.
Conversely, four out of the 10 studies evaluated the participants’ communication skills using an informal assessment procedure prior to conducting a preference assessment (see Table 3 for a summary of the studies that met the inclusion criteria under this heading). Similar to the studies using formal assessment procedures, results from these four studies (Clevenger & Graff, 2005; Graff & Gibson, 2003; Groskreutz & Graff, 2009; Nuernberger, Smith, Czapar, & Klatt, 2012) indicated that preferred stimuli were successfully identified following informal communication assessments. Specifically, these four studies conducted informal assessments of the participants’ ability to follow instructions and make independent selections through matching tasks (picture-to-object and object-to-picture). Likewise, all except one study (Nuernberger et al., 2012) provided a description of each participant’s expressive communication skills, including the modalities used (e.g., use of symbols).
In addition to the aforementioned results, Ms. Bell and Ms. Davis discovered that in all of the preference and reinforcer assessments conducted in the 10 studies, preferred stimuli were always successfully identified, whether the participants’ communication skills were formally or informally evaluated. For the most part, these stimuli also functioned as reinforcers. In other words, the stimuli that were initially described as being preferred by the participants were further assessed to see if the participants were willing to earn the stimuli by completing tasks. Preferred stimuli were considered a reinforcer if the participants completed the task demands in order to receive the preferred stimuli. After completing this process, Ms. Bell and Ms. Davis noted that preference assessment outcomes across studies were similar in their findings. However, these findings are tenuous at best, as the quality ratings of these studies were weak overall.
The Evidence-Based Decision
In order to formulate an evidence-based decision based on the literature, Ms. Bell and Ms. Davis revisited their clinical question: Should practitioners conduct a formal or informal communication assessment prior to completing a preference assessment to accurately identify preferences for students with severe developmental disabilities? The findings suggested that there is no difference in the outcomes of preference assessments following formal or informal communication skills assessments. However, given that the 10 studies had methodological flaws, the accuracy of these findings was highly questionable.
As a result, Ms. Davis and Ms. Bell decided to evaluate Kristin’s communication skills through a combination of formal and informal assessment strategies for two reasons. First, they concluded that formally evaluating Kristin’s communication skills was in her best interest given that she was a new student in the school. Ms. Bell and Ms. Davis felt that they did not know her well enough to rely solely on informal assessments. By using formal assessments, they could potentially gather baseline information about Kristin’s overall communicative abilities.
Second, Ms. Davis and Ms. Bell determined that informal assessments would provide additional information that might be missed if they relied only on formal assessments. They noticed that matching was a key skill evaluated in the studies that used informal assessments (see Table 3). That is, all participants in these studies were able to match objects to picture and pictures to object. Ms. Bell and Ms. Davis also planned to evaluate Kristin’s matching ability prior to conducting the preference assessment. By doing so, they could be more confident that Kristin could not only follow instructions during the preference assessment process, but that she also had the ability to indicate her preferred choices.
Ultimately, Ms. Bell and Ms. Davis discovered that there was little empirical evidence to guide their decision on the best form of assessment to use with Kristin prior to a preference assessment. They understand that with minimal or unreliable evidence, practitioners should supplement the evidence with professional judgment and student needs to make the best decisions for their students. Given the information gathered, Ms. Bell and Ms. Davis were optimistic that conducting both a formal and informal communication assessment prior to Kristin’s preference assessment would not only help Ms. Davis in developing Kristin’s communication goals and objectives, but would also provide Ms. Bell with important information to support Kristin’s educational needs.