Editor's Note: This text is a transcript of the course, Evidence-Based Practice Decisions in Child Language Assessment, presented by Katrina Fulcher-Rood, PhD, CCC-SLP.
Learning Outcomes
After this course, participants will be able to:
- Describe current assessment practices in school settings.
- Explain the complexity of child language diagnostic decisions.
- Identify evidence-based best practice recommendations for child language assessment.
Introduction
I am honored and privileged to be speaking at the 12th Annual Nancy McKinley Lecture. Today, I'm going to be talking about what we know about evidence-based practice specifically for assessment considerations. The work that I've been doing for about seven years has not been done in a vacuum. I have an amazing research team with Dr. Anny Castilla Earls who is an associate professor at the University of Houston, as well as Dr. Jeff Higginbotham, who is a professor at the University at Buffalo. We have been so lucky to work with various undergraduate and graduate students who have helped us at almost every single point of these various research studies that I'm going to be talking about today. I also want to recognize all of our study participants. So many school-based SLPs have agreed to work with us and we get to learn from those experts in the field. This course would not be possible without my team, my students, and amazing school-based SLPs.
Previous Work in Diagnostic Decision Making
To get started, It is important that we look at previous research in this area of diagnostic decision-making. What is happening in the field around making diagnostic decisions? I'm primarily going to focus on school-based SLPs, but as we're talking about other tools and strategies, I don't think this is isolated to school-based SLPs. However, that is the population of SLPs that we have worked with the most. We know that school-based SLPs use a combination of assessment tools that include standardized testing, as well as a variety of informal measures when assessing the language performance of children. What stands out is that even though a combination of tools is being used, there appears to be a heavy reliance on standardized testing. Standardized testing seems to be a way that SLPs can have a quick, comprehensive view of a child's language performance. These tools, specifically standardized testing, are usually what is most readily available and accessible for school-based SLPs.
While we have some research indicating what assessment tools SLPs are using during their diagnostic practices, we don't have that much research on how SLPs actually make diagnostic decisions. How do we actually go about deciding that Child A has typically-developing language skills and Child B is presenting as a child with a language disorder? That might seem simple to answer, but we're finding that it is a highly complex and detailed decision-making process. Some research is also indicating that there are other influential factors that affect school-based SLPs' decision-making.
As you know, there are state level, district level, and federal regulations and policies that dictate specific assessment tools that you may need to use throughout your evaluation process. Many school districts provide policies and guidelines that standardized testing is a mandatory component of an evaluation. There are even some specific criteria, or specific standardized scores, that a child would need to have in order to qualify for speech and language intervention services in the schools. We've also found that sometimes these policies that are set at a state or a district level, will not allow for informal measure data to be used to make these qualifying decisions. So, while you may feel that you might get richer data, or that data from naturalistic observations or parent and teacher questionnaires, might be better than standardized testing, your employment setting may not allow you to use that to make qualifying decisions.
There is a study published in 2019 that shows that there is a large impact of these guidelines on decision-making. During a large decision-making task, Sellen and her colleagues found that when they told SLPs to not think about the guidelines at their state or district level, they actually diagnosed children with language impairments at a higher rate than what we see in schools. If these guidelines were lifted, potentially, this is showing us that school-based speech-language pathologists would make different decisions. That's very eye-opening for us. In addition, influences of evidence-based practice and your own clinical expertise will also influence your diagnostic decisions.
As a result, about seven years ago, I was lecturing on how to write treatment goals to an undergrad class. I was using the PICO model framework from Asher and I was talking about how to make sure that goals are objective, how many cues you should add and scaffolding, etc. A student raised her hand and said, "Yes, Dr. Fulcher, this is great, but where's the evidence, or who do I cite when I want to say that I am writing this treatment goal or providing these cues?" I actually paused because that is a great question. She knows I would ask her to cite her part of her decision making. She went on to ask, "So why do you make your goals like that? Who told you? What researcher?" I said, "A researcher didn't tell me to do this. I do it this way because this is what I learned in graduate school. This is what I learned in my CFY experiences. This is what I do when I see a client. But I can't give you a citation." My answer bothered me so much. Then I thought about assessment and some of those decisions that I made. Again, they were decisions primarily based on my clinical expertise and experience. So that led me to go to my colleagues, Anny Castilla Earls, Jeff Higginbotham, and others, to ask them why do they make these decisions? I was finding that the common answer was, "This is how I do it."
It occurred to me that we have to look at how clinicians make decisions and learn from experts in the field and those who are doing this every day in order to break down this assessment process. Once we understand the expertise, the steps, or the procedures that are used, then we can ask, "What do we need to research? What do we need to? What needs to be validated? What do we know that needs to be reliable? What other tools do SLPs need?
Our Guiding Questions in Examining Diagnostic Decision Making
We started guiding this EBP decisions project based on three primary questions:
- What assessment procedures do school-based SLPs utilize for their evaluation protocols?
- Why do school-based SLPs select these specific diagnostic procedures?
- How do school-based SLPs make diagnostic decisions?
The first question focused on documenting what assessment procedures, assessment tools, or strategies SLPs use when they're completing evaluations. We could then corroborate or compare those procedures to previous research studies. We could also see if tools have changed over time. Then our next two questions have had very little evidence in the field: why do SLPs use these specific diagnostic procedures and how do school-based SLPs make diagnostic decisions? If you decide that you're going to do a language sample, a play-based observation, or a classroom observation, why do you do that? What data are you hoping to get and how does that data help you make a diagnostic decision? The third question is one of the most complex questions that we're trying to answer - how do you actually go about making a diagnostic decision?
For part of this course, I will present the results of the research that we currently have to answer these three questions. Then we will see how can we bring those results into our assessment practices. We want to make sure that our evaluations or assessment procedures are based on all three points of evidence-based practice.
Current Work
We have conducted a variety of studies that you can see here:
- Phone Interviews
- Fulcher-Rood, Castilla-Earls, and Higginbotham (2018)
- School-Based SLPs’ perspectives of diagnostic decision making
- Fulcher-Rood, Castilla-Earls, and Higginbotham (2020)
- Perspectives of evidence-based practice and its influence on decision making
- Fulcher-Rood, Castilla-Earls, and Higginbotham (2018)
- Survey
- Retrospective Review of Clinical Files
- Simulated Assessment Task
- Fulcher-Rood, Castilla-Earls, and Higginbotham (2019)
- New and in progress:
- Assessment practices of SLPs working with bilingual populations
- Examination of diagnostic report writing
First, we've done two series of qualitative semi-structured phone interviews. For the first phone interview, we had approximately 39 school-based SLPs across the United States work with us. We did a semi-structured phone interview asking various questions about the what, the why, and the how. We used those three guiding principle questions.
We also did a follow-up study with those same 39 SLPs, asking them about their perspective of evidence-based practice. How do they define evidence-based practice? How did they go about finding evidence-based resources? How do they actually go about using evidence-based practice in their daily clinical practice? We also conducted a survey and had about 700 respondents. From that survey, we compared speech-language pathologists who work in school-based settings to SLPs who still work in the childhood setting, but not necessarily in school districts. These were SLPs who worked in agency-based settings, outpatient clinics, maybe universities, or center-based clinics to compare assessment tools and strategies across those SLPs.
We also did a retrospective review of clinical files. There were a variety of agencies and centers in the Buffalo, New York area that allowed us to collect about 100-120 de-identified diagnostic reports. We are currently in the process of analyzing those diagnostic reports to see what tools SLPs use, if they're providing a rationale for their assessment tool selection, as well as to see if we can use these as artifacts of decision-making. The survey and the retrospective review of clinical files should be ready for publication within the year.
Additionally, we conducted a simulated assessment task, which I'll be talking about more in-depth in a few minutes as well as some new and up-and-coming work. Specifically, we are taking those initial interview questions that we use in the Fulcher-Rood 2018 study, and working with school-based SLPs who work with the bilingual population or with children who are identified as English language learners, to see what assessment tools they're using and why they use them. We are interested in knowing if you are in a school setting, does your assessment procedure change or differ if you're working with a different population of students.
We're also examining diagnostic report writing. We're looking at the readability of diagnostic reports. We've conducted semi-structured phone interviews with approximately 60 SLPs about their reasoning for diagnostic report writing, who they write for and why to look at this diagnostic process as a whole.
What Assessment Tools Do School-Based SLPs Use for Child Language Evaluations?
Let me share what we found when we look across these studies. Let's start with the first guiding question, what assessment tools do school-based SLPs use during their diagnostic protocols? Similar to previous research that I mentioned earlier, we found that school-based SLPs use a combination of formal measures (e.g., standardized testing) as well as informal measures when assessing students for suspected language impairment. We're finding this to be true across all of our different studies. We found this finding in the phone interviews for the survey, when we reviewed the clinical files and during the simulated assessment task. That's a pretty solid finding that we found across multiple studies and contexts. It is also in conjunction with previous work in this area.
Also, eighty percent of school-based SLPs report using two or more standardized tests during their evaluation protocol. These findings are from the survey we conducted. That is different from SLPs in other settings, such as an agency or a center-based, or a private practice setting. SLPs in those settings are using standardized testing, but not as many standardized tests as our school-based SLPs. In regards to informal measures, the most frequently used informal measures that we have found are either parent and teacher interviews or questionnaires. We've seen this finding across three of our studies, and this has also been a finding in previous research.
Why Do School-Based SLPs Select These Specific Diagnostic Procedures?
Having a good idea of the tools that SLPs are using most frequently, we then asked SLPs specifically in the phone interview, "Why do you use these tools? Why do you put them in your assessment? What are you hoping that they're going to do for you? We looked across our two categories of tools, standardized testing and informal measures. SLPs report that they've used standardized testing because they are comprehensive, quick, and available. When SLPs say that standardized tests are comprehensive, they're saying that they feel these standardized tests look at language as a whole. They can get receptive language, expressive language, as well as look at the variety of language components, such as semantics, syntax, and morphology. They feel it's a great benefit that they can do this comprehensive assessment in a very time-efficient manner. The fact that they are quick and easy to administer is definitely a pro for our school-based SLPs. Also, standardized tests are readily available in their employment setting. In the phone interviews, as well as in the survey, these SLPs are stating that there are specific guidelines or policies set up by their district or the state that requires them to use standardized testing. What was really interesting is when we looked at the survey that was comparing school-based SLPs to non-school-based SLPs, school-based SLPs frequently answered the question with greater frequency, "Yes, there are policies that dictate what I have to do." They strongly agreed with that statement, versus our non-school based SLPs who were sometimes neutral on that question.
Moving to the second category of informal measures, we found that primarily SLPs are using informal measures, such as parent-teacher interviews and questionnaires, observations, and language samples because they feel that those informal measures allow them to collect data about how that child communicates in a naturalistic setting. Those informal measures are getting a snapshot of daily or everyday communication.
How Do School-Based SLPs Make Diagnostic Decisions?
What do we know about that third guiding question, "How do SLPs actually make these very complex diagnostic decisions?" I want to talk about this across three different studies and then try to bring these three studies together for some overall conclusions about what we're finding with how school-based SLPs make these decisions. In the phone interviews, our qualitative analysis revealed four primary themes or mechanisms that SLPs use to make diagnostic decisions:
- Holistic
- Standardized testing
- Classroom performance
- Combination of standardized testing and classroom performance
The first category is the holistic category. For these SLPs, they looked at all of the assessment data as a whole. They compared and contrasted that assessment data to see what was the common narrative or what were the common findings. Again, they were looking across all data points. Another group of SLPs stated that they determined if there was an impairment based on standardized testing performance only. A third group of SLPs used performance in the classroom or what they considered the child's academic success only, as a way to make a diagnostic decision. Then there was a fourth group of SLPs that used a combination of the performance on standardized testing, as well as the student's current academic success to determine a diagnosis.
What I want to point is that regardless of the four themes or mechanisms that SLPs used to make diagnostic decisions, each of these four themes has standardized testing included. The only one that doesn't use standardized testing is that classroom performance group. So we are seeing heavy reliance on standardized testing. It was also interesting to note that when determining treatment recommendations and the severity of an impairment, those decisions were exclusively made based on standardized testing data and not data from other measures that may have been administered.
How does the survey compare to these results from the phone interview? In general, we're seeing a trend that is comparable to the phone interviews. Sixty-seven percent of our 350 SLPs in the study used a combination of tools. But what was also interesting is that 67% also said that their specific employment agency has criteria that would determine treatment and severity eligibility. Again, we see this idea of a combination of tools being used, but there are outside factors that impact the diagnostic decision-making process.
Let's add in one more layer. Let's look at another study that we conducted to examine the question of, "How do school-based SLPs make decisions?" Figure 1 is a table from our 2019 study that had 12 SLPs complete a simulated assessment task.
Figure 1. Diagnostic decision-making by SLPs.
SLPs reviewed five different cases. These cases are all real children who were pulled from a previous study where we collected many child language evaluations. We chose these five cases for specific reasons. If you look across all of the different measures, Case 1 is what we consider a congruent case. Meaning all of the assessment data is leading you to make a specific decision. We administered the CELF P-2, the PLS-5, the SPELT-3, the KBIT for non-verbal IQ, a parent questionnaire and a teacher questionnaire, as well as two language samples. One was a picture elicitation task and the other was a story retell. For Case 1, all of the testing scores are within normal limits, the parent and the teachers do not show any concerns, and the MLU, as well as the percentage of grammatical units, were all in the average range. This is a congruent case and one that we were assuming all SLPs would state that the language performance was within normal limits.
Case 2 is also a congruent case. But what you can see from the standardized testing scores, as well as our informal measures, all of the standardized testing scores are below normal limits. They are below that cutoff of 85, the parents and teachers have concerns, there were concerns about grammatically as well as the mean length utterance when we examined the language samples.
Then we have some interesting cases, what we call the in-congruent cases, where we thought we would see how this diagnostic decision-making is made in the real world. When I say incongruent, I mean that there is not a comparison. There's no equality between our standardized measures and our informal measures. With Case 3, you will see above-average scores for the CELF and PLS. However, when we get to the parent concerns and the teacher concerns, they're both expressing concerns about some errors with speech and hearing and some parts of speech. When we examine the language sample both in the picture elicitation and the story retell, this child is highly ungrammatical. So what will our SLP do or what would you in this type of case? Case 4 is another in-congruent case. With this child, we see that formal measures are below normal limits, but the informal measures are not noting any concerns. Case 5 is a congruent case. However, you can see this is what we've labeled as the "borderline case" where all scores are on the cusp between normal and mildly impaired.
Each of our SLPs went through all of these cases. They could select and ask to see any assessment data that they wanted. We did not dictate to them that they had to look at all of the standardized tests or that they had to look at language samples or parent and teacher questionnaires. It was up to them to ask us for what assessment data they would like to review. The SLPs told us when they were ready to make a diagnostic decision; when they felt like they had enough data to provide a diagnosis.
Case 1 and 2. What did our SLPs do? Let's start with those first two cases. Case 1 and Case 2 are the congruent cases. For Case 1, all of the measures are within normal limits and for Case 2, all of our measures were below normal limits. We had 100% consensus across all of our SLPs that Case 1 was a child who was typically developing. Likewise, for Case 2, there was 100% consensus that the child would be diagnosed with a language impairment.
These were very similar results to a previous study and diagnostic decision making conducted by Records and Tomblin, 1994. They also found that when SLPs were given standardized scores that were clearly within normal limits or clearly below normal limits, they received great consensus - almost up to 100%. But that's what we were expecting because there was a clear delineation and a clear path that all of the assessment data was leading them to.
Case 3 and 4. But what about Cases 3 and 4? Remember these are incongruent cases. Case 3 was incongruent in that the informal measures were showing concern or were below normal limits (i.e., parent and teachers have concerns and the language samples were highly ungrammatical), however, standardized testing scores were within normal limits. We had 14 SLPs in this study. Thirteen SLPs, or 93%, concluded that this case had typical language impairment. They would not diagnose the child with any language impairment if they were provided the data for Case 3. Remember SLPs looked at a combination of tools for this simulated assessment task. All SLPs reviewed at least one standardized test and one informal measure. So even though they had informal data available to them for this case, they tended to make decisions that coincided with the performance on the standardized test.
What happened for Case 4, which was also an incongruent case? Remember, Case 4 had standardized testing, or formal measures, that were below normal limits. But informal measures were within normal limits. The parents and teachers did not have any concerns, MLU was average and the child was grammatical. In this case, twelve speech-language pathologists, or 86%, stated that if this was a real child, they would diagnose Case 4 with a language impairment. Two SLPs said that this child was typically developing. Again, for the majority of our SLPs, even though they are looking at a combination of assessment data, their diagnostic decisions coincided with the performance on standardized testing.
Case 5. We had one last case, Case 5, which was a congruent case. However, it was a borderline case between low average-average to maybe mild, depending on your assessment process. Ten SLPs, or 71%, concluded that this was a case with typically developing language. Of these 10 SLPs, there were two who stated that they would monitor and potentially do a follow-up assessment since there were some borderline skills. This compares to 29% of our SLPs who stated that Case 5 presented with a language impairment.
So, we're seeing the most disagreement with this final case. What does this tell us? When we compare our results from the phone interviews, the survey, the simulated assessment task, and our review of clinical files, standardized testing is pervasive and is the most relied upon tool in diagnostic decision-making. I don't think that this is a pervasive issue because this is what SLPs are using. I'm an SLP that also uses standardized testing, so I'm also in this boat. But when you look across information from national associations textbooks, supervisors, graduate school, CFY guidelines, and employment setting, they're all pointing us to use standardized testing first. I think it's very interesting that that's the case. In some ways, because it's standardized, it's normed, and there's a specific way that you're supposed to do things, then we think that's more objective than other informal measures. Even the word "informal", what does that mean? What about evidence-based practice? How was that talked about or used?
What surprised us when we conducted the 2018 study with the phone interviews when we asked SLPs why they use these assessment tools none of them mentioned evidence-based practice as a rationale or a guiding principle for why they selected and used specific measures. That response led us to our rationale for doing the follow-up study with those SLPs and asking them about their perspectives of evidence-based practice. We thought that maybe the discussion of evidence-based practice would happen naturally. When it didn't, we decided that we have to go back and ask.
That study showed us that SLPs will tend to use external scientific evidence under two conditions. One, when they feel uncertain or uncomfortable about a client. For example, maybe it's a client that they aren't familiar with or they have a diagnosis that they haven't seen in a while. That will spur them on to look for external scientific evidence. The second condition in which external scientific evidence would be used is if the SLP needed to provide justification or rationale for why they were already using something. For example, maybe a parent, a school psychologist, or a special education chairperson asks them why they were doing something. Then they would seek out external evidence, which shows that external scientific evidence is being used in various specific ways.
What was also interesting about this study is when these SLPs were talking about evidence-based practice, they were primarily talking about treatment. Out of the 39 SLPs, we had 25 SLPs participate in the follow-up study. All of those SLPs mentioned using evidence-based practice for treatment and did not mention it for assessment, which is another reason or the impetus for this course.
Why Is This Important?
We know that standardized testing is being used. It's pervasive and it's most likely not just happening in the school settings. I think that this is something that we're seeing across many different employment settings. There are some concerns with standardized testing. For example, what if a child was having a bad day? What if that child wasn't included in the norming sample? What if the child needs more exposure to English? Is that standardized testing? Was that actually measured for diagnostic accuracy?
Before moving on, I want to be clear that I am not saying to throw out standardized testing and never use it. That is not the point. We're all SLPs and we use standardized testing in our research protocols and our clinical protocols. Standardized testing can provide us with great important information. However, when selecting standardized testing, we want to be critical reviewers and examiners, and make sure that each standardized test has quality psychometric standards. We need to understand the specific cutoff scores and what we can do to go beyond just a standard score. What are the other scores that standardized tests give us that may be more helpful and accurate in decision-making? Again, we're not throwing out standardized testing, but there may be other ways to use them to make sure that we can feel very confident in the diagnostic accuracy.
Principles of Standardized Test Interpretation
Let's discuss the principles of standardized tests and how to ensure we're using them in a more evidence-based manner. We want to use tests with good diagnostic accuracy. What that means is when a test was developed, did it take into consideration and do some examination or research to make sure it differentiated between children with language disorders versus children with typical skills. Typically, you see these measures of diagnostic accuracy as measures of sensitivity or specificity.
Measures of Diagnostic Accuracy
Sensitivity is when you have a disorder and I give you a test, the test gives me a positive. Meaning, yes, you do indeed have that disorder. For example, we have ten children that we know have language impairments. We give them a standardized test and seven are identified as having a language impairment. That is a 70% sensitivity, meaning that three children were incorrectly identified. They should have been diagnosed with a language impairment, but they were missed.
The other measure of diagnostic accuracy is specificity. That's the other side of the coin. I do not have a disorder, you give me a test, and the test is negative. Meaning, I don't have the disorder. The test should identify you as typically developing and it does a correct job in saying, yes, indeed, you are typically developing. For example, we take another 10 children. who are typically developing. We give them a standardized test and eight of those 10 children are diagnosed with typically developing language skills. That would mean that that test has 80% specificity. That also means that two children were incorrectly identified. So two children who were typically developing were given a diagnosis of language impairment and would get intervention when they actually don't need it. So, it is crucially important that these measures are accurate because they are helping us make sure that we're not over or under identifying.
Consider the following graphs. The intersection lines in the graph (figure 2) is the standard cutoff score of 85. The axis labeled "measure of interest" could be the CELF or the PLS. The other axis is the "gold standard measure" which is a measure that everybody agrees does a great job at identifying, like the SPELT.
To earn CEUs for this article, become a member.
