03Hotz_CH06(P2)
Evidence-Based Practice (Technical Analyses)
Examiners need to know that the interpretations they make are supported by data-based evidence. In this chapter, we review the types of questions examiners should ask about the nature of any test, which will inform their conclusions about test performance. Therefore, this section covers validity and reliability information as well as information concerning analysis of test bias. We continue to use the concept of constrained and unconstrained skills (cf. Paris, 2005) to interpret results. To review, constrained subtests tap skills that develop relatively early in development (e.g., knowing one’s name, naming common items). Unconstrained subtests tap skills that tend to show continued refinement over a wide range of ages. The PTBI contains both types of subtests, and the statistical results are presented separately for each type. A question and answer framework is used to describe the evidence base related to various clinical questions about the nature of the PTBI.
Technical Notes
Question 1: How do I interpret the criterion categories?
The categories of performance for each age are based on the performance of the group of children without brain injuries selected to represent the general population. The percentages of these children whose scores fell into each of the four criterion categories is provided in Table 6.1. As this table shows, most scores fell into the High performance category. The differences in the percentages of children classified into the different performance categories are highlighted in the findings.
EVIDENCE FOR CONSTRUCT-IDENTIFICATION VALIDITY
| Performance criterion categories | ||||
|---|---|---|---|---|
| Subscale | Very Low | Low | Moderate | High |
| 1. Orientation | 0% | 6% | 19% | 75% |
| 2. Following Commands | 7% | 8% | 2% | 83% |
| 3. Word Fluency | 3% | 20% | 25% | 53% |
| 4. What Goes Together | 2% | 1% | 19% | 78% |
| 5. Digit Span | 2% | 6% | 30% | 62% |
| 6. Naming | 0% | 4% | 3% | 93% |
| 7. Story Retelling-Immediate | 2% | 10% | 28% | 60% |
| 8. Yes/No/Maybe | 0% | 5% | 18% | 77% |
| 9. Picture Recall | 0% | 5% | 35% | 60% |
| 10. Story Retelling-Delayed | 0% | 3% | 20% | 77% |
Technical Notes
Ninety-nine children’s scores were compared for this source of evidence. Thirty-three children were included in each of the three groups (TBI, ABI, typical). These children were matched by age (mean age of 10 years, 7 months; range 6–16 years) and gender (14 males, 19 females per group). A variety of racial/ethnic groups were included (TBI: 26 white, 7 minority; ABI: 17 white, 16 minority; Typical: 22 white, 11 minority). The majority of the children in each group were right-handed.
Table 6.2. Raw score means (standard deviations) for matched groups of children with Traumatic Brain Injury (TBI; n = 33), other acquired brain injury (ABI; n = 33), and typically developing children (Typical, n = 33).
| Category | Constrained skills | Unconstrained skills | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Category | 1. Orientation | 2. Following Commands | 6. Naming | 3. Word Fluency | 4. What Goes Together | 5. Digit Span | 7. Story Retelling—Immediate | 8. Yes/No/Maybe | 9. Picture Recall | 10. Story Retelling—Delayed | |
| TBI | $30.8^{a,b}$(8.25) | $24.94^{a,c}$(6.75) | 14.7(1.14)$ | 13.65^{c}$(2.45) | 12.18^{a,b}$(0.79) | $11.27^{a,c}$(2.35) | $20.37^{a,b}$(9.11) | $14.97^{a,c}$(9.60) | $49.51^{b}$(23.59) | $52.62^{c}$(25.32) | |
| ABI | $36.02^{b,c}$(5.29) | $14.90^{c}$(0.54) | $12.40^{b,c}$(0.38) | $27.27^{b,c}$(7.74) | $69.02^{b,c}$(19.83) | $40.75^{b,c}$(20.40) | $61.33^{b,c}$(34.53) | $24.03^{b,c}$(7.37) | $34.42^{b,c}$(8.31) | $43.90^{b,c}$(19.91) | |
| Total possible | 38.0 | 15.0 | 12.5 | No limit | 100.5 | 82.5 | (1) 51.5 (2) 142.5 (3) 138.5 | 39.0 | 45.5 | (1) 51.5 (2) 83.0 (3) 111.0 | |
| aSignificant difference between typical and TBI and ABI groups ($p<.05$). | |||||||||||
| bSignificant difference between TBI and Typical groups ($p<.05$). | |||||||||||
| cSignificant difference between ABI and Typical groups ($p<.05$). |
Technical Notes
Exploratory factor analysis based on the ability scores of 257 children was completed. Factors were rotated using a Varimax procedure, and the three-factor solution provided a more meaningful grouping of subtests.
Question 7: How do I know whether change is due to real improvement or just fluctuation that would occur from one test session to the next?
To answer this, one must understand test–retest reliability values, which range from 0 to 1.00, where 1.00 indicates identical scores obtained by all children at two different test sessions. Note that the test–retest reliability coefficient was the basis for calculating the standard error of measure (SEM).
EVIDENCE FOR RELIABILITY
| Constrained skills | Unconstrained skills | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1. Orientation | 2. Following Commands | 6. Naming | 3. Word Fluency | 4. What Goes Together | 5. Digit Span | 7. Story Retelling—Immediate | 8. Yes/No/Maybe | 9. Picture Recall | 10. Story Retelling—Delayed | |
| Point-to-point agreement | 99% | 99% | 100% | 100% | 98% | 100% | 100% | 100% | 99% | 100% |
| Correlation | .98 | 1.00 | 1.00 | 1.00 | .99 | .99 | 1.00 | 1.00 | .99 | 1.00 |
Conclusion
This chapter guided examiners through the questions to ask to determine test performance and supported the analysis of test reliability and validity.