linder2_01ch01.001_034

WHAT DATA SUPPORT USING TPBA?

Research on the original TPBA

Numerous studies have looked at the advantages and disadvantages of the TPBA model. In the following discussion, studies that have been conducted on TPBA to determine reliability and validity are examined. Reliability, as defined by Gall, Borg, and Gall (2006), refers to the level of internal consistency or stability of a measuring device over time. Reliability of the TPBA process across time and raters has been studied (Al-Balhan, 1998; Cornett & Farmer-Dougan, 1998; Friedli, 1994; Linder, Green, & Friedli, 1996; Myers, McBride, & Peterson, 1996) and was well supported in most test–retest and interrater conditions.

Interrater Reliability

Interrater reliability generally refers to the degree of agreement between two or more raters on the point value of responses to specific test items. Interrater reliability was studied by Myers and colleagues (1996), who noted higher percentages of interrater agreement when using TPBA as compared with standardized assessments. In their study, they compared each rating on every developmental profile item with ratings made by other team members and parents on the same item. Higher percentages of agreement were found using transdisciplinary play-based assessments as compared with standardized assessments. For TPBA, 11 of 15 developmental domains had higher percentages of exact agreement, and 12 domains had higher percentages of agreement within one point. For TPBA, there was a mean agreement of 51.8% for staff and parents’ developmental ratings with a range of 28.8%–84.4%. At the same time, the standardized assessment had a mean agreement of 46.5% for staff and parents’ developmental ratings, with a range of 38.4%–77.3%. In a second measure of interrater reliability, staff members’ mean ratings were correlated with parents’ ratings. Higher correlation coefficients were found between parents and professionals for transdisciplinary play-based assessments (r 0.70, p 0.001) than with standardized assessments (r 0.67, p 0.001) when examining developmental profile ratings. A Fisher’s z-test indicated a slightly higher, although nonsignificant, difference in correlation between parents and professionals for transdisciplinary play-based assessments. The interrater reliability of TPBA was also examined by Friedli (1994) using video recordings of TPBA sessions, with independent raters assessing each child. She revealed that the reliability for the language, motor, and combined domains met the same criteria for standardized tests used to make eligibility decisions. Interrater reliability was also examined by Al-Balhan (1998). In his study of training and implementation of TPBA in Kuwait’s kindergartens, he compared the TPBA results of newly trained kindergarten teams with the results of specialists in each developmental domain who were also newly trained in TPBA. He found that the mean overall percentage of agreements between kindergarten team members and specialists in the field ranged from 68% to 82%.

Test–Retest Reliability

Test–retest reliability refers to overall stability of a measure across a specified time interval. In 1994, Friedli studied the stability of TPBA assessments for 10 children between the ages of 3 and 6 years. Each child studied was assessed twice using TPBA within a 6-week period of time. Her results showed that chi-square tests of association between the first and second test was significant at p < 0.0001 for all domains.

Validity

Validity can take several forms including content validity, criterion-related validity, and construct validity. Validity studies on TPBA have shown strong support for validity across all of these dimensions (Al-Balhan, 1998; Friedli, 1994; Karr, 1998; Linder & Green, 1995; Myers et al., 1996).

Content Validity

Gall et al. (2006) describe content validity as the degree to which the sample of test items represents the content that the test is designed to measure. In a study of content validity, Friedli (1994) found that the TPBA guidelines were supported by early childhood professionals most likely to use the process, such as psychologists, educators, speech-language therapists, and motor specialists. These experts rated the developmental domains and subcategories for relevance, clarity, and comprehensiveness using a Likert scale ranging from 1 to 7. All of the subcategories of the TPBA guidelines were judged favorably (higher than 4), with most ratings between 6 and 7. Linder and Green (1995) surveyed 40 professionals across the country (N 10 for each domain) concerning the content validity of each domain, using a Likert scale of 1–7. All subcategories for all domains were rated between 6 and 7 on the scale, indicating strong support for content validity. Myers et al. (1996) also examined staff perceptions of the amount of information that they learned from the TPBA and found most favorable results. The authors concluded that the TPBA provided a better representation of young children’s development than standardized measures within professional training.

Criterion-Related/Concurrent Validity

Gall et al. (2006) state that criterion-related or concurrent validity is determined by relating the test scores of a group of subjects to a criterion measure administered at the same point in time, or within a short interval of time. The concurrent validity of TPBA was measured by comparing the outcomes of play-based assessment to traditional standardized and norm-referenced tests for children with and without disabilities. Friedli (1994) found that TPBA was as accurate as standardized measures for determining whether a child was eligible for services as compared with the results of the Battelle Developmental Inventory (BDI-2; Newborg, Stock, Wnek, Guidubaldi, & Svinicki, 1984). She also found that the two assessments produced similar profiles regarding the examinee’s strengths and needs. In addition, TPBA was actually more accurate in identifying one child with social-emotional concerns. Karr (1998) used a sample of typically developing children and compared results scores, and significant correlations between the two approaches were found. A study by Kelly-Vance, Needelman, Troia, and Ryalls (1999) with at-risk 2-year-olds modified the TPBA and compared only the results from the TPBA cognitive section with the Bayley Scales of Infant Development-II (BSID-II; Bayley, 1993). Comparison of age equivalents (AE) by averaging the subdomains for the cognitive domain on TPBA with the AE on the mental scale of the BSID-II, revealed higher cognitive AE on the TPBA than on the BSID-II.

Social Validity

Social validity refers to the ecological integrity of assessment information, the acceptability of the assessment methods, and the importance of the results to families and professionals (Neisworth, 1990). Myers and colleagues (1996) compared a multidisciplinary standardized model with the transdisciplinary play-based approach through random assignment of 40 children under the age of 3 referred to one of the models. They then examined consumer feedback (parent and professional) on the approaches, time spent on evaluations, and evaluation of written discipline reports. Through questionnaires with 17 statements describing positive aspects of the assessment and report process, parents were asked to rate their degree of agreement or disagreement with each of the statements using a 5-point Likert scale. Myers et al. (1996) found that the means for 13 of the 17 items, along with the overall total for TPBA, were higher, although not statistically significantly higher, than those for standardized assessments. Specifically, parents appeared to feel more comfortable seeking information from professionals during TPBA, and they also perceived the goals identified as a result of the TPBA as important. Myers and colleagues also found that speech-language pathologists and school psychologists rated TPBA as significantly more useful than standardized assessments for identifying a child’s strengths and weaknesses and for developing program planning. Functional utility, which refers to the clarity, completeness, and usefulness of information, was also studied by Myers et al. (1996). Myers and colleagues examined the functional utility of TPBA assessment reports. Compared with reports written after traditional testing, the reports resulting from TPBA rated higher in their study, particularly for ease of obtaining an overview of the child’s abilities, ease of determining which developmental areas were of concern, the number of developmental areas discussed in the report, the report being written in jargon-free language, the integration of discipline-specific information, and the objectives being clearly based on the child’s strengths and weaknesses. When all of the items were combined, the mean scores for TPBA assessment reports were significantly higher than the mean for the standardized assessment reports.

Research on TPBA2

Construct and content validity of the revised Transdisciplinary Play-Based Assessment (TPBA2) was analyzed in a study with national and international experts (Linder, Goldberg, & Goldberg, 2007). The first aspect of the study was to look at the validity of the transdisciplinary concept. Twelve experts, three for each domain, were provided a survey related to all of the content. Each expert was asked to rate how frequently a problem in one subcategory directly contributed to problems in other subcategories. These questions were posed in the format “How often do problems in (Subcategory A) directly contribute to problems in (Subcategory B)?” with the answer choices of never, rarely, sometimes, frequently, and always. These frequencies were later converted into numerical scores using the experts’ rating of the frequencies on a scale of 0 (never) to 10 (always). An algorithm was then used to look at the relationships within and across domains. Across the 28 subcategories, the total number of possible bidirectional transdisciplinary influences is 588.

Interrater Reliability

Several interrater reliability studies were conducted with TPBA2 (Linder, 2005) to determine interrater agreement with regard to individual domains. These studies included participants from numerous disciplines involved in 1) a 2-day initial training on TPBA2; 2) an initial 2-day training, time to practice, and then a 2-day follow-up training; 3) graduate students in Child, Family, and School Psychology in a 10-week course on TPBA2; and 4) professionals who have used TPBA2 during the past several years. Several of the studies involved watching videotapes of four different children. Cuts from four tapes were shown of children in TPBA sessions, and professionals were asked to rate each tape in relation to one area of development.

Table 1.1. Consolidated influences between developmental domains

Table 1.2. Reliability of observations of TPBA with video recordings

Table 1.3. Impact of training on reliability

Validity

In another study, an expert TPBA team evaluated a child and each team member, plus 10 student observers rated the child across all four domains. The only disagreement was in the area of emotional development, which was a relative strength for the child.

Conducting concurrent validity with TPBA2 and other instruments is difficult since there are characteristics of TPBA (i.e., informed clinical opinion, discussion with team members about cross-disciplinary influences, inclusion of parental perceptions) that are unique compared to traditional testing.

DeBruin (2005) conducted a study to determine whether TPBA demonstrated a precise picture of assessment outcomes and revealed that children received TPBA first; this provided a better understanding of child development. They also compared perceptions of both primary caregivers and assessment team members of the assessment experience using a 5-point Likert scale for evaluating their perception of the assessment's accuracy and effectiveness.