Validating the Assessment, Evaluation and Programming System (AEPS) in International Contexts

Authors

Jennifer Grisham, Ed.D. - University of KY

Lema Kabashi, Ph.D. - University of Wisconsin-LaCrosse

Overview of Assessment, Evaluation, and Programming System (AEPS)

AEPS-3 is a:

• Curriculum-based and criterion-referenced assessment, used to ...
• Develop individualized goals and outcomes for children, in order to ...
• Design appropriate instruction for individual children and groups of children, and then ...
• Measure whether the teaching/intervention is effective or not.

Assessment, Evaluation, and Programming System for Infants and Children, Third Edition (AEPS®-3), Bricker, Dionne, Grisham, Johnson, Macy, Slentz, & Waddell. © 2022 Brookes Publishing Co. All rights reserved.

Field Test of AEPS-3 in U.S.

Content

• 36 Professionals from each field (OT, PT, SLP, Psychology, Education)
• Completed survey about relatedness of items

Validity

• Clarity of items and criterion
• Analysis: Agreement among professionals

Utility

• 11 teachers who had used the AEPS-2
• Participated in online training
• Completed AEPS-3 on 2-4 children
• Completed survey about the scoring, items, and usefulness of the items
• Analysis: Descriptive Statistics

Interrater Reliability

• ALL teachers who collected data for the field test
• Prior to collecting data on children, trained on AEPS-3
• Completed Inter-rater reliability test
• Analysis: Inter-rater agreement

Concurrent Agreement

• 50 children (25 males and 25 females)
• 8 teachers
• Teachers completed training and Inter-rater Reliability test
• Teachers completed AEPS
• Within 2 weeks, teacher also gives criterion measure - Battelle Developmental Inventory–Second Edition (BDI-2) (Newborg, 2005)
• Analysis – Pearson’s correlations

Scale Evaluation and Cut Scores

• 200 Teachers (all completed online training and took IRR test)
• Minimums of 4 children for each teacher (n=874)
• Children with disabilities (52.9%) and without disabilities (47.1%)
• Analyses: Rasch Rating Scale Model, Cut-Score Development, Classification Analysis

Curriculum Validity

• 12 teachers
• 2-3 children for each teacher
• Teachers completed AEPS on each child
• Teachers completed the family report
• Family and teachers identified the goals for the child
• After 3 months of implementation, focus groups were held to determine validity of curriculum
• Analysis: Qualitative

References

• Toland, M., Grisham, J., Waddell, M., & Crawford, R. & Deuber, D. (2022). Scale evaluation and eligibility determination of a field test version of the Assessment, Evaluation, & Programming System 3rd edition. Topics on Early Childhood Special Education, 42(2), 15-161.
• Grisham, J., Waddell, M., Crawford, R. & Toland, M. (2021). Psychometric properties of Assessment, Evaluation, and Programming System – 3 Edition. Journal of Early Intervention, 43(1), 24-37.
• Macy, M., Bricker, D., Dionne, C., Grisham-Brown, J., Johnson, J., Slentz, K., Waddell, M., Behm, M., & Shrestha, H. (2016). Content validity analysis of qualitative feedback on the revised Assessment, Evaluation, and Programming System for Infants and Children test. Journal of Intellectual Disability -Diagnosis & Treatment, 3(4), 1-10.

Rationale for Validating AEPS in Other Countries

• Early Intervention Systems in other countries often adhere to a medical model in terms of assessment and service delivery.
• Professionals in many countries are trying to promote recommended practices including authentic assessment, family-centered practices, and inclusion.
• Assessment and curriculum systems that promote recommended practices do not exist in many parts of the world; the AEPS is one system that does so.
• In order to establish the validity of the system, research is necessary in the culture and in the language of the country.

Field-Test of the Assessment, Evaluation, and Programming System (AEPS) for Infants and Children-3rd Edition in Kosovo

Purpose

• Publication of AEPS-3 in Albanian language.

Long-Term Goal

• Implementation of AEPS-3 in Kosovo.

Project Significance

• Assessment instruments, existing or new, cannot be used by early childhood professionals if they do not have defensible psychometric properties addressing validity, reliability, and utility.
• Kosovo doesn’t have a criterion-referenced curriculum-based assessment that can be used with children from birth to 6 years old.
• Ultimate goal-provide Kosovo with increased sustainable resources and practices to improve early childhood education.

Field-Test

• Content validity
• Utility
• Interrater Reliability
• Concurrent Validity
• Curriculum Validity
• Psychometrics of Test and Cutoff Scores.

Timeline: Key study events:

• Preparation: Summer 2021
• Study: October-November 2021
• 35 expert reviewers from different disciplines reviewed the AEPS-3 and completed the survey about the items to validate the content.
• Changes were made for any items that professionals considered necessary; only one item was eliminated.

Study 1: Content Validity

Content validity refers to the degree to which an assessment measures what it is designed to measure.

Study 2: Utility

Utility refers to usefulness of an assessment for different purposes (e.g., developing individualized education programs, monitor progress, etc.).

Study 3: Interrater Reliability

Reliability refers to the stability and accuracy of assessment results.

Study 4: Curriculum Validity

Curriculum validity refers to the effectiveness of the test items in teaching the curriculum.

Study 5: Psychometrics of Test and Cut Scores

The psychometrics evidence and cut scores refer to an assessment that accurately and reliably discriminate between typically developing children and those whose development is significantly delayed or deviant and requires intervention.

Study 4: Concurrent Validity

Concurrent validity refers to the degree an assessment’s results are similar to those generated from a norm-referenced assessment.

Implementation of AEPS-3 in Kosovo

Study of the effectiveness of the AEPS-3.

Adaptation of the AEPS Scale to Spain

• Family-Centered Services in Natural Environments.
• Early Intervention.
• Inclusion.
• Individualized Education Program.

The Practices in Spain

• Two or more service providers work with the child at separate.
• Parents inside or outside of the room but they only observe the professionals working with the child to later on imitate those activities at home.

Why the validation?

We needed to change our practices so we needed instruments to support us to use authentic assessment to:

1. Determine children’s level of functioning
2. Write Individual Goals for Children
3. Plan Intervention
4. Monitor Performance.

Utility Study

Study Objectives

• Validation of the AEPS 3rd Edition in Spain.
• To gather feedback about the usefulness of the AEPS in Spain.

Instrument: AEPS 3rd Edition Utility Survey. The questions address 3 areas of feedback:

1. Clarity and ease of scoring;
2. Function and clarity of items and criteria;
3. Usefulness of the AEPS for its intended purposes.

Method - Participants

The present study included a total of 16 Spanish providers:

• 14 female
• 2 male
• 2 Teachers
• 1 Teacher Assistant
• 13 Early Intervention/ Early Childhood Special Education Specialists.

Data Analysis

AEPS is easily administered in authentic environments (e.g., child’s classroom, home, or in any other natural environment).

Results - Continued

| | Media | Desviación Típica | Mínimo | Máximo | | 25th percentile | 50th percentile | 75th percentile | | --- | --- | --- | --- | --- | --- | --- | --- | | V2 | 3.563 | 0.512 | 3.000 | 4.000 | | 3.000 | 4.000 | 4.000 |

• AEPS items provide useful information for summarizing individual child strengths and writing present levels of development.

Conclusions:

Usefulness of AEPS for its intended purposes. We revised scoring criteria to reflect culturally relevant experiences and norms. Examine each AEPS item and change the content to ensure our cultural appropriateness for the community. For example, when eating, the children cannot choose the food and they can’t bring the food from home. In Spain, the culture has an oral tradition of sharing information rather than relying on written documents, so the participants said, "we need many times to complete the scale."

Next Steps

Use the scale in many schools with different providers and collect more data.

Discussion

• International research, such as this, requires collaboration at many levels including between researchers and with other entities (e.g., universities, ministry of education, NGOs).
• Although authors attempted to develop a "universally designed" assessment, the current research shows that it is impossible to use an assessment developed in the U.S. in other countries without adaptations (aside from just translation).
• The potential for AEPS-3 to support the development of early intervention systems or promote the use of recommended practices within existing systems in other countries is promising.