Validating the Assessment, Evaluation and Programming System (AEPS) in International Contexts

Jennifer Grisham, Ed.D.
University of KY
Lema Kabashi, Ph.D.
University of Wisconsin-LaCrosse

Overview of Assessment, Evaluation, and Programming System (AEPS)

AEPS-3 is a curriculum-based and criterion-referenced assessment, used to:

• Develop individualized goals and outcomes for children.
• Design appropriate instruction for individual children and groups of children.
• Measure whether the teaching/intervention is effective.

Assessment, Evaluation, and Programming System for Infants and Children, Third Edition (AEPS®-3), Bricker, Dionne, Grisham, Johnson, Macy, Slentz, & Waddell. © 2022 Brookes Publishing Co. All rights reserved.

Field Test of AEPS-3 in U.S.

• 36 Professionals from each field (OT, PT, SLP, Psychology, Education)

Content

• Completed survey about relatedness of items, validity and clarity of items and criteria.
• Analysis: Agreement among professional.

Utility

• 11 teachers who had used the AEPS-2.

• Participated in online training.

• Completed AEPS-3 on 2-4 children.

• Completed survey about the scoring, items, and usefulness of the items.

• Analysis: Descriptive Statistics.

• All teachers who collected data for the field test.

Interrater Reliability

• Prior to collecting data on children, trained on AEPS-3.
• Completed online training.
• Completed Inter-rater reliability test.
• Analysis: Inter-rater agreement.

Concurrent Agreement

• 50 children (25 males and 25 females).
• 8 teachers; teachers complete training and Inter-rater Reliability test.
• Teachers complete AEPS within 2 weeks and also give criterion measure - Battelle Developmental Inventory–Second Edition (BDI-2).
• Analysis: Pearson’s correlations.

Scale Evaluation and Cut Scores

200 Teachers (all completed online training and took IRR test). Minimums of 4 children for each teacher (n=874):

• Children with disabilities (52.9%) and without disabilities (47.1%).

• Analyses: Rasch Rating Scale Model, Cut-Score Development, Classification Analysis.

• 12 teachers, 2-3 children for each teacher.

Curriculum

• Teachers received training on curriculum.
• Family and teachers identified the goals for the child and identified AEPS curriculum activities/routines and strategies for teaching the goals.
• After 3 months of implementation, focus groups were held to determine validity of curriculum.
• Analysis: Qualitative.

Rationale for Validating AEPS in other Countries

• Early Intervention Systems in other countries often adhere to a medical model in terms of assessment and service delivery.
• Professionals in many countries are trying to promote recommended practices including authentic assessment, family-centered practices, and inclusion.
• Assessment and curriculum systems that promote recommended practices do not exist in many parts of the world; the AEPS is one system that does so.
• In order to establish the validity of the system, research is necessary in the culture and in the language of the country.

Field-Test of the Assessment, Evaluation, and Programming System (AEPS) for Infants and Children-3rd Edition in Kosovo

Lema Kabashi, Ph.D.

Project Significance

• Assessment instruments, existing or new, cannot be used by early childhood professionals if they do not have defensible psychometric properties addressing validity, reliability, and utility.
• Kosovo doesn’t have a criterion-referenced curriculum-based assessment that can be used with children from birth to 6 years old.
• Ultimate goal: provide Kosovo with increased sustainable resources and practices to improve early childhood education.

Funding

AAUW & Peja CARITAS Swiss EMRG Board Donations Municipality:

• €14.300 euro
• $4000
• $3000
• €800

Timeline: Preparation and Study

• Preparation: Summer 2021
• Study: October-November 2021

Key study events:

• 35 expert reviewers from different disciplines reviewed the AEPS-3 and completed the survey about the items to validate the content.
• Results: Changes were made for any items that professionals considered necessary; only one item was eliminated.

Study 1: Content Validity

Content validity refers to the degree to which an assessment measures what it is designed to measure.

Study 2: Utility

Utility refers to the usefulness of an assessment for different purposes (e.g., developing individualized education programs, monitoring progress, etc.).

Study 3: Interrater Reliability

Reliability refers to the stability and accuracy of assessment results.

• 54 teachers were trained in the AEPS-3; they administered AEPS-3 with 1-2 children & completed a survey about the utility of the AEPS-3.

Study 4: Curriculum Validity

Curriculum validity refers to the effectiveness of the test items in teaching the curriculum.

• 38 teachers were trained in how to administer AEPS-3 to both children with and without disabilities.

Study 5: Psychometrics of Test and Cut Scores

The psychometric evidence and cut score refer to an assessment accurately and reliably discriminate between typically developing children and those whose development is significantly delayed or deviant and requires intervention.

Conclusion

Usefulness of AEPS for its intended purposes:

• Revised scoring criteria reflect culturally relevant experiences and norms.
• Examine each AEPS item and change the content to ensure cultural appropriateness for the community.

Discussion

• International research requires collaboration at many levels including between researchers and with other entities.
• While attempting to develop a “universally designed” assessment, adaptations are necessary for different countries.
• The potential for AEPS-3 to support the development of early intervention systems or recommend practices in other countries is promising.

Conclusive Summary

• Assessment and intervention can adapt to various cultural and linguistic contexts to ensure efficacy and relevance.
• Future implementation of AEPS-3 should consider local conditions and cultural norms.