Technical Appendix

2012 ADDENDUM TO THE TECHNICAL APPENDIX

Since the publication of the ELLCO Pre-K in 2008, psychometric analyses of the ELLCO Pre-K were performed using data collected from 2008 through 2010 as part of a U.S. Department of Education–funded Early Reading First project: Reading to Nurture Excellence in Worcester (RENEW). ELLCO Pre-K observations were conducted twice annually in 35 classrooms over the course of 3 years. Due to teacher turnover, the total number of classroom observations conducted during this 3-year period equals 203. The RENEW project, and Early Reading First projects in general, are concerned with improving language and literacy outcomes for disadvantaged children. The data in these analyses were therefore collected in classrooms that serve a low-income, at-risk population.

Interrater Reliability

Research use of the ELLCO Pre-K is predicated on the appropriate training of observers. Observers are selected based on knowledge of early childhood classrooms and experience in conducting observations. Prospective observers participate in a daylong training session on using the ELLCO Pre-K, which includes background information on language and literacy development, explanation of how to use the instrument, and scoring practice using written examples and video footage. Observers then participate in a second day of training that consists of supervised practice using the tool in a classroom setting followed by independent scoring and discussion of scoring decisions. When observers are trained and supervised appropriately, we have achieved an average interrater reliability of 74%.

Table A.1. Descriptive statistics for the ELLCO Pre-K (n 203)

General Statistics

The ELLCO Pre-K comprises five sections: Classroom Structure, Curriculum, The Language Environment, Books and Book Reading, and Print and Early Writing. These five sections are grouped into two main subscales: the General Classroom Environment subscale, which consists of the Classroom Structure and Curriculum sections, and the Language and Literacy subscale, which comprises The Language Environment, Books and Book Reading, and Print and Early Writing sections. Table A.1 reports descriptive statistics for ELLCO Pre-K data gathered as part of the RENEW project (n 203).

Reliability Analysis

Reliability analysis was conducted to examine the internal consistency of the ELLCO Pre-K. Table A.2 shows alphas obtained for the five ELLCO Pre-K sections as well as for the two subscales. Cronbach’s alphas for the five sections were high, ranging from .723 for the Curriculum section to .894 for the Print and Early Writing section. Item-total correlations for each section were moderate to high and ranged as follows: Classroom Structure section from .519 for Item 4, Personnel, to .657 for Item 1, Organization of the Classroom; Curriculum section from .487 for Item 7, Recognizing Diversity in the Classroom, to .610 for Item 5, Approaches to Curriculum; The Language Environment section from .492 for Item 11, Phonological Awareness, to .645 for Item 9, Opportunities for Extended Conversations; Books and Book Reading section from .623 for Item 13, Characteristics of Books, to .795 for Item 15, Approaches to Book Reading; Print and Early Writing section from .728 for Item 19, Environmental Print, to .861 for Item 17, Early Writing Environment. Cronbach’s alpha of .864 for the General Classroom Environment subscale.

Table A.2. Cronbach’s alpha for the five sections and two subscales that compose the ELLCO Pre-K (n 203)

Measuring Stability and Change

Using data collected from intervention and comparison group classrooms from the RENEW project, we report on the ability of the ELLCO Pre-K to measure both stability and change over time (see Table A.3). The intervention group consisted of Head Start teachers from the Worcester Community Action Council in Worcester, Massachusetts, who engaged in an intensive professional development process designed to improve language and literacy instruction and classroom environments. The comparison group was composed of Head Start teachers from nearby communities who were not engaged in a professional development intervention. In both groups, classrooms were observed biannually, in the fall and in the spring, over the course of 3 years. The pre- and postobservations for each year were paired to create a dataset of 128 fall and spring cases (69 intervention cases, and 59 comparison cases).

Although means for RENEW classrooms were consistently higher than those for comparison classrooms, comparison means remained relatively stable from fall to spring; difference-of-means analysis (t-tests) indicate a statistically significant difference between the fall and spring scores for RENEW classrooms across all sections and subscales of the ELLCO Pre-K. The lack of statistically significant difference between fall and spring scores for the comparison group illustrates the stability of the ELLCO Pre-K and is a good indicator of the tool’s test–retest reliability. Conversely, the statistically significant difference between fall and spring scores for the RENEW intervention classrooms illustrates the intervention's effectiveness.

Table A.3. Stability and change scores: mean (standard deviation) for the ELLCO Pre-K (n 69 RENEW; 59 comparison)

ELLCO Pre-K is both stable and sensitive to interventions that target literacy in ways that are consistent with its assumptions about what constitutes appropriate early literacy practices.

2008 TECHNICAL APPENDIX

This technical appendix reports data that were collected from 1997 to 2002 as part of the development of the ELLCO Toolkit, Research Edition, as well as additional data that were collected from 2002 to 2007 using the Research Edition. As described in Chapter 1, on the basis of this data, along with feedback from the field, we made numerous changes that serve to make the ELLCO Pre-K easier to use and score. The most significant changes are the integration of the Literacy Environment Checklist and the Literacy Activities Rating Scale into the architecture of the observation, and the inclusion of detailed descriptive indicators for each of the five scale points. Specific psychometric analyses on the current ELLCO Pre-K will be reported as the tool is used. For reasons that we outline at the end of this appendix, however, we believe that the ELLCO Pre-K will prove to be as reliable, if not more reliable, than the Research Edition.

PSYCHOMETRIC PROPERTIES OF THE LITERACY ENVIRONMENT CHECKLIST

The psychometric properties presented for the Literacy Environment Checklist (ELLCO Toolkit, Research Edition) are based on data from Year 4 of the NEQRC project combined with data from Years 1–3 of the LEEP project. Data from the NEQRC project were collected during the winter of 1998–1999 (n 29). The data from Year 1 of the LEEP project were collected in the fall of 1998 (n 26) and the spring of 1999 (n 26). Data from Year 2 of the LEEP project were collected in the fall of 1999 (n 42) and spring of 2000 (n 38). Data from Year 3 of the LEEP project were collected in the fall of 2000 (n 47) and spring of 2001 (n 47). Together, the projects resulted in a total sample size of 255, although the actual subsample sizes vary depending on the analyses conducted. Many of the classrooms included were in Head Start programs. Unlike the Classroom Observation, the Literacy Environment Checklist and the Literacy Activities Rating Scale have been used for research only in preschool classrooms and were designed specifically to help identify the impact of our literacy intervention in those classrooms. They have not been used to predict children’s growth; rather, they have been used in conjunction with the Classroom Observation to pinpoint the specific effects of a literacy intervention.

Measuring Stability and Change

Using the data collected from the LEEP classrooms, we reported preliminary findings on the ability of the Literacy Environment Checklist to measure both stability and change over time (see Table A.6). When one looks at mean scores across the 3 years of the LEEP project, the fall scores of the intervention are slightly higher on the three dimensions of the Literacy Environment Checklist than the comparison group. (For the fall scores in the LEEP study, differences in means were statistically significant for the Writing subtotal only; t = -2.62, p < .05.) In the spring, the comparison group showed significant change on the Total score as well as on the Books subtotal yet remained stable on the Writing subtotal. As hoped, the intervention group scores changed significantly from fall to spring in all categories. These changes resulted in intervention group scores that were statistically significantly different from the comparison group scores in every category and statistically significantly different from the intervention group fall scores in every category.

Table A.6. Stability and change in Literacy Environment Checklist scores (ELLCO Toolkit, Research Edition), fall and spring means, for Years 1–3 of the Literacy Environment Enrichment Program (LEEP)

PSYCHOMETRIC PROPERTIES OF THE CLASSROOM OBSERVATION

Like the other parts of the ELLCO Toolkit, Research Edition, the Classroom Observation has been used for research for the NEQRC and LEEP. The Classroom Observation also has been used as a part of a school improvement project in the Philadelphia public school system in classrooms that range from kindergarten through grade 3. It has also been introduced to school systems in Connecticut and Maine. In these settings, it is being used both to collect data on and to provide a basis for discussions about classroom quality. The psychometric properties presented in the sections that follow come from various analyses of data from Year 4 of the NEQRC research project combined with data collected from Years 1–3 of the LEEP project. Data from the NEQRC project were collected during the winter of 1998–1999 (n 29). The data from Year 1 of the LEEP project were collected in the fall of 1998 (n 27) and the spring of 1999 (n 27). Data from Year 2 of the LEEP project were collected in the fall of 1999 (n 42) and spring of 2000 (n 38). Data from Year 3 of the LEEP project were collected in the fall of 2000 and spring of 2001 in New England (fall: n 34; spring: n 37) and North Carolina (fall: n 37; spring: n 37). Together, the projects resulted in a total sample size of 308 classrooms, though the actual subscale size varies depending on the analyses conducted. As with the other parts of the ELLCO Toolkit, the data reported here for the Classroom Observation come from centers and classrooms in lower income communities.