Meta-Analysis Study of the Effects of Teacher Professional Development

Meta-Analysis Study of the Effects of Teacher Professional Development
Improving Evaluation of Professional Development

Final Report

Effects of Teacher Professional Development on Gains in Student Achievement: How Meta Analysis Provides Scientific Evidence Useful to Education Leaders

This report is based on a meta analysis focused on completed studies on the effects of professional development for K-12 teachers of science and mathematics. Specifically, the study and its report addressed two questions important to education leaders:

  1. What are the effects of content-focused professional development for on improving teacher knowledge and skills and on improving student achievement?
  2. What professional development program characteristics explain the degree of effectiveness, and are the findings consistent with existing research on effective professional development, e.g., content focus, duration, coherence, active learning, and collective participation of teachers?

The meta analysis found that 16 studies reported significant effects of teacher development on improving student achievement. These studies reported effect sizes for student achievement gains for a treatment group as compared to a control group and the studies provided adequate data and documentation for the CCSSO research team to compute or re-analyze effect sizes. The large majority (12 of 16) studies were focused on analyzing mathematics teacher professional development and effects on student achievement in mathematics. CCSSO also reviewed the professional development program designs and learning goals documented in the 16 studies. Several common patterns emerged including program designs with strong emphasis on teachers learning specific subject content as well as pedagogical content for how to teach the content to students, multiple activities to provide follow-up reinforcement of learning, assistance with implementation, and support for teachers and duration of professional development activities lasting six months or more.

Based on the study, CCSSO have several recommendations for how the results and processes from the meta-analysis can be useful to researchers, evaluators, and state education leaders:

Meta Analysis Coding Forms

The document review process for the meta analysis study is aided by coding forms that coders and reconcilers complete in order to record systematically how they determined a document is included into the pool of studies to be analyzed. The systematic review is conducted with at least two coders in mind, with a third person as a reconciler. The coding forms are Excel file documents composed of multiple spreadsheets that 1) assist in determining whether a document in question is a candidate to be included in the analysis and 2) aid in the extraction of key data needed for the analysis. Each coder completes one form per document independently from his/her partner coder. The reconciler completes another similar form that combines the information from both coders by bringing their entered information side-by-side. Below are two types of coding forms: the coding form and the reconciler form.

Meta Analysis Coder Form
Meta Analysis Reconciler Form

The reconciler form is very similar to the coder form but with the added feature of having side-by-side sections for coders’ responses.

Both forms are organized in the same manner. The first spreadsheet presents the scaffolded guide that assists a coder through the process of determining the viability of a document for entry into the pool of studies to analyze. At certain decision junctures, the coder is forced to consider whether the document should continue to the next round of reviews or should be rejected. A document could be rejected any of the decision points of the review process. Coders and the reconciler are asked to complete the subsequent spreadsheets to record the following:

  • Table 1a to record student outcome measures and constructs to determine the validity and reliability of these instruments ·
  • Table 1b to record teacher outcome measures and constructs to determine the validity and reliability of these instruments ·
  • Figure 1a to document and calculate the number of teachers participating in the study, by treatment and control groups ·
  • Figure 1b to document and calculate the number of students participating in the study by treatment and control groups ·
  • Table 2 to log number of teachers matching various teacher characteristics such as education background, teaching experience, and results from outcome measures mentioned in Table 1b, by treatment and control groups ·
  • Table 3 to log number of students matching various student characteristics including grade level, gender, disability, poverty level, race/ethnicity, urbanicity, classroom type, achievement level and results from outcome measures mentioned in Table 1a, by treatment and control groups ·
  • Table 4 to record characteristics of the professional development initiatives, and ·
  • Tables 5a, b, c, and/or d to record and calculate estimates of treatment effects (effect sizes) by student outcomes using standard formulas. Note that cases arise where non-standard formulas not provided in these tables must be used to calculate effect sizes.

The meta analysis coder and reconciler forms are modified from the coding form and reconciliation forms designed by Kwang Suk Yoon  of the American Institutes for Research AIR in a systematic document review they conducted on how teacher professional development affects student achievement.