Rubrics

Overview

This review focused specifically on alignment to the Common Core State Standards (CCSS) in mathematics and English Language Arts. As Washington’s newly adopted state learning standards in these subjects (July 2011), the CCSS represent a significant shift for classroom teachers’ instruction and, more significantly, in the nature and use of instructional material and resources. OSPI has myriad resources to support educators in the transition to the CCSS. These can be found on the OSPI CCSS website.

Through the intentional development and sequencing within the CCSS, it is critical that educators and curriculum developers consider new and existing instructional materials through a different lens when looking at their alignment with student learning standards. Traditionally, judging alignment has been approached as a crosswalking exercise. But, crosswalking can result in large percentages of “aligned content” while obscuring the fact that the materials in question do not address the spirit of the standards. As such, alignment of materials to the CCSS is emerging work. Since one comprehensive instrument does not exist, OSPI and other states recommend the combined use of several instruments designed intentionally for the CCSS by CCSS developers and state/national curriculum experts. The Washington OER review was grounded in the use of these specialized instruments:

  • Publishers’ Criteria for the Common Core State Standards (for English Language Arts, K-2 and 3-12, and Mathematics, K-8 and high school) developed by CCSS authors
  • Educators Evaluating Quality Instructional Products (EQuIP) Rubric based on the Tri-State Rubric and modified by Achieve, Inc.
  • Rubrics for Evaluating OER Objects developed by Achieve, Inc.

In some cases, we used the rubrics “as is;” in others, we discovered overlap and made adaptations to eliminate duplicated information reporting from multiple sources. In addition to the above rubrics, we added two additional review instruments:

  • CCSS worksheet
  • Overall reviewer comments

Details on each of the instruments follow and copies of all the rubrics distributed to reviewers may be found on the OSPI OER Project website.

CCSS Worksheets

These worksheets - specific to Algebra 1, Integrated Math, and ELA - listed relevant “standards clusters” for mathematics and “target standards” for ELA to verify content inclusion. Although the worksheet was not scored, it helped create a structured review of the materials. This work provided a strong foundation, supporting the completion of other rubrics which asked specific questions regarding the extent of CCSS coverage.

The “standards clusters” determined for an Algebra 1 or Integrated Math 1 course were adapted from the PARCC Model Content Framework for Mathematics since this document reflected the single course year nature of the material under review.

The “target standards” were taken directly from the reading and writing strands within the CCSS ELA document. As outlined in the OER selection criteria, writing standards 4 through 10 and select reading standards from literature or informational text were examined.

Publishers’ Criteria for the Common Core State Standards

The Council of Chief State School Officers (CCSSO), National Governors Association (NGA), and lead writers of the Common Core Standards collaborated with partner organizations, teachers, researchers and other stakeholders to create the Publishers’ Criteria documents in both mathematics and ELA. These documents guide publishers and curriculum developers in understanding what must be comprehensively covered in curricular materials in order to align with the CCSS. Additionally, they help states and districts as they evaluate instructional materials or work to modify existing resources. They provide a broad overview of curricular materials and are best used to review whole courses.

The OER review committee used the criteria to help gauge whether a resource reflected the spirit of the key instructional shifts in the CCSS. Below are the documents in their entirety:

Publishers’ Criteria for Common Core Standards in English Language Arts and Literacy, grades 3-12 (pdf)

High School Publishers’ Criteria for the Common Core State Standards for Mathematics (pdf)

In order to turn the large documents into a viable response form to collect reviewer data, the OER project made several adaptations to the narrative structure of the source documents based on input from the directors of the OSPI Mathematics and ELA departments.

  • Major criteria in the documents were identified and a 4-pt Likert scale (Strongly Disagree to Strongly Agree) was applied.
  • Mathematics: Three of eight major criteria dealt with meaningfully connecting content standards and practice standards. For the reporting instrument, they were combined.
  • Mathematics: Several criteria had subcategories. In those cases, the subcategory scores were averaged to report a score for the major criteria.
  • ELA: The reporting instrument was adapted from an existing resource used as part of Tennessee’s statewide instructional materials review process in 2012 (the Tennessee Literature Review Instrument). The only substantive change was to note that resources reviewed with the Publishers’ Criteria at the unit level may be limited in their range of texts. Reviewers were instructed to determine if the balance was appropriate for the intention of the unit.

EQuIP Rubric

Achieve is a bipartisan, non-profit organization that partnered with the CCSSO and NGA on the CCSS initiative. Achieve convened educators from a multi-state collaborative to develop the EQuIP (Educators Evaluating Quality Instructional Products) rubric to measure CCSS alignment of lessons and units. It looks at four areas, including:

  • Alignment to the rigors of the CCSS
  • Key areas of focus in the CCSS
  • Instructional supports
  • Assessment

This rubric was unchanged from its original format for this review process. Since the EQuIP rubric was not intended for full course review, only one unit in each mathematics course was reviewed using this instrument.

Achieve Rubrics for Evaluating OER

To help states, districts, teachers, and other users determine the degree of alignment of OER to the CCSS, and to determine aspects of quality of OER, Achieve developed eight rubrics in collaboration with leaders from the OER community. These rubrics provide a structure for systematically, purposefully and comprehensively evaluating an online resource.

Rubric I. Degree of Alignment to Standards
Rubric II. Quality of Explanation of the Subject Matter
Rubric III. Utility of Materials Designed to Support Teaching
Rubric IV. Quality of Assessment
Rubric V. Quality of Technological Interactivity
Rubric VI. Quality of Instructional Tasks and Practice Exercises
Rubric VII. Opportunities for Deeper Learning
Rubric VIII. Assurance of Accessibility

Though they may be used with many types of resources (from digital textbooks to videos or interactive simulations), the rubrics are also designed to be modular in nature so that resources smaller in grain size than units or lessons may still be evaluated. Rubrics which do not apply to a particular resource, since it may not have been created to address that particular purpose, may be omitted.

Although none of the rubrics was adapted for the purpose of this review, only four were used: Rubric II, V, VI, and VII. This was due to overlap with questions addressed in the EQuIP rubric. In these areas of overlap, the EQuIP rubric assessed CCSS alignment in greater depth.

Reviewer Comments

As the final step in the evaluation process, reviewers were asked to discuss the focus, coherence, rigor, and balance of the resource. Specifically, they were instructed to cite evidence from the material that reflected alignment or sections that need adaptation. Also requested was a rating of how likely they would be to use the full resource in their classroom and an estimate of how much work would be required to bring the material into alignment.