Review Instruments and Rubrics
This review focused specifically on alignment to the Common Core State Standards (CCSS) in mathematics and English Language Arts. As Washington’s adopted state learning standards in these subjects (July 2011), the CCSS represent a significant shift for classroom teachers’ instruction and, more significantly, in the nature and use of instructional material and resources. OSPI has myriad resources to support educators in the transition to the CCSS. These can be found on the OSPI CCSS website.
Through the intentional development and sequencing within the CCSS, it is critical that educators and curriculum developers consider new and existing instructional materials through a different lens when looking at their alignment with student learning standards. Traditionally, judging alignment has been approached as a crosswalking exercise. However, crosswalking can result in large percentages of “aligned content” while obscuring the fact that the materials in question do not address the spirit of the standards. As such, alignment of materials to the CCSS is emerging work. Since one comprehensive instrument does not exist, OSPI recommends the combined use of several instruments designed intentionally for the CCSS by CCSS developers and state/national curriculum experts. The Washington OER review was grounded in the use of these specialized instruments:
- Instructional Materials Evaluation Tool (IMET) for high school mathematics (developed by Student Achievement Partners)
- Educators Evaluating Quality Instructional Products (EQuIP) Rubric (based on the TriState Rubric and modified by Achieve, Inc.)
- Rubrics for Evaluating OER Objects (developed by Achieve, Inc.)
In some cases, we used the rubrics “as is;” in others, we discovered overlap and made adaptations to eliminate duplicated information reporting from multiple sources. In addition to the above rubrics, we added two additional review instruments:
- CCSS Worksheet
- Overall Reviewer Comments
Details on each of the instruments follow and copies of all the rubrics distributed to reviewers may be found on the OSPI OER Project website.
These worksheets—specific to Geometry, Integrated Math 2, and ELA—listed relevant “standards clusters” for mathematics and “target standards” for ELA to verify content inclusion. Although the worksheet was not scored, it helped create a structured review of the materials. This work provided a strong foundation, supporting the completion of other rubrics that asked specific questions regarding the extent of CCSS coverage.
The Course Focus documents for a Geometry or Integrated Math 2 course were used to provide guidance on course content and focus for the material under review.
The “target standards” for ELA units were taken directly from the reading and writing strands within the CCSS ELA document. As outlined in the OER selection criteria, writing standards 1 through 10 and select reading standards from literature or informational text were examined.
Instructional Materials Evaluation Tool (IMET)
The Instructional Materials Evaluation Tool (IMET) is a resource used to evaluate a comprehensive textbook or textbook series for alignment to the Common Core State Standards (CCSS) in mathematics and English Language Arts/literacy. There are separate versions for use with ELA/literacy materials for grades K–2 or 3–12 and for mathematics materials K–8 or high school.
Student Achievement Partners, a non-profit organization founded by some of the lead writers of the Common Core State Standards to support CCSS implementation, created this rubric in collaboration with many state and education organization partners. IMET is based on the Publishers’ Criteria generated by Council of Chief State School Officers (CCSSO) and National Governors Association (NGA) in collaboration with partner organizations, teachers, researchers and other stakeholders. The Publishers’ Criteria document guides publishers and curriculum developers in understanding what must be comprehensively covered in curricular materials in order to align with the CCSS. In the 2013 OER review, the OER project adapted the narrative Publishers’ Criteria document into a reviewer scoring form.
The use of the IMET rubric represents a shift from the 2013 OER Review. The IMET rubric reorganizes the criteria from the listed order in the Publishers’ Criteria, weights different components of the criteria, and includes additional quality indicators that are referenced toward the end of the Publishers’ Criteria document. This change was based on feedback from directors of mathematics and ELA at the OSPI and the increasing use of IMET in textbook adoption committees across states who have adopted CCSS. For instance, the IMET was recently used in the Louisiana 2013-2014 math and ELA instructional materials review.
Read the math IMET document in its entirety: IMET Mathematics High School (pdf)
There are several important points to note regarding the IMET rubric:
- The IMET was designed for evaluation of comprehensive materials and is not appropriate for evaluating supplemental materials. Because of the nature of this tool, the WA OER review only used IMET with full-course math resources and not with the individual ELA units.
- The IMET was written for use by district adoption committees. Certain criteria are “non-negotiable” and if not met, IMET suggests stopping the review process. Since the purpose of the OER review is to provide feedback for improvement/adaptation, we did not want to make any items “non-negotiable” and stop reviewing. Therefore, we used a 0–3 point Likert scale to rate each element of the rubric.
Achieve is a bipartisan, non-profit organization that partnered with the CCSSO and NGA on the CCSS initiative. Achieve convened educators from a multi-state collaborative to develop the EQuIP (Educators Evaluating Quality Instructional Products) rubric to measure CCSS alignment of lessons and units. Washington teachers and content experts were part of this group. It looks at four areas, including:
- Alignment to the rigors of the CCSS
- Key shifts in the CCSS
- Instructional supports
View the complete EQuIP rubrics for both math and ELA.
This rubric was unchanged from its original format for this review process. Since the EQuIP rubric was not intended for full course review, only one unit in each mathematics course was reviewed using this instrument.
Achieve Rubrics for Evaluating OER
To help states, districts, teachers, and other users determine the degree of alignment of OER to the CCSS, and to determine aspects of quality of OER, Achieve developed eight rubrics in collaboration with leaders from the OER community. These rubrics provide a structure for systematically, purposefully, and comprehensively evaluating an online resource.
Rubric I. Degree of Alignment to Standards
Rubric II. Quality of Explanation of the Subject Matter
Rubric III. Utility of Materials Designed to Support Teaching
Rubric IV. Quality of Assessment
Rubric V. Quality of Technological Interactivity
Rubric VI. Quality of Instructional Tasks and Practice Exercises
Rubric VII. Opportunities for Deeper Learning
Rubric VIII. Assurance of Accessibility
Though they may be used with many types of resources (from digital textbooks to videos or interactive simulations), the rubrics are also designed to be modular in nature so that resources smaller in grain size than units or lessons may be evaluated. Rubrics that do not apply to a particular resource, since it may not have been created to address that particular purpose, may be omitted.
Although none of the rubrics was adapted for the purpose of this review, only four were used: Rubric II, V, VI, and VII. This was due to overlap with questions addressed in the EQuIP rubric. In these areas of overlap, the EQuIP rubric assessed CCSS alignment in greater depth.
We did not use Rubric V with ELA units. The intent of this rubric is to measure interactivity and how the resource responds to the user, not just technology in general. Thus, a simulation that could be manipulated to view variable outcomes would qualify as interactive but merely opening a PDF file would not. In some of the ELA resources, there were multimedia elements like video, graphics collections, and websites, but none of these seemed to meet the criteria of “technological interactivity” described by the rubric.
As the final step in the evaluation process, reviewers were asked to discuss the focus, coherence, rigor, and balance of the resource. They were instructed to cite evidence from the resource that supported their comments about areas needing adaptation. Additionally, they were asked to provide suggestions for changes that would help improve alignment.
Reviewers clarified the” ideal use” scenario for each reviewed resource and estimated the amount of work that would be required for a small group to make adaptations to bring the resource into CCSS alignment. They also selected the ways they would use the resource in both its current and adapted form from the options below:
- Textbook replacement
- Unit replacement
- Portion of unit
- Supplemental material
- Would not use
Changes from 2013 Process
Based on reviewer and facilitator feedback from the 2013 review and newly available review instruments, we made several changes to the 2014 review process:
- Used IMET instead of adapted Publishers’ Criteria instrument to evaluate full-course resources.
- Did not use IMET with ELA review as it did not meet the full-course requirement.
- Did not use Achieve Rubric V with ELA since these units did not utilize interactive elements as defined by the rubric.
- Added more structure to the Reviewer Comments feedback form to provide more consistent user responses.
- Asked reviewers to find evidence of all their noted strengths/challenges and provide suggestions as well as identify ideal usage scenarios.
This report is licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License.