Math Findings

General Observations

This is the second review of mathematics OER materials performed by OSPI. For this review cycle, seven mathematics courses were reviewed. Five of the resources were Geometry, one was Integrated Math 2, and one was an Algebra 1 resource that was unavailable during the 2013 review cycle covering that content area.

DeveloperFull TitleShort TitleType
CK–12 Foundation CK–12 Geometry Concepts CK12 Geo Cncpts Geometry
CK–12 Foundation CK–12 Geometry Honors Concepts CK12 Hon. Geo. Cncpts Geometry
Curriki Curriki Geometry Curriki PBL Geometry Geometry
Mathematics Vision Project Secondary Two Mathematics: An Integrated Approach MVP Integrated Math 2 Integrated Math 2
EngageNY/Common Core New York State Common Core Mathematics Curriculum: Algebra 1 ENY Algebra 1 Algebra 1
EngageNY/Common Core New York State Common Core Mathematics Curriculum: Geometry (Module 1) ENY Geometry Geometry
Saylor Foundation The Saylor Foundation K–12 Geometry Course Saylor Geometry Geometry

The materials were reviewed with a specific goal of looking at how well they address CCSS shifts, rather than evaluating their quality by previous standards. The CCSS in mathematics are very different from previous K–12 state learning standards. In particular, there are several key shifts:

  1. Focus: focus strongly where the standards focus
  2. Coherence: think across grades and link to major topics within grades
  3. Rigor: in major topics, pursue conceptual understanding, procedural skill and fluency, and application with equal intensity

Overall, the findings show that the reviewed 2014 math OER curricula showed far more alignment to the CCSS than resources reviewed in 2013. Several of the reviewed resources show significant promise as a viable selection now and several more could be considered with minor-to-moderate adaptation. Four resources, EngageNY Algebra, Engage NY Geometry, MVP Integrated Math 2, and CK12 Honors Geometry Concepts, consistently received an overall average score of 2 or higher (on a 0–3 point scale) across most rubric criteria. For the most part, the other products showed potential in some areas, but their comprehensive scores were lower, and a majority of the reviewers did not recommend the full course for use, though their use as supplemental material or a portion of a unit was frequently recommended in the Reviewers’ Comments documents. As resources are being developed to address the fundamental shifts in teaching and learning inherent in the CCSS and not just re-purposed, greater alignment is being achieved.

It is important to note that this review process was not intended to rank or endorse the materials reviewed. As such, there are few comparative graphs in this report. It is also important to note that the materials reviewed are not the only OER resources available—others exist. The OER mathematics review process was limited in scope and solely examined six full-courses in Geometry or Integrated Math 2 and one Algebra 1 course. This review should be viewed as a gap analysis and as an opportunity to provide input on the changes necessary to bring these OER resources into closer alignment with the CCSS.

Finally, this review process represents a point in time. More so than print materials, digital resources with an open license can be freely modified, so all the products that were reviewed can be and are frequently updated.

IMET Rubric

The Instructional Materials Evaluation Tool (IMET) is a resource used to evaluate a comprehensive textbook or textbook series for alignment to the Common Core State Standards (CCSS). It is based on the Publishers’ Criteria documents, created to guide publishers and curriculum developers in understanding what must be comprehensively covered in curricular materials in order to align with the CCSS.

We used the IMET specific to high school mathematics materials. The IMET review instrument separates criteria into seven sections:

  1. Focus in High School: In any single course, students and teachers using the materials as designed spend the majority of their time developing knowledge and skills that are widely applicable as prerequisites for postsecondary education.
  2. Consistent, Coherent Content: Each course’s instructional materials are coherent and consistent with the content in the Standards.
  3. Rigor and Balance: Each grade’s instructional materials reflect the balances in the Standards and help students meet the Standards’ rigorous expectations, by helping students develop conceptual understanding, procedural skill and fluency, and application.
  4. Practice–Content Connections: Materials meaningfully connect the Standards for Mathematical Content and the Standards for Mathematical Practice.
  5. Standards for Mathematical Content
  6. Standards for Mathematical Practice
  7. Other Indicators of Quality: Supports the use of technology; problems or exercises have a purpose and are given in an intentional sequence; variety in pacing and grain size of content coverage, as well as variety in what students produce; separate teacher materials that support teacher study; manipulatives connected to written methods and faithful representations of the math objects they represent; materials are reviewed by qualified individuals; visual design isn’t distracting; support for special populations is thoughtful.

Since the purpose of the OER review is to provide feedback for improvement/adaptation, we adapted the rubric to include a Likert scale from 0–3 to rate each element of the rubric (Strongly Disagree – 0, Disagree – 1, Agree – 2, Strongly Agree – 3).

  • All average ratings for each category trended above the midpoint of the scoring range with some categories having average scores in the upper third of the range (figure 1). This is in contrast to the 2013 review where the majority of the categories measuring similar areas were at or below the scale midpoint.
  • Three resources scored in the Agree to Strongly Agree range for most all of the IMET categories – EngageNY Algebra 1, Engage NY Geometry, and MVP Integrated Math 2.

Publishers' Criteria

Figure 1. Average IMET ratings for all seven math combined - 31 total reviews.

EQuIP Rubric

The EQuIP rubric measures overall quality of alignment to the CCSS by examining a single unit from the full course in depth. One unit from each mathematics resource was chosen to review with this instrument. The units all covered the same topical area—Congruence/Transformations. Reviewers considered four dimensions described below:

  • Alignment to the Rigors of the CCSS: the unit targets a set of grade level mathematics standards, Standards for Mathematical Practice that are central to the lesson are identified, and the unit presents a balance of procedures and conceptual understanding inherent in the CCSS.
  • Key Shifts in the CCSS: the unit reflects evidence of key shifts in focus, coherence, and rigor.
  • Instructional Supports: the unit is responsive to varied student learning needs, provides guidance to support teaching and learning of the targeted standards, and provides appropriate level and type of scaffolding, differentiation, intervention, and support for a broad range of learners.
  • Assessment: the unit regularly assesses whether students are mastering standards-based content and skills through direct, observable evidence, via accessible and unbiased methods.

Each dimension had a number of criteria that were considered. The number of criteria for each dimension that were met was rated on a scale from 0–3 (None – 0, Few – 1, Many – 2, All – 3). The rubric also provides an Overall rating for the resource based upon the sum of each of four dimensions. Scores from 11–12 are considered Exemplar, 8–10 are Exemplar if Improved, 3–7 are in the Revision Needed category, and scores 2 and below are Not Ready to Review.

  • Results for all seven resources averaged well above the midpoints of most of the scales, trending towards “many” criteria being met (figure 2).
  • The majority of the 31 reviews had an Overall rating of Exemplar if Improved or Exemplar:
    Exemplar7 reviews
    Exemplar if Improved10 reviews
    Needs Revision12 reviews
    Not Ready to Review2 reviews
  • Average Alignment for all resources combined was 2.1, indicating that Many to All of the CCSS criteria were met. Two resources, EngageNY Algebra 1 and EngageNY Geometry, received an All Alignment criteria met score on each of their reviews.
  • On average, the reviewed resources met Many to All of the criteria for Key Shifts:
    All17 reviews
    Many11 reviews
    Some2 reviews
  • For many of the resources that were evaluated, the Assessment scale showed a lower average score than others. Reviewer comments indicated that many of the products under review had few or no assessment components. Some resources, such as MVP Integrated Math 2 do offer assessment components for a small fee.

EQuIP
Figure 2. Average EQuIP ratings for all resources combined – 31 total reviews.

Achieve OER Rubrics

The Achieve OER rubrics are specifically designed to be used with digital resources as opposed to print media. They also examine other aspects of OER quality, may be used with any standards, and are designed to evaluate resources that may be smaller in grain size than units or lessons.

The Achieve instrument has eight different smaller rubrics, several of which significantly overlap the EQuIP instrument. Since the EQuIP instrument was developed specifically to consider alignment to the CCSS, it was used in this review in lieu of the overlapping Achieve OER rubrics in order to minimize duplicative measurement scales. The four Achieve rubrics used for this review process are:

  • Rubric II. Quality of Explanation of the Subject Matter
  • Rubric V. Quality of Technological Interactivity
  • Rubric VI. Quality of Instructional Tasks and Practice Exercises
  • Rubric VII. Opportunities for Deeper Learning

Achieve OER
Figure 3. Average Achieve OER ratings for all resources – 31 total reviews.

Each rubric was scored independently of the others using a 0–3 scale that describe levels of potential quality, usefulness, or alignment (Weak – 0, Limited – 1, Strong – 2, Superior – 3).

Resources scored well on most of these rubrics, with overall averages tending to fall in the Strong or Superior category.

Of all the rubrics, Rubric V: Quality of Technological Interactivity, tended to score the lowest. In this rubric, interactivity is not defined as technology in general but rather a measure of how the object responds to the user and behaves differently based on what the user does. Resources from CK12, Curriki, and Saylor scored well on this scale. Using this particular rubric with the math review posed a challenge related to grain size of the resource. While Rubric V works perfectly well with one interactive element, it is challenging to apply to a unit where there are multiple elements, with varying degrees of interactivity. To complicate matters, most often these elements were not created by the same group that developed the base curricula, instead being aggregated from multiple sources.

Reviewer Comments

Reviewers were asked to write a short narrative providing an assessment of each of the resources they reviewed. They were instructed to cite evidence from the resource that supported their comments about areas needing adaptation. Additionally, they provided suggestions for changes that would help improve alignment.

As part of their professional assessments, reviewers clarified the “ideal use” scenario for each reviewed resource and estimated the amount of work that work that would be required for a small group to make adaptations to bring the resource into CCSS alignment. Finally, reviewers selected all the ways they would use the resource in both its current and adapted form. Below are some of the highlights, but for an in-depth look at comments for each resource, please visit the OER Project reviewed materials library.

Reviewer Comments
Figure 4. Number of times out of 31 reviews that each potential use was cited. EngageNY Geometry was removed from the Textbook Replacement question since only one module was complete at time of review.

  • Out of 31 reviews, 7 stated they would use a resource as a textbook replacement “as is” in its current state. That number jumped to 13 if suggested adaptations were made.
    Engage NY Algebra 1 (3 as is/4 adapted)
    Mathematics Vision Project (2 as is/4 adapted)
    CK12 Honors Geometry Concepts (1 as is/3 adapted)
    CK12 Geometry Concepts (1 current/2 adapted)
  • Five out of five reviewers would use EngageNY Geometry as a unit replacement in its current state. Two stated they would use the resource as a textbook replacement, however we did not count those scores as only one module was complete at time of review.
  • Only two reviews out of 31 stated that they would not use a resource in some capacity.

While the intent of this report is not to rank the products based upon their overall average scores, comparing the performance of the resources on certain scales or items provides meaningful information. The charts that follow show how the resources compared with each other based upon selected scales or items.

Focus in High School
Figure 5. IMET. Students and teachers spend the majority of their time developing knowledge and skills that are widely applicable as prerequisites for postsecondary education. ENY Geometry was not included in this category since the full course was not complete.

Consistent, Coherent Content
Figure 6. IMET. Instructional materials are coherent and consistent with the content in the Standards.

Rigor and Balance
Figure 7. IMET. Material reflects the balances in the Standards and helps students meet the Standards’ rigorous expectations by helping students develop conceptual understanding, procedural skill and fluency, and application.

Standards for Mathematical Content
Figure 8. IMET. Material is based on content specified in the Standards. They foster coherence through connections throughout the course and preserve the focus, coherence, and rigor the Standards.

Standards for Mathematical Practice
Figure 9. IMET. Materials connect practice standards with content, attend to the full meaning of all practice standards, and support the Standards emphasis on mathematical reasoning.

Alignment to the Depth of the CCSS
Figure 10. EQuIP. Alignment of a selected unit to the letter and spirit of the CCSS.

Key Shifts in the CCSS
Figure 11. EQuIP. Evidence of key shifts reflected in the CCSS in one unit of the curriculum.

Instructional Supports
Figure 12. EQuIP. Examines whether a unit is responsive to varied student learning needs.

Assessment
Figure 13. EQuIP. Unit regularly assesses whether students are mastering standards-based content and skills.

Deeper Learning
Figure 14. Achieve OER. Measures the unit’s ability to engage learners in one or deeper learning skills, including think critically and solve complex problems, reason abstractly, construct viable arguments and apply discrete knowledge and skills to real-world situations.

Quality of Explanation of Subject Matter
Figure 15. Achieve OER. Rates how thoroughly the subject matter is explained or otherwise revealed in the object.

Quality of Technological Interactivity
Figure 16. Achieve OER. One of the true benefits of an OER is the ability to leverage technological interactivity. Note that opening PDF files or web content does not constitute technological interactivity.

Detailed Findings

For detailed information on each reviewed mathematics resource, including scores on all rubrics, extensive reviewer comments, and supplemental metadata, visit the OSPI OER Project Materials Review website.