Math Findings

General Observations

This is the first review of mathematics OER materials performed by OSPI. For this initial process, seven mathematics courses were reviewed. Six of the resources were Algebra 1, and one was Integrated Math 1.

DeveloperFull TitleShort TitleType
Georgia Virtual LearningCCGPS Algebra 1Georgia VirtualAlgebra 1
CK-12 FoundationCK-12 Algebra 1, 2nd Ed.CK-12Algebra 1
CurrikiCurriki Algebra 1CurrikiAlgebra 1
Mathematics Vision ProjectIntegrated Math 1MVPIntegrated Math 1
Monterey Institute/ NROCNROC Algebra 1 – Open TextbookNROCAlgebra 1
Open High School of UtahOpen High School of Utah Algebra 1(A)Utah OpenAlgebra 1
Saylor.orgSaylor Algebra 1SaylorAlgebra 1

The CCSS in mathematics are very different than previous K-12 state learning standards. In particular, there are several key shifts:

  1. Focus: focus strongly where the standards focus
  2. Coherence: think across grades and link to major topics within grades
  3. Rigor: in major topics, pursue conceptual understanding, procedural skill and fluency, and application with equal intensity

See www.achievethecore.org for more information.

The materials were reviewed with a specific goal of not evaluating their quality by existing standards but looking at how well they address CCSS shifts. Like most of the currently available commercial textbooks, many of these OER materials were not designed specifically with the CCSS in mind. Thus, the review process examined materials against target standards that developers were not originally aiming for at material creation. Additionally, some of the products reviewed were not considered complete by the developers but were provided, at OSPI request, to participate in the review process.

Overall, the findings were that CCSS aligned OER are emerging platforms for mathematics curricula. They show significant promise as a viable selection in the future but are still evolving and not yet mature. This parallels the current state of much published CCSS aligned material. When all rubrics were combined MVP scored favorably, with reviewers stating they would use it in their classroom. Curriki shows promise, but didn’t garner quite the same level of desire to be used in the classroom. For the most part, the other products showed potential in some areas, but their comprehensive scores were lower, and a majority of the reviewers did not recommend the full course for use.

It is important to note that this review process was not intended to rank or endorse the materials reviewed. As such, there are few comparative graphs in this report. It is also important to note that the materials reviewed are not the only OER resources available –others exist. The OER mathematics review process was limited in scope and solely examined seven full-courses in Algebra 1 or Integrated Math 1. This review should be viewed as a gap analysis and as an opportunity to provide input on the changes necessary to bring these OER resources into closer alignment with the CCSS.

Finally, this review process represents a point in time. More so than print materials, digital resources with an open license can be freely modified, so all the products that were reviewed can be and are frequently updated.

Publishers’ Criteria

The Publishers’ Criteria rubric examined six areas using a scale from “Strongly Disagree” to “Strongly Agree”:

  • Focus – achieving greater mastery of a smaller set of prerequisites versus shallow exposure to a wide variety of topics
  • Rigor & Balance – pursuing with equal intensity conceptual understanding, procedural skill and fluency, and applications
  • Consistent Content – ensuring that materials are based on content specified in the CCSS, that students have extensive work with course-level problems (versus review of previous-grade materials), and relating concepts explicitly to prior knowledge
  • Coherent Connections – making mathematics make sense, where powerful knowledge results from reasoning with a small number of principles
  • Practice Standards – averaged three sections of the Publishers’ Criteria review instrument:
    • Practice-Content Connections - meaningfully connecting content standards and Standards for Mathematical Practice (Practice Standards)
    • Focus and Coherence via Practice Standards – promoting focus and coherence by not connecting practice standards with content in a mechanistic or random way
    • Careful Attention to each Practice Standard – attending to the full meaning of each practice standard
  • Emphasis on Mathematical Reasoning – prompting students to construct viable arguments, engaging students in problem solving, and explicitly attending to the specialized language of mathematics.

The Publishers’ Criteria rubric measures broad alignment to the intent of the CCSS. It addresses the big shifts in thinking, focus, and rigor, and attends to effective Standards for Mathematical Practice. While overall average ratings trended towards the midpoint of the scoring range, many reviewers noted that the vast majority of traditional textbooks would likely have similar scores when compared to CCSS, because most textbooks have not yet been re-written to address the fundamental shifts in teaching and learning inherent in the CCSS.

Publishers' Criteria

Figure 1. Average Publishers' Criteria ratings for all resources combined.

EQuIP Rubric

The EQuIP rubric measures overall quality of alignment to the CCSS by examining a single unit from the full course in depth. Reviewers considered four areas using two alignment scales described below. The four areas considered included:

  • Alignment to the Rigors of the CCSS – the unit targets a set of grade level mathematics standards; Standards for Mathematical Practice that are central to the lesson are identified, and the unit presents a balance of procedures and conceptual understanding inherent in the CCSS.
  • Key Areas of Focus in the CCSS – the unit reflects evidence of key shifts in focus, coherence and rigor.
  • Instructional Supports – the unit is responsive to varied student learning needs, provides guidance to support teaching and learning of the targeted standards, and provides appropriate level and type of scaffolding, differentiation, intervention, and support for a broad range of learners.
  • Assessment – the unit regularly assesses whether students are mastering standards-based content and skills through direct, observable evidence, via accessible and unbiased methods.

The rubric also provides an overall assessment rating for the resource based upon the four domains listed above. One unit from each mathematics resource was chosen to review with this instrument. The units all covered the same topical area (Linear and Exponential Functions).

EQuIP
Figure 2. Average EQuIP ratings for all resources combined.

Note that in the graph, two measurement scales are used. The bottom axis shows the scale values for the four areas of focus, and the top axis shows the ratings for the Overall recommendation for the resources. EQuIP is designed to be used at the resource unit level, rather than the full course, to get a more detailed picture of the resource quality. Overall results for all seven resources averaged below the midpoints of the scales. The Overall rating for all of the products ranged from Not Ready for Review/Not Recommended to Exemplar, with a majority of the products receiving an Overall rating of Not Recommended (20 of 35 total responses) or Needs Revision (10 of 35). For many of the resources that were evaluated, the Assessment scale showed a lower average score than others. Reviewer comments indicated that many of the products under review had few or no assessment components.

Achieve OER Rubrics

The Achieve OER rubrics are specifically designed to be used with digital resources as opposed to print media. They also examine other aspects of OER quality, may be used with any standards, and are designed to evaluate resources that may be smaller in grain size than units or lessons.

The Achieve instrument has eight different smaller rubrics, several of which significantly overlap the EQuIP instrument. Since the EQuIP instrument was developed specifically to consider alignment to the CCSS, it was used in this review in lieu of the overlapping Achieve OER rubrics in order to minimize duplicative measurement scales. The four Achieve rubrics used for this review process are:

  • Rubric II. Quality of Explanation of the Subject Matter
  • Rubric V. Quality of Technological Interactivity
  • Rubric VI. Quality of Instructional Tasks and Practice Exercises
  • Rubric VII. Opportunities for Deeper Learning

Achieve OER
Figure 3. Average Achieve OER ratings for all resources.

Initially, there was wide variance in the Interactivity rubric. We discovered that reviewers had different opinions about what constituted technological interactivity. Some individuals felt that opening a PDF or a web content link was sufficient enough to deem the resource technologically interactive. However, when the reviewers were provided additional guidance using the definition of technological interactivity according to the Achieve OER rubric, the scores in this section normalized, and variance was reduced.

REVIEWER COMMENTS

Reviewers were asked to write a short narrative providing an assessment of each of the resources they reviewed. As part of their professional assessments, they were asked to identify the amount of work they felt was necessary to bring the product into alignment with the CCSS. They were also asked to identify their level of agreement with the statement, “I would use this resource in my classroom.” The overall results are shown in Figure 4.

Reviewer Comments
Figure 4. Overall results for Reviewer Comments, showing averages for all resources combined.

While the intent of this report is not to compare or rank the products based upon their overall average scores, comparing the performance of the resources on certain scales or items provides meaningful information. The charts that follow show how the resources compared with each other based upon selected scales or items.

I would use this in my classroom
Figure 5. Reviewer Comments. Note that reviewers indicated they would use MVP and Curriki in their classrooms. It is also important to note that only the open source NROC book was reviewed. NROC has a much more robust product offered through Hippocampus.org, but it did not meet all the criteria for consideration in the review.

Amount of Alignment Work
Figure 6. Reviewer Comments. This item measures the amount of work necessary to complete the alignment of the resource to the CCSS. It does not reflect the degree of alignment, which may be different. This is a reverse scale item. A longer bar is better, in that less work is necessary to bring the course into alignment with the CCSS.

Alignment
Figure 7. EQuIP. This scale looks at the alignment of a selected unit in the materials to the CCSS.

Assessment
Figure 8. EQuIP. Curriki and Georgia Virtual Academy demonstrated many of the criteria identified in the assessment scale of the EQuIP rubric.

Rigor & Balance
Figure 9. Publishers' Criteria. This scale measures whether the materials pursue with equal intensity conceptual understanding, procedural skill and fluency, and applications. MVP was designed from the ground up to be more aligned with shifts in thinking, including rigor and balance.

Deeper Learning
Figure 10. Achieve OER. This scale measures the unit’s ability to engage learners in one or deeper learning skills, including think critically and solve complex problems, reason abstractly, construct viable arguments and apply discrete knowledge and skills to real-world situations.

Quality of Explanation of Subject Matter
Figure 11. Achieve OER. The quality of explanation of subject matter is in the mid-range for most products. This is an area where Curriki and CK12 performed well in comparison to other resources.

Quality of Technological Interactivity
Figure 12. Achieve OER. One of the true benefits of an OER is the ability to leverage technological interactivity. Georgia Virtual and Utah Open Learning had the highest average scores on this scale, while NROC and MVP did not offer any interactive components. Note that opening PDF files or web content does not constitute technological interactivity.

Detailed Findings

For detailed information on each reviewed mathematics resource, including scores on all rubrics, extensive reviewer comments, and supplemental metadata, visit the OSPI OER Project Materials Review website.