## Math Findings

#### General Observations

This is the third review of mathematics OER materials performed by OSPI. For this review cycle, ten mathematics courses were reviewed. Four of the curricula were for sixth grade, three for seventh grade, and three for eighth grade.

Developer | Full Title | Short Title |
---|---|---|

EngageNY | Grade 6 Mathematics | EngageNY Gr 6 Math |

Federal Way Public Schools Internet Academy | 6th Grade Common Core Math | FWPS 6th Gr CC Math |

Georgia Virtual Learning | MS Math 6th Grade | GVL Math 6th Gr |

Saylor.org Academy | Math Grade 6 | Saylor Math Gr 6 |

CK–12 | Middle School Math Concepts–Grade 7 | CK-12 Concepts 7 |

EngageNY | Grade 7 Mathematics | EngageNY Gr 7 Math |

Utah Middle School Math Project | Middle School Math 7th Grade | Utah 7th Gr Math |

CK–12 | Middle School Math Concepts–Grade 8 | CK-12 Concepts 8 |

Georgia Virtual Learning | MS Math 8th Grade | GVL Math 8th Gr |

Utah Middle School Math Project | Middle School Math 8th Grade | Utah 8th Gr Math |

The materials were reviewed with a specific goal of looking at how well they address CCSS shifts, rather than evaluating their quality by previous standards. The CCSS in mathematics are very different from previous K–12 state learning standards. In particular, there are several key shifts:

- Focus: focus strongly where the standards focus
- Coherence: think across grades and link to major topics within grades
- Rigor: in major topics, pursue conceptual understanding, procedural skill and fluency, and application with equal intensity

Reviewers found that four of the reviewed resources (EngageNY Grade 6 and 7; Utah Middle School Math Project Math 7 and 8) show significant promise as a viable selection now and several more could be considered with adaptation. These four mathematics resources consistently received an overall average score of 2 or higher (on a 0–3 point scale) across most criteria. For the most part, the other products showed potential in some areas, but their comprehensive scores were lower, and none of the reviewers recommended the full courses for use, although their use as supplemental material or a portion of a unit was well documented.

It is important to note that this review process was not intended to rank or endorse the materials reviewed. As such, there are few comparative graphs in this report. It is also important to note that the materials reviewed are not the only OER resources available—others exist. The OER mathematics review process was limited in scope and solely examined ten full-courses in middle school mathematics. This review should be viewed as a gap analysis and as an opportunity to provide input on the changes necessary to bring these OER resources into closer alignment with the CCSS.

Finally, this review process represents a point in time. More so than print materials, digital resources with an open license can be freely modified, so all the products that were reviewed can be and are frequently updated.

#### IMET Rubric

The Instructional Materials Evaluation Tool (IMET) is a resource used to evaluate a comprehensive textbook or textbook series for alignment to the Common Core State Standards (CCSS). It is based on the Publishers’ Criteria documents, created to guide publishers and curriculum developers in understanding what must be comprehensively covered in curricular materials in order to align with the CCSS.

We used the IMET specific to K-8 mathematics materials. The IMET review instrument separates criteria into six sections:

**Freedom from Obstacles to Focus:**Materials must reflect the content architecture of the Standards by not assessing the topics named before the grade level where they first appear in the Standards.**Focus and Coherence:**Materials must focus coherently on the Major Work of the grade in a way that is consistent with the progressions in the Standards.**Rigor and Balance:**Materials must reflect the balances in the Standards and help students meet the Standards’ rigorous expectations.**Standards for Mathematical Practice:**Materials must demonstrate authentic connections between content Standards and practice Standards.**Access to the Standards for All Students:**Materials must provide supports for English Language Learners and other special populations.**Indicators of Quality:**Lessons are thoughtfully structured; include both problems and exercises that have a purpose and are given in an intentional sequence; teacher materials that support teacher study; manipulatives are faithful representations of the mathematical concepts; include a variety of assessments with aligned rubrics, answer keys and scoring guidelines; unbiased assessment; materials evaluated by qualified individuals; visual design supports students in engaging thoughtfully with the subject; materials engage parents in appropriate ways.

**IMET - Average Scores**

Figure 1. Average IMET ratings for all ten math resources combined – 40 total reviews.

Since the purpose of the OER review is to provide feedback for improvement, we adapted the rubric to remove the non-negotiable gatekeeper criteria and included a Likert scale from 0–3 to rate each element of the rubric (Strongly Disagree – 0, Disagree – 1, Agree – 2, Strongly Agree – 3).

- When averaged, all categories were near the scale midpoint; however, four resources scored in the Agree to Strongly Agree range across the board – EngageNY Grade 6 and 7, Middle School Math Project Grade 7 and 8 (fig 1).
- These same four resources demonstrated authentic connections between the standards for mathematical practice and the content standards (fig 11).
- Though the two EngageNY curricula did provide supports for English Language Learners and other special populations, this access for all students was a deficiency in the other reviewed resources (fig 12).

The comparison chart below shows IMET averages when the four highly aligned resources are split out. We suggest that there are may be several reasons for the differences in scores. Two of the developers we examined, Georgia Virtual Learning and Saylor.org Academy, were specifically designing online courses. These are best used as self-directed practice for students or as supplemental material by teachers in a traditional classroom. They may also work well in a homeschool or alternative learning environment. As such, they do not neatly fall into the comprehensive textbook category targeted by the IMET. Additionally, some resources were created pre-Common Core and adapted to increase alignment to the new standards. Such adaptation is most often less effective than resources created from scratch with the CCSS as guidance.

**IMET – Breakdown Scores**

Figure 2. Comparison of IMET averages when broken down into two scoring tiers

#### EQuIP Rubric

The Educators Evaluating Quality Instructional Products (EQuIP) rubric measures overall quality of alignment to the CCSS by examining a single unit from the full course in depth. One unit from each mathematics resource was chosen to review with this instrument. The units all covered the same topical area. The areas for each grade were:

**Grade 6:**Ratios and Unit Rates**Grade 7:**Ratios and Proportions**Grade 8:**Linear Functions

Reviewers considered four dimensions described below:

**Alignment to the Rigors of the CCSS:** the unit targets a set of grade level mathematics standards, Standards for Mathematical Practice that are central to the lesson are identified, and the unit presents a balance of procedures and conceptual understanding inherent in the CCSS.

**Key Shifts in the CCSS:** the unit reflects evidence of key shifts in focus, coherence, and rigor.

**Instructional Supports:** the unit is responsive to varied student learning needs, provides guidance to support teaching and learning of the targeted standards, and provides appropriate level and type of scaffolding, differentiation, intervention, and support for a broad range of learners.

**Assessment:** the unit regularly assesses whether students are mastering standards-based content and skills through direct, observable evidence, via accessible and unbiased methods.

Each dimension had a number of criteria that were considered. The number of criteria for each dimension that were met was rated on a scale from 0–3 (None – 0, Few – 1, Many – 2, All – 3). The rubric also provides an **Overall** rating for the resource based upon the sum of each of four dimensions. Scores from 11–12 are considered *Exemplar*, 8–10 are *Exemplar if Improved*, 3–7 are in the *Revision Needed* category, and scores 2 and below are *Not Ready to Review*.

**EQuIP**

Figure 3. Average EQuIP ratings for all resources combined – 40 total reviews.

- Results for all ten resources averaged above the midpoints of most of the scales, trending towards “many” criteria being met (fig 3).
- Reviewers gave the following Overall evaluations:

Exemplar 3 resources Exemplar if Improved 1 resources Needs Revision 4 resources Not Ready to Review 2 resources - Five out of ten resources met Many to All of the criteria for the Key Shifts (fig 14).
- For many of the resources that were evaluated, the Assessment scale showed a lower average score than others. Reviewer comments indicated that many of the products had few or no assessment components.

As with the IMET, four resources, EngageNY Grade 6 and 7 and Utah Middle School Math Project Grade 7 and 8, met many to all of the criteria. Their scores are broken out below.

**EQuIP – Breakdown Scores**

Figure 4. Comparison of EQuIP averages when broken down into two scoring tiers

#### Achieve OER Rubrics

The Achieve OER rubrics are specifically designed to be used with digital resources as opposed to print media. They also examine other aspects of OER quality, may be used with any standards, and are designed to evaluate resources that may be smaller in grain size than units or lessons.

The Achieve instrument has eight different smaller rubrics, several of which significantly overlap the EQuIP instrument. Since the EQuIP instrument was developed specifically to consider alignment to the CCSS, it was used in this review in lieu of the overlapping Achieve OER rubrics in order to minimize duplicative measurement scales. The four Achieve rubrics used for this review process are:

- Rubric II. Quality of Explanation of the Subject Matter
- Rubric V. Quality of Technological Interactivity
- Rubric VI. Quality of Instructional Tasks and Practice Exercises
- Rubric VII. Opportunities for Deeper Learning

Each rubric was scored independently of the others using a 0–3 scale that describe levels of potential quality, usefulness, or alignment (Weak – 0, Limited – 1, Strong – 2, Superior – 3).

**Achieve OER**

Figure 5. Average Achieve OER ratings for all resources – 40 total reviews.

Of all the rubrics, Rubric V: Quality of Technological Interactivity, was the most challenging to review. In this rubric, interactivity is not defined as technology in general but rather a measure of how the object responds to the user and behaves differently based on what the user does. Resources from CK–12 and Georgia Virtual Learning scored well on this scale. Using this particular rubric with the math review posed a challenge related to grain size of the resource. While Rubric V works perfectly well with one interactive element, it is challenging to apply to a unit where there are multiple elements, with varying degrees of interactivity. To complicate matters, most often these elements were not created by the same group that developed the base curricula, instead being aggregated from multiple sources.

In the breakout chart below, you can see that although resources in the first group scored well in the other categories, they had no interactive elements. On the other hand the second group did have interactivity but was limited in deeper learning and quality of instructional tasks and exercises.

**Achieve OER – Breakdown Scores**

Figure 6. Comparison of Achieve OER averages when broken down into two scoring tiers

#### Reviewer Comments

Reviewers were asked to write a short narrative providing an assessment of each of the resources they reviewed. They were instructed to cite evidence from the resource that supported their comments about areas needing adaptation. Additionally, they provided suggestions for changes that would help improve alignment.

**Reviewer Comments**

Figure 7. Number of times out of 40 reviews that each potential use was cited.

As part of their professional assessments, reviewers clarified the “ideal use” scenario for each reviewed resource and estimated the amount of work that work that would be required for a small group to make adaptations to bring the resource into CCSS alignment. Finally, reviewers selected all the ways they would use the resource in both its current and adapted form. Below are some of the highlights, but for an in-depth look at comments for each resource, please visit the OER Project reviewed materials library.

- Out of 40 reviews, 9 stated they would use a resource as a textbook replacement “as is” in its current state. That number jumped to 16 if suggested adaptations were made.

EngageNY Grade 6 Math (1 current/2 adapted) EngageNY Grade 7 Math (3 current/4 adapted) Utah Middle School Math Project 7th Grade Math (2 current/3 adapted) Utah Middle School Math Project 8th Grade Math (3 current/3 adapted) CK-12 Math Concepts 7 (1 adapted) CK-12 Math Concepts 8 (2 adapted) Saylor.org Academy (1 adapted) - Only 3 reviewers out of 40 stated that they would not use a resource in some capacity.

While the intent of this report is not to rank the products based upon their overall average scores, comparing the performance of the resources on certain scales or items provides meaningful information. The charts that follow show how the resources compared with each other based upon selected scales or items.

**Freedom From Obstacles to Focus**

Figure 8. IMET. Materials must reflect the content architecture of the Standards by not assessing the topics named before the grade level where they first appear in the Standards.

**Focus and Coherence**

Figure 9. IMET. Materials must focus coherently on the Major Work of the grade in a way that is consistent with the progressions in the Standards.

**Rigor and Balance**

Figure 10. IMET. Materials must reflect the balances in the Standards and help students meet the Standards’ rigorous expectations

**Standards for Mathematical Practice**

Figure 11. IMET. Materials must demonstrate authentic connections between content Standards and practice Standards

**Access for All Students**

Figure 12. IMET. Materials must provide supports for English Language Learners and other special populations.

**Alignment**

Figure 13. EQuIP. This scale looks at the overall alignment of the resource to the CCSS.

**Key Shifts in the CCSS **

Figure 14. EQuIP. Evidence of key shifts reflected in the CCSS in one unit of the curriculum.

**Instructional Supports**

Figure 15. EQuIP. Examines whether a unit is responsive to varied student learning needs.

**Assessment**

Figure 16. EQuIP. EQuIP. Unit regularly assesses whether students are mastering standards-based content and skills.

**Deeper Learning**

Figure 17. Achieve OER. Measures the unit’s ability to engage learners in one or deeper learning skills, including think critically and solve complex problems, reason abstractly, construct viable arguments and apply discrete knowledge and skills to real-world situations.

**Quality of Explanation of Subject Matter**

Figure 18. Achieve OER. Rates how thoroughly the subject matter is explained or otherwise revealed in the object.

**Quality of Technological Interactivity**

Figure 19. Achieve OER. One of the true benefits of an OER is the ability to leverage technological interactivity. Note that opening PDF files or web content does not constitute technological interactivity.

#### Detailed Findings

For detailed information on each reviewed mathematics resource, including scores on all rubrics, extensive reviewer comments, and supplemental metadata, visit the OSPI OER Project Materials Review website.

This report is licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License.