Scoring


Back: Criteria | Page 1 of 4 | Next: Study: Avoiding Bias

Study

Scoring

You will score each of the 54 criteria against the evidence provided. The critical questions you created will help guide you in understanding how to score each piece of evidence to ensure that the full criteria are addressed. Please read the initial review process and scoring information to ensure your understanding of the process, as well as how the critical questions and the following Scoring Protocol will assist you in analyzing the provided evidence.

Understanding the Scoring Protocol

In order to provide a consistent review and valuable feedback to providers, this scoring protocol will help you determine how many points to score a given criterion based on the evidence provided.

For each of the criteria review the evidence as follows:

  1. Is this a demo-reliant criterion?
    1. If yes, open course demos and go to Step 2.
    2. If no, go to Step 3.
  2. Did the provider give you a location in the demo to locate the specific criterion?
    1. If yes, find the location specified in the demo and go to Step 5.
    2. If no, go to Step 3.

    Note: provider cannot receive a full point if evidence is not found in demo.

  3. Is there supporting evidence submitted by the provider?
    1. If yes, go to Step 5.
    2. If no, go to Step 4 or mark as zero (0) and note in comments that 'No evidence was submitted.'
  4. Did you find the evidence elsewhere?
    1. If yes, go to Step 5.
    2. If no, mark as zero (0) and note in comments that "Looked at provided documentation and was unable to locate evidence. Provider failed to specify location of evidence relevant to this criterion."
  5. Using the criterion-level rubric we generated during the reviewer training, analyze the supplied evidence (including the demo courses and any additional narrative or documentation supplied for the criterion) against the criterion as follows:
    1. All points of criterion are met in the provided evidence. Score full point (1) and provide comments of where evidence was found if location was not specified by provider in comments.
    2. For demo-reliant criteria, evidence was found within an actual full course demo (i.e., not in screen shots, or in the submitted evidence).

    3. Evidence only supports a portion of the criterion. Score half point (.5) and provide comments outlining which portions of the criterion were met and which parts of the criterion were not met by the evidence provided.
    4. For demo-reliant criteria, evidence was found in other sources, but not directly in the demo.

    5. Evidence does not support the criterion. Score zero (0) and provide comments outlining why the evidence provided did not meet the criterion; be specific.

Back: Criteria | Page 1 of 4 | Next: Study: Avoiding Bias