Page Location:Admin > Review Process Setup > Head to Head Reviews 
Playbook Section:Review & Decide


Before using H2H for the crowd, it's important to understand that all of your participants may not have an opportunity to contribute. Here's why:

The maximum number of crowd reviewers that can participate in a challenge depends on the total number of reviews completed. As the number of pairs reviewed by a crowd reviewer increases, the number of allowable crowd reviewers decreases. To obtain a statistically accurate result, the maximum coverage for a review occurs when the (total number of reviewed pairs) is greater than (10 x number of ideas).

Setting a maximum cap on the number of crowd reviewers encourages timely participation in the review process. Without a maximum number of crowd reviewers, the Administrator is responsible for determining how many reviews are required to yield a statistically accurate result.

Recommendations for structuring your review for The Crowd are also found within the Admin Menu in Crowd Sourcing Tools & Review Team Tools. Based on the current information in the system, Innovation Central generates the number of Reviews and Reviewers. For example, in the sample below, there are 22 ideas in this challenge, Innovation Central recommends you have a minimum of 10 crowd Reviewers from The Crowd who complete a minimum of 15 pairs each. If the challenge is ever “in danger” of not achieving a statistically valid result, then a red warning displays and offers guidance on what to do next.


In the example below, if each crowd reviewer reviewed 15 pairs, then the maximum coverage allowed would be 15 reviewers.  Another calculation to estimate the maximum number of crowd reviewers is (the number of ideas x 10) / (pairs completed per crowd reviewer).  In our example, using 22 ideas and 15 pairs per crowd reviewer, the maximum allowable number of crowd reviewers is estimated to be (22 x 10) /15 = 15.