Interpretation of Review Results
The Ranking Table and 2×2 Charts are the primary results used when a review team is looking at the results of the H2H Reviews, comparing options and discussing recommendations.
The leftmost column shows the overall weighted score (a number from 100 to 0) colored red (“hot”) to blue (“cool”). The center columns show the scores for each idea against the various factors (the sliders in the review process). Idea author name and title complete each row; the idea can also be shown by clicking on the triangle to reveal the idea.
The sliders above the big table encourage you to explore your data. They default to equal weight across factors but itʼs easy to change by simply moving the sliders left or right. The large table recalculates the overall weighted score to reflect the choices.
Clicking any total score circle highlights it, and this highlight stays with the idea no matter how the weighting is changed.
Ranks, Scores, and Bins
Sometimes neat tables and graphs suggest that there’s more precision in data than really exists. This is a cautionary note about how the table is constructed, what the top-to-bottom order really means, and the usual intention of a review
process.
H2H reviewing results in a full top-to-bottom for a set of ideas; it is intrinsically a relative judgement. It does not actually assign a numeric score, just an ordering. We show a numeric 100-to-zero score because that helps teams identify and discuss ideas, but it is just the weighted average (according to how you set the factor sliders) of the rank order of ideas against each factor. If you had just one factor and five ideas, their scores would be 100-75-50-25-0; there is no finer granularity or interpretation.
There can always be nearly-identical ideas. A table has to have a sequence so close ties aren’t always apparent. The scores displayed will be close if the differences are spread across a couple of factors. Also note (see the picture at the right) that we highlight significant jumps in the scores with a thicker dividing line — which shows that the reviewers had a significant change in their thinking across all factors between the separated ideas.
What Reviews Are, and Are Not
In most cases a review process is intended to give knowledgeable advice and to focus attention on some items more than others. But it does not give automatic decisions. For example, don’t assume that the top 3 ideas on the list should immediately be funded with no further consideration. It is important to discuss the top ideas within the context of the current business environment.
2×2 Charts
Clicking the blue Show Chart button on the right side of the screen will produce the 2×2 charts
H2H gives you the ability to make decisions with complex information across multiple factors by using the 2×2 charts.
The 2×2 Charts are exportable under the “More Tools” menu. Using the + – buttons, you can zoom-in to show details such as idea titles. 2×2 Charts display up to four dimensions of information as x, y, color, and dot size. The information displayed can be chosen from the H2H scoring results or any of the selections and tags associated with the ideas themselves. The 2×2 Charts also reflect the weighting selections made in the sliders.
Trends and Outliers
The concept of trends and outliers is the main reason to create the 2×2 Charts. This example goes into a little more detail in the context of reviews against the factors “Market Appeal” and “Revenue Potential”.
It makes sense that Market Appeal and Revenue Potential should be correlated — if people want something more, they’ll probably pay for it. As expected there’s a general diagonal trend to the ideas in this event.
But notice that there are five ideas in the lower right corner, definitely off the trend. These are outliers and they always tell us something interesting. In this case, if the reviewers are right, these are ideas with high appeal but low monetary value. For example, they might be ideas to add product features that competitors already offer, which are therefore important to our products but which may not justify a price or volume increase (though they might avert a loss!). Of course there can be positive outliers too: ideas which offer more-than-typical benefit for less-than-typical cost or risk. Explore carefully on both sides of any trend line you see, and click ideas to highlight them on all graph and table views.
Outliers are rarely bad ideas, but often healthy challenges to “business as usual” thinking. When designing an event, and certainly when selecting the H2H Review factors, think carefully about these types of learnings and the careful language that will elicit them.