After attending an excellent Vistage meeting on “Winning with Data”, a discussion on AI and ML (Machine Learning), I read an article on “data-driven decision making” which posed the challenge of “The Illusion of Insight”. I thought I’d share the article and the highlights.
Vikram Joshi notes that “The illusion doesn’t arise from a lack of data, but rather from selective use of it. The cost can be steep: flawed strategies, poor investments, broken trust and wasted resources.”
When data is cherry-picked to support a predetermined conclusion, it can create the appearance of truth while potentially leading us away from it. We may be misled by “confirmation bias” in order to pressure sales, impress stakeholders or justify sunk costs.
Similarly, when we’re looking to justify the use of features in a product and turn to power users for their feedback, the sample might not be representative of the total addressable market. The same inconclusive result could result if we discover that the ML model we’re using was trained on a set of different set of benchmarks (e.g. men) when we’re focused on a broader market (men and women).
To develop greater date integrity, he recommends project leaders use these five practices to safeguard against cherry-picking:
- Visualize the full picture; don’t focus only on highlights
- Identify the context; context changes everything
- Incentivize truth over narrative; accept uncertainties and flaws.
- Use follow-up questions for clarity: “what are we not seeing?”
- Focus on exploration not confirmation; start with questions, not conclusions.
Using these approaches can lead to a clear benefit; avoiding mistaken correlation for causation.
Now go forth and use ML to win more deals with data.