“Not everything that counts can be counted, and not everything that can be counted counts.”
- William Bruce Cameron
In the evolving landscape of Knowledge Management (KM), organisations increasingly rely on data-driven strategies, especially for evidence driven decision making. Metrics and written information, while crucial, can lead organisations astray when misused. Two critical concepts highlight these risks. I have spoken about the McNamara Fallacy before, and this week I came across its end result: the Tyranny of Metrics.
McNamara Fallacy: Measuring Only the Measurable
Tyranny of Metrics: When Metrics Drive Dysfunction
Coined by Jerry Z. Muller, the Tyranny of Metrics describes how excessive reliance on numerical targets can distort organisational behaviours, leading to gaming, stifling innovation, and prioritising short-term goals at the expense of long-term strategic value. Jerry isn't focusing on just one off acts. The Tyranny of Metrics describes the overall cultural impact that infiltrates all decision-making over time, - including the ridicule of those that call for more qualitative or soft insights and factors to be considered. It describes a corporate workspace where only hard (and usually output) metrics are considered acceptable.
Generative AI: Bias Towards Explicit Knowledge
Whether through training, or via RAG-style run-time context, Generative AI (Gen-AI), particularly Large Language Models (LLMs), predominantly learn from textual data. This inherent bias favours explicit knowledge, potentially marginalising tacit knowledge obtained through human interaction. This limitation risks reinforcing existing biases found in data and reducing contextual understanding critical for nuanced decision-making. Capturing meeting transcriptions and using multi-model LLMs that consider meeting recordings, etc, goes a small way to solving this bias, but we all know that often key decisions are made around a water cooler, over personal phone calls and often before the official meeting where the decisions are officially recorded.
Mitigation Strategies
To navigate these pitfalls, executives and knowledge managers consider:
- Balance Quantitative and Qualitative Metrics: Adopt holistic knowledge frameworks incorporating both data-driven insights and qualitative employee feedback. Live feedback through stakeholder codesign or sensemaking initiatives are even better.
- Promote Tacit Knowledge Capture: Utilise storytelling, mentoring, and collaborative platforms to capture valuable experiential insights. I am usually critical of the focus on Capture when it comes to Tacit Knowledge, but in this case that is exactly what is required;
- Human-AI Integration: Ensure that AI complements human expertise, fostering hybrid systems that leverage the strengths of both.
- Consider how you monitor effectiveness in complex environments: Using evaluation approaches like Adaptive Management and Outcome Mapping, you can ensure the end result is part of the original design.
- Ethical AI Frameworks: Establish governance practices promoting transparency, fairness, and accountability in AI deployments.
Avoiding a pitfall can be a lot cheaper than climbing out of one later
As you take your first steps into using Gen-AI for your KM or decision-support work, make it a point to keep the McNamara Fallacy on the agenda for design and strategy sessions. Besides helping you create a much more sustainable and accurate solution, it may just save you a tonne of money down the track trying to account for context and expertise that you never bothered capturing in the first place. Even if AI isn't on your roadmap yet, start thinking now about capturing the conversations, critical knowledge and operational stories that make your organisation really work. Your future self will thank you a thousand times over.
0 Comments