Beware the Metrics System – Part 2
“Beware the Metrics System – Part 1” presented potential advantages of implementing a metrics system, metric classifications, and warnings of potential pitfalls. This installment will provide examples from diverse industries and recommendations for development and management of metrics systems.
Examples from Diverse Industries
In general, “The Third Degree” focuses on manufacturing and common service industries. This post is no different, as is evident from the example metrics and categories discussed. However, metrics systems are pervasive in modern society. The following examples demonstrate the ubiquity of metrics and unintended consequences.
Doctors and hospitals are reimbursed according to their activities (e.g. number of blood tests performed) instead of the effectiveness of those activities (i.e. Did it lead to a proper diagnosis?) or their relevance to the patient’s condition (Does a drug overdose warrant a cardiac stress test?) Insurance companies want healthcare providers to be effective, but pay them to be busy. Providers can achieve their objectives (e.g. high reimbursements) whether or not the patients’ needs are met.
Evaluation of teachers’ performance is based increasingly on students’ standardized test scores. This incentivizes narrowing the scope of material presented to students and “teaching to the test.” The metrics system, typified by the infamous “No Child Left Behind” Act, ostensibly targets a better-educated student body. However, it arguably achieves the opposite outcome by focusing students on the test instead of deep learning and long-term retention.
A university may boast of its incoming class’ high average SAT score, buoyed, no doubt, by students taught to perform well on standardized tests. Unfortunately, it says little about the academic performance that this class is likely to achieve. It is one indicator – the reliability of which is oft debated – among many factors that influence a student’s academic achievement. A disconnect exists between the metric (SAT score) and the objective (high academic performance).
Armed forces track numerous measures of their operations – number of troops deployed, number of missions, number of bombs dropped and targets destroyed, number of rounds fired and enemy combatants killed, and so on. None of these metrics, however, are true measures of progress toward the goal: victory or an end to hostilities. The metrics system generates a great deal of data, but little information.
Police departments increasingly utilize metrics to manage their limited resources. By comparing the frequency of crimes in different neighborhoods, patrols can be scheduled for greatest deterrent effect, minimum response time, or greatest probability of capturing perpetrators. Public disclosure of crime statistics can encourage misuse of the metrics system. This occurs, for example, when arrests are “down-classified” to minor offenses to create the illusion that a jurisdiction is safer than it really is, or that crime rates are declining.
Web hosting services monitor a number of measures to assist site owners in evaluating the effectiveness of their web pages. Website metrics include page views, click-through rate, bounce rate, time-on-site, link referrals, and more. A site’s metrics effect its search rankings; the aim of Search Engine Optimization (SEO) is to improve a site’s metrics, allowing it rise in search rankings.
This metrics system has also been abused, exploited and manipulated by competitors or others that would like to limit a website’s audience. Specific online behaviors lower a site’s rankings, thereby reducing its visibility. Fortunately, the same analytics can serve to identify a saboteur, though it may require substantial effort to do so.
IT Help Desk/Customer Service Hotline
The “IT Help Desk” and customer service hotline are quintessential examples of metrics system failure. To ensure that customers receive prompt attention and resolution of their issues, the length of time “tickets” remain open and the duration of telephone calls are monitored. Instead of prompt attention and resolution of issues, customers get abruptly disconnected calls and tickets demoted to a low-priority status so that they will be judged according to a lower standard of performance. If you want to learn how to misuse and abuse a metrics system, look no further than the No-Help Desk!
While there is no perfect metrics system, careful consideration of an organization’s circumstances during development will increase the probability of creating a successful one. Whether closely following an existing framework or creating a unique system, there are universal ideals that developers should strive to attain.
Metrics Administration (System Management)
As you can see, development of a productive and efficient metrics system requires a great deal of cogitation. Complaints about an organization’s metrics system typically stem from haphazard design or execution; perhaps both. As is true of all organizational initiatives, leadership commitment is required to maintain an efficacious metrics system. Without it, the system will be fraught with fabricated or otherwise distorted data and “orphaned” metrics – those for which no one accepts responsibility. In this scenario, any decisions supported by the metrics will be ill-advised and may lead to very unfavorable results. Summing it up, do it right or don’t do it at all.
If you find yourself in the middle of a metric minefield, JayWink Solutions can help. Together, we can construct a metrics system that effectively monitors the health of your organization without excessive burden. Contact us today for a consultation.
[Link] “To Measure is to Know,” Susan Leister and Suzanne Tran. Quality Progress, November 2015.
[Link] “Psychological Impact of Metrics,” Duke Okes. The Journal for Quality & Participation, January 2013.
[Link] The Tyranny of Metrics, Jerry Z. Muller. Princeton University Press, 2018.
[Link] “The Balanced Scorecard – Measures that Drive Performance,” Robert S. Kaplan and David P. Norton. Harvard Business Review, January-February 1992.
[Link] “Time-Relevant Metrics in an Era of Continuous Improvement: The Balanced Scorecard Revisited,” Richard J. Schonberger. Quality Management Journal, Vol. 20, No. 3, 2013.
[Link] “Are Manufacturers’ KPI Reporting Practices Keeping Up?” Holly Lyke-Ho-Gland. IndustryWeek, June 4, 2019.
[Link] “It’s All in the Numbers – KPI Best Practices,” Lee Schwartz. IndustryWeek, March 24, 2015.
[Link] M. Schrage and D. Kiron, “Leading With Next-Generation Key Performance Indicators,” MIT Sloan Management Review, June 2018.
[Link] “Does Management by Objectives Stifle Excellence?” John Dyer. IndustryWeek, December 17, 2013.
[Link] “The OPTIMAL MBO: A model for effective management-by-objectives implementation,” Sharon Gotteiner. European Accounting and Management Review, Vol. 2, No. 2, May 2016.
[Link] “OKRs.” Medium.com, November 28, 2014.
[Link] “OKR: Should you use them for setting goals?” Claire Lew. Knowyourteam.com, July 18, 2019.
[Link] “On the folly of rewarding A, while hoping for B,” Steven Kerr. The Academy of Management Executive, February 1995.
[Link] “Goodhart’s law,” Wikipedia.com.
[Link] “Goodhart’s Law and Why Measurement is Hard,” David Manheim. Ribbonfarm.com, June 9, 2016.
[Link] “Overpowered Metrics Eat Underspecified Goals,” David Manheim. Ribbonfarm.com, September 29, 2016.
[Link] “Our Metrics Fetish – And What to Do About It,” David Shaywitz. Forbes, June 23, 2011.
[Link] “We Are Not a Dashboard: Contesting the Tyranny of Metrics, Measurement, and Managerialism,” David Shaywitz. Forbes, December 24, 2018.
[Link] “The burden of checklists and the importance of core metrics,” Bill Gardner. Theincidentaleconomist.com, May 20, 2015.
[Link] “Five Categories to Focus Your KPIs,” Michael Schrage. MIT Sloan Management Review, September 21, 2018.
[Link] “Manufacturing KPIs: How Do Yours Compare?” Jill Jusko. IndustryWeek, June 24, 2019.
[Link] Selecting the Right manufacturing Improvement Tools: What Tool? When? Ron Moore. Elsevier, 2007.
[Link] “Standing Out from the Start,” Lindsay Scott. PM Network, June 2019.
Jody W. Phelps, MSc, PMP®, MBA
JayWink Solutions, LLC
Your comment will be posted after it is approved.
Leave a Reply.
If you'd like to contribute to this blog, please email email@example.com with your suggestions.
© JayWink Solutions, LLC