Beware the Metrics System – Part 2

     “Beware the Metrics System – Part 1” presented potential advantages of implementing a metrics system, metric classifications, and warnings of potential pitfalls.  This installment will provide examples from diverse industries and recommendations for development and management of metrics systems.

Examples from Diverse Industries
     In general, “The Third Degree” focuses on manufacturing and common service industries.  This post is no different, as is evident from the example metrics and categories discussed.  However, metrics systems are pervasive in modern society.  The following examples demonstrate the ubiquity of metrics and unintended consequences.
 
Healthcare
     Doctors and hospitals are reimbursed according to their activities (e.g. number of blood tests performed) instead of the effectiveness of those activities (i.e. Did it lead to a proper diagnosis?) or their relevance to the patient’s condition (Does a drug overdose warrant a cardiac stress test?)  Insurance companies want healthcare providers to be effective, but pay them to be busy.  Providers can achieve their objectives (e.g. high reimbursements) whether or not the patients’ needs are met.

Education
     Evaluation of teachers’ performance is based increasingly on students’ standardized test scores.  This incentivizes narrowing the scope of material presented to students and “teaching to the test.”  The metrics system, typified by the infamous “No Child Left Behind” Act, ostensibly targets a better-educated student body.  However, it arguably achieves the opposite outcome by focusing students on the test instead of deep learning and long-term retention.
     A university may boast of its incoming class’ high average SAT score, buoyed, no doubt, by students taught to perform well on standardized tests.  Unfortunately, it says little about the academic performance that this class is likely to achieve.  It is one indicator – the reliability of which is oft debated – among many factors that influence a student’s academic achievement.  A disconnect exists between the metric (SAT score) and the objective (high academic performance).

Military
     Armed forces track numerous measures of their operations – number of troops deployed, number of missions, number of bombs dropped and targets destroyed, number of rounds fired and enemy combatants killed, and so on.  None of these metrics, however, are true measures of progress toward the goal:  victory or an end to hostilities.  The metrics system generates a great deal of data, but little information.
 
Law Enforcement
     Police departments increasingly utilize metrics to manage their limited resources.  By comparing the frequency of crimes in different neighborhoods, patrols can be scheduled for greatest deterrent effect, minimum response time, or greatest probability of capturing perpetrators.  Public disclosure of crime statistics can encourage misuse of the metrics system.  This occurs, for example, when arrests are “down-classified” to minor offenses to create the illusion that a jurisdiction is safer than it really is, or that crime rates are declining.

Website Analytics
     Web hosting services monitor a number of measures to assist site owners in evaluating the effectiveness of their web pages.  Website metrics include page views, click-through rate, bounce rate, time-on-site, link referrals, and more.  A site’s metrics effect its search rankings; the aim of Search Engine Optimization (SEO) is to improve a site’s metrics, allowing it rise in search rankings.
     This metrics system has also been abused, exploited and manipulated by competitors or others that would like to limit a website’s audience.  Specific online behaviors lower a site’s rankings, thereby reducing its visibility.  Fortunately, the same analytics can serve to identify a saboteur, though it may require substantial effort to do so.

IT Help Desk/Customer Service Hotline
     The “IT Help Desk” and customer service hotline are quintessential examples of metrics system failure.  To ensure that customers receive prompt attention and resolution of their issues, the length of time “tickets” remain open and the duration of telephone calls are monitored.  Instead of prompt attention and resolution of issues, customers get abruptly disconnected calls and tickets demoted to a low-priority status so that they will be judged according to a lower standard of performance.  If you want to learn how to misuse and abuse a metrics system, look no further than the No-Help Desk!

Best Practices
     While there is no perfect metrics system, careful consideration of an organization’s circumstances during development will increase the probability of creating a successful one.  Whether closely following an existing framework or creating a unique system, there are universal ideals that developers should strive to attain.
 
Metric Selection

• Choose a broad mix of metrics that allows managers to monitor the organization from multiple viewpoints, at various levels, and with different time horizons.  A Balanced Scorecard approach can be very helpful, especially if you or your organization is new to building formalized metrics systems.
• Limit the number of metrics to a manageable set.  A common recommendation is to choose 3 – 5 metrics in each of 3 – 5 categories.  The Balanced Scorecard identifies four perspectives (categories):  financial, customer, internal business, and innovation and learning.
• Focus on measuring outcomes.  Once desired outcomes are achieved consistently, inputs can be measured to determine efficiencies and identify potential improvement projects.  The military example above demonstrates the futility of measuring inputs alone.
• Resist the allure of “easy” metrics.  Ease of data collection or presentation is not indicative of its utility or relevance.  Also, too much “low-hanging fruit” in your metrics system – goals that are easy to reach – will add to members’ workload with little payoff for the organization.
• Use objective, not subjective measures.  Subjectivity in metrics invites “easing up” on standards to boost results.  If consistency in evaluation cannot be assured, it is not a useful metric.
• Link metrics to success and risk factors.  For each metric, ask:
  • Does it identify success or failure?
  • Does it inform the organization of the level of risk it faces?
  • Are appropriate reactions to changes in the metric understood?

         Answers in the affirmative justify retention in the pool of potential metrics.

• Choose metrics that can be tracked for multiple locations, departments, etc. within the organization.  This will facilitate knowledge-sharing throughout the organization by creating internal benchmarking opportunities.  It can also be expected to limit data manipulation, as the scrutiny of benchmarking and knowledge-sharing activities exposes it.
• Balance the cost of measurement with the value provided by the metric.  Costs to consider include process instrumentation, data transmission and storage, time required to analyze and present the results, and opportunity costs of consuming resources.  The cost of tracking a low-information-value metric can quickly exceed the benefit of doing so.
• Structure the metrics system to support the organization, not the reverse.  If major changes are needed to operate the system, an additional review is worthwhile.  Be certain that the chosen path is in the best interest of the organization before proceeding.

 
Metrics Administration (System Management)

• Use the metrics system “to lead and to manage” the organization.  Many metrics are used to inform managerial decision-making, but they can also be aspirational.  Leaders do not have to choose between visionary abstractions and objective measurements; there is room for both in inspiring messages.
• Use metrics to guide improvement efforts, not for disciplinary purposes.  The potential for negative consequences invites data manipulation and responsibility-dodging.
• Limit the number of metrics for which any individual is responsible.  More than three is almost certain overload when added to “normal” job duties.
• Ensure that those assigned responsibility for performance per any metric have all that is needed to successfully monitor and improve the metric – access to data, analysis tools, and other resources.
• Reassign responsibility for metrics that create adversarial relationships (see SQA vs. PQA example above) to a single entity.  This way, energy that would otherwise be expended on attempts to deflect responsibility can be used to optimize both metrics.
• Periodically review and evaluate the metrics system.  Have the chosen metrics driven desired behaviors?  Have they driven any undesired behaviors?  Have valued, but unmetricated, activities suffered due to reallocation of resources?  Has the system driven efficiency gains in non-value-added activities – that is, has waste been streamlined?
• Monitor standards of acceptability to ensure that improving measures are due to genuine progress, not lowered standards.  Also, “outlier” claims should be thoroughly investigated.  Otherwise, it will be too easy – and too tempting – to artificially inflate performance measures by removing unfavorable results from the dataset.
• Resist the urge to add more metrics in response to reviews.  More is not always better.  Vigorously justify each new metric; consider retiring an existing metric in exchange.
• Leaders should allow and encourage risk-taking and innovation.  Though a metric may suffer in the short-term, step-change improvements require experimentation.  If the metrics system is used to instill fear, the status quo will likely be the pinnacle.
• Avoid succumbing to “short-termism.”  The metrics system should be used to secure long-term sustainability, not just an acceptable quarterly review, bonus check, or promotion.
• Seek to fully understand the interrelationships among metrics.  It will help to identify redundancy in the tracked metrics and to detect data manipulation (“gaming”).  Do not blindly follow the metrics without validating sources of data and understanding the meaning of results.
• Use dashboards judiciously.  Displaying too much information at once can overwhelm the average user and nullify the metrics system; limit the amount of information displayed so it can be easily processed and understood.  For example, display only those metrics that relate to a specific viewer’s area of influence, or one Balanced Scorecard perspective.  Customizable dashboards provide the greatest utility by allowing each user to selectively view the information most useful to them at any given time.
• Develop trust in the metrics system by maintaining transparency.  Data sources, collection and analysis methods, the purpose of each metric, and targets should be freely shared with all those affected.  Open communication and consistent application of standards foster trust in the system; visible results foster belief in its utility.

 
     As you can see, development of a productive and efficient metrics system requires a great deal of cogitation.  Complaints about an organization’s metrics system typically stem from haphazard design or execution; perhaps both.  As is true of all organizational initiatives, leadership commitment is required to maintain an efficacious metrics system.  Without it, the system will be fraught with fabricated or otherwise distorted data and “orphaned” metrics – those for which no one accepts responsibility.  In this scenario, any decisions supported by the metrics will be ill-advised and may lead to very unfavorable results.  Summing it up, do it right or don’t do it at all.
 
     If you find yourself in the middle of a metric minefield, JayWink Solutions can help.  Together, we can construct a metrics system that effectively monitors the health of your organization without excessive burden.  Contact us today for a consultation.

References
[Link] “To Measure is to Know,” Susan Leister and Suzanne Tran.  Quality Progress, November 2015.
[Link] “Psychological Impact of Metrics,” Duke Okes.  The Journal for Quality & Participation, January 2013.
[Link] The Tyranny of Metrics, Jerry Z. Muller.  Princeton University Press, 2018.
[Link] “The Balanced Scorecard – Measures that Drive Performance,” Robert S. Kaplan and David P. Norton.  Harvard Business Review, January-February 1992.
[Link] “Time-Relevant Metrics in an Era of Continuous Improvement:  The Balanced Scorecard Revisited,” Richard J. Schonberger.  Quality Management Journal, Vol. 20, No. 3, 2013.
[Link] “Are Manufacturers’ KPI Reporting Practices Keeping Up?” Holly Lyke-Ho-Gland.  IndustryWeek, June 4, 2019.
[Link] “It’s All in the Numbers – KPI Best Practices,” Lee Schwartz.  IndustryWeek, March 24, 2015.
[Link] M. Schrage and D. Kiron, “Leading With Next-Generation Key Performance Indicators,” MIT Sloan Management Review, June 2018.
[Link] “Does Management by Objectives Stifle Excellence?” John Dyer.  IndustryWeek, December 17, 2013.
[Link] “The OPTIMAL MBO:  A model for effective management-by-objectives implementation,” Sharon Gotteiner.  European Accounting and Management Review, Vol. 2, No. 2, May 2016.
[Link] “OKRs.” Medium.com, November 28, 2014.
[Link] “OKR:  Should you use them for setting goals?” Claire Lew.  Knowyourteam.com, July 18, 2019.
[Link] “On the folly of rewarding A, while hoping for B,” Steven Kerr.  The Academy of Management Executive, February 1995.
[Link] “Goodhart’s law,” Wikipedia.com.
[Link] “Goodhart’s Law and Why Measurement is Hard,” David Manheim.  Ribbonfarm.com, June 9, 2016.
[Link] “Overpowered Metrics Eat Underspecified Goals,” David Manheim.  Ribbonfarm.com, September 29, 2016.
[Link] “Our Metrics Fetish – And What to Do About It,” David Shaywitz.  Forbes, June 23, 2011.
[Link] “We Are Not a Dashboard:  Contesting the Tyranny of Metrics, Measurement, and Managerialism,” David Shaywitz.  Forbes, December 24, 2018.
[Link] “The burden of checklists and the importance of core metrics,” Bill Gardner.  Theincidentaleconomist.com, May 20, 2015.
[Link] “Five Categories to Focus Your KPIs,” Michael Schrage.  MIT Sloan Management Review, September 21, 2018.
[Link] “Manufacturing KPIs:  How Do Yours Compare?” Jill Jusko.  IndustryWeek, June 24, 2019.
[Link] Selecting the Right manufacturing Improvement Tools:  What Tool?  When?  Ron Moore.  Elsevier, 2007.
[Link] “Standing Out from the Start,” Lindsay Scott.  PM Network, June 2019.

Jody W. Phelps, MSc, PMP®, MBA
Principal Consultant
JayWink Solutions, LLC
jody@jaywink.com