Insight from Agora Consultants

Bernoulli and Choosing Metrics


“One should not appraise human action on the basis of its results.” -  Jakob Bernoulli (1654-1705)

This statement has significant implications on devising metrics for Performance Management. In fact, at first glance it could be interpreted as why should we measure at all. Let’s look at the background on Bernoulli’s comment and see how we can use his concepts to derive better metrics.

Jakob Bernoulli was one of the founding fathers of probability theory. In particular, he developed the concept now called Bernoulli trials, whose result can be either of two possible outcomes, “success” or “failure”. Situations like the following are considered Bernoulli trials:

  • Did the project complete on time?
  • Did the sales team hit their sales targets?
  • Did my child get over 90% on his exam?
  • Did the CEO qualify for a bonus?
  • Did the Toronto Maple Leafs win the game on Saturday?

Each of these situations has a probability of success (and a probability of failure which is 1 - the probability of success). For each of these situations, no matter how hard the person (or group) tries and no matter what their skill there is still a chance of failure. Conversely, no matter how much the lack of effort or skills of a person (or group) there is still a probability of success. We have seen many times in real life a CEO getting an underserved bonus, a star pupil blowing a test and a weaker team winning the championship (Leaf fans…don’t get excited, the Stanley cup is still far away).

To understand how to minimize chance we need to go to Bernoulli again. Bernoulli’s Law of large numbers states that over time, the average will trend towards the expected value. Looked at another way, with a greater number of observations (e.g. number of games, number of tests), the average will be more representative of the underlying person or group’s skills.

Unfortunately, people tend to behave in the opposite fashion. People often assume a small sample is representative of the underlying situation when in fact it is too small to be reliable. This has been observed so many times it has sarcastically been called the Law of small numbers. It has also been called Hasty Generalization. This behaviour reinforces the importance of creating metrics that remove this human bias.

Considering what Bernoulli has taught us, let’s look at characteristics of good metrics.

Closely aligned with the desired action. Increased sales is what you desire from a sales rep but it is not directly tied to the actions or behaviours you want from the sales rep. If you know that number of sales calls in a week is a behaviour that leads to increased sales it should be considered as the metric. Number of sales calls in a week is a better metric because it is closely tied to the desired action.

High number of observations. Repeated observations (measurements) increases the representation of the underlying situation. Looking at many projects delivered by a project manager rather than just one is more representative of his ability to delivery on time.

Variety of independent metrics. In addition to minimizing chance by averaging across observations, chance can be reduced by measuring a number of independent metrics. The independent aspect is important otherwise correlation between the metrics will distort the true picture. Measuring a student through multiple methods such as class participation, oral communication and a written test measures comprehension better than a single multiple choice exam.

The desire by people to look at results based on small observations is strong. It requires courage to change the way that results are looked at. Hopefully drawing on a mathematical foundation for metric design will remove some human bias. I wonder what Bernoulli would have thought if he knew his ideas from over 300 years ago are being applied to devising metrics to running organizations today.

Comments are closed