This is the first of what will likely be many posts related to topics we are covering in our book on heavy-tails (which I discussed in an earlier post).
I figure I’ll start with a topic on heavy-tails that is near and dear to my heart — the catastrophe principle. This is, in my mind, a crucial and defining property of heavy-tailed distributions that far too many people aren’t aware of…
A thought experiment
Suppose you are in a class with 50 other students, and the professor does an experiment. She records the heights and the number of twitter followers of every student in the class. Interestingly, it turns out that both the sum of the heights and the sum of the numbers of followers are unexpectedly large, meaning that they are significantly larger than they would have been if each person had the average height and number of followers. The question the professor then asks the class is “What led to the unexpectedly large sums?”
Of course, there are many possible explanations, but they fall into two general categories:
- There are lots of students in the class that have slightly larger average heights and number of followers
- There are a few people in the class that have extremely large heights or numbers of followers.
Perhaps, the more intuitive of these explanations is the first: that the sums are large because of lots of slightly larger than average values.
In fact, this is true in the case of heights; the most likely reason that the sum of the heights is large is that there are a lot of tall people in the class, e.g., perhaps the basketball and volleyball teams are taking the class. It is certainly not likely that the sum of the heights is tall because a giant is taking the class!
However, in the case of twitter followers, the most likely explanation is different. The most likely explanation for why the sum is large is that there is one person in the class that is extremely popular on twitter and, further, it is likely that the sum is dominated by the followers of just this one person.
Said in different terms, the most likely explanation for a large sum of heights is a “conspiracy” where many people are slightly taller than average, while the most likely explanation for a large sum of twitter followers is a “catastrophe” where one person has an extremely large number of followers.
Catastrophes and Conspiracies
This contrast is quite jarring and, of course, it is not just a contrast between heights and number of twitter followers. The fundamental reason for the difference is that the distribution of heights is light-tailed, and the distribution of twitter followers is heavy-tailed, and this same contrast between conspiracies and catastrophes holds more broadly across light-tailed and heavy-tailed distributions. In particular, heavy-tailed distributions tend to follow a “catastrophe principle,” while light-tailed distributions tend to follow a “conspiracy principle.”
Consider another example of a heavy-tailed distribution — the damage caused by hurricanes. Suppose we sum up the estimated cost of the damages from each hurricane during a given year, and that the sum is unexpectedly large. Of course, we would want to understand why. A priori, one might expect that this unexpectedly large amount of damage probably came about because there were a lot of hurricanes that caused larger than typical amounts of damage. However, the catastrophe principle for heavy-tailed distributions tells us that this is not the case. Instead, as with the case of twitter followers, we should expect that the most likely reason that the sum is unexpectedly large is because there was one hurricane that caused an extremely large amount of damage. In fact, we should expect that there was one hurricane that caused nearly all of the damage caused during the year, i.e., there was a catastrophe.
This feels like a counter-intuitive explanation, but when the underlying distributions are heavy-tailed, the catastrophe principle does indeed provide the most likely explanation. In some sense, it is a form of Occam’s razor — the simplest explanation for a large sum is that one large event happened, not that a conspiracy of many slightly-larger-than-expected events happened.
However, because of its counter-intuitive nature, the catastrophe principle is one of the more mysterious properties associated with heavy-tailed distributions. It is completely contrary to what happens under the “typical” light-tailed distributions that we learn about in introductory probability courses, e.g., the Normal and Exponential.
To make a little sense of the contrast between catastrophes and conspiracies, let’s try to build some intuition. An overly simplistic, but still useful, characterization of light-tailed distributions is that the samples never differ too much from the mean of the distribution. Thus, all the samples are of a similar size, i.e., on the same scale. Note that the conspiracy principle is quite natural given this view of light-tailed distributions — since all samples are similar, any large sum must be a combination of many slightly larger than average samples.
In contrast, an equally simplistic, but useful view of heavy-tailed distributions is that they are made of “many mice and a few elephants.” That is, heavy-tailed distributions have a lot of small samples (many mice), and when there are large samples, they are very large (a few elephants). This view of heavy-tailed distributions provides an extremely clear explanation of the catastrophe principle — if the sum is unexpectedly large, it must be because an elephant arrived. Further, since there are only a few elephants, it is unlikely that two elephants arrived, so the sum is probably dominated by a single elephant. (This is sometimes referred to as “the principle of a single big jump.”)
Making things formal
While I have been informal in my descriptions of conspiracies and catastrophes in this post, these concepts are actually precise, formal, principles. Thus, they are useful not only in intuitively reasoning about heavy-tailed and light-tailed distributions; they are important analytic tools, as well.
In the next post on this topic, I’ll describe a bit about how to formalize and use this principles. For example, we’ll connect the catastrophe principle to a general class of heavy-tailed distributions termed “subexponential distributions.”