Rethinking Markets

Geoffrey James of Inc. magazine Reports:

A few years back, I interviewed some of the most successful CEOs in the world in order to discover their management secrets. I learned that the “best of the best” tend to share the following eight core beliefs.

I dislike articles like this, books like this, talks like this, thoughts like this. Aside from being lazy, they betray a consistent misunderstanding of how data, causality, and basic logic work. Inc. is terrible about this kind of stuff (9 qualities of successful entrepreneurs!, secret traits of successful entrepreneurs!). But they are not nearly alone: There are the 7 Habits of Highly Successful People, or What the Best College Teachers Do, or, well Jack Welch’s Winning and the ensuing implementations of Six Sigma management techniques. The list is long, and the criticism is consistent.

At its heart, my complaint is simple: you can not establish causality by sampling on the dependent variable.

What does this mean?

What the author wants you to believe, and why people read these books/articles over and again, is that having these eight core beliefs will lead to extraordinary success. Successful traders see signal where others see noise. Successful runners sleep on their left sides and drink two glasses of Gatorade. The implicit message here is that you too could be a successful entrepreneur/trader/runner/parent by cultivating these habits/traits/beliefs.

And it may be true!

But alas, maybe not. Here are some problems:

1. People with the traits of ‘successful’ people may not themselves be successful. Or lead to success. This can be because those traits are necessary but insufficient. That is, hard work may be necessary, but it may be the combination of hard work and luck that leads to entrepreneurial success. Or hard work and a trust fund. Time management plus a sense of humor. In other words, there are potentially additional, unmeasured traits that are also necessary for success.

   It may also be the case that you picked a sample of people who have both success and the traits you are looking at. Perhaps you didn’t notice all the other people who relentlessly seek new experiences, are empathetic, etc., but who did not achieve success. What is important to note is that you simply cannot know if the traits cause success if you start with success. The best you can say is that of people who are successful, here are traits they share. You may also have these traits! But your success is not guaranteed.

2. It is also possible that people without the traits in question are also successful. When someone says they interviewed ‘some of the most successful CEOs’, your bullshit radar should start beeping. This is problematic because it is possible (and in this case, likely) that the sample is biased: think about those CEOs, or hedge fund managers, or mothers, or entrepreneurs who are willing to talk to reporters and researchers. And what they might or might not say. And so, there may be successful people, without these traits, who are simply outside the purview of the researcher/reporter/journalist/writer. You may not have these traits! But your success is not precluded.
3. It is also possible that the traits/beliefs/habits of successful people may be relatively more or less causal in leading to success as some other set of traits/beliefs/habits. Let’s say that people with the cluster of characteristics you find are successful 50% of the time. Wow! But what if people without those traits are successful 45% of the time? Hrm, then it’s less interesting. And what if people with another cluster of characteristics are successful 85% of the time? Then we can start to think that maybe your characteristics are causal, but not relatively so.

The thing is, demonstrating causality is hard. You have to account for the fact that idiots may become successful, and then after they become successful, they acquire a set of characteristics. Like ultra-rich people being arrogant. Arrogance may lead to richness, but it’s also likely that arrogance comes from being rich.

It’s also the case that people who are successful have a personal stake in explaining their success through a combination of plucky characteristics and hard work. It is actually hard to emphasize this enough – people are better justification machines than they are explanation machines. People are better justification machines than they are explanation machines. Michael Lewis, in a smart, old, 2007 profile of investor Blaine Lourd, points to a speaker at Dimensional Fund Advisors. DFA is a company founded almost in opposition stock pickers:

…two speakers discuss how, knowing what we now know, anyone could present himself as a stock-picking guru. “If you put a thousand people in barrels and push them over Niagara Falls,” one of them says, “some of them will survive. And if you take those guys and push them over again, some of them will survive. And they’ll write books about how to survive being pushed over Niagara Falls in a barrel.”

That’s it in a nutshell.

Ok, so what? Well, a couple things.

First, determining correlation is relatively easy. Causality is hard. To do it, you need three elements. 1) A correlation between what you think is causing a phenomenon and the phenomenon that is being caused. The explanandum is what needs to be explained (success!!), and the explanans is what explains it (7 Secret Habits!). 2) Prior temporality of the potentially causal variable(s). The habits have to come before the success. And 3) a plausible explanation for how or why the the variables could and would cause the outcome.

Second, be a better consumer of these kinds of books and articles. The vast majority of the business trade press suffers from massive selection on the dependent variable problems. You can understand What the Best College Teachers Do, because it is interesting! And there are ideas in there that might be useful for you. But there is no reason to believe that doing what the best college teachers do will make you even a better college teacher.

But know that the implicit promise that X leads to Y is wrong. If a survey of an appropriately random sample (or possibly an appropriately non-random sample) of college teachers shows that a cluster of traits are both associated with success, that these traits are temporally precedent to the success, and that have some viable explanation for the relationship, then you can say X causes Y. Approximating these conditions lead to better claims of causality; but as a general rule association is a much lower bar than causation.