Itamar Rosenn, Facebook

Itamar conveyed how Facebook’s Data Team used R in 2007 to answer two questions about new users: (i) which data points predict whether a user will stay? and (ii) if they stay, which data points predict how active they’ll be after three months?

For the first question, Itamar’s team used recursive partitioning (via the rpart package) to infer that just two data points are significantly predictive of whether a user remains on Facebook: (i) having more than one session as a new user, and (ii) entering basic profile information.

For the second question, they fit the data to a logistic model using a least angle regression approach (via the lars package), and found that activity at three months was predicted by variables related to three classes of behavior: (i) how often a user was reached out to by others, (ii) frequency of third party application use, and (iii) what Itamar termed “receptiveness” — related to how forthcoming a user was on the site.

Now at least you know why Facebook is obsessed with the ‘You haven’t spoken to X in a while. Drop them a line!’

I am mostly R-illiterate, and like LaTex, Emacs, and all other things Kieran, I suspect that having passed graduate school without these tools, I’ve missed my window for ever really incorporating them into my habitus…As Edward Lewis once said, People’s reaction to R is very dramatic. They love it or they hate it. If they love it, they will always love it. If they don’t, they will learn to appreciate it, but it will never become part of their soul.

Scenario 1:
In the first case, we have a series of attributes attached to a person, and then we can make arguments (empirical, theoretical) about how these attributes lead to outcomes. A person who is value-conscious is likely to pass on high-ticket items; a person who is sexually adventurous is likely to seek out partners to participate in kinky sex; a person who likes crappy romantic comedies is likely to see Miss Congeniality.

To make these kinds of arguments, we would have to seek out attributes to are causally related to the outcomes in question. In this case, if we want to predict what kinds of parties a person would want to go to, we would ask questions about their introvertedness/extrovertedness and the results of these questions would have a positive/negative/null effect on the kinds of parties they either go to or report going to. That is, the empirical and theoretical difficulty is in finding variables related to the outcomes we want to know about.

Scenario 2:
In the second scenario, the argument is that we should give primacy to a likeness or correspondence analysis to help understand what kinds of parties you want to go to – and a whole host of other things.

In this case, we rely less on the causal relationship between introvertedness and the kinds of parties you would want to go to, and more on the fact that whatever you want to do, another person who is just like you would also want to do those same things. If this were true, then what’s important is not so much to find a causal link between what kind of person you are and what kinds of parties you want to go to. Instead, the challenge is in finding what makes you different from one person and similar to another. If we can somehow ‘clump’ all the similar people, we would be more likely to know what kinds of things they like and do, regardless of their specific characteristics.

In this case, a person who is value-conscious, sexually adventurous, or likes crappy romantic comedies is likely to like the same kinds of things as other people who are value-conscious, sexually adventurous, or who have questionable movie tastes.

I think that approach #2 is behind much of the recommendation engine work that has emerged of late, but I’m wondering in a more and more pointed fashion which of these is a more reasonable approach to understanding what people like. What do you lurkers think?

Their latest discussion is about how they managed a question on their support forms. I want to drop some research methodology on this problem. While their discussion is about how to design a feedback form, it is also about the kinds of questions you should ask on a survey. And it would benefit from a discussion of categorical/nominal variables, ordinal variables, and interval-level variables. Yep.

So, terms. Categorical variables (also called nominal variables) are those variables with 2 or more ‘states’, but without an intrinsic ordering. Male/female is categorical, as is eye or hair color, race, what school you went to. Ordinal variables are those variable with two or more states that have an ordering to them. Low/medium/high are ordinal variables. Less than HS, HS, some college, BA is ordinal. Interval-level variables are ordered, and the distance between categories is evenly spaced. Income, height, and years of education are all interval-level variables.

The difference between categorical/nominal variables and ordinal variables is the hierarchical ordering of the latter. What school you went to is nominal, but tiered ranking of what school you went to is ordinal. Tiered ranking may be ordinal, but amount of school endowment is interval-level. And quantitative variables are a hint that the variable is ordinal or interval, but not decisive (zip codes are nominal, for instance).

So, back to 37 Signals. What they want is some variable that would be easy to understand (from the customer perspective) and helpful to process (from the company perspective). Their first attempt looked something like this:

It takes time to think through what my state of mind is, because the items are almost ordered, but not really. Confused, worried, upset, and panicked are not points on a continuum, they are just different states of being. The question is asked as a categorical variable question. But it is one that they reallywanted to be an ordinal variable question. They tried to solve the problem by formalizing an ordinal variable, and putting a numbered ordering system on it to make it clearly so:

This is easier to deal with as a customer, since you can sort of pick up your relative state of panic. In other words, you pick up that there is a rough ordering very quickly, and the numbers help a lot in this respect. But alas, what’s good for the customer was no good for 37Signals. The reason is that while they began by wanting to know the subjectivity of the customer, what they really really wanted to know was, ‘how important is this problem for our company?’. Reasonable, but different. This is their final solution:

They have a reason for this, and it is a decent one:

Now, if something’s broken, we can spot it and fix it right away. A system failure is much more important to us (and our customers) than a feature request or general feedback. This method lets us prioritize these queries accordingly, instead of treating them like they’re all the same.

However, this final solution kind of sucks, I think. The problem is that it moves priority from the customer to the company, while giving an illusion of giving control to the customer. That is, the variable is categorical/nominal for the customer, but ordinal for the company. In other words, what is important to the company is more important than what is important to this particular customer. This is probably even more true for those customers for whom everything for them is the most important thing in the world. And yet.

I think perhaps a better solution splits the question into two, which provides space for both ‘urgency for the customer’ and ‘urgency for the organization’.

At the risk of adding yet another item to your survey/form, you have solved both problems with some cognitive ease: customers are defaulted to medium (which is easy for people to just ignore/skip over or shift with little cognitive difficulty), which would allow the organization to give its own priority to the categorical variable. If the customer changes the default to ‘not urgent’, this still stands. If the customer makes their own priority ‘urgent’, then the organization has some discretion on whether to treat this as ‘urgent for us’ or not, but at least has a sense of the panic level for the customer.

I’m sure there is an aesthetic here as well, but the general lesson should be two-fold. 1) Consider carefully the meanings behind your survey variables. Categorical variables often require thinking about, particularly as the category options become large. Ordinal and interval variables (which create an ordering) are easy, until they are too refined. Let’s say you are at a hospital, and a doctor asks you ‘how do you feel?’ Sorting through a list of adjectives that describe your feelings sucks. And assessing your level of pain between 1 and 7 is easier than between 1 and 1000.

The thing is, sometimes what is important to you is not what’s important to the doctor. If you feel throbbing, it’s not lethal. If you feel numbness, it is. In this case, the options are categorical to the patient, but ordinal to the doctor.

Which leads to 2) If you want your customers’ opinions, it may behoove you to give them a way to tell these to you. In the medical example, what kind of pain and how much does it hurt are two questions; patients care more about the second, even if doctors care more about the first. So don’t ask what kind of pain without asking how much does it hurt. Even if this saves the patient’s life, they will still be pissed at you for dismissing their subjective reality.

From CoreFiling’s insight blog: “It won’t be very long before it is those documents – the bar-coded financial disclosures – that will be the primary materials consumed by financial market systems to help analysts and investors make decisions about the best way to invest. This is vastly more sophisticated than today’s processes that rely on slow and inaccurate re-keying of a subset of the financial information published by companies.”

This is commensuration more than just standardization, since the tags are designed to be specific to a particular business enough so that everyone is not required to give the same information, yet the tags are standardized enough that everyone is required to give information that can be made comparable. The pitch for companies (other than, because otherwise we’ll fine you and take away your business license) is that XBRL will make their financial reporting less costly, less prone to error, and ultimately more efficient.

Personally, I think this is a flat out misrepresentation of what’s going on here. XBRL helps one group of people orders of magnitude more than anyone else: investors. And the trade-off between increased government efficiency and business streamlining of compliance data on the one hand, and increased ability for data-gatherers for banks, hedge funds, and the investor class is totally totally off the charts. What this will end up doing is: 1) creating a standard way for companies to report financials; 2) creating some increased efficiency for government entities to keep tabs on the finances of these organizations; and 3) create a massive additional datastream for financial services and investment firms to work with. If you think it is a challenge for public firms to resist making short-term decisions based on financial analysts’ quarterly reports of earnings now, wait until this information is directly readable by quant trading models.

This would be an amazing dissertation topic. I would track: a) the creation of the standard; b) the adoption of the standard around the world; c) how XBRL is being incorporated into financial modeling; d) the before-and-after effects of XBRL on market prices for firms; and e) qualitatively, what gets excised from XBRL, or rather, what remains incommensurable about firms, governments, etc.

UBMatrix
XBRL’s main site
US SEC’s ‘Interactive Data Viewers’
Microsoft uses XBRL
US GAAP XBRL Taxonomy (GAAP is the accounting standard in the US)
CoreFiling