Rethinking Markets

Over at 37 Signals, they have a regular series detailing their design decisions. It is an insightful feature and an insightful blog.

Their latest discussion is about how they managed a question on their support forms. I want to drop some research methodology on this problem. While their discussion is about how to design a feedback form, it is also about the kinds of questions you should ask on a survey. And it would benefit from a discussion of categorical/nominal variables, ordinal variables, and interval-level variables. Yep.

So, terms. Categorical variables (also called nominal variables) are those variables with 2 or more ‘states’, but without an intrinsic ordering. Male/female is categorical, as is eye or hair color, race, what school you went to. Ordinal variables are those variable with two or more states that have an ordering to them. Low/medium/high are ordinal variables. Less than HS, HS, some college, BA is ordinal. Interval-level variables are ordered, and the distance between categories is evenly spaced. Income, height, and years of education are all interval-level variables.

The difference between categorical/nominal variables and ordinal variables is the hierarchical ordering of the latter. What school you went to is nominal, but tiered ranking of what school you went to is ordinal. Tiered ranking may be ordinal, but amount of school endowment is interval-level. And quantitative variables are a hint that the variable is ordinal or interval, but not decisive (zip codes are nominal, for instance).

So, back to 37 Signals. What they want is some variable that would be easy to understand (from the customer perspective) and helpful to process (from the company perspective). Their first attempt looked something like this:

It takes time to think through what my state of mind is, because the items are almost ordered, but not really. Confused, worried, upset, and panicked are not points on a continuum, they are just different states of being. The question is asked as a categorical variable question. But it is one that they reallywanted to be an ordinal variable question. They tried to solve the problem by formalizing an ordinal variable, and putting a numbered ordering system on it to make it clearly so:

This is easier to deal with as a customer, since you can sort of pick up your relative state of panic. In other words, you pick up that there is a rough ordering very quickly, and the numbers help a lot in this respect. But alas, what’s good for the customer was no good for 37Signals. The reason is that while they began by wanting to know the subjectivity of the customer, what they really really wanted to know was, ‘how important is this problem for our company?’. Reasonable, but different. This is their final solution:

They have a reason for this, and it is a decent one:

Now, if something’s broken, we can spot it and fix it right away. A system failure is much more important to us (and our customers) than a feature request or general feedback. This method lets us prioritize these queries accordingly, instead of treating them like they’re all the same.

However, this final solution kind of sucks, I think. The problem is that it moves priority from the customer to the company, while giving an illusion of giving control to the customer. That is, the variable is categorical/nominal for the customer, but ordinal for the company. In other words, what is important to the company is more important than what is important to this particular customer. This is probably even more true for those customers for whom everything for them is the most important thing in the world. And yet.

I think perhaps a better solution splits the question into two, which provides space for both ‘urgency for the customer’ and ‘urgency for the organization’.

At the risk of adding yet another item to your survey/form, you have solved both problems with some cognitive ease: customers are defaulted to medium (which is easy for people to just ignore/skip over or shift with little cognitive difficulty), which would allow the organization to give its own priority to the categorical variable. If the customer changes the default to ‘not urgent’, this still stands. If the customer makes their own priority ‘urgent’, then the organization has some discretion on whether to treat this as ‘urgent for us’ or not, but at least has a sense of the panic level for the customer.

I’m sure there is an aesthetic here as well, but the general lesson should be two-fold. 1) Consider carefully the meanings behind your survey variables. Categorical variables often require thinking about, particularly as the category options become large. Ordinal and interval variables (which create an ordering) are easy, until they are too refined. Let’s say you are at a hospital, and a doctor asks you ‘how do you feel?’ Sorting through a list of adjectives that describe your feelings sucks. And assessing your level of pain between 1 and 7 is easier than between 1 and 1000.

The thing is, sometimes what is important to you is not what’s important to the doctor. If you feel throbbing, it’s not lethal. If you feel numbness, it is. In this case, the options are categorical to the patient, but ordinal to the doctor.

Which leads to 2) If you want your customers’ opinions, it may behoove you to give them a way to tell these to you. In the medical example, what kind of pain and how much does it hurt are two questions; patients care more about the second, even if doctors care more about the first. So don’t ask what kind of pain without asking how much does it hurt. Even if this saves the patient’s life, they will still be pissed at you for dismissing their subjective reality.