A number of tools are around to improve decision-making by providing information. The latest, announced by IBM, is “stream computing“, a somewhat mysterious-sounding technology that aims to improve the performance of single-threaded applications on multi-core chips. The seemingly most important application is that stream computing can aid in analyzing digital data as it comes into a database, rather than waiting for it to be stored. It’s an improvement in data-mining, the idea being that you don’t have to wait to analyze data – you can do it in real-time. Think search-engines, and more interestingly, think financial services.
What’s interesting about this is the convergence of this technology and an increasingly important practice in financial services – black-box trading. There is a proximate phenomenon here, and a more general one. Proximately, black box programs are designed to take advantage of smaller and smaller increments of trading disparities, in real-time. So for arbitrage, this would mean small differences between two similar products trading at slightly different prices (like, say, the varied information trading sites and presidential contenders), but it could also mean algorithms based on historical variations in volatilities, relationships between products, and the like. Stream computing would allow one to not wait for a database to test and mine data, but would be able to do it in real time, as data came in. This would likely provide an advantage over other, slower players.
At what point, though (and this is the general question), does information mining actually replace decision-making. In sociology, C. Wright Mills made the argument a half-century ago, this is leads to the vacuousness of abstracted empiricism. In business decision-making, it leads to an abdication of decision-making in favor of empirical data mining. The problem here (following James March) is that the world is not just uncertain, it is ambiguous. If the world were simply uncertain, reduction of uncertainty via the aggregation of more and better information might prove just the ticket. But what happens when a decision has to be made between qualitatively different options? When more information does not provide a clear direction to go? Or when decision-making could actually increase, decrease, or change in fundamental ways the options themselves? At this point, information-mining actually becomes harmful to the extent that it replaces rather than augments real decision-making. Worse, making decisions is a skill, that needs to be flexed, and used. Understanding when and how information-mining would be useful seems to me a more important ability than even knowing how to manage the information-mining itself.