It was the bold title of a conference this month at the Massachusetts Institute of Technology, and of a widely read article in The Harvard Business Review last October: “Big Data: The Management Revolution.” At the M.I.T. conference, a panel was asked to cite examples of big failures in Big Data. No one could really think of any. Soon after, though, Roberto Rigobon could barely contain himself as he took to the stage. Mr. Rigobon, a professor at M.I.T.’s Sloan School of Management, said that the financial crisis certainly humbled the data hounds. “Hedge funds failed all over the world,” he said.
The problem is that a math model, like a metaphor, is a simplification. This type of modeling came out of the sciences, where the behavior of particles in a fluid, for example, is predictable according to the laws of physics.
In so many Big Data applications, a math model attaches a crisp number to human behavior, interests and preferences. The peril of that approach, as in finance, was the subject of a recent book by Emanuel Derman, a former quant at Goldman Sachs and now a professor at Columbia University. Its title is “Models. Behaving. Badly.”
Claudia Perlich, chief scientist at Media6Degrees, an online ad-targeting start-up in New York, puts the problem this way: “You can fool yourself with data like you can’t with anything else. I fear a Big Data bubble.” She is worried about a rush of people calling themselves “data scientists,” doing poor work and giving the field a bad name. Indeed, Big Data does seem to be facing a work-force bottleneck.
“We can’t grow the skills fast enough,” says Ms. Perlich.
“Models do not just predict, but they can make things happen. That’s not discussed generally in our field.” Models can create what data scientists call a behavioral loop. A person feeds in data, which is collected by an algorithm that then presents the user with choices, thus steering behavior.
Personally, my bigger concern is that the algorithms that are shaping my digital world are too simple-minded, rather than too smart.
Read more at the New York Times
