The algorithmic hand is replacing the invisible hand. But Hayek still applies.

Last week Adam Ozimek posted an interesting piece Big Data Versus Hayek. He noted how centralized algorithmic price setting, for real estate, airlines and hotels, was displacing local and decentralized decision making. Because it works better and is more profitable. And how this development was very “un-Hayekian.” Friedrich Hayek of course famously arguing centralized planning and pricing could never do as well as the man-on-the-spot in taking into account local conditions.

Ozimek concludes with:

So is big data displacing Hayek? In some places, I would say yes, for sure. But we should not get ahead of ourselves in declaring the death of decentralized knowledge and decision-making. As a prediction, pricing, and choice-making device, algorithms and big data still have a long way to go in many contexts. And rather than being left in the dust, markets themselves are evolving too.

As you can see, it’s a balanced piece. And a provocative line of thought. In the machine learning era now upon us, the algorithmic hand is indeed replacing the invisible hand. But I would counter that the algorithmic hand’s domain is still narrow enough that the main thesis from Hayek’s 1945 essay “The Use of Knowledge in Society” (which Ozimek quotes) still stands. Albeit with some updates and concessions. So while I may not fundamentally disagree with Ozimek’s position, I want to make the case for Hayek.

Let’s start with a pure commodity market like the stock exchange. The London Stock Exchange moved to electronic trading in 1986, and the New York Stock Exchange in 2006. First, we must concede it is now possible for a computer to know all the raw data at once, and use a centralized algorithm to determine a best price. Hayek’s 1945 essay is out of date and now just wrong on this point. Nonetheless, there’s a big distinction between centralized algorithms finding a market clearing price versus having a predetermined and centralized plan setting that price. Obviously the latter is not happening. Stock prices remain as volatile and unpredictable as ever.

Let me quote from Hayek’s essay:

We must look at the price system as such a mechanism for communicating information if we want to understand its real function—a function which, of course, it fulfils less perfectly as prices grow more rigid…. The most significant fact about this system is the economy of knowledge with which it operates, or how little the individual participants need to know in order to be able to take the right action. In abbreviated form, by a kind of symbol, only the most essential information is passed on and passed on only to those concerned. It is more than a metaphor to describe the price system as a kind of machinery for registering change, or a system of telecommunications which enables individual producers to watch merely the movement of a few pointers, as an engineer might watch the hands of a few dials, in order to adjust their activities to changes of which they may never know more than is reflected in the price movement.

The marvel is that in a case like that of a scarcity of one raw material, without an order being issued, without more than perhaps a handful of people knowing the cause, tens of thousands of people whose identity could not be ascertained by months of investigation, are made to use the material or its products more sparingly; i.e., they move in the right direction. This is enough of a marvel even if, in a constantly changing world, not all will hit it off so perfectly that their profit rates will always be maintained at the same constant or “normal” level.

I have deliberately used the word “marvel” to shock the reader out of the complacency with which we often take the working of this mechanism for granted. I am convinced that if it were the result of deliberate human design, and if the people guided by the price changes understood that their decisions have significance far beyond their immediate aim, this mechanism would have been acclaimed as one of the greatest triumphs of the human mind. Its misfortune is the double one that it is not the product of human design and that the people guided by it usually do not know why they are made to do what they do. But those who clamor for “conscious direction”—and who cannot believe that anything which has evolved without design (and even without our understanding it) should solve problems which we should not be able to solve consciously—should remember this: The problem is precisely how to extend the span of our utilization of resources beyond the span of the control of any one mind; and therefore, how to dispense with the need of conscious control, and how to provide inducements which will make the individuals do the desirable things without anyone having to tell them what to do.

This seems exactly right for prices in a stock or other commodity market, even when an algorithm calculates the price of exchange.

Next consider Ozimek’s examples of real estate, airlines and hotels. There are a few differences. In these cases businesses use variable pricing models to extract maximum profit by segmenting the market. For example by selling airline seats to business users at higher prices than family vacationers. With that said, the algorithm’s goal is similar to the stock exchange algorithm’s goal: clear the market. In the first case by matching buyers and sellers, in the latter by setting a profit maximizing (variable) price. For both cases though, the algorithm responds to the market. It’s not setting prices to any preconceived plan. Hence both algorithms follow Hayek’s core insight: prices communicate information and allow coordination to unpredictable change. The algorithmic hand facilitates, remaining fairly light.

Let’s move on to when a customer has come to an online store, and algorithms suggest a product. For example Netflix recommending movies, Pandora recommending songs, or Amazon recommending products. Here the algorithm is taking on the role of salesman. Suggesting and enticing the customer to buy. Ultimately I don’t see an algorithm replacing a salesman as that big of change. The strongest case here is probably that Amazon presents different prices to different users to maximize profit. But of course this is merely the variable pricing model strategy already discussed, now applied to commoditized products.

Now consider when a potential customer uses Google search to find a product. In this case Google has incentive to steer potential customers to its own shopping sites, which is precisely why Google is getting sued in Europe on antitrust charges. This will get even worse when the starting point for search shifts from typing to voice interaction with Google Now or Apple Siri. My post here. Clearly the algorithmic hand weighs heavy. But the reason the algorithmic hand weighs heavy is due to one of the oldest economic problems in the book, natural monopoly. If there were multiple search companies that had equal market share, it’s not clear if this would be a problem at all. There have always been middlemen in economic transactions. So I’m not sure I would go so far as to say algorithmic abuse of monopoly is un-Hayekian. But it’s certainly a new twist on an old problem.

Finally, consider matching markets. Examples of matching markets are labor markets or choosing schools. With marriage being perhaps the ultimate matching market. In matching markets, both sides have to choose each other, so there is no single market price. Let’s walk through Alvin Roth’s example of parents choosing schools in the Boston school system. Parents force rank their preferred schools, but principals could see those choices and potentially reject parents who insulted the principal by ranking their school second. So there was an incentive for parents to rank as first not their preferred school, but the one they thought they were most likely to get into. This gaming dynamic resulted in non-optimal school matching and anger. Roth’s solution was to use a deferred acceptance algorithm to computer match the students. This allowed matching without the school’s themselves learning where the parents ranked them. What’s fascinating here is this is an example of consciously designing a market using an algorithm. Similar examples would be sharing economy companies like Uber or Airbnb, which use algorithms to build trust between market participants. They are algorithmic market makers. I’d recommend Ben Thompson’s recent post for more on the sharing economy. Going in, I would have rated matching markets as the most un-Hayekian case of the algorithmic hand.  Above algorithmic monopoly abuse. But in Roth’s recent interview by Russ Roberts on the Econtalk podcast, Roth surprised me by quoting Hayek.

So let me end by quoting Nobel Prize winner Alvin Roth, who in turn quotes Nobelist Friedrich Hayek. From Econtalk (great episode by the way):

But let me read you the quote from Hayek that I included in my book. This is a quote from his free-market manifesto, The Road to Serfdom. And he wrote, “There is, in particular, all the difference between deliberately creating a system within which competition will work as beneficially as possible and passively accepting institutions as they are.” So, that was Hayek. He understood that what makes a market free is that it has rules that allow it to work freely. And one of the metaphors I use in the book is of a wheel that can rotate freely. It’s not rotating in a vacuum. It has an axle and it has well-oiled bearings. And over time–people have been designing markets for millennia. And often the process of trial and error leads to better and better markets. But it can be a lengthy process of trial and error. And as we better understand what is required for marketplaces to help markets work freely we can sometimes intervene. And, you said ‘top down,’ but earlier you talked about Uber and Airbnb. Those are marketplaces that are not top down. People have been designing marketplaces forever. It’s what we do.

______________________________________________________

APPENDIX WITH LINKS:

• Note: Hayek clearly says markets are not “designed.” But I would read the intent of Roth’s final statement as saying the rules of marketplaces can be iteratively tinkered with and experimentally reviewed. That caveat could keep us within Hayekian territory. It’s too bad they ended there as it would have been interesting to delve into how Hayekian Roth’s beliefs really are.
• Adam Ozimek’s post that inspired this one Big Data Versus Hayek
• Frederich Hayek’s original 1945 essay The Use of Knowledge in Society
• Alvin Roth on the Econtalk podcast hosted by Russ Roberts Alvin Roth on Matching Markets
• Ben Thompson on the sharing economy and trust Airbnb and the Internet Revolution. Also Netflix and the Conservation of Attractive Profits.
• My post on Talking computers pose a threat to current Apple versus Google market segmentation. Beyond Peak Google.