TAKE a look at this chart, showing an index of Phoenix, Arizona home prices over the past two decades:

From this, it looks as though a bout of temporary insanity broke out in 2004, lasted about two years, and then came to an abrupt and painful end. It may seem hard to square market gyrations like this with any reasonable conception of consumer rationality. How, in any sensible accounting, could a home that went for one price in 2004 and 2009 go for more than twice that price for a short time in between? The 2006 level looks absurd. And yet Phoenix home prices rose 23% in the year to January. Three more years like that (and home-price momentum only appears to be building) and values would be right back at bubble highs. Is another round of madness following directly on the heels of the last, or is something else going on?

In a new NBER working paper, Edward Glaeser examines America’s long history of property booms and busts and reckons that the assumption of irrationality actually fits the data rather poorly. Wild property price swings are more consistent with great uncertainty about the future. This week’s Free exchange column explains:

The Chicago boom of the 1830s reached even greater extremes. At the time, water access was critical to trade. In 1816 it cost as much to move goods 30 miles over land as to ship them across the Atlantic Ocean. Land near key ports and shipping routes was therefore extremely valuable. The Erie Canal led to economic booms around the Great Lakes, and Chicago’s proximity to the Mississippi river system made it an attractive bet. In 1830 Chicago land went for a song at $800 per acre (in 2012 dollars). In just six years the value soared to $327,000 per acre, with some plots fetching $1m. Tighter international credit conditions led to panic in 1837. By 1841 prices had fallen back to $38,000 per acre.

Yet this was more a product of unpredictability than irrationality. Given the risk that Chicago might fail to become a great metropolis, values immediately after the crash look low but justifiable. Prices at the peak were also consistent with reasonable views. At the time Chicago’s prospects looked uniquely bright. Land values in 1836 made sense given the defensible assumption that Chicago prices would rise to a fourth of those in New York city. And, Mr Glaeser notes, people who bought and held land through the crash prospered over the next two decades: average annual returns through to 1856 were about 9%.

If there is a weakness in market thinking it is not necessarily irrationality, but instead a tendency to underestimate the supply response to high prices:

If investors are not irrational, they may nonetheless fall prey to a dangerous nearsightedness. Over the long run high prices lead to more supply, a dynamic often forgotten in the heat of a boom…

New York and Chicago boomed in the early 20th century. Adjusted for inflation, land values in New York City leapt by more than 50% in the 1920s, buoyed by high rents and new high-rise technology that boosted the earning potential of a single property. Investors were slower to appreciate skyscrapers’ effect on supply. The erection of 50-storey buildings on just half of Chicago’s Loop business district would have generated a tenfold increase in its 1933-era square footage. In practice, it took less construction than that to send prices down.

This is what makes booming values in Phoenix so mysterious. Over the long run, Mr Glaeser notes, the price of something should converge toward the cost of producing it. Home prices in Phoenix were well above construction costs, and Phoenix seemed to have little trouble adding new supply. Arizona is hardly running out of desert to convert into new residential development. Why were prices able to rise so much and why are they rising once more? Maybe the failure to consider supply growth could have generated one boom and bust, but could it also produce another immediately after?

These patterns suggest that something else could be going on—perhaps that housing, as a credit-driven sector, is inherently bubbly. Karl Smith has made this point. Rising home values can make both lending and borrowing more attractive, encouraging more people to buy first homes or sell old ones and buy new ones. In housing, in other words, rising prices can increase demand, at least for a while. The same dynamic occurs on the way down; falling prices make people reluctant to enter the market and cause lenders to tighten credit standards, reducing overall demand. Meanwhile, supply responds on a lag. Construction may ramp up during the boom, but it also collapses during the bust, during which time population continues to grow. Eventually, supply tightens enough to raise rents and prices, touching off another boom. Unless, that is, credit standards are prevented from tightening and loosening in a pro-cyclical manner, or unless supply can be made to respond quickly and substantially enough to dampen the price spiral.

We don’t have to rely on that mechanism entirely to explain the odd example of Phoenix, however. As Mr Glaeser notes, some inland cities with lots of supply growth may have nonetheless seen prices rise because buyers were comparing prices across cities. Just as soaring Chicago prices looked reasonable given the assumption that property values would rise to a quarter of those in New York City, Phoenix home prices didn’t seem crazy given the assumption that they might move toward prevailing values in California (a source, it’s worth pointing out, of many Arizona buyers and investors).

That still looks a little crazy. Coastal California cities are hemmed in by mountains and ocean, household incomes are generally much, much higher in Los Angeles and San Francisco than in Arizona, and the climate is generally considered to be much superior on the coast than in the interior desert. But while we might always expect coastal California homes to have a climate premium, the other factors are less certain. Geography constrains California cities, but as the Chicago example demonstrates small areas with tall buildings can add enormously to housing and office supply—if zoning codes allow. Were the city of San Francisco built at Manhattan density, it could accommodate about 8m people, or roughly 7m more than it currently houses. Zoning trumps geography where supply is concerned.

And higher incomes on the California coast aren’t due to any inherent economic advantage but to the fact that cities there are home to extremely productive agglomerations of human capital. The people living in the Bay Area would be roughly as productive as they are now if they were transported to some other American location.

And so in recent price gyrations we have another major source of uncertainty. If one believes that sky-high costs in productive cities will eventually put pressure on governments to change zoning rules, leading to a boom in new housing and office supply, then high Phoenix values look absurd. But if one thinks that California’s unrepentant NIMBYs will never allow coastal cities to grow as markets would prefer, then the picture changes. Inland migration may well continue, or even accelerate. If one assumes that such migrations will lead to the development of productive agglomerations with their own attractive gravity in inland cities, that too offers reason to anticipate more price convergence in places like Phoenix. And if you think that zoning in Phoenix will come to look ever more like that in California, then all the desert in the world might not matter. Supply growth in places like Phoenix will slow even as values rise.

To put things another way: if it seemed as though housing in one city were perfectly substitutable for housing in another, then so long as supply were flexible somewhere we wouldn’t expect wild price gyrations to come one after the other. Maybe you get one boom and bust, but thereafter it should be clear that high values in coastal cities aren’t sustainable, because they just shift demand to Phoenix or Houston.

But housing across cities isn’t perfectly substitutable. One might say that (for our purposes) there are two types of markets. There is a Type I market where demand is very high and supply growth is constrained. And there is a Type II market where demand is moderate but supply is almost perfectly flexible. In a boom phase prices rise rapidly in Type I cities and population grows rapidly in Type II cities, while population growth is flat in Type I cities and price growth is flat in Type II cities. But given enough population growth, a Type II city may flip to Type I status, as skilled workers generate a high productivity cluster while NIMBYs worried about congestion issues begin to fight new growth. “Flipping” means a big jump in the housing price level. And so to some extent, the behaviour of values in cities like Phoenix may be a measure of market expectations of the probability of an eventual flip.

Markets may turn out to be wrong, or they may turn out to be right. Given uncertainty, however, it would be wrong to call them out-and-out irrational.