Betting the Business

Monday’s post discussed a proposal by Vikram Pandit, the CEO of Citigroup, calling for a comparison of the results produced by risk models across different banks when evaluating a standardized “hypothetical” portfolio of assets. Exercises like this are standard fare in many research fields where modeling encompasses a broad array of complicated issues, and there can be wide disparity in both structural and parameter choices.

Let’s look at the economics of controlling greenhouse gas emissions. A number of major research institutions have developed economic models to assess the costs of mitigation, including one of the MIT centers where I work. The results from these models play a useful role in public discussion about what should be done.

But these models are inherently very complicated. Greenhouse gases are produced in many different sectors of the economy, in particular in the energy sector which is, in turn, an input to many other sectors, so an economy wide model is required. The greenhouse gas problem operates at long time scales, so technological evolution over many decades is key. Finally, greenhouse gases are a global public goods problem, so a global economic model is required. Building a model to meet these demands is a heroic effort, demanding many judgment calls on major issues.

Understanding the different assumptions and structural choices and how they impact the results is useful, and can shape how the results are read. During the Bush administration, the U.S. government, which underwrites much of the research in this area, ran a comparison exercise on these economic models similar to what Vikram Pandit is proposing for bank risk models. They took 3 of the leading models, and had them generate a suite of diagnostic results when analyzing a common set of policies. They then published an analysis of the different results and the underlying modeling choices that generated the differences. This was done as a part of the Climate Change Science Program and the full results can be found here.

To illustrate the variation across models, here’s just one of the diagnostics, the forecasted change in the price of natural gas. The three models produced strikingly different results. One model produces a forecast that increases more than 800% by the end of the century under modest emissions constraints, while another forecasts increases less than 200%.

Seeing such widely variant results is an eye opener. Novices in any research area often take modeling results for granted. Seasoned researchers are more attuned to the weaknesses and uncertainties and range of different opinions across the scientific community. Comparison exercises, like the one done by the U.S. Climate Change Science Program, or like the one Citibank’s Vikram Pandit is proposing, shine light on differences that the public needs to be better attuned to.

Of course, knowing that there are differences isn’t the end of the process, and doesn’t solve all of the problems. But it’s a useful contribution.