A comprehensive review of a comprehensive review

This is a free-to-view edition of Unpacking Climate Risk. Become a paying subscriber today to get articles sent direct to your inbox three times a week.

If I was offering a comprehensive review of a new film, what would you expect to read?  You’d presumably want me to discuss the performances of the actors and the quality of the screenplay.  I’d bet you’d also appreciate an exploration of the more detailed elements of the production, covering the costumes, lighting and special effects.  

Most critical in determining the film’s greatness, though, will be whether it fulfills the mission the producers set out to achieve.  As an overall package, does the film work?  If it’s a popcorn movie it would have to be enjoyable, fast-paced, and distracting.  If it’s a potential Best Picture Oscar winner it will most likely need to be profound and affecting.  

I’ve seen lots of films that were beautifully shot but were ultimately a complete waste of celluloid (or whatever the digital equivalent might be).  Some of my favorite films, by way of contrast, would be considered fairly agricultural by the latte-swilling, beret-wearing movie-buff set.  I once saw a New York Times review that described “Dude, Where’s My Car?” as the pinnacle of its genre (though I remember the article distinctly, alas I can’t find a link).  I enjoyed the flick immensely but it wasn’t particularly profound.  

The Times reviewer in question got it spot on.  But I digress.

So when Louie pointed me to “A Comprehensive Review of Global Supervisory Climate Stress Tests”, recently published by the UN Environment Programme Finance Initiative (UNEP FI), I expected to find a detailed critique of the exercises that have drawn so much of our attention over the past few years.  Are the producers of these exercises actually achieving what they set out to do?  

Furthermore, given that the UENP FI review covers an entire genre, as opposed to an individual movie, I also expected that it would identify one or more “Dude, Where’s My Car?”-style stress tests: in other words, those that were crude but effective.  I was sadly disappointed.

In fact, I found the Comprehensive Review to be less than comprehensive.  A much more useful roundup was provided by the participants of the US Fed’s recent climate scenario analysis, who highlighted a large number of deficiencies in the overall methodology.  It doesn’t take long to realize that UNEP FI is not interested in comprehensively reviewing the practice of climate stress testing.  

The abstract for the report starts thusly (my emphasis):

“The use of climate stress tests to evaluate the vulnerability of financial institutions to climate change impacts is gaining momentum globally. As supervisory authorities, regulators, central banks, and policymakers aim to understand the financial risks posed by rising global temperatures, these exercises have become crucial in assessing systemic vulnerabilities. Many financial supervisors have already conducted climate stress tests or are planning to do so.

Climate stress testing is a crucial forward-looking approach for assessing the potential impact of climate change on financial institutions and identifying climate risks as systemic risks. Robust data plays a key role in enabling financial institutions to manage and identify climate-related risks and opportunities, facilitating more informed decision-making amid evolving climate challenges. However, the development of climate stress testing and its integration into the strategy of firms has been hindered by a lack of standardized data with the necessary quality to effectively assess and model the potential financial impact of climate risks. Current data limitations often lead to underestimation and only partial measurement of climate-related risks.”

I included a long quote to ensure I didn’t mis-contextualize.  I think this passage should be unpacked in detail because it goes to the heart of what the authors of the report are setting out to achieve.  The bolded phrases will be the main focus of my own comprehensive review.

There’s no doubt that bank regulators love scenarios and that the technique is gaining in momentum.  Scenarios are cheap and easy to produce and they’re impossible to refute, making them the perfect tool to use for many regulatory applications.  

But when you say that regulators are using them with the aim of understanding financial climate risk, it’s important that you question how well this aim is being achieved. 

So, what is the most profound insight into financial climate risk unearthed by several years of scenario analysis?  If, as you say, the exercises have been “crucial” in assessing system vulnerabilities, can you point to some revealed weaknesses that would otherwise have been overlooked had scenario-based stress testing not been used?  

I like to make fun of the fact that the only insight the Fed’s major study revealed was that insurance helps to mitigate losses from physical risk.  This notion was well understood by Babylonian King Hammurabi in the second millennium BC.  It was inscribed on a basalt stele that UNEP FI researchers could inspect on a summer visit to the Louvre.

The UNEP FI review highlights a number of other alleged critical insights.  In 2022, the Australian Prudential Regulatory Authority (APRA) found that some locations are more exposed to flood and wildfire risk than others and that diversification can help to ameliorate risk.  The ancient Mesopotamians apparently knew about the risk-reducing benefits of diversification and I suspect they also knew that some locations were more appropriate than others for building houses. 

The ECB in 2022 independently rediscovered the secrets of the Mesopotamians by also working out that physical risks may rise as global temperatures increase.

Is this it?  Surely there’s something more profound to be wrung from more than half a decade’s worth of scenario analysis?

Perhaps not.  I would posit that the exercises conducted thus far have yielded no meaningful insights.  Instead, they have been used to apply a veneer of “analysis” to the prior views rigidly held by their designers.  If the UNEP FI authors disagree with this assessment they can always provide a list of the most significant ways that regulators’ views have changed since the stress testing projects kicked off.   Personally I can’t see that their views have moved one iota.

Bear in mind that any useful insights would actually be gleaned via the diligent background empirical work undertaken by researchers working in the field.  Some will suggest that this provides a justification for the use of scenario analysis generally.

The problem is that the production of portfolio-level scenario projections always requires the designers to make a number of untestable assumptions and apply subjective overlays.  If these are done in a neutral way that is consistent with the empirical evidence, it may not cause a problem.  But if the overlays are bold – and in my experience they always are – then any insights gleaned from the empirical exploration can only be obscured by the requirement to produce portfolio-level scenario projections.

The UNEP FI authors continue by suggesting that the scenario approach is crucial (that word again) in assessing the potential impact of climate on financial institutions.  I’ve been grappling with the inclusion of the word “potential” – the purpose of stress tests is not to assess the actual impact of climate change on banks, but the potential impact?  This again seems like an acknowledgement that the scenario-based methodology is somehow removed from the underlying evidence.  The potential impact could be a nothingburger, or it could be the complete breakdown of the global financial system.  That’s the beauty of potential.  

Moreover, identifying only the potential impact is too easy and too vague a goal to be useful. I’m more interested in learning about the actual impact of global warming and the transition response on the financial system.  If scenario analysis isn’t trying to uncover this, then what’s the point?

The next phrase suggests that stress testing is “crucial” for “identifying climate risks as systemic risks.”  This proved a head-scratcher.  Of course we have to draw a sharp distinction between systemic and systematic risk.  The first is generally viewed as the possibility that a shock could lead to banks falling like dominoes, causing a full-blown financial crisis.  The second is more mundane – the situation where shocks can be transmitted, for better or worse, across multiple financial organizations.

Right now, in an environment where we can’t yet identify a single climate-related bank failure, it seems premature to raise the alarm about system failure.  I suspect, though, that if climate risk ultimately develops and that financial losses start to become excessive, there’s a strong possibility that the nature of the process will have systemic features. 

But scenario analysis is “crucial” for identifying this?  Surely it’s possible to maintain the idea that climate risks are systemic while conducting a traditional empirical exploration of the underlying data?   Once financial losses start to mount we will have time to empirically explore them and test theories about how systemic risks might develop.  We have to remember that humanity will survive the next banking crisis. The urgency that should be applied to resolving the climate crisis is not required for questions related to the financial system.

Finally, we get to perceived data inadequacies and the apparent underestimation of climate-related financial risk.  It’s important to point out here that climate stress tests seem to only ever focus on the losers from climate change, making no effort to account for the likely winners.  These winners will be those smart or lucky enough to own assets that are sheltered from climate impacts. The value of these assets will rise as the physical costs of climate change, borne by others, trigger cascading supply shocks.  This, of course, suggests that financial losses, if anything, are over-predicted by climate stress tests – but we’ll set this aside for now.

Focusing only on the losers, does the evidence suggest that predicted losses are being lowballed?  The UNEP FI review’s insistence on underestimations implies that there have been cases where the predictions made by past tests have fallen short.  But climate-related losses have generally remained very low, so a failure of prediction has not occurred.  How do the authors know the forecasts are underweight?  Or is this an untested assertion?

Ultimately, the UNEP FI report does an excellent job of describing the cinematography, lighting, costume design, special effects and acting performances underlying all the major climate stress tests that have been conducted to date.  If you’re happy to assume that scenario-based exploration really is crucial to our understanding of climate risk, the report provides a comprehensive discussion of the details that have emerged in previously conducted tests.

But the report fails to mention whether the overall story holds together, or whether the director was able to produce a coherent movie that achieves the lofty goals that were established at the outset.  

I still don’t know who was responsible for the “Dude, Where’s My Car?” of the climate stress testing genre.

Two stars.