Risk modelling firm Risk Management Solutions (RMS) was the
subject of a scathing article in Florida’s
Herald Tribune in November. The article said that
following Hurricane Katrina in 2005 RMS aggressively pushed a
new short-term model.
The lurid article, headlined "Florida insurers rely on
dubious storm model", stated: "RMS, a multimillion-dollar
company that helps insurers estimate hurricane losses and other
risks, brought four-hand picked scientists together in a
Bermuda hotel room. There, on a Saturday in October 2005, the
company gathered the justification it needed to rewrite
hurricane risk. Instead of using 120 years of history to
calculate the average number of storms each year, RMS used the
scientists’ work as the basis for a new crystal
ball, a computer model that would estimate storms for the next
five years. The change created an $82bn gap between the money
insurers had and what they needed, a hole they spent the next
five years trying to fill with rate increases and policy
The article goes on to say that some hurricane specialists
are now sceptical of RMS’ claimed "scientific
consensus" for its new model. "Today, two of the four
scientists present that day no longer support the hurricane
estimates they helped generate," the article says. "Neither do
two other scientists involved in later revisions. One says that
monkeys could do as well."
The article adds that as a result of RMS’s
model change the cost to insure a home in parts of Florida hit
(Full disclosure – Reactions and RMS share an
ultimate parent in the Daily Mail & General Trust).
A number of points should be made in response to this
article. Firstly, it is not risk modellers that set rates
– insurance and reinsurance firms do that. Secondly,
no one is forcing the reinsurance and reinsurance firms to use
the RMS model. It is only one of several modelling firms.
Thirdly, all the models were way off on Katrina, so the loss
estimates were always going to increase (the question was by
how much). Lastly, it is ludicrous to blame all of the rate
increases after hurricane Katrina on a model change. Capital
was depleted – the laws of demand and supply kicked
in, pushing up pricing. The effects of model changes came on
top of that.
The problem for RMS is the higher losses its new model
predicted never came, making it look like the near-term model
was not needed.
RMS responded in a letter to the editor of the Herald
Tribune. It, not unreasonably, pointed out that the models
merely deliver probabilistic forecasts not deterministic
predictions. It also said that there is a widespread agreement
among scientists that the number of North Atlantic hurricanes
has increased since the 1970s and that since 1995 activity has
been "significantly higher than the long-term average since
1900". The question, RMS said, is how much higher is the
frequency and how will it impact hurricanes making landfall in
RMS said: "The scientists were deliberately kept at a
distance from the commercial implications of the
recommendations. In our annual review of medium-term activity
rates (the next five years), we have worked with a total of 17
leading experts, representing a broad spectrum of
It added: "There is no commercial advantage for us to
overstate the risk."
But the article does raise an important issue – the
over-reliance of the industry on models. The Herald
Tribune article stated: "RMS continues to promote its
short-term model as the preferred option for its customers. A
survey by Bermuda officials shows it is the dominant model for
Bermuda reinsurers, the most crucial source of private
hurricane protection for Florida."
This is further evidence that insurers and reinsurers need
to question what the models tell them. This is not the
modellers’ fault. The models will never be perfect
and they will always be wrong. Ironically, the models normally
come in too low.
But equally, modellers should be careful not to overstate
the accuracy of their models also. Karen Clark, CEO of risk
modelling consulting firm Karen Clark & Company and founder
of AIR Worldwide, an RMS rival, believes this is a problem. "I
thought it was a pretty good article," Clark told me after the
Herald Tribune article came out. "I agree with the
gist of what it was saying."
Clark says three things were on the minds of insurance
companies after Katrina: the industry had just had two years in
a row of big cat losses; there was a sense that all models had
been too low on Katrina; and there was a tremendous amount of
press around Katrina, some even saying Katrina was caused by
"There was pressure to come out with models," says Clark.
"What happened is RMS came out with a near-term model that said
over the next five years hurricane frequency would increase by
40%. That is fine if there is some science behind it to support
an increase in frequency but quite another thing to say, OK, we
throw out the model we have been using and move to the
Clark says too much emphasis was placed on the near-term
model by RMS.
"They were kind of oversold," says Clark. "It was money to
real people. A small insurer may have to buy a lot more
reinsurance and not be able to afford it. So the question is:
is the science enough to justify it? A handful of scientists
came up with that number and they never really had that use in
mind for it."
Clark says modellers overemphase the outcomes of their
models, which can give answers down to the cent. She says the
data behind the models is simply too uncertain to be that
precise, and that a range should be provided instead.
"As humans we fundamentally don’t like
uncertainty," she says. "CEOs value a number. The problem is
that the modellers do oversell the answer the models give. When
models change it is just new research changing the number. So
we have to change that; we have got to start talking about the
unknown, rather than the known. We need to move to recognising
the uncertainty, both as modellers and as users."
In short, the market needs to stop taking the
models’ output as gospel and modellers need to
stop acting as if their models are more than an educated
Some executives have also urged caution when using the
models. Former Hannover Re CEO Wilhelm Zeller was fond of using
the phrase, 'A fool with a tool is still a fool,’
in reference to risk models. Tad Montross, CEO of Gen Re, has
also highlighted models shortcomings while adding that they are
still the best thing the market has to help assess its
Writing in our CEO Risk Forum 2010 earlier this
year, Montross said: "Probabilistic cat models have been around
for 30 years and have brought much greater discipline and focus
to the management and quantification of catastrophe exposures.
This has been a positive development for the industry. Having
said that, the actual track records of the models have not been
good. The one thing we can all agree on is that the model
estimates are wrong. Just last year the initial estimates for
Hurricane Ike were 50% to 60% off the mark. So the actual to
modelled variance can be huge – suggesting a large
margin of safety is appropriate when using these tools to
measure capital at risk.
"Particularly, in extreme events, the variance can be even
larger since the calibration is more difficult. Why do we
invest so heavily and spend so much time using cat models to
measure and manage our accumulations? Simply because
while imperfect, they are the best tool we have.
"While many industry reports and analysts speak to the 1% or
0.2% loss amounts, few qualify their statements with supporting
information on how the model was actually used and
parameterized. The judgments, with respect to occurrence vs
aggregate loss amounts, VAR vs. TVAR, storm surge, medium vs
long term frequencies, loss amplification, secondary
uncertainty and data quality/resolution can produce wildly
different loss estimates. In some cases the range can be
twofold. Startling, given the aura of precision the EP
(exceeding probability) curves project."