It’s time epidemiological models faced the same scrutiny as banks
After the global financial crisis of 2009, Basel, the international standard setting body for banking regulation, conducted a thorough analysis of banks’ internal models.
Banks had been allowed considerable discretion in the way in which they modelled their risks to justify the amount of capital they should hold against them. Basel’s analysts found that the internal models were complex and opaque, with worryingly large variations in outcomes. A bank’s reported capital ratio could vary as much as 50 per cent for the same hypothetical portfolio.
Basel did not propose banning banks from using their own models to assess the amount of capital they should hold above the required minimum. They could continue to use them, but only if they set up a validation process which had to be entirely separate from the model developers and the commercial risk-taking functions.
Its staff have to have all the relevant knowledge and skills, including understanding the business to which the models apply. The rules involve checking the plausibility of the model’s outputs, the quality of the databases and quality assurance of the computer code.
It does not stop there. Senior management must review the monthly stress testing of the models and take action to mitigate any risks arising from any vulnerabilities of the model which come to light. The decision to use or continue to use particular models has to be taken by the board. In addition, the models have to be validated by external auditors and the regulators. The models themselves may be open to criticism by academics and other experts in risk modelling.
It is clearly vital for banks to be obliged to meet stringent standards for their internal models, since they are fundamental to public policy and the functioning of the economy.
Epidemiologists play a similarly significant role in the wellbeing of the nation, as we have seen since the start of the pandemic. As such, their models should be subject to similar rigorous scrutiny.
It’s true that academic and other institutions may not be able to set up the same kind of internal independent function that banks are obliged to establish. So instead, there should be an independent body, staffed not only by academics in the same discipline (given the dangers of group-think) but by specialists in a wider set of areas, including those with a broad knowledge and experience of people’s living conditions and the economic implications of certain strategies.
Just as banks are obliged to test their models frequently and to document fully any issues with a model, so epidemiologists should be required to do the same and update or reject a particular model. The UK’s Science and Technology Committee would do well to look at such checks and balances.
On that score, the infamous model from Professor Neil Ferguson on which Britain’s lockdown policies were based is likely to have failed. From all accounts, his modelling seems to have produced inaccurate results in the past, such as the predictions of 200 million deaths worldwide from Bird Flu in 2005 (when just 282 people died between 2003 and 2009).
Ferguson has provided some details of his model since his original document was released on 16 March. It was based on an undocumented 13-year-old computer code for a feared influenza pandemic. The simulation of social contact data was based on a model built in 2005 to see what would happen in Thailand if HSN1 avian flu mutated to a version that spread between people.
Once the dramatic figures in Ferguson’s worst-case scenario were released to the media, lockdown was the only politically viable solution. This release precluded any more considered response or examination of the resultant damage to so many people’s lives of a nationwide lockdown.
Ferguson’s original model has never been released to the public. It has, however, been released to GitHub, the software development company owned by Microsoft, which describes its status as being in active development and subject to significant code changes to “enable modelling of different intervention scenarios and to improve performance”. In other words, it is by no means definitive, and there is significant room for improvement.
All of this underlines the need for the independent scrutiny of models on which politicians depend to shape public policy in such difficult times. We need an independent commission reviewing the models, whose membership should include a broader range of experts such as engineers and risk officers from financial institutions, who understand that the misuse of a model can have serious and immediate repercussions.
Bankers and their regulators have had to learn the hard way. Their models are continually updated in the light of new data, and then considered in a wider context. When the decisions they make affect a country’s entire economy and every aspect of the lives of people living there, such analysis and inspection is essential.
If the epidemiological models had been subject to such scrutiny, I have no doubt that we would have found other ways to protect those who were seriously at risk, and would not be in the situation we are in now, with so many lives destroyed and the economy at breaking point.
Main image credit: Getty