The BFD | Computer Models: Garbage In = Garbage Out.

Angus Aardvark

Computer Models, Garbage in = Garbage out. Image credit The BFD.

Computer modelling is very useful: I developed models and used them for many years of commercial prediction work. They are a really powerful tool.

But ONLY if you have plenty of feedback and constantly update the model so it can tweak itself to match the real world.

Essentially it’s quite simple: you tell the computer, ‘every 100 people with Covid will infect 105 more within two weeks’ or whatever. And the computer does the maths and can accurately calculate what the infection rate will be at Christmas. The obvious problem is the assumed 1.05 infection rate: if that is wrong then all the projections will be wrong with it. Garbage in = Garbage out.

So commercial modelling software always (ALWAYS …. because decades of experience have shown that models are useless without it), has feedback mechanisms where the actual observed infection rates and other factors like weather, seasons, lockdown levels, vaccination levels, mask-wearing rates and many others, are constantly fed back into the model so it can tweak itself to match the observed results. The model will also have data from a large number of similar situations to compare to its own results to see how they worked out. Anything other than primitive software will constantly run scenarios on the various factors, to identify the weightings that give the most accurate result. E.g. how much does 50% of people wearing masks affect the infection rate? -5%, no change, or +5%? The computer doesn’t care, it has no agenda (unless instructed to have one), it just dispassionately crunches the numbers to match the observed real world.

The model developer must always accept that the real world is correct. If the model disagrees with the world, then the model is wrong… not the world. No matter how clever the algorithms or how powerful the computer is, if it doesn’t match the real world then it’s wrong. Any commercial modeller who was out by a multiple of 5 or 10, and whose performance did not rapidly improve, would be told to get a job better suited to their talents… like flipping burgers at Maccas. You produce accurate and improving predictions, or goodbye.

Sadly, I see no sign that the academic computer models in use for this pandemic are anywhere near the rigour of commercial modelling. They are just so wrong, so often. Perhaps because the failed developer is not ‘down the road’ but just writes a paper explaining why it was wrong, and keeps their job.

The climate change models are worse in a way: they are complex and run on powerful computers, taking into account hundreds, perhaps thousands, of factors. But nearly all the climate models (some of the Russian ones being honourable exceptions) have been consistently wrong for many years, and always in the same direction. This can only happen if some human is leaning on the levers: left to itself, a competent model will quickly oscillate around the observed world as it tries to get closer.

Computer modelling is acquiring a bad name from the huge waste of money and lives when inadequate models are followed by politicians, apparently driven by academic experts who are certain they are right. This is unfortunate, as the commercial models are working away accurately in the background making sure that the goods are on the supermarket shelf when needed in the right quantity, and that the electricity is generally there when you flip the switch. Any of the people who work on these systems would be able to do better than the Covid and climate predictions I’ve seen from academic experts, just by accepting that the observed world is always right. Flipping a coin would give better results. Commercial developers would have been flipping burgers rather than coins if their work were that bad. You would only get forgiven once for predicting a 3 degree temperature rise when a 0.1 degree occurred.

I’m not sure what the reasons are for this. Perhaps it’s because the academics are the ‘go to’ people for politicians wanting expert advice to help them make a decision. And the experts may be often wrong, but they are never in doubt; that’s what makes them ‘experts’. Perhaps it’s due to the media wanting soundbites and clickbait: ask 10 academic modellers and headline the most frightening prediction. Fear sells. Perhaps the academics just don’t realise how unacceptable it is to make wildly incorrect predictions, and that the commercial world has moved way beyond them?