Consultation Response #1: Back to School – What can pensions learn from the exam algorithm crisis?
As children headed back to school earlier this month, spare a thought for the teachers.
Over the Spring and Summer, they had the unenviable task of estimating grades; then had their ability and capacity to make that judgement challenged in front of a national audience; finally, to have their views accepted after all!
“Teachers know me, not algorithms” was one of many banners displayed by students. Perhaps there’s something the pensions industry can learn from this?
Take stochastic models – often these can be portrayed (or interpreted, or both) as providing a well-informed ‘quasi-scientific’ prediction of a scheme’s future to help reduce the level of judgement in decision making. But, far from eliminating the role of judgement, the outcomes of stochastic models depend on the assumptions about how different asset classes and liabilities are expected to behave in different scenarios (rather than how they will actually behave which – somewhat obviously – cannot be known).
Seemingly the more complex a given model and our associated language, the more we might seek to confer it ‘predictive power’ but, sadly, the world simply doesn’t care what our models say – e.g. pension schemes were not uniformly modelling pandemic scenario pre-COVID.
Now, that’s not to say these models don’t have their place. They can be a useful tool, but they need to be used with care and their limitations perhaps better acknowledged. It can become quite easy to place more faith in these models than they really deserve and for decisions to be made which apply too great a weighting to what ‘the model is telling us’.
The same goes for some models that seek to describe sponsor covenant as a single number, expected probability of failure, or of achieving a particular outcome. These models are re-emerging as part of the emerging consolidation market and usefully contributing to this debate. Whether using a basket of financial metrics or credit default rates, the models can again be a useful tool, but they need to be used with care and their limitations better acknowledged by our industry.
Stochastic / covenant models might need a more obvious health warning or we will be making life changing decisions for scheme members based on the exam algorithm (stochastic / covenant models) rather than teacher estimates (trustee/advisor/sponsors weighing up all relevant factors).
The inconvenient truth is that many difficult judgements will need to be made in the coming years and decades that simply cannot be modelled. The COVID-crisis (and resulting exam crisis) might have taught us this if nothing else.
Darren Redmayne, CEO