Solving the challenges of predictive model degradation (Part 2)

They serve us well

Machine-learning and predictive modelling have formed the foundations of fraud detection solutions for years across the financial services and insurance sectors. They allowed these organisations to make great use of the wealth of customer data at their disposal and to automate decisions with limited manual intervention.

These approaches have proven incredibly beneficial both financially and operationally. No longer do teams of people need to sit and work through each customer application, instead they are able to prioritise and focus only on those that need further investigation. From a customer experience perspective, genuine customers are happier as they can be boarded much more quickly and smoothly. The results: more genuine customers, boarded more quickly, generating more money faster.

Time for a change, well several actually

Supervised machine-learning is a very common technology used to automate decision-making. The model is trained against pre-determined characteristics identified in a set of historic data for a specific scenario (in this case fraud).

Though unlike other semi-supervised and unsupervised machine-learning techniques, the models built and trained by supervised machine-learning are more prone to degradation over time.

This can happen when the original characteristics that were used to train the model are no longer valid or certainly less valid than when they were first used. This is often the result of behaviours changing over time.

If we think about this in a fraud scenario, fraudsters often have to adapt their techniques and approaches in order to bypass and continue exploiting an organisation’s existing fraud defences. When this change happens, it becomes more difficult for the model to accurately function as it once did as the characteristics that allowed it to operate so successfully at first have now changed. We call this model drift.

To help prevent inaccuracies creeping in over time the models require recalibration. This ensures that you can be confident in the decisions you are automating, whether it be accepting or declining new customers.

Could other technologies work better?

Yes and no. As standalone technologies, probably not. In our experience, the most effective fraud defence is a multi-layered one. Utilising a range of fraud prevention technologies that work holistically to strengthen each other.

That being said, the likes of Nationwide and esure have achieved some truly incredible results when they deployed our supervised machine-learning models even as a standalone machine-learning proposition alongside their usual data-matching fraud defences.

Within just two months of Nationwide working their usual volume of applications, they experienced a 20% uplift in third-party adverse identified.

Likewise, esure realised some incredible efficiency and performance gains. Following live deployment of their predictive model they saw a 50% reduction of investigations, whilst simultaneously increasing the amount of fraud identified by 30%.

Though, there is the danger of these models degrading and consequently the performance and efficiency gains.

In contrast, unsupervised machine-learning techniques such as anomaly detection are less vulnerable to model degradation as they are able to identify and adapt on-the-fly to changing behaviours and characteristics. Unlike supervised machine-learning they self-discover any patterns or trends rather than rely on information to label, flag or score the data when training the model.

It’s not one or the other

Supervised & unsupervised – it’s not one or the other. It’s both.

To build a truly pro-active fraud defence relies on multiple technologies. Utilising both supervised and unsupervised machine-learning will help mitigate the inherent weaknesses of either system and leave you less vulnerable to model degradation.

We can help you create the best fraud detection and prevention solution. Please get in touch so we can discuss your requirements.