Speaking at the 2026 IT Indaba, Lesedi Modibane, head: data, analytics and AI at MakwaIT discussed the implications of AI incorrectly flagging people when it comes fraud scoring, claims, credit and eligibility.
In a session titled When the algorithm says NO: fair, explainable AI in high-stakes decisions, Lesedi reflected on the limitations of AI and how to remediate its mistakes. It starts with a claim, a flag and a letter with no reasons, he said. While the model to assess a claim is generally working, the process that happens afterwards is what matters most.
“If the model gets it wrong, as a client or consumer, what sort of recourse do you have?” Lesedi asked. If you’re rejected, you’re now left unable to get a new cell phone, car or even home loan. Therefore, when it comes to gaps in the governance around this, we need to have a way of filling these.
Why do we score
With 26 percent more cases of fraud detected in 2024 when compared to the previous year, this is undeniably a growing issue. “Fraud is real and it’s rising,” Lesedi said. Considering that out of 100 000 claims, we assume that one percent are fraudulent, having an AI model to determine their veracity through a client scoring system is vital. Due to scoring, R1.4 billion in fraud and dishonesty was prevented in 2024.
Yet even a good model can flag the wrong people. While models catch up to 90 percent of fraud, they wrongly flag five percent of claims. “We need detection, but the real question is what detection costs honest customers,” Lesedi said.
Legal gaps
The challenge we’re facing in tackling this dilemma in South Africa is our current laws. “While Popia section 71 says that no decision with legal or substantial may rest solely on an automated decision, we're still currently using models that are fully automated to make a decision on whether we give people credit or not,” said Lesedi.
While on paper, a token human sign-off moves a decision outside the rule, it changes nothing in practice.
“We're going for efficiency which means automation and automation means we close our eyes and trust the machine,” Lesedi said. “But there has to be some give and take. Whoever is granting credit should protect the applicant and if you are the lender, you should protect yourself.”
With Popia not doing enough, we need to ask if there’s a law that addresses its limitations. That’s where the Draft National AI Policy, 2026, comes in, which aims to define how we are going to implement AI in South Africa. “These are the governing laws from a customer perspective or consumer to a supplier perspective,” Lesedi said. "You cannot simply rest and trust a system just because it’s AI. Ask questions around what the machine uses as input."
In filling legal gaps that South African law leaves open, organisations also must turn to international legislation in the form of the NIST Cybersecurity Framework, a voluntary set of guidelines created by the US National Institute of Standards and Technology. Sector neutral and widely adopted, it offers one vocabulary guideline for boards, regulators and vendors. It assesses where a wrong flag can hurt someone, measures how often and who it impacts and manages the issue by acting in determining how to put this right.
Overseeing AI systems
“You can’t be automating channels without looking at the core of the business model and how you operate,” Lesedi said. “We should be able to explicitly explain to users why their claim was declined and what aspects were flagged.”
Businesses must maintain a detailed inventory of all AI systems, including what they do, how they connect to other systems, and how much decision-making authority they have. This must be supported by a decision register that records each decision and the level of human oversight required.
A clearly identified business owner must be accountable for the outcomes, Lesedi added. This must include designated people responsible for overseeing decisions and handling cases where the AI makes decisions such as rejecting a credit application.
