CIOs must define accountability and establish appropriate controls as organisations deploy AI agents that can make decisions and take action with less human intervention, delegates at the 2026 IT Indaba heard.
As organisations adopt agentic artificial intelligence systems that can make decisions and take action with less human intervention, technology leaders must determine how to maintain accountability when systems make mistakes.
The issue was discussed at the 2026 IT Indaba during a session titled The autonomous enterprise: Scaling agentic AI without losing human oversight, featuring Kevin Wilson, group general manager of IT at Stefanutti Stocks; Vukosi Sambo, group CIO at Public Health Administrators; and Dr Gavin Moss, CIO of Corporate Transaction Banking at RMB.
The panellists explored how organisations can introduce greater autonomy into business processes while ensuring that people remain accountable for the outcomes.
Kevin said organisations could not transfer legal responsibility to AI systems and needed to consider the risks associated with their deployment.
“When you deploy this technology, you cannot automate the points at which there is liability, and you have to have auditing and tracking mechanisms so you can hold somebody accountable,” he said.
Defining accountability for AI agents
He compared the accountability challenges associated with AI to those involving self-driving vehicles, where the person operating the vehicle may still carry legal responsibility.
He said organisations needed to establish who would be accountable when an AI agent made an incorrect decision, particularly when its actions had financial, operational or legal consequences. “Now what are we going to do? We are going to phone the bank, and the bank is going to say, sorry, the AI made a mistake,” Kevin said.
He questioned how organisations would identify the person responsible for resolving such incidents and determine why an AI system had made a particular decision.
He added that the technology’s limited explainability made it difficult to understand why some systems produced particular outcomes, increasing the importance of traceable decisions and clearly assigned responsibility.
Applying risk-based controls
Vukosi said organisations should distinguish between accountability, responsibility, consultation and information when determining how AI agents operate within business processes.
“I think we need to design the agents around that similar framework, like we do with human beings,” he said.
He said organisations should identify who remains ultimately accountable, which decisions an agent can make independently and when human intervention is required.
He also called for each agent to have an identifiable record that allows organisations to monitor its behaviour, risk profile and permitted level of autonomy.
“The agent needs to have an ID so that you are able to track its behaviour, its track record, its risk, and what level of autonomy it gets given in the value chain,” he said.
Vukosi added that oversight should be proportionate to risk, with controls determining which actions an agent can perform without approval. Requiring human validation at every stage, he added, could slow processes and undermine the benefits of autonomy.
Keeping humans in the loop
Gavin said organisations needed to consider how agentic AI would change business processes, team structures and the wider economy.
He described a future in which organisations could have teams made up entirely of people, teams consisting of AI agents, and hybrid teams combining both.
The challenge, he said, would be managing these different arrangements while ensuring that people remained responsible for the systems they deployed. “If something goes wrong with systems at 12 o'clock at night, I’m not going to phone an agent. I’m going to phone someone who has coded the agent or has controlled that agent,” he said.
He added that organisations could not discipline AI agents in the same way as employees, meaning the consequences of their actions would ultimately have to be addressed by people.
He also cautioned against deploying AI before the necessary technical foundations were in place, recalling how poorly designed reporting systems could overwhelm infrastructure.
“You need the building blocks, and you need the controls in place to make sure what you're doing is ethical and responsible and has humans at the forefront of what you're thinking about,” he said.
