Governance is moving inside the machine
When the ChatGPT moment arrived, most of us reached for policy. We wrote responsible AI principles, stood up governance committees, classified use cases by risk and told employees what they could and could not do with generative AI. The governance lived outside the technology, and for a while that was enough.
It made sense when AI mostly generated content, answered questions and assisted humans. It becomes far less adequate the moment AI can act.
Microsoft’s recently published a draft Humanist AI Code of Conduct – released for public consultation – offers an early glimpse of what comes next. It says Microsoft’s future in-house MAI models should remain subordinate to people, accept correction and shutdown, respect the boundaries of their operating environment and never pursue their objectives at the expense of human control.
Microsoft AI CEO Mustafa Suleyman has described it as “a constitution of sorts” for future models.
The significant development here isn't that another technology company has published another set of principles. It’s where governance is heading – and the direction is unmistakable. AI governance is moving from policy into architecture. From something we tell systems about, to something systems are built inside.
From responsible AI to executable governance
Enterprise AI governance has passed through two broad phases so far. The first was policy governance, which asked: what are people allowed to do with AI? Organisations answered with acceptable-use policies, privacy requirements, responsible AI principles and restrictions around sensitive information.
Then came use-case governance, which asked: which AI systems are we prepared to deploy? Enterprises answered with risk assessments, AI registers, model reviews, data assessments and cybersecurity controls, plus extra scrutiny for high-risk applications.
Agentic AI introduces a third phase – executable governance – and it asks a fundamentally different question: what is the AI itself technically permitted to do?
That distinction matters. A policy can tell an employee not to delete a customer record without authorisation. But once an AI agent has access to that customer system, a policy document is no longer the control. The agent must be technically unable to perform the action unless the right conditions are met. The rule has to exist in the system, not the slide deck.
An agent is defined by what it can do
The risk an AI agent presents isn't determined simply by how intelligent the underlying model is. It's increasingly determined by the authority we give it.
A model that can read a maintenance report presents one level of risk. Give the same model access to a maintenance management system and let it create work orders, and the risk changes. Let it approve expenditure and it changes again. Connect it to operational technology and allow it to initiate physical actions, and you are dealing with something else entirely.
A useful way to think about these levels of delegated autonomy – rather than as a formal industry maturity model – is a progression that runs roughly.
Observe, recommend, decide, act, act autonomously
At the first level, AI observes information. At the second, it recommends an action but a human decides. At the third, it makes the decision but needs human approval to execute. At the fourth, it executes authorised actions within defined boundaries. At the fifth, it operates autonomously for a defined period or objective.
Every step to the right raises the stakes on identity, permissions, monitoring, reversibility and accountability. And that is exactly where the architecture of AI governance starts to look familiar.
Zero trust may offer the blueprint
Cybersecurity went through a similar evolution. Organisations once relied on policies telling employees how systems should be protected, then learnt that policy alone was insufficient. Security became embedded in architecture: identity, access, encryption, network segmentation, least privilege and continuous monitoring all became controls rather than instructions.
The principle behind zero trust is simple – never assume access should be granted merely because something is already inside the environment. Agentic AI demands an equivalent philosophy. An agent should have an identity and an accountable human owner. It should receive only the permissions its task requires, with tool access explicitly authorised and high-impact actions gated behind additional approval. Its behaviour should be observable and auditable, its autonomy bounded, and there must always be a way to stop it.
And this isn't one company's idea. The same direction is visible in the emerging governance standards: both NIST's AI Risk Management Framework and the ISO/IEC 42001 AI management-system standard push organisations to build governance into how AI is operated and controlled, not merely to write it down.
Nor is it one vendor's product. Microsoft provides one prominent example, but the same architectural direction is emerging across the industry. Its emerging enterprise agent architecture is moving in this direction, with technologies across its ecosystem addressing agent identity, scoped permissions, authorisation, human approval, isolation and auditability. AWS is moving the same way: Amazon Bedrock AgentCore gives agents distinct identities, scoped least-privilege permissions and a managed consent layer for authorising the external tools an agent is allowed to call.
Note the distinction that runs through all of it: a code of conduct sets out principles for how a model should behave, while the surrounding enterprise technologies are what actually enforce controls over what an agent can do. That separation between the constitution and the machinery is precisely the point.
The pieces of an AI control plane are beginning to appear.
Human control cannot mean human approval of everything
There is a tension worth naming. The obvious response to autonomous AI is to insist humans stay “in the loop” – but taken literally, that defeats the purpose of autonomy. If an enterprise deploys thousands of agents performing millions of small actions, humans cannot review every decision, and nor should they. We don't manually approve every packet crossing a corporate network; we build the identities, permissions and monitoring that determine what is allowed to happen.
AI may follow the same trajectory. The objective isn't to put a human inside every AI decision. It's to embed human authority inside every AI system – a fundamentally different idea. That authority can be expressed through permission boundaries, financial thresholds, authorised tools, escalation rules, stopping conditions, operating windows, data boundaries and actions that always require sign-off.
The machine may make the decision. But humans set the constitutional boundaries within which it is allowed to operate.
The physical economy raises the stakes
This matters most in industries built around physical assets. In banking, an autonomous agent might move money. In healthcare, it might influence treatment. In mining, manufacturing, chemicals and energy, AI increasingly has the potential to touch machinery, maintenance systems, production environments and operational technology directly.
The gap between an AI that recommends changing a mill's operating parameter and one authorised to change it is enormous – and on a plant, that gap can be measured in safety incidents, not error logs. That is why industrial AI cannot simply inherit the governance model built for chatbots. As AI moves closer to physical operations, autonomy has to be treated almost like privileged access: the greater the potential consequence of an action, the narrower the operating envelope should become.
Some decisions in safety-critical environments may always remain human. Others may eventually become autonomous, but only once the system can demonstrate it operates safely inside clearly defined limits.
The question, then, isn't whether we trust AI. It's what authority we are prepared to delegate, under what conditions, and how quickly that authority can be withdrawn.
Governance must become machine-readable
There's a further implication. Plenty of organisations have excellent AI principles sitting in PowerPoint. AI agents cannot execute PowerPoint.
If an organisation says an agent may spend up to R10,000 without approval, that rule eventually has to live in the transaction architecture. If an agent may read customer information but not export it, that restriction has to live in its permissions. If an autonomous process must stop after three failed attempts, the orchestrator has to enforce the limit. If certain decisions require a human, the workflow has to make bypassing that checkpoint technically impossible.
Governance, in other words, has to become machine-readable and machine-enforceable. That may be one of the biggest shifts agentic AI brings to enterprise technology – but it comes with a caveat worth stating plainly.
Encoding the rules doesn't dissolve responsibility; it concentrates it in whoever writes them. Executable governance moves power towards the engineers, architects and vendors who translate principle into code, which makes how those rules are set, reviewed and challenged a governance question in its own right.
The people configuring an agent's permissions, thresholds and stopping conditions are no longer merely implementing governance. They are participating in it.
The AI constitution is coming
Microsoft's Code of Conduct will not solve AI governance. It's a draft, and hard questions remain about how a principle like “human control” survives contact with systems operating at enormous scale and speed. But the direction is what counts. As AI grows more autonomous, governance has to move closer to the point where decisions and actions actually occur.
Policies will remain necessary. Governance committees will remain necessary. Risk assessments will remain necessary. But none of them can stop an autonomous agent at runtime.
Architecture can.
The enterprises that grasp this early will stop asking only whether they have an AI policy. They'll start asking whether their AI architecture can actually enforce it.
Because when AI can act, governance cannot stay a document.
It has to become code.
















