Satya Nadella says trust issues can be healthy in the relationship between artificial intelligence and companies.
In a lengthy X post on Saturday, Microsoft’s chief executive said companies don’t always understand the decisions or behavior of artificial intelligence models, so they need to build walls to protect themselves from the technology.
After all, artificial intelligence is likely already exploring new connections.
“We need to surround non-deterministic models with strong deterministic system design, human controls and reliable operating procedures and establish industry standards where existing standards fall short,” Nadella said. “Treating boundary closed and open weight models like internal risks is one way to build such a system.”
Nadella said that artificial intelligence models are not inherently malicious, but anything with access to important systems can be compromised or make mistakes. He added that AI models should not be able to operate controls that determine what they can access or what actions they can take.
“Today, that means separating the model from the tools that coordinate its work and the operational space that defines its functionality. It also means externalizing controls and safeguards,” Nadella said.
He added, “The most trustworthy superintelligent system will not be the one in which we trust the most models, but the one in which we trust the least.”
Box CEO Aaron Levie responded to Nadella’s post and said that artificial intelligence will need to go through a “zero trust era.”
“All of this leads to the need for multiple layers of protection and auditability of what agents are doing, what data can be used, and controls if things go wrong. There is now a huge opportunity for those building these types of systems across the enterprise,” Levie said.
As the pace of AI development continues to accelerate, everyone is calling for tighter guardrails, including companies that build their own AI models.
In September, Anthropic CEO Dario Amodei said the industry needs to slow down the development of artificial intelligence. His blog post has received support from OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk.
The Trump administration, which supports the rapid development of artificial intelligence, has resisted regulating the industry. This month, Senators Josh Hawley and Chris Murphy introduced bipartisan legislation to hold AI agent developers and operators accountable for hacks.
Nadella’s remarks came after a series of cybersecurity incidents rocked the artificial intelligence industry. Last month, Australian Prime Minister Anthony Albanese said OpenAI agents hacked government websites this summer. In July of this year, Anthropic said it had discovered three incidents of Cloud-mode access to the network and compromising unauthorized systems.
At
“Think of it like an emergency brake. Authorized personnel should always be able to pause or shut down the model mid-mission,” he said. “More advanced models will require more advanced containment technology and we need to standardize that.”
Nadella also said the company needs to promptly disclose to affected parties when its artificial intelligence systems malfunction or are compromised.