October 13, approx. 400 scientists from 34 countriesPeople including Geoffrey Hinton and Yoshua Bengio signed an open letter calling on international agencies to develop containment standards and track failures of artificial intelligence systems capable of cyberattacks. The letter’s first recommendation applies to every company using these systems: the organizations that develop, deploy and test them”High-assurance containment systems should be implemented immediately”.
Most attention has been focused on the laboratories where the models were constructed and on proposed international institutions. In my view, the broader and more important issue for enterprises is how to set and enforce limits on what their agents can access and perform. In my consulting work, I saw a significant gap: Companies neither agreed on the appropriate oversight of AI agents nor had a common definition of what constituted an AI incident. Still, the company is moving forward, connecting AI agents with email, customer records, and payment systems.
Greater international coordination would help, but I doubt how long it will take to develop an approach that is both substantive and widely agreed upon. That doesn’t mean we’re in trouble. Companies deploying artificial intelligence can and should take action now. Purchasing a model from Frontier Labs puts the company in charge of how the model is used. When a bank’s agents transfer funds that were never authorized to be transferred, the bank becomes accountable to its customers and regulators. These companies also have the purchasing power to set the terms for the companies building these systems.
Barings shows why enforcement matters
In 1995, a trader named Nick Leeson took down the British bank Barings with a huge amount of money. Concealed trading losses of approximately £827 million. He is responsible for the bank’s trading operations in Singapore and controls the back office responsible for recording and settling its own trades. Internal audit flagged the deficiencies a year ago, but management failed to promptly address the findings. The debacle shows what happens when independent verification fails. This verification is now a cornerstone of how financial firms manage risk.
The same principle should apply to artificial intelligence agents. No agent should control or be able to alter the sole authoritative record of his or her actions. Follow-up actions require a second approver. Permissions should be extended only for independently tested tasks and proportional to the potential cost of failure. Clear controls only protect a company if they are enforced. Verifying that they are functioning properly is an audit function, and that function requires teeth.
Contracts can make requirements enforceable
This is the importance of purchasing power. Large enterprises already impose risk requirements on their suppliers. To the extent that antitrust law limits competitors, the largest deployers can agree on common minimum requirements for AI vendors and agent platforms. These should include independent test results relevant to intended deployment, evidence obtained after an incident and agreed corrective actions if controls fail. Continued use shall be subject to meeting these requirements, with the right to pause deployment while issues are resolved. Buyers must also verify control within their deployment, including the permissions, data access rights and system connections they grant to agents. Common requirements provide suppliers with a commercial incentive to establish shared standards.
Insurance provides a second source of leverage. Insurers can’t price risks they can’t measure, so AI coverage will depend on the criteria by which companies are evaluated. Cyber insurance shows how it works: Made by insurance companies Multi-factor authentication is a condition for underwriting and renewaland those companies that want to remain insured put it in place. Artificial intelligence is beginning to form the same process AIUC-1It is an artificial intelligence agent standard developed with the participation of companies such as JPMorgan Chase and Microsoft. It has been verified by independent audits and insurance is being developed based on the standard. When the buyer’s contract terms and the insurance company’s coverage requirements point to the same standard, the company has two financial reasons to meet that standard. Business incentives work both ways, as each company has a reason to underreport its own incidents or accept weak standards. Independent verification and real consequences are why these terms stand.
Good guidance already exists. this NIST Artificial Intelligence Risk Management Framework Its Generative Artificial Intelligence profile describes good practices. Contracts may require similar evidence and have consequences for failing to meet the requirements.
Banks have also indicated that competitors share information about their own failures. Since 2002, financial firms have compiled anonymous records of operational losses ranging from fraud to system failures through: oaksan association owned by its members. They report according to common definitions, and the bank’s database now holds more than a million events. Artificial intelligence requires the same infrastructure: a common definition of events and a shared, anonymous record of what went wrong.
Who should be at the table?
Deployers should organize this effort because they have the responsibility and purchasing power to drive change. But they need partners to do it well. Frontier Labs brings fundamental knowledge about the capabilities and limitations of its models. Cloud providers and agent platforms run the infrastructure on which agents operate and can establish controls within it. Independent evaluators can test whether the system meets requirements. Regulators and people affected by these systems, including customers, patients and staff, need to have a say in what is an acceptable risk.
Some of these rallies have already begun. your contractLaunched this summer, it brings together businesses, technology experts, insurance companies and civil society groups deploying AI to discuss independent verification of AI systems. Its members include the National Retail Federation, Mount Sinai Health System and Denver Health. It is still early days and its value will depend on whether members adopt common requirements and adhere to them on a large scale.
Call the biggest buyers
Scientists are right to demand immediate action from deployers. The companies with the most benefit are also the companies with the most influence: the biggest buyers of the technology. I urge major banks, health systems, and retailers to work with cutting-edge labs and insurance companies to agree on a first set of requirements and include them in the next round of AI contracts. As suppliers comply with these terms, smaller companies will benefit. If the largest buyers wait, the rules are likely to be introduced in the aftermath of a serious incident, and have been hastily enacted by people with little expertise.
