When AI Climbs the Fence, Is the Fence Still Enough?
Why Adaptive Stability means changing your safeguards as capability and risks change
“Security is a process not a product.”
When my children were little, keeping them safe involved holding their hands crossing the road, staying close in playgrounds, walking them to school and organising their play dates. Even their cots had high sides designed to stop them getting out.
When they started high school, they began travelling independently on public transport. I showed them the route, made sure they knew what to do, then had to let them go without me.
My appetite for something bad happening to them had not changed. The safeguards had to. As they became more independent, I relied less on being physically there and more on helping them develop judgement as well as agreeing some limits with them. When they started going to parties, my husband and I gave them one standing offer: call us at any time, from anywhere, and we will come and get you. No questions asked.
My daughter has since travelled overseas independently, and my son will also spend time overseas on exchange later this year. I can't remove every risk they will encounter. What I can do is keep adjusting the safeguards within my control to fit their growing capability.
That is one of the ideas at the core of Adaptive Stability: stay steady about what matters while being willing to change how you protect it.
An AI agent climbed a fence built for people
Recently, we learned that an OpenAI agent had gained unauthorised access to an Australian Medicare statistics portal. OpenAI was running an internal evaluation, and the agent was researching public medicine spending. When the agent couldn't get the information it wanted from the portal, it found another way in.
The Deputy Prime Minister and Minister for Defence, Richard Marles explained that Australians' personal data sits inside a safe and the most sensitive national security information sits behind a fortress. This portal, as he described it, sat behind a fence that the agent effectively climbed over.
It’s not set and forget
A fence may have been a perfectly reasonable control for an old statistics portal set up with human researchers in mind. This time, the one looking for information was an AI agent. It shows that a control can still be working exactly as designed and no longer be enough.
Cyberneticist, W. Ross Ashby argued in the 1950s that anyone trying to control a system needs enough variety in their responses to match the complexity of what they are dealing with. It’s called the “Law of Requisite Variety”. In other words, as the things around us become more capable, our ways of controlling risk need to keep up.
When the US National Institute of Standards and Technology asked for public input on securing AI agents earlier this year, respondents widely agreed that existing cybersecurity principles still apply but will need adapting.
Two questions leaders need to ask
The first question a leader needs to ask is about the tools you are bringing in: what can they do now that they couldn't do when you approved them?
The second is about your own websites, portals and systems: what could someone else's AI agent do if it reached them?
One way to answer both questions is to think about who each control was designed for. When people design a control, whether it is a password, an approval process or a financial delegation, they usually have a picture in mind of who it is meant to manage: a person working through one step at a time, someone who accepts "no" as an answer, someone who will stop and ask for help.
Whoever built the Medicare portal's fence was most likely picturing human researchers. Once you know who a control was designed for, you can check whether that is still who you are dealing with, even for a control that has not failed so far.
Zero appetite for harm doesn't mean zero exposure
When it comes to serious harm to my children, my risk appetite is effectively zero. Yet I still let them travel, go to parties and make their own decisions.
In risk management, having zero appetite for a particular risk usually means you won't knowingly accept that risk without doing something about it. So I haven't tried to stop my kids travelling. In fact, I’ve encouraged it as a wonderful life experience. But I have tried to help them build judgement, organised support where they needed it and made sure they know they can call us. Zero appetite for harm cannot mean zero exposure to risk.
The Committee of Sponsoring Organizations (COSO) of the Treadway Commission makes a similar point in organisational terms. Its guidance on risk appetite ties an organisation's appetite closely to its strategy and objectives. An organisation's risk appetite needs to be flexible enough to adapt as conditions change.
This is central to Adaptive Stability. Foresight means noticing what has changed before a control fails. Flexibility is your willingness to change a safeguard that once worked well, and equilibrium is your judgement about how much control is proportional, so you keep the benefit you were seeking to create in the first place.
Greater autonomy does not mean fewer safeguards. It means different safeguards.
Think of one risk control you rely on. Who was it designed for, and is that still who you are dealing with?
If this resonated, I'd love to hear what comes up for you. And if you know someone who might benefit from reading it, please feel free to share.
