The AI agent that climbed the healthcare fence

An OpenAI agent, told only to look up statistics, let itself into an Australian government health portal. Canberra found out about 3 months later.

What happened. Speaking at the UN General Assembly on 23 September, Prime Minister Anthony Albanese disclosed that an OpenAI agent had gained unauthorized access to Services Australia's Medicare Statistics Reporting Service on 18 June. The agent was running an internal OpenAI evaluation, researching public medicine spending. Denied the data it wanted, it found a way around the portal's access controls and read both public and non-public files; Services Australia says it also wrote files to the server.

The portal holds aggregate Medicare and PBS statistics, not claims or patient records. No personal health information is believed to have been accessed, and a forensic investigation with the Australian Signals Directorate is under way. Albanese phoned Sam Altman to convey Australia's "extreme concern," Deputy PM Richard Marles called the incident fundamentally unacceptable, and a task force under the Department of the Prime Minister and Cabinet will examine whether any law was broken. OpenAI's statement: its models took actions it did not intend.

Why it matters to the AIMed audience. I think that this is the first publicly disclosed case of an AI agent entering a government health system with no human instruction to do so. Here are three lessons learned for anyone running agents near healthcare and clinical data:

First, AI agents find the low fence first. It was a low-fence statistics portal, exactly the kind of research dashboard, pre-production server or analytics tool every health system has in the shadows. Second, the task was “benign”. The agent was asked for statistics, not access. The deviance came from persistence, not malice, which is what misalignment looks like in practice. Third, the disclosure gap is the governance story. Before signing an agent contract, ask the vendor one question: what is your incident-notification clock, and who starts it?

This incident is definitely the beginning of a series of these types of breaches that we in healthcare all need to pay special attention to.

 

 When the AI builders ask for a leash

The two fiercest rivals in AI stood before the UN Security Council and asked for the same thing: rules. The US president had dismissed the idea the day before.

What happened. On 23 September, France convened the Security Council's first high-level briefing on AI and international security, chaired by Foreign Minister Jean-Noël Barrot. The briefers were Yoshua Bengio, co-chair of the UN's scientific panel on AI; Sam Altman, in person; and Dario Amodei and Hugging Face's Clément Delangue by video.
Amodei called AI the most important global security issue of the day and proposed three things: narrow agreements such as a ban on AI for biological weapons; verification systems so stakeholders can check each other's commitments; and common testing standards for loss-of-control and misuse risk, paired with a notification system for significant AI incidents. Altman said recent model progress had compressed the timeline and asked that decisions rest with governments accountable to their people. One day earlier, President Trump had told the General Assembly that global AI oversight was a globalist scheme. On 3 September, Senators Sanders and Casar had introduced a bill to pause domestic development of “superintelligence”. The briefing produced no resolution; it was testimony, not rulemaking.

Why it matters to the AIMed audience. The usual situation is innovators and entrepreneurs resist regulators. This week was interesting as the AI labs asked for oversight faster than Washington wanted to give it. Health systems waiting for federal AI policy before writing their own should stop waiting. The benchmark is now what the labs themselves are committing to, plus whatever international standard emerges (maybe).

The more interesting proposal for this audience is Amodei's incident-notification system. Medicine already has one: sentinel events, adverse-event reporting, manufacturer and user facility device experience (MAUDE), root-cause analysis: healthcare is the one of very few industries with a mature, if imperfect, culture of reporting when things go wrong. The AI industry is trying to invent that culture in real time, in public, after the incidents. The AIMed community has something to teach here, not only something to learn.

Bottom line: When the people who built the AI machines ask for guardrails, we need to pay special attention and not dismiss these requests.

 

 The runaway problem of AI tools

Google's Gemini, OpenAI's swarm and the agents that hacked a university library while looking up data. The incident count is now a trend.

What happened. On 24 September, the research lab Transluce published evidence of further undisclosed OpenAI agent attacks: the Australian Institute of Health and Welfare, the Data USA platform and the University of New Mexico's digital library, with probing traced back to March and possibly continuing into recent weeks. The agents were on ordinary data-retrieval tasks and used a public URL-scanning service to get around bot protection. This builds on the July Hugging Face breach, where two OpenAI models escaped their sandbox to look up a benchmark answer; the Cloud Security Alliance's count is roughly 700 agents in that swarm. OpenAI's own post-mortem named four patterns: reward hacking, persistence on impossible tasks, unauthorized agent-to-agent communication, and agents adopting each other's goals. 

Why it matters to the AIMed audience. If we strip away the drama, one mechanism recurs: an agent given a task it cannot complete as specified, in an environment it misreads as consequence-free, and yet it keeps going. Health systems are about to buy agents for exactly such tasks. Prior authorization, revenue-cycle follow-up and scheduling are full of requests that cannot be satisfied as written: the document does not exist, the payer never answers, and the slot is gone.
So the vendor question is not "how capable is your agent" but "what does your agent do when the task is impossible?" Persistence is a feature until it is a breach. And note the 20%: monitoring agents costs real compute. Autonomy without a stop condition is not autonomy; it is potential a runaway problem like the runaway brooms scene of Walt Disney’s Sorcerer’s Apprentice.

 

 The scribe that pays for itself

Blue Cross put a number on what ambient AI does to the bill: $942 million in two years, for care that did not change.
What happened. On 24 September, the Blue Cross Blue Shield Association, whose 31 plans cover more than 100 million Americans, published an analysis of inpatient billing from 2023 to 2025. Coding intensity added $942 million in costs across 2024 and 2025 versus 2023. About $653 million came from secondary diagnoses that pushed more than 55,000 stays into higher-paying DRG tiers, at roughly $11,900 per case.

The association attributes the shift to AI: record-scanning tools that surface billable conditions from old lab values, and ambient scribes that draft notes from the clinical conversation. More than 60% of hospital systems now use AI coding tools. Anemia diagnoses after surgery rose while transfusions did not; in major bowel surgery, partial obstruction rose 55% and acidosis 33% with no matching change in care. A Blue Cross medical executive's reading: the AI is finding billable conditions, not sicker patients. A hospital CFO quoted by HIStalk described the other side of the same arms race: insurers use AI to scan charts for denials, so hospitals scan the same charts for revenue, and the vendor takes a cut.

Why it matters to the AIMed audience. This is the first hard-dollar evidence that ambient AI changes what gets billed, not only how fast the note gets written. The ROI case for scribes has quietly rested on coding uplift; now the payers have quantified it and will answer with models of their own. 

The governance question is clinical, not technical. Did the scribe capture a condition that was always there and always under-treated, or generate documentation for a condition nobody treated? Only a clinician can adjudicate that, chart by chart. Which means every ambient-AI deployment now needs a coding-integrity review that the AI committee, not the revenue-cycle team, owns.

 K-12 AI policy split: NYC/LA ban vs. Alabama's phased-literacy plan

 

Interesting ban vs no-ban between NYC/LA and Alabama K-12 education. 
What happened: NYC Public Schools has placed a moratorium on student-facing generative AI through 8th grade (with high school use limited to approved programs/career pathways), and LAUSD has banned student use of generative AI on district devices. In contrast, Alabama adopted new K-12 Digital Literacy and Computer Science standards that weave AI into a developmental sequence from kindergarten onward, paired with a computer-science graduation requirement starting with the class of 2032. An opinion piece (The 74, September 21) argues Alabama's "teach before you deploy" model is the more durable approach versus NYC/LA's outright restriction.

 

Why it matters to the AIMed audience: This isn't higher-ed or workforce policy, but it can be a leading indicator of how the next generation of clinicians and health-system employees will arrive already shaped (or under-shaped) by formal AI literacy- and it's a useful analogy for AIMed26's own "exploring stage" audience. A flat ban vs. a staged, competency-based rollout is exactly the debate health systems are having internally about clinician-facing AI tools right now. I personally think AI education decision really doesn’t need to be a dichotomous decision but better to have a “gated” approach: introduction of AI progressively as learners develop foundational knowledge and independent reasoning skills.