Agentic AI in US Health Systems: A Governance Checklist Before You Scale

Most health systems already have an agentic AI pilot running somewhere — a coding assistant, a denial-appeal drafter, a documentation copilot. Very few have decided, in writing, what happens when that pilot asks to touch a claim, a chart, or a patient message on its own.

That gap is wider, and more common, than most leadership teams realize. A 2026 survey of healthcare AI strategy leaders found 28% already running agentic AI in production and another 44% piloting it — but 72% admitted some AI tools are deployed without formal IT approval, even as 86% described themselves as confident they could fully govern what their agents do. Fewer than one in ten organizations has a formal AI oversight board. That gap between confidence and actual control is where scaling stalls. Not the model quality — the governance model.

Not every agent needs the same leash

The mistake most governance efforts make is treating "agentic AI" as one category needing one policy. A chatbot that summarizes a chart for a clinician to read carries a different risk profile than a system that files an appeal or messages a patient without review. The useful first move is classifying every live or piloted use case by how much unsupervised authority it actually has — not by what the vendor calls it.

Three questions before the second use case gets funded

Once a use case has a tier, three questions decide whether it’s actually ready to scale, not just technically working in a demo.

• Which actions can the agent take unsupervised at its assigned tier, and which require a named human sign-off before anything ships? Tier creep — a Tier 2 pilot quietly behaving like Tier 3 in production — is the most common way governance breaks down.
• What is logged, and can a reviewer reconstruct why the agent did what it did, six months later, when audited? If the honest answer is "we’d have to ask the vendor," that’s a Tier 3 or 4 system being run with Tier 1 audit capability.
• Who owns the model and data pipeline itself as a risk surface — the same way compliance already owns coding accuracy and documentation completeness? Models drift, vendors update them, and "it worked at go-live" is not a governance posture.

The regulatory floor is already higher than most pilots assume

This isn’t a hypothetical compliance exercise. ONC’s HTI-1 final rule already requires developers of AI and predictive algorithms embedded in certified health IT to meet baseline transparency requirements — disclosing enough about how a model works for a clinician or health system to assess its fairness, validity, and safety before relying on it. If an agentic tool touches certified health IT or clinical decision support, that obligation doesn’t go away because the system is described as "AI" rather than "software." Treating agentic AI governance as a new, separate initiative — instead of an extension of the compliance function that already owns HIPAA, coding accuracy, and documentation completeness — is the most common reason governance programs stall before they start.

Before your next pilot scales

1. Classify every live and piloted use case into one of the four autonomy tiers — not by vendor marketing, but by what it’s actually permitted to do unsupervised.
2. Stand up a lightweight AI oversight function now, even informally. The gap between 86% confident and 10% with a formal board is the single biggest governance exposure in the industry today.
3. Run a shadow-AI inventory alongside the formal one — ask department leads what their teams are already using, not just what IT provisioned.
4. If any agent touches certified health IT or clinical decision support, map it against HTI-1’s transparency requirements before it scales, not after an audit asks.
5. Require an audit trail as a go-live condition, not a retrofit — "we’ll add logging later" is how Tier 3 systems end up running with no real oversight.

None of this is exotic. It is the same discipline HIPAA already expects of any system touching PHI, extended to a system that also makes decisions. Health systems that treat agentic AI governance as an extension of existing compliance — rather than a new, separate initiative — get through their next audit cycle without surprises.

Frequently asked questions