Vendor Due Diligence Checklist for AI Agents in Medical Practices

AI agents built for medical practices do not simply extend the capabilities of the software already running the back office. They execute multi-step workflows across EHRs, payer portals, scheduling platforms, and billing systems on their own, making routine decisions and escalating exceptions without waiting for a staff member to trigger the next step. Traditional software procurement asks whether a product integrates, what it costs, and what the contract term runs. None of those questions address how an agent actually touches protected health information, how it navigates systems using staff credentials, or how it behaves when a workflow fails halfway through. An agent typically sits on top of the systems a practice already runs, the EHR, the billing platform, the scheduling tool, without replacing any of them, so the vendor relationship begins before any conventional integration work starts, and due diligence has to begin there too. The sections that follow walk through the six dimensions a practice administrator needs to evaluate, in the order they need to be worked through: how the agent touches your systems, what compliance posture it can prove, whether it owns a full workflow or just a piece of one, what going live actually demands from staff, and how the vendor measures performance once the agent is running.

How the agent actually interacts with your systems, and what that determines

Before a practice asks whether an agent integrates with its EHR, it needs to ask how that agent interacts with every system inside the workflow it is meant to automate. That question matters because the two dominant interaction models carry meaningfully different implications for what a practice can safely deploy.

Screen-operating agents read screens, click, type, and navigate the way a staff member would, which lets them work across any system without requiring that system to expose an API. That includes payer portals, and many payer portals lack a modern API. API-dependent agents, by contrast, can only reach systems that have built a connection for them. When a payer portal, a legacy billing platform, or some third-party tool inside the workflow lacks a supported API, an API-first agent cannot touch it. The manual work stays exactly where it was, in the cases that most needed automation.

The agents that deploy safely and quickly are the ones that sit on top of a practice's existing systems and follow its existing standard operating procedures, rather than requiring the practice to redesign its workflow around the agent's limitations. A practice evaluating a vendor should ask directly whether the agent operates via screen interaction, API calls, or both, and which one it falls back to when an API is unavailable. It should ask whose session credentials the agent uses when it navigates a payer portal as a screen-operating bot, staff credentials shared with humans, or isolated machine credentials built for the purpose. It should ask how the vendor handles credential rotation, portal lockouts, and mid-session failures, and who gets notified when one of those happens. Finally, it should ask the agent to demonstrate its workflow against the specific systems the practice runs today, not a generic demo environment built to look impressive in a sales call. The answers to these questions constrain everything a practice can responsibly evaluate afterward: compliance posture, workflow scope, deployment timeline, and performance measurement all depend on which systems the agent can actually reach and how it reaches them.

Compliance posture: what HIPAA and SOC 2 Type II actually require a vendor to demonstrate

An agent that navigates a payer portal, reads a denial letter, and updates an EHR record is touching protected health information at every one of those steps, often across systems owned and operated by entirely different parties. A practice needs to understand how PHI is governed at each handoff in that chain, not only at the level of the agent platform itself. Every deployment requires permissions scoped to the specific task at hand, a full audit trail on every action the agent takes, and clear control over where the data lives. These are baseline requirements any agent touching PHI has to meet before it goes near a live workflow.

A practice should confirm whether the vendor holds a current SOC 2 Type II attestation and a signed data protection agreement covering protected health information, and whether the vendor can produce both before any contract is signed. SOC 2 Type II is an attestation issued following an audit against the AICPA's trust services criteria, not a certification in the formal sense, so a vendor should be able to produce the actual attestation report rather than a badge or a claim. A practice should also ask where PHI is stored, for how long, and under what access controls, both during a workflow run and after it completes. It should ask whether every agent action is logged to a full, retrievable audit trail, including actions the agent takes inside payer portals and other third-party systems it does not own. It should ask how permissions are scoped, whether the agent can only access what a given workflow strictly requires or whether it operates with broader system access than the task calls for. And it should ask how the vendor handles a PHI breach or a compliance audit, including the notification timeline and the evidence the vendor is prepared to produce.

Practices operating in California have one more layer to confirm. On September 30, 2026, the state signed two health-AI bills that require deployers of clinical decision tools to support clinician authority and to monitor and mitigate biased impacts, while placing the documentation burden on developers, who must record the steps they took to detect and mitigate those impacts. The law takes effect January 1, 2027. A California practice should confirm whether a vendor's administrative agents fall within the scope of that law and what documentation the vendor is prepared to provide.

Whether the agent can handle the full workflow or only an isolated task inside it

The decisive question in evaluating scope is whether the agent owns the complete workflow, from the moment it triggers to the moment it resolves, across every system that workflow touches. When only one step is automated, the handoffs before and after it remain manual, and the bottleneck a practice was trying to eliminate simply relocates.

Prior authorization makes the distinction concrete. An agent that owns the full workflow discovers whether a given order requires authorization directly from the EHR, gathers the clinical documentation the payer will require, submits the request through whatever pathway is available, an electronic API where one exists, structured fax or phone outreach where it does not, polls for a decision, and escalates to a human staff member only at true exceptions. Claims workflows follow the same pattern. An agent that owns the full workflow reads the visit documentation, maps it to billing codes, checks completeness against payer-specific rules, submits the claim, reads the denial letter if one arrives, categorizes the reason for denial, assembles the corrected documentation, and resubmits, or routes the case to a human with full context already assembled.

A vendor that automates claim submission but not denial follow-up, or that automates inbound scheduling calls but not the EHR updates and referral coordination that come after, is delivering task automation dressed up as workflow automation. The gap tends to be invisible in a demo, because a demo shows the automated step working well. The practice absorbs the remaining manual work after go-live, often without realizing during procurement that it was never part of the deal. A practice evaluating scope should map the full workflow out on a whiteboard with the vendor present and identify exactly where the agent hands off to a human and why. It should ask what the agent does when a payer has no electronic API: does it stop, generate an alert, or continue through an alternative pathway such as structured fax or phone outreach? It should ask how the agent handles exceptions, whether it escalates with full context assembled or simply drops the task back onto a staff member's desk. And it should ask what happens when a mid-workflow system goes down, whether that is portal downtime, an EHR maintenance window, or a payer timeout.

What "Going Live" Requires From Your Staff

A vendor's technical capability means little to a practice that cannot reach go-live without a six-month integration project and a workflow redesign that displaces the staff the agent was supposed to support. The agents that deploy fastest are the ones that sit on top of existing systems and follow existing standard operating procedures, which means no rip-and-replace of current platforms and no staff retraining required before deployment begins. The practice's current workflow becomes the starting point for deployment, not something that has to be rebuilt first.

Most teams still need workflow design work done before time savings become visible, and the real question is who does that work: the vendor, through structured implementation support, or the practice's own staff, working from scratch with nothing but software access. The gap between a deployment measured in weeks and one measured in months is not primarily a function of which AI model powers the agent. It comes down to whether the vendor has a repeatable method for converting a practice's existing staff workflows into deployable automations without manual development overhead at every step.

A practice should ask what the vendor needs from it to start deployment, and who on each side owns each item on that list. It should ask for a realistic timeline from contract signing to the first live workflow, with named milestones and accountable parties on both sides. It should ask whether the vendor provides workflow design support as part of implementation, or whether the practice is expected to define its own workflows and hand them over fully formed. It should ask what going live actually requires from staff: credential sharing, workflow documentation, training, IT involvement. It should ask how the vendor handles the period between go-live and stable performance, including whether there is a defined hypercare period and who monitors it. For multi-location groups and larger organizations managing portfolios of practices in particular, the deployment process must scale from one site to many, and the fifth deployment across a portfolio should be meaningfully faster than the first.

How the Vendor Defines and Measures the Agent's Performance

A vendor who measures an agent against generic benchmarks, tasks completed, uptime percentage, interactions handled, rather than against the workflow-specific KPIs that reflect a practice's actual bottleneck, is offering a metric nobody can be held accountable to. Scheduling workflows have standard KPIs for a reason: conversion rate on inbound contacts and no-show rate are the two most commonly cited measures in the field. Claims workflows have their own standard pair: clean claim rate and days in accounts receivable. Referral intake has its own as well, with time from referral receipt to appointment booked serving as a central measure, alongside time-to-contact and conversion rate as intake-specific companions. A vendor's willingness to be measured against the KPI standard for the specific workflow it is automating, rather than a generic activity metric, is a signal worth weighing.

Revenue cycle management workflows carry a particular accountability risk. Payers are deploying automated denial systems at scale: one physician survey found that a majority of physicians reported payer AI systematically driving up denial rates, and a nationwide survey found most health plans now use AI for prior authorization, with some using it to support batch denials with minimal human review. A vendor whose agent automates clean claim submission without maintaining current intelligence on payer policy can report strong submission numbers while a practice's denial rate quietly worsens behind that dashboard. A practice evaluating an RCM agent needs to ask specifically how the vendor tracks changes in payer denial patterns and how quickly the agent's behavior adjusts in response.

The non-negotiables here are the same ones that governed the compliance conversation earlier: a full audit trail on every action, and clear control over where the data lives. Those audit trails do double duty. They are what makes performance measurement honest, because they let a practice verify what the agent actually did rather than relying on whatever a vendor's dashboard chooses to report. A practice should ask what the KPIs for its deployment are, and insist that those KPIs are defined in the contract itself. It should ask how the vendor reports on performance, what data the practice receives, how often, and in what format. It should ask whether the agent's action log is available to the practice on demand or only surfaced when the vendor decides to share it. It should ask how the vendor identifies and responds when performance degrades, whether that degradation comes from a portal UI change, a payer rule update, or an EHR version change. For RCM workflows specifically, it should ask how the vendor monitors shifts in payer denial policy and how fast those shifts get reflected in the agent's behavior. And it should ask what the escalation path looks like when the agent hits something it cannot resolve on its own, and whether that handoff reaches staff with full context intact or forces them to start over.

A vendor who answers these questions clearly, with contract language, documentation, and a working demonstration against the practice's own systems, is a vendor capable of being held to the deployment it promises. A vendor who resists defining KPIs before the contract is signed is telling a practice, in the clearest way it can, that it does not expect to meet them.