Where HIPAA Meets AI: What Changes When Your Model Touches PHI

Last updated: September 2026
By: Kevin Yamazaki, Partner and CEO at Sidebench

HIPAA did not change when AI arrived. Your risk surface did. The moment an AI tool sees protected health information without a Business Associate Agreement, that disclosure is impermissible, and HIPAA presumes it is a breach unless you can show otherwise. Building HIPAA compliant AI starts with treating the model vendor as a business associate rather than a software subscription.

Healthcare CTOs and compliance leaders have shipped software through every regulatory season. AI adds power and complexity, but the old rules still apply. The difference is that unpapered model access to PHI moves the exposure to the data pipeline itself. Our view: most AI-related HIPAA issues in 2026 will come from unpapered data flows, not model magic.

The practical question is rarely whether HIPAA applies to an AI feature. It does. The question is where in your architecture the obligation actually lands, and which of those obligations you can contract away versus which stay yours no matter who you buy from. That is what the rest of this is about.

---

What Actually Changes When Your Model Touches PHI

Nothing in HIPAA changed for AI. Your obligations did not shrink. What changed is the surface HIPAA must cover. When a model or its operators can access PHI, your BAA, data flow diagrams, and technical safeguards must extend to that model context and its operations, not just your core application.

Here is the operator view:

• The AI pipeline becomes part of your regulated system of record. That includes prompts, grounding data, embeddings, intermediate stores, logs, feature stores, and outputs.
• Your access-control stack must bind identities and roles across app, data, and model services.
• Your incident scope must include prompt injection and model extraction routes, not just network intrusion.

HIPAA Security Rule technical safeguards live at 45 CFR 164.312. The CFR text is not AI-specific, which is the point. You still need access control, audit controls, integrity, person or entity authentication, and transmission security. Stitching consistent identity and audit primitives across AI services and your existing stack takes real work.

If your PHI touches AI, treat the full chain as production, not a lab. That means quality gates, rollout controls, and rollback plans equal to your core EHR-bound features.

---

The BAA Is The Line

Under 45 CFR 164.502(e), you may disclose PHI to a business associate only after obtaining satisfactory assurance that it will safeguard the information, documented in a written contract that meets 164.504(e). That contract is the BAA. Without one, sending PHI to an AI tool is an impermissible disclosure. With one, you can share PHI under defined controls.

OCR has not published an AI-specific BAA template and I would not wait for one. These are the clauses I now push for in every AI exhibit:

• Role-based access controls, periodic access reviews, and access limited by business need across vendor personnel.
• Clear breach definitions that include unauthorized access through the AI system itself, including successful prompt injection or model extraction events.

The penalty exposure is real. For penalties assessed on or after January 28, 2026, HHS set the Tier 4 range at $73,011 to $2,190,294 per violation, with a $2,190,294 cap on a calendar year of violations of the same provision (91 FR 3665). Your board will want that quantified and gated by contract. If a vendor refuses a BAA while asking for PHI exposure, walk.

Some vendors’ standard terms still treat AI as an analytics add-on. You’ll need to push for BAA paper and AI-specific exhibits.

---

The Four Questions A 2026 BAA Has To Answer For An AI Vendor

A 2026-ready BAA must address four AI-specific points: whether PHI may be used to train or fine-tune models, which cloud environments and regions are permitted, what de-identification standard applies before any secondary analytics, and what access controls and audit rights you have over the vendor’s staff and systems.

Here is how we advise clients to frame each:

1) PHI for model training or fine-tuning

• Must the vendor only process PHI for your project, or may they train general models with it under any circumstance
• If fine-tuning is allowed, is it on a tenant-isolated instance, with documented data retention and deletion
• Are model artifacts, prompts, and embeddings scoped to your BAA and explicitly excluded from vendor reuse

Training rights are the clause I see negotiated hardest and drafted loosest. Default to no reuse outside your project unless you have a strong, documented reason.

2) Cloud environments and regions

• Which clouds and services are permissible
• Are regions, KMS, and data residency pinned in contract

Cloud choice is where the contract and the architecture have to agree, and they often don’t on the first pass. Aligning vendor stacks with your allowed services list sometimes takes negotiation cycles; budget time for it.

3) De-identification standard for secondary analytics

• If data will be used for analytics beyond your direct treatment, payment, or operations purpose, what de-identification standard applies
• Who attests and what process is used before the vendor may proceed
• How are re-identification risks monitored and reported

HIPAA gives you two de-identification routes and no third: Safe Harbor, which strips 18 identifier categories, or Expert Determination, at 45 CFR 164.514. Name which one applies in the BAA, and name who attests to it.

4) Access controls, periodic reviews, and audit rights

• Require role-based access controls, periodic access reviews, and access limited to defined business need for vendor personnel
• Define the audit trail you’ll receive, the cadence, and what triggers an exceptional review
• Clarify your right to conduct or commission audits tied to the AI environment

The fastest way to stall a project is a vague access clause that both legal teams read differently. Nail it early.

---

AI Vendor With A BAA vs Without vs De-Identified-Only

Your operating stance changes with the contract and the data you send. With a BAA, you can process PHI under agreed controls. Without a BAA, you cannot send PHI at all, full stop. With de-identified data only, you still need clarity on the standard used and proof that re-identification risk is managed.

Scenario	What you can send	Contractual must-haves	Risk posture	OCR scrutiny (my read)	Practical impact
AI vendor with a signed BAA	PHI within minimum-necessary scope	AI exhibit covering training rights, cloud environment, de-identification standard, RBAC, access reviews, breach definition including prompt injection/model extraction	Managed contractual risk	Lower if controls are explicit and monitored	Build with live PHI in production-like settings with proper gates
AI vendor without a BAA	No PHI, only non-PHI or publicly available data	Terms must bar model reuse and logging of sensitive inputs, but still no PHI	High if PHI is transmitted; the disclosure is impermissible and presumed a breach	High if PHI is shared at any point	Prototype with synthetic or public data only; block PHI at the edge
De-identified data only	De-identified datasets, consistent with your policy	Define the de-identification standard and attestation; prohibit re-identification	Moderate; re-identification risk remains if standard is weak	Focus on de-identification process and controls	Useful for analytics and some model tasks; validate utility carefully

Teams often overestimate what de-identified data can do for LLM use cases that hinge on patient-specific context.

---

What Stays Yours Even With A Signed BAA

A BAA does not outsource your HIPAA Security Rule obligations. You still own minimum-necessary use, your application-layer controls, your access management, and your incident response. The BAA gives you a lawful basis and shared duties. It never replaces your program.

Your responsibilities include:

• Application-layer controls. Access control, audit logging, and transmission security remain anchored at 45 CFR 164.312. Read the rule on ecfr.gov to align technical safeguards with your architecture.
• Minimum necessary. Limit prompts, context windows, and retrieval scopes to the minimum needed for the task. Hard controls beat policy here.
• Prompt and output governance. Build redaction at ingress, PHI pattern checks, toxicity filters, and output classifiers before anything writes back to a medical record.
• Data mapping and lineage. Maintain current data flow diagrams covering prompts, grounding stores, vector indices, caches, and observability tools.
• Encryption and key management. Bring your own keys, separate duties, and rotate keys on vendor personnel changes that touch your environment.
• Identity and access. Enforce SSO, MFA, conditional access, and least-privilege roles that bind across app, data, and model surfaces. Schedule periodic access reviews and revocation SLAs with the vendor, and put the cadence in the contract.
• Monitoring and alerting. Instrument prompt ingress, RAG retrieval events, model calls, and write-backs. Alert on out-of-policy prompts, high-entropy output patterns, and unusual retrieval joins.
• Incident response. Define playbooks that treat successful prompt injection or model extraction as potential security incidents, with notification paths consistent with your BAA.
• Vendor oversight. Run regular control attestations and tabletop exercises with the vendor team. Treat them like an extension of your SOC.

For a deeper rationale on why app-layer controls are decisive, see Sidebench’s write-up on why HIPAA compliance starts at the application layer, not the cloud: https://sidebench.com/why-hipaa-compliance-starts-at-the-application-layer-not-the-cloud/

Aligning product roadmaps with these controls can feel slower at first, but it pays back in predictability during audits and investigations.

---

The Unsettled Questions: Model Weights And Prompt Injection As Breach

HIPAA already presumes that an impermissible disclosure of PHI is a breach, and a prompt injection that surfaces PHI would start from that presumption. What remains unsettled is the rest: whether extracting a model trained on PHI counts, and whether those weights must be destroyed when a contract ends. Your BAA has to answer both.

The presumption is written into 45 CFR 164.402. An impermissible use or disclosure counts as a breach unless you can show a low probability that the PHI was compromised, based on a risk assessment of at least four factors: the nature and extent of the PHI, including how likely it is to be re-identified; who received it; whether it was actually acquired or viewed; and how far the risk has been mitigated. That is the obligation you inherit the moment PHI goes somewhere it should not, which is why it belongs in the BAA as a defined event rather than a surprise.

This is where legal language meets ML reality:

• Define breach to include any route where an attacker coerces the system to reveal PHI or extracts proprietary model artifacts containing PHI-derived signals. That includes prompt injection on RAG pipelines and model-stealing.
• Include notification timelines and forensic cooperation duties for AI-specific incidents.
• Decide and document the model-weight policy. If models were trained or fine-tuned with PHI, your BAA should state whether weights must be deleted or retrained, and under what verification protocol.

I would not assume a regulator will accept silence on either point. Verification of model-weight destruction or isolation is a live technical challenge.

If you cannot verify post-incident model hygiene, don’t permit PHI for vendor-managed general model training in the first place.

The financial stakes bear repeating. Tier 4 penalties now run to $2,190,294 per violation, and the same figure caps a calendar year of violations of one provision. That number will shape how your board views AI risk acceptability.

---

Where The PHI Actually Goes: The Architecture Decision Behind The Contract

The contract question and the architecture question are the same question asked twice. Teams solve it one of three ways: sign a direct BAA with a model vendor, run inference inside a cloud platform already covered by an existing BAA, or de-identify before data leaves your boundary. The third is the cheapest to defend and the hardest to get right.

The first route is the one most teams try first, and it is the slowest. The major model vendors will now sign a BAA, OpenAI, Anthropic and Google Cloud among them, so the question has moved from whether they sign to what the signature covers. Each limits its BAA to named services and configurations, and the training-rights conversation still happens on their paper, per vendor, every time you add one.

The second route collapses that problem into a contract you probably already have, and it is the one we reach for first. Amazon Bedrock is our preferred stack for this. AWS lists Bedrock on its HIPAA eligible services reference, so a single AWS Business Associate Addendum covers the service and you are not negotiating separately with each model provider. Model and region availability vary, so confirm both for the specific model you plan to use before PHI goes near it. Per AWS, your data is not shared with model providers and is not used to improve the base models.

One caveat matters more than the rest. HIPAA-eligible means AWS supports using that service with PHI under a BAA. It does not make your workload compliant. Under the shared responsibility model, your configuration, your access controls and your logging are still yours to get right, and that is where audits actually land.

Route	What you gain	What you still own	Where it gets stuck
Direct BAA with a model vendor	Access to a specific model	Everything above the API, plus training-rights terms per vendor	Most vendors will not sign, and each new model restarts the negotiation
Managed platform under your existing cloud BAA	One contract covering multiple models, provider isolation	Configuration, access control, logging, retention, and the whole shared-responsibility half	Allowed-services lists and region pinning need settling with security early
De-identify before it leaves your boundary	The narrowest possible exposure	The de-identification standard itself, and proving it holds	Teams underestimate what has to stay identifiable for the feature to work

One of our own builds runs on exactly this reasoning. A hospice technology company came to us with documentation that arrived as scanned paper and unstructured notes, and needed it turned into something structured enough to work with. The pipeline combines OCR with language models, and every design decision in it was set by where the PHI was allowed to travel, well before anyone asked what the models could do. That is usually the order in regulated work.

The third route deserves the hardest thinking, because the question is rarely “can we strip the names”. It is which fields this feature genuinely needs to be identifiable to do its job. A summarization feature usually needs far less identity than the spec assumes. A feature that writes back to a chart needs all of it. Work that out before you design the data flow, because it decides which of the three routes you are actually on.

---

Wellness-Adjacent AI vs Clinical AI: Obligation Differences

Wellness features and clinical features live under different expectations. Wellness-adjacent AI can often run on de-identified or non-PHI datasets. Clinical AI that affects diagnosis or treatment will carry higher HIPAA controls and may trigger FDA pathways. Treat the claims you make as the switch that sets the regulatory bar.

Dimension	Wellness-adjacent AI (coaching, general education, admin helpers)	Clinical AI (diagnosis support, treatment guidance, medical record write-backs)
Product claims	Lifestyle, wellness, admin efficiency	Clinical benefit, diagnosis, treatment, medical decision support
Data category	Often non-PHI or de-identified; can avoid PHI entirely	PHI almost certain; minimum necessary must be engineered
BAA requirement	Possible to avoid if no PHI is shared	BAA required for any vendor touching PHI
FDA considerations	Usually outside SaMD scope	May trigger SaMD; plan for evidence, risk management, labeling
Security obligations	Still strong, but can scope out PHI controls	Full HIPAA technical safeguards across app and AI surfaces
EHR integration	Optional or read-only	Often bi-directional; strong audit and rollback needed
Monitoring	Drift, toxicity, and privacy checks	All of the left, plus clinical safety monitoring and override behavior
Incident response	Privacy and reputational	Privacy, patient safety, and regulatory reporting timelines

Regulated software raises the bar on evidence, change control, and postmarket surveillance, and none of that can be retrofitted cheaply. If your claims lean clinical, staff the regulatory thread on day 1.

It is tempting to keep claims vague to avoid FDA scope, but ambiguity can backfire if your marketing or UI implies clinical use.

---

If You Already Have A Spec, How To Judge A Build Partner

Pick the team that treats AI as part of your regulated system, with a BAA-ready process, application-layer controls, and clear data contracts. Ask for proof in production settings, not slideware. The right partner will show how strategy, engineering, and compliance line up from first sprint to audit.

A practical checklist:

• BAA readiness for AI. Do they have AI exhibits that address training rights, cloud environments, de-identification, RBAC, breach definitions including prompt injection and model extraction, and model-weight disposition
• Application-layer controls. Can they show reference implementations for 45 CFR 164.312 safeguards in AI flows: identity-binding across model calls, prompt and output guards, audit trails, and redaction
• Data governance. How do they map and tag data across prompts, embeddings, and outputs; what lineage tools are in place; what de-identification standard can they support with evidence
• Cloud posture. Can they meet your allowed services list, and how do they pin regions and KMS
• EHR integration depth. Ask for proof of read and write integrations and rollback strategies that protect the medical record
• Model operations. How do they guard against prompt injection, data poisoning, and model exfiltration; what drift and bias monitors do they run; how do they test and stage changes
• Incident response. Do they have AI-specific tabletop playbooks that include vendor coordination and regulator-ready timelines
• Regulatory planning. If your claims are clinical, what is their SaMD experience; how do they design evidence generation into the build
• Outcome track record. After launch, what happened in prior programs that look like yours

Sidebench has shipped 60+ healthcare implementations over 14 years, with HIPAA-compliant architecture and EHR integration depth behind them. For the detail on specific programs, our case studies are the better read than anything we could summarize here.

---

FAQ

Does HIPAA have special AI rules

No. HIPAA’s Security Rule technical safeguards sit at 45 CFR 164.312. The same access control, audit, integrity, authentication, and transmission security requirements apply to AI systems. Your job is to extend those controls to your AI data flows and vendors.

If I don’t send output back to the EHR, do I still need a BAA

If the AI vendor sees PHI at any point, you need a BAA. 45 CFR 164.502(e) only permits the disclosure once satisfactory assurances are documented in a written contract, so without one the disclosure is impermissible.

Can my AI vendor use our PHI to train their general model

Only if your BAA explicitly permits it, in words, not by silence. Many covered entities prohibit vendor reuse of PHI for general training and confine it to tenant-isolated fine-tuning with deletion requirements.

What cloud environments are acceptable for AI processing with PHI

Your BAA should name the cloud services PHI is allowed to touch. Pin regions, key management, and data residency in the contract rather than in a solution document that can be revised later.

What does de-identified mean in practice for AI

Define the de-identification standard in your BAA before any secondary analytics, and name whether it is Safe Harbor or Expert Determination under 45 CFR 164.514. Require attestation, documented process, and monitoring for re-identification risk.

How should breach be defined for AI systems

Include unauthorized access through the AI system itself in the breach definition, such as successful prompt injection or model extraction. Your BAA should set notification timelines and forensic cooperation for these events.

If a model was trained with PHI and there’s a breach, must weights be destroyed

This is unsettled, and nobody has ruled on it. The open question is whether model weights trained on PHI must be destroyed after a breach or contract end. Address it explicitly in your BAA with a verification plan. Verification can be difficult.

What Security Rule controls matter most at the AI layer

Tie identity to model calls, restrict prompts to minimum necessary, encrypt in transit and at rest, log prompts and outputs, and monitor for abnormal retrieval and output patterns. Align your design with 45 CFR 164.312 and document it.

Can I use de-identified data to skip the BAA

Possibly, if no PHI is shared and the data truly meets your de-identification standard under 45 CFR 164.514. Your BAA with other vendors and your internal policy must allow it. Expect your de-identification process to be examined if there is ever a complaint.

What are the financial stakes of getting this wrong

For penalties assessed on or after January 28, 2026, Tier 4 runs from $73,011 to $2,190,294 per violation, with a $2,190,294 calendar-year cap for repeated violations of the same provision (91 FR 3665). That doesn’t count remediation costs, reputational damage, or contract terminations. Don’t carry this risk for the convenience of skipping a BAA.

---

What Separates The Teams That Ship From The Teams That Stall

The teams that ship AI features in regulated healthcare settle three things before the first sprint: which route the PHI takes, what the BAA says about training rights and de-identification, and which controls stay theirs under the shared responsibility model. The teams that stall discover all three in security review, six weeks before a launch date.

None of that is technically hard. It is slow, and it is unglamorous, and it needs legal, security and product in the same conversation rather than in sequence. Teams that sequence it lose a quarter. Teams that parallelize it lose a fortnight.

The competitive consequence is worth naming. As managed platforms bring more models under a single cloud BAA, the contractual barrier that used to slow everyone down evenly is falling away. What separates teams now is whether the data architecture underneath was designed for this or is being retrofitted to it.

If you are about to put PHI through a model, we can review your data flows and your draft BAA and come back with the exhibit language and the architecture route that actually fit your risk appetite. See how Sidebench approaches product strategy and discovery, or send the spec and we will start there.

---

Cited sources

• HIPAA Privacy Rule, disclosures to business associates, 45 CFR 164.502(e), and business associate contract requirements, 45 CFR 164.504(e)
• HIPAA breach definition and the four-factor risk assessment, 45 CFR 164.402
• AWS, HIPAA Eligible Services Reference and Amazon Bedrock security and compliance
• OpenAI, Business Associate Agreement for the API; Anthropic, BAAs for commercial customers; Google Cloud, HIPAA compliance
• HHS, Annual Civil Monetary Penalties Inflation Adjustment, 91 FR 3665, published January 28, 2026. Tier 4: $73,011 to $2,190,294 per violation; $2,190,294 calendar-year cap
• HIPAA de-identification standard, 45 CFR 164.514: Safe Harbor and Expert Determination
• HIPAA Security Rule technical safeguards at 45 CFR 164.312

---

---

About the author