Shipping AI features in regulated domains

I spent the last few years leading a team that builds HIPAA-compliant healthcare software used by more than 200 medical practices. It is the kind of environment where an AI feature has to clear the data boundary before anyone writes a prompt. Most "add AI to your product" advice assumes you can send anything to a model and ship on Friday, and with protected health information (PHI), that assumption is the bug.

Shipping AI under those constraints is entirely doable anyway. Here is what actually changes, drawn from leading a re-architecture of how a live healthcare platform handles PHI, from shipping AI in production with Pagewise, and from the early days of Kenda, the AI FinOps startup I just co-founded with Sebastian Lujan.

The short answer: treat the data boundary as the design, not a checkbox. Map exactly what can reach the model before writing a prompt, send the minimum and redact the rest, validate every output like untrusted input, keep a human in the loop wherever a wrong answer is expensive, and instrument the feature well enough that a quietly degrading prompt shows up on a dashboard instead of in a support ticket.

Start from the data, not the model

Every AI feature in a healthcare product has to start the same way: not with a prompt, but with a map of what data could reach the model and whether it is allowed to.

With PHI the default answer is "not as-is." That forces a few decisions before any AI work begins:

• Send the minimum. HIPAA has a name for this instinct, the "minimum necessary" standard, and it is a good engineering principle even where it is not the law. If the feature summarizes a claim status, the model does not need the patient's name, date of birth, or address to do it.
• Redact or tokenize before the boundary. Strip identifiers before the payload leaves your system, and rehydrate them after the response comes back. The model sees [PATIENT], your UI shows the real name, and the provider never holds both.
• Know your provider's paperwork. If PHI does touch a model API, you need the right agreement in place with the vendor (for HIPAA, a signed BAA) and you need to know their retention and training policies, not assume them.
• Keep payloads out of your logs. Log that a request happened, its latency, its token counts, its outcome. Never the prompt or completion text, because the moment PHI lands in your logging pipeline, the pipeline is in scope too.

None of this is exotic. It is data flow analysis, done before the fun part instead of after the incident.

Treat the model as untrusted input

A model's output is non-deterministic and occasionally confidently wrong. So I wrap it the way I would wrap any untrusted source, like user input or a third-party webhook:

• Validate structure before anything downstream trusts it. If the feature expects JSON with three fields, prove it:

// Don't trust the shape, prove it.
const result = ResponseSchema.safeParse(modelOutput);
if (!result.success) return fallback();

• Constrain the task. Grounded generation over documents you supply beats open-ended generation, both for accuracy and for auditability. The narrower the contract, the easier it is to say what "wrong" looks like and to catch it.
• Take prompt injection seriously. The documents and messages your feature reads are an attack surface. A pasted referral note that says "ignore previous instructions" should be a funny log line, not a privilege escalation, which means model output never triggers privileged actions directly. There is a reason injection sits at the top of the OWASP LLM Top 10.

The mental model that works for me: the model is a brilliant, fast, occasionally wrong contractor. You do not give a contractor production credentials on day one.

Keep a human where wrongness is expensive

In consumer apps a bad completion is a shrug and a retry. In healthcare, a wrong answer can change a claim, a bill, or in the worst case a decision about care. So the question for every feature is not "how accurate is the model" but "what does one wrong answer cost, and who catches it?"

The pattern that survives review in a regulated setting is draft, not decide. The model prepares the summary, flags the anomaly, or pre-fills the form; a person approves anything that has consequences. That is not an admission that AI is not ready. It is the same separation of duties we already apply to deploys and database migrations, applied to a new kind of actor.

The nice side effect: every human correction is labeled evaluation data for making the feature better.

Make it observable before it ships

The failure mode that worries me most is not the crash, it is the silent one. A model provider updates something, a prompt tweak regresses an edge case, and quality drifts down for weeks while the error rate stays at zero.

So an AI feature should ship with the same discipline as the rest of the stack, plus a few extras:

• A fixed eval set of real (de-identified) cases with known-good answers, run on every prompt or model change. Treat prompt changes like schema migrations: reviewed, versioned, and tested before they land.
• Tracing without payloads. Latency, token counts, refusal rates, validation failures, per feature and per model version. You want the metadata, never the PHI.
• Alerts on drift, not just errors. A rising validation-failure rate or a falling approval rate from the humans in the loop is your early warning, and it fires within hours instead of at month-end.
• Watch the spend. Token costs are the one production metric that can quietly 10x without a single error in the logs. Runaway loops and context bloat show up on the invoice before they show up anywhere else. This one bothered me enough that I just co-founded Kenda around it, and it deserves a post of its own.

Frequently asked questions

Can you send PHI to an LLM at all?

Yes, with conditions. In the US that generally means a signed BAA with the provider, minimum-necessary data in the payload, and clarity on retention and training policies. De-identifying the data before it leaves your boundary is the stronger position when the feature allows it, and HHS publishes concrete guidance on what de-identification means.

Do you need to self-host a model for compliance?

Usually not. The major providers offer enterprise agreements that cover regulated workloads, and a self-hosted model does not remove your obligations, it just moves them onto infrastructure you now have to secure yourself. Self-hosting is a real answer for some data-residency and risk postures, but it is not the default I would start from.

How do you test something non-deterministic?

Pin what you can and measure the rest. A fixed eval set with expected outputs catches regressions; schema validation catches malformed responses at runtime; sampling with human review catches the qualitative drift that assertions miss. You are not asserting byte-for-byte equality, you are asserting "still within the contract."

Is prompt injection actually a problem in internal tools?

Yes, because "internal" data is not trusted data. Referral notes, patient messages, and uploaded documents are all authored outside your organization, and any of them can carry instructions aimed at your model. If model output can trigger an action, injection is in your threat model.

The short version

Shipping AI in a regulated domain is mostly ordinary good engineering, applied with less room to be wrong. Map the data before the prompt. Send the minimum, redact the rest, and get the paperwork right. Validate output like untrusted input and keep it away from privileged actions. Put a human wherever a wrong answer is expensive. Instrument for drift and for cost, not just for errors.

No heroics required. Just decide, before the demo, that the boring parts are the feature.

If you are shipping AI somewhere regulated (healthcare, fintech, or anywhere else the stakes are real), I'd like to compare notes. Reach out.

Further reading

• HHS: the HIPAA Privacy Rule
• HHS: guidance on de-identification of protected health information
• OWASP Top 10 for LLM Applications (GenAI Security Project)