In healthcare and government, the burden of proof flips
AI in healthcare & the public sector
The question: What changes when AI decisions touch patients and citizens — and what does responsible deployment look like there?
Healthcare and public-sector AI hold the highest stakes and the strictest duties: EU AI Act high-risk categories, medical device regulation, administrative law. They are also where AI's most meaningful wins live — which is exactly why the discipline matters.
What the lesson covers
The pattern that works in healthcare today is documentation and triage, not diagnosis: ambient scribes that draft clinical notes (clinician reviews and signs), discharge summaries, coding assistance, and imaging triage that ORDERS the worklist (urgent-first) rather than deciding. The professional stays the decision-maker; AI compresses the paperwork that burns them out.
Your healthcare-encounters data (in the business-datasets pack) shows the analytics side: readmission risk, length-of-stay drivers, resource planning. The fairness stakes sharpen here — a risk model trained on historical access to care learns historical inequity (the infamous US care-management algorithm used cost as a proxy for need, under-serving Black patients). Proxy vigilance is a clinical safety issue.
Public sector: eligibility triage, document processing, citizen service bots. Administrative law adds duties AI must inherit — reasons for decisions, appeal rights, equal treatment. The Dutch childcare-benefits scandal (algorithmic fraud scoring that wrongly ruined thousands of families) is the cautionary tale every government AI programme must study: opaque risk scores + weak appeal + institutional deference = catastrophe.
The governance stack is familiar but stricter: EU AI Act high-risk duties (both sectors appear in Annex III), medical-device conformity where applicable, DPIAs, human oversight that is real (not rubber-stamp — Article 22), and public transparency. The flip: in low-stakes domains you justify caution; here you justify automation.
One further asymmetry decides how these systems are received, and it is about who bears the error rather than how often it happens. In commerce a false positive costs a discount; in health or benefits it costs somebody their treatment or their income, and the person bearing it is almost never the person who chose the threshold. That is why documentation, appeal and human review are not bureaucratic overhead here but the mechanism by which the cost is returned to whoever imposed it. The practical version: for every automated decision, name the person who can reverse it, the time within which they will, and how the affected person learns that the route exists. An appeal nobody knows about is not an appeal.
Key points
- Winning healthcare pattern: scribes, summaries, coding help, worklist triage — clinician decides.
- Proxy bias is a safety issue: cost-as-need under-served the underserved; audit proxies clinically.
- Public-sector AI inherits administrative law: reasons, appeals, equal treatment — the Dutch scandal is the syllabus.
- In high-stakes domains the burden of proof flips: you justify automation, not caution.
- The error is borne by somebody who did not choose the threshold. Name **who can reverse a decision, how fast, and how the affected person learns the route exists** — an appeal nobody knows about is not one.
Framework — Assist → Order → Never decide (yet)
High-stakes AI earns its place by assisting documentation and ordering work by urgency/risk — while decisions about people stay with accountable professionals, with reasons and appeals attached.
The lab
Design a high-stakes deployment that would survive both a regulator and a journalist.
Open this lesson, its lab and its quiz
Sources and further reading
- EU AI Act — Annex III (high-risk categories) — AI Act Explorer
- Dissecting racial bias in a health algorithm — Obermeyer et al., Science 2019