AI can reduce routine work and
make patterns easier to see. It should not quietly become the
decision-maker for consent, safety, care or a person’s rights.
Artificial intelligence is already entering home and community care
through scheduling tools, documentation assistants, remote-monitoring
systems, risk scores, chat interfaces and software that recommends what
someone should do next.
The useful question is not whether a product uses AI.
The useful question is:
What work is the system doing, who could be affected and who
remains responsible for the result?
An AI feature may help organize information without making a
decision. Another may influence which person receives attention first. A
third may draft a care note that becomes part of a record. Those uses do
not carry the same risk, and they should not receive the same level of
review.
The safest operating principle is simple: use AI to assist
people, not to make human responsibility disappear.
Start with the
decision, not the technology
Before reviewing a product demonstration, write down the real
workflow:
- What problem are we trying to solve?
- What information will enter the system?
- What output will it produce?
- Who will rely on that output?
- What happens if the output is wrong, late or unavailable?
- Who can challenge, correct or override it?
This prevents a broad promise—“AI will improve care
coordination”—from replacing a concrete use case.
“Draft a reminder using an approved template” is a use case. “Predict
who is becoming unsafe at home” is a different and more consequential
use case. The second requires stronger evidence, qualified review,
clearer escalation and careful attention to the tool’s intended use.
A practical
framework: Assist, Review or Stop
This framework is not a legal classification. It is a starting point
for deciding how much human control a workflow needs.
Assist: bounded
work a person can readily verify
AI can be helpful when the task is limited, reversible and easy for a
person to check.
Examples may include:
- drafting a routine message from approved facts;
- summarizing a non-urgent meeting for review;
- suggesting schedule options under defined rules;
- organizing an approved list of open tasks;
- translating plain administrative information with human
verification; - identifying missing fields in a form; or
- grouping recurring workflow problems for quality review.
Even here, the person using the output should know it was generated
or assisted by AI when that knowledge matters. The organization should
define what data may be entered, which sources the system may use and
who approves the final result.
AI should reduce avoidable work—not create a second invisible process
that no one understands.
Review:
AI may influence action, but a person must decide
Human review becomes essential when an output may change how care,
attention, staffing or communication is handled.
Examples include:
- drafting a care-related note;
- highlighting a possible change from a person’s usual pattern;
- prioritizing follow-up;
- suggesting a staffing or visit assignment;
- summarizing a transition or referral;
- detecting a possible documentation inconsistency; or
- recommending a next workflow step.
“A human reviewed it” is not enough. The reviewer needs the time,
authority, information and competence to disagree.
Define the review:
- What must the reviewer check?
- What source information is available?
- What limitations are visible?
- What is the confidence or uncertainty, if meaningful?
- How is disagreement recorded?
- When must the output be escalated instead of approved?
If staff are expected to approve hundreds of outputs quickly, human
review may exist on paper while automation still controls the work.
Stop: keep AI out of final
authority
Some decisions should not be delegated to an AI system.
Keep a qualified person in charge of:
- emergency recognition and response;
- diagnosis or treatment decisions;
- starting, stopping or changing medication;
- consent and decisions about who may receive information;
- final care-plan approval;
- decisions that restrict a person’s choices or services;
- eligibility, discipline or employment action;
- resolution of conflicting instructions; and
- final communication of serious, sensitive or life-changing
information.
AI may sometimes provide information that supports these workflows.
It should not become the accountable decision-maker.
For an urgent or severe concern, follow the person’s care plan,
organizational procedure and appropriate professional or emergency
route. Do not wait for an AI alert or chatbot response.
Ask what the
system knows—and what it does not
Responsible use requires more than a feature list.
Ask the supplier or internal team:
- What is the system’s intended purpose?
- What was it trained or configured to do?
- Which populations, settings and languages were represented in
development and evaluation? - What performance measures were used?
- Under what conditions does performance weaken?
- How are updates tested and communicated?
- Can users see the source information behind an output?
- How are errors, overrides and complaints recorded?
The ONC HTI-1 rule offers a useful example of the kinds of
transparency questions that matter for certain predictive
decision-support interventions in certified health IT: design,
development, training, evaluation, fairness and risk-management
information. Those specific requirements do not apply to every AI tool,
but the questions are broadly useful.
Protect
privacy before entering the first record
Do not test an AI tool with real care information merely because
access is convenient.
Before use, determine:
- what information the system receives;
- whether the information is retained;
- whether it is used to train or improve a model;
- where it is stored and processed;
- who can access it;
- how it is deleted;
- whether a contract covers the intended use; and
- what law, policy or consent requirement applies.
HIPAA protects certain health information held by covered entities
and business associates. It does not cover every consumer app, family
caregiver or technology relationship. “HIPAA compliant” should not be
treated as a complete privacy review.
Use the minimum information necessary for the approved purpose, and
never place identifiable care information into an unapproved public AI
service.
Test the workflow, not
only the model
A technically accurate system can still fail in practice.
Test ordinary and difficult conditions:
- missing or conflicting information;
- different languages and communication needs;
- a person who declines the recommendation;
- an unavailable reviewer;
- a false alert and a missed alert;
- a system outage;
- a change in the person’s usual pattern; and
- a complaint or request for correction.
Measure more than speed. Track whether people can understand the
output, whether overrides are respected, whether some groups experience
more errors and whether the workflow closes safely.
NIST’s AI Risk Management Framework emphasizes managing risk across
the AI lifecycle. Approval is not a one-time event. Performance, context
and consequences can change.
Tell people when AI
materially affects them
Transparency should be useful, not ceremonial.
When AI meaningfully shapes a decision or communication, people
should be able to learn:
- that AI was involved;
- what role it played;
- who made the final decision;
- how to ask a question or request correction; and
- how to reach a person.
WHO’s guidance on AI for health places ethics and human rights at the
center of design, deployment and use. In home care, that means dignity,
autonomy and participation must remain visible even when the technology
works as designed.
Use a human-in-charge record
For every approved use case, record:
| Question | Required answer |
|---|---|
| Purpose | What bounded problem may AI assist with? |
| Data | What information may enter the system? |
| Output | What does the system produce? |
| Reviewer | Who checks the output and what must they verify? |
| Authority | Who makes the final decision? |
| Escalation | What triggers a human-only route? |
| Evidence | How will performance and harm be monitored? |
| Exit | How can the organization pause or stop the use case? |
If any answer is missing, the use case is not ready.
The standard is
not “human somewhere in the loop”
The standard should be a person with real authority at the point
where judgment, consent, safety or rights are at stake.
AI can help a team see work, reduce repetition and find patterns. It
cannot accept moral, professional or organizational accountability. That
responsibility stays with people.
Use the Human-in-Charge AI Review before approving an AI use case.
This resource provides general educational and operational
information. It is not medical, legal, privacy, compliance, procurement
or emergency advice. Requirements depend on the tool, intended use,
organization, jurisdiction, contracts and professional duties. For
urgent or severe concerns, use the appropriate professional or emergency
route rather than relying on AI.
Sources
