Designing for Reliance, Not Just Adoption
Trust is not a feature you can ship.
Yet the current wave of AI product development routinely treats it as one — a checkbox in a responsible AI framework, a reassuring word in a product demo, a commitment deferred to the next sprint. Teams measure adoption instead: sign-ups, active users, workflows run. Adoption tells you people tried a product. It doesn’t tell you whether they can rely on it.
Human factors research offers a more useful goal. John Lee and Katrina See (2004) framed it as appropriate reliance: people depending on an automated system as much as it deserves — no more, and no less. I design AI products for scientists and other domain experts, and much of my work happens in the gap between adoption and that kind of reliance. Closing it isn’t about asking people to trust AI more. It’s about giving them the means to check it.
Why experts can’t take AI on faith
In science, misplaced reliance has a price. Lab validation is slow and expensive, so a biologist who acts on a confident but wrong gene-function prediction loses real time and money. Errors that survive the lab can spread through published literature. And in multi-agent systems, where one agent’s output becomes another’s input, it can be hard to trace how a system reached its recommendation. The same questions run through my current work on AI decision support for commercial aviation.
Lisanne Bainbridge named the deeper problem in 1983. In “Ironies of Automation”, she argued that the more advanced an automated system becomes, the more crucial its human operator’s contribution — even as automation erodes the skills and attention that contribution depends on. Designing for reliance means taking that irony seriously. Experts stay in the loop not as a formality, but because their judgement is what makes a system safe to depend on.
What reliance asks of a product
In practice, I think designing for reliance asks four things of a product: that it shows its work, keeps the expert in charge, leaves an honest record, and is candid about its limits and costs.
Showing the work means putting evidence where the decision happens, not in documentation. In Sciansa, the multi-agent AI platform whose design I led at CSIRO, every workflow step carries its own context, outputs and notes, plus an explanation of what the agents did. This is seamful design — deliberately revealing some of a system’s workings rather than hiding them (Chalmers and MacColl, 2003). My test for a seam is whether people can understand an automated decision, configure it and contest it. A tooltip that explains a decision the user still can’t change is not a seam.
Keeping the expert in charge means the system proposes and the scientist decides. Sciansa drafts a workflow from a plain-language request, but the scientist reviews each step before anything runs. Researchers quickly built a mental model of what the platform could do — and accurate mental models are what appropriate reliance depends on.
Leaving an honest record means what ran stays fixed. Once a Sciansa step has run, its prompt becomes read-only, so downstream steps rest on a stable foundation and scientists can reproduce the workflow.
Being candid starts with naming costs. Reviewing a plan first means more steps before a first result; freezing a prompt means less freedom to revise a step after it runs. I name those costs deliberately: a product that hides its trade-offs asks for the same blind faith it claims not to need.
Reliance depends on context
None of this works as a universal checklist, and my own research shows why. Before Sciansa, I led the design of Operationalising Responsible AI, a CSIRO project to turn Australia’s AI Ethics Principles into checks that science teams could act on while building AI systems. Its brief was to reach technology readiness level 3: proof the idea could work, not a product.
We interviewed scientists across agriculture, astronomy and water research, and the clearest finding was that ethical considerations for AI in science are highly specific to each use case, and sometimes in conflict. That’s why my method starts from real systems rather than principles, and from the people who will depend on them. The findings fed into an empirical study, Investigating responsible AI for scientific research.
How the work gets done
Reliance can’t be retrofitted, so method matters. On Sciansa, we worked with more than 100 domain experts through co-creation and user research over two years, and what they told us shaped the decisions above.
Early in a product’s life, I use provotypes — deliberately provocative prototypes (Boer and Donovan, 2012) — to surface hidden assumptions about AI before a team builds the wrong thing. I also treat context engineering, the authored instructions, data and examples that steer an AI system, as a design material. Sciansa’s notes feature is a small example: it lets researchers control what context carries into the next step, and testing showed controls like it were critical to usability.
And I try to make the stance reusable. The Seamful Design Audit is a Figma skill that finds where a design hides automated decisions; the Critical Design Engineering Manifesto sets guardrails for designing with agentic tools.
Why now
This matters more each month. Agentic features keep moving decisions below the interface: software increasingly ranks, sorts, corrects, completes and pre-decides on people’s behalf, often with no seam where they could see the choice, change it or contest it.
Arvind Narayanan and Sayash Kapoor (2025) argue for seeing AI as “normal technology” — that its impact will depend less on what models can do than on the slower work of fitting them into real institutions and practices. I agree, and I think much of that slower work is design work. Bainbridge described the irony in 1983. What’s new is the speed and the scale.
Adoption is not the finish line
By March 2026, more than 3,000 scientists had run over 12,000 workflows on Sciansa. That’s adoption. The signal I value more is quieter: a scientist telling us the platform “directly aids in answering my research questions”. That’s closer to what reliance sounds like.
I don’t think designing for reliance is a problem you solve once. The systems are too complex, and the stakes too dependent on context. What I do think is that it starts with taking the question seriously — at the beginning of a product, not the end — and being honest about what each decision costs. That’s the orientation I bring to every project. If you’re working on the same problem, I’d like to compare notes.