Working Framework

Agent Usability Heuristics

What are the qualities of an API or CLI interface that make it easier or harder for an agent to reason about and use effectively?

Mark Hazlewood, Staff Product Designer, HubSpot Developer Product Group
v0 — May 2026 · Source

Why this framework

Nielsen wrote his 10 usability heuristics in 1994 for human users of GUIs. There is no equivalent canon for agents as consumers of APIs, CLIs, and tool surfaces.

Two things changed at once in 2025-2026. Agents became real consumers of API and tool surfaces (MCP, tool-use connectors, multi-step agent workflows). And platforms began committing to full API parity, meaning every product capability needs an API surface. That combination means APIs now have two distinct readers: human developers and AI agents. The constraints on each are different.

Some of what makes a surface usable for humans helps agents (consistent naming, good errors, clear scopes). Some doesn't (visual hierarchy, terse human-readable copy). Some things help agents specifically and barely register for humans (typed errors over prose, lean response shapes, self-describing schemas). This framework names the differences.

The 10 heuristics

Each heuristic includes a rule, why it matters for agents specifically, and what failure looks like.

H1 Discoverability without trial-and-error Reason-about

The agent can find the right capability from a description of intent, not by trying things until something works.

Why for agents: Humans skim and recover from wrong choices visually. Agents recover by retrying, which is expensive in tokens and dangerous in side effects.

Fails when: Two endpoints with overlapping responsibilities, no clear signal which is canonical. Names don't match the agent's natural framing of intent.

Prior art, distinct contribution, empirical hooks

Prior art

Nielsen H1 (visibility of system status) and H6 (recognition over recall). Stylos & Myers (2008) on method placement affecting discoverability. Anthropic "choose tools thoughtfully." MetaTool operationalizes tool selection as measurable behavior.

Distinct contribution

The retry-cost asymmetry: humans recover from wrong choices visually (cheap), agents recover by retrying (expensive in tokens and dangerous in side effects). This makes discoverability more load-bearing for agents, not just differently load-bearing. Existing API-usability work doesn't reckon with this because human retry is essentially free.

Empirical hooks

  • MetaTool "whether-and-which tool" accuracy on a real tool catalog
  • τ-bench pass^k reliability — discoverability problems show up as pass-rate drop from pass^1 to pass^8
  • Synthetic agent runs: given N realistic intents, what % land on the canonical tool first try?
H2 Predictable shapes Reason-about

Inputs and outputs are typed, deterministic, and consistent. The same concept has the same shape across the surface.

Why for agents: Agents pattern-match on shapes more than humans do. A contact that looks one way in one endpoint and another way three calls later is a hallucination magnet.

Fails when: Pagination differs between resources. Date formats vary. ID fields named inconsistently.

Prior art, distinct contribution, empirical hooks

Prior art

Bloch (2006) "Names matter" + "minimize mutability." Cognitive Dimensions: Consistency and Role-expressiveness. BFCL operationalizes syntactically/semantically valid calls. StableToolBench studies shape drift over time.

Distinct contribution

Cross-endpoint shape consistency as a distinct usability dimension. Most prior work measures shape correctness per call. v1 names cross-call consistency because agents pattern-match across calls in a session — a contact that looks different in a list response vs a get-by-id response is a hallucination magnet.

Empirical hooks

  • BFCL AST-correctness on a domain-specific subset
  • Schema-similarity score across endpoints returning the same conceptual type
  • Agent confusion rate after a shape change in a synthetic eval
H3 Self-describing surface Reason-about

Every endpoint, tool, or command returns enough metadata to be understood without out-of-band documentation.

Why for agents: Agents have no prior context unless it was injected into the prompt. A self-describing surface lowers the cost of cold-start use. Schemas, examples, annotations, and inline help are first-class.

Fails when: OpenAPI spec exists but isn't loadable by the agent. MCP tool descriptions are one-liners.

Prior art, distinct contribution, empirical hooks

Prior art

Robillard (2009) five factors of API documentation (intent, examples, scenario-matching, penetrability, format). Anthropic "return meaningful context" + "prompt-engineer descriptions." MCP capability negotiation. Bloch's "every API is a little language."

Distinct contribution

The schema-as-prompt insight: for agents, there is no separation between "API contract" and "instructions to the model." Schema text is consumed as natural language. Documentation-as-afterthought is a design failure for agents in a way it isn't for humans, who can fall back on tutorials or asking a teammate.

Empirical hooks

  • Token-efficiency at fixed pass rate on τ-bench (more self-describing surfaces should reach the same pass rate with fewer tokens)
  • Agent recovery rate when passed schema-only vs schema-plus-docs
  • Description-quality eval per "Learning to Rewrite Tool Descriptions" methodology
H4 Errors that are machine-actionable Use-effectively

Errors are typed, coded, and include structured recovery information. Prose alone fails.

Why for agents: Humans read "Sorry, that didn't work" and infer intent. Agents read it literally and either retry blindly or hallucinate a fix. An error is a second-chance prompt: a structured error that names the problem, lists valid options, and points to relevant docs enables near-100% recovery.

Fails when: All errors are 500s with a string body. Validation errors don't identify the offending field.

Prior art, distinct contribution, empirical hooks

Prior art

Nielsen H9 (help users recognize, diagnose, recover from errors). Anthropic guidance on error responses as agent recovery surfaces. 12-Factor Agents "errors as tokens." Robillard (2009) error documentation as API learnability factor.

Distinct contribution

The "error as second-chance prompt" reframe. Errors aren't just observability — they're another input to the agent's next decision. Existing error-design work treats error messages as user feedback; v1 treats them as part of the prompt the agent reads on the next turn. Shifts what counts as a good error: not "what went wrong," but "what would help the agent succeed on retry."

Empirical hooks

  • Recovery rate after error injection (invalid arg, unauthorized, rate-limited) — % of traces that recover within N retries
  • Error-content quality: does the error name the field, list valid values, point to docs? (Static measurement)
  • Tokens-to-recovery: how many tokens does an agent spend after an error before a successful retry?
H5 Composability Use-effectively

Outputs of one operation match inputs of related operations without translation.

Why for agents: Multi-step agent workflows chain calls. Every translation step (IDs, formats, references) is a place to fail. A composable surface lets the agent move data between calls without reshaping.

Fails when: "List" returns IDs in one shape, "Get" wants a different shape. Reference fields aren't directly callable.

Prior art, distinct contribution, empirical hooks

Prior art

Bloch (2006) on small surfaces that compose. Cognitive Dimensions: Hidden dependencies and Viscosity. 12-Factor Agents "structured outputs." Anthropic Code Execution with MCP — the argument that agents writing code over tool outputs is more efficient than chaining direct calls.

Distinct contribution

Cross-surface composability as a usability dimension. v1's territory is API + CLI + MCP, and composability across surfaces (CLI output flows into API input flows into MCP tool call) is a property of coherent platform design. No existing framework names cross-surface composability as a specific heuristic. Note: "without translation" is the most underspecified part of v0 — v1 should sharpen this.

Empirical hooks

  • Multi-step task success on τ²-bench / MCP-Bench
  • "Translation overhead": count places in agent traces where the agent reshapes data between calls
  • Static analysis: what fraction of "outputs of X" are valid "inputs to Y" without translation?
H6 Idempotency and safe retry Use-effectively

Operations declare their idempotency. Agents can retry on failure without guessing at side effects.

Why for agents: Agents retry. A lot. If the platform doesn't tell the agent which operations are safe to retry, the agent will either over-retry (duplicates) or under-retry (give up too easily).

Fails when: No idempotency keys on writes. POST that's actually upsert, not documented as such.

Prior art, distinct contribution, empirical hooks

Prior art

Cognitive Dimensions Viscosity (cost to recover from a wrong move). HTTP method semantics (RFC 9110). Stripe API design retrospective on idempotency keys as first-class affordance. 12-Factor Agents assumes retry-safe operations.

Distinct contribution

The agent-retry argument: agents retry much more than humans, and their reasoning is itself non-deterministic. Combining stochastic retry with non-idempotent operations compounds rather than cancels uncertainty. Existing API design treats idempotency as developer convenience; v1 frames it as a usability necessity. Novel claim: idempotency declaration belongs in the schema (machine-readable), not just the docs.

Empirical hooks

  • Static: what % of write operations declare idempotency in machine-readable form?
  • Duplicate-side-effect rate under controlled fault injection (force a 500 mid-write, count duplicates)
  • Survey: how often do agent builders write retry wrappers? (Revealed-preference signal)
H7 Token economy Use-effectively

Responses are lean by default and expandable on request. No mandatory bloat.

Why for agents: Every byte of response goes into a context window the agent has to pay for. A 50-field object when the agent needed two fields is taxation by API design.

Fails when: No field selection. Default response includes deeply nested associations. Pagination defaults to absurdly large pages.

Prior art, distinct contribution, empirical hooks

Prior art

Anthropic "optimize for tokens." Anthropic Code Execution with MCP — token economy as the argument for code-execution architectures. LangChain context engineering (write/select/compress/isolate). Karpathy's "context window as RAM." Cognitive Dimensions Diffuseness/terseness.

Distinct contribution

The "taxation by API design" reframe. Existing token-economy guidance treats verbosity as inefficiency; v1 frames mandatory bloat as a usability tax — the surface forcing the agent to spend context on unwanted data, structurally analogous to a UI requiring you to read a paragraph to find a button.

Empirical hooks

  • Tokens per successful call at fixed pass rate (BFCL token-efficiency variants)
  • Default-response size distribution per endpoint family
  • "Useful tokens" ratio: in a returned payload, what fraction was used by the agent in its next decision?
H8 Scoped intent Use-effectively

Authorization is granular and intelligible up front. The agent knows what it's allowed to do before it tries.

Why for agents: A 403 mid-flow is a debugging nightmare for an agent. Agents need to know the boundaries of their authority at the start, in machine-readable form, and react to scope changes cleanly.

Fails when: Scopes aren't visible to the agent. Scope names don't match capability names. Failure mode is generic 403 with no hint about which scope is missing.

Prior art, distinct contribution, empirical hooks

Prior art

OAuth 2.0 scope design. OWASP LLM Top 10 (LLM06, LLM08). Willison's lethal trifecta — narrow scopes as structural mitigation. Anthropic Building Effective Agents implies scope visibility for agentic patterns.

Distinct contribution

Scope as a usability dimension, not just a security one. Existing OAuth/IAM design treats scopes as security primitives; v1 argues that machine-readable scope visibility (agent can query its own grants, not just discover them via 403s) is a usability precondition. Novel claim: a 403 mid-flow is a usability failure, not just an auth failure.

Empirical hooks

  • Scope-mismatch error rate per task type (% of traces that hit 403 mid-flow)
  • Static: does the surface expose granted scopes to the authenticated agent in machine-readable form?
  • Scope-to-capability mapping coverage: are scope names self-describing relative to the capabilities they gate?
H9 Naming as IA Reason-about

Names carry meaning. Similar things are named similarly. The lexicon agents see in tool/endpoint names matches the lexicon humans see in docs.

Why for agents: Agents lean on names harder than humans do because they often pick a tool from a name alone. Naming is the first and most-used affordance. Empirically, tool names have stronger influence on agent selection than descriptions (ToolTweak, 2025).

Fails when: The same concept has three names. Internal jargon leaks into public names.

Prior art, distinct contribution, empirical hooks

Prior art

Bloch (2006) "Names matter." Cognitive Dimensions Role-expressiveness. ToolTweak (2025) — empirical proof that tool names dominate selection. "Learning to Rewrite Tool Descriptions" — description revision improves accuracy. Stylos & Myers (2008) on method placement. Anthropic "namespace for clarity."

Distinct contribution

Two moves: (1) Tying naming explicitly to information architecture, not just per-tool naming — categorization across the surface affects agent groupings. (2) Empirical companion via the API IA Study (card-sort agreement between agent-derived and human-developer groupings). This heuristic has the strongest competitive moat for v1 — the combination of Bloch + ToolTweak + the in-flight IA study + HCI background is a position nobody else has assembled.

Empirical hooks

  • ToolTweak resistance: replace tool names with generic identifiers, measure selection-accuracy delta
  • Tool-selection accuracy on MetaTool with a real tool taxonomy
  • Card-sort agreement between agent-derived groupings and human-developer groupings
H10 Observability and audit Use-effectively

Every action an agent takes is traceable, attributable, and visible to the customer in real time.

Why for agents: Agents operate at scale. One mistake propagates fast. Customers and admins need to see what an agent did, when, and on whose authority, both for trust and for recovery.

Fails when: Agent actions aren't distinguishable from user actions in audit logs. No per-agent attribution.

Prior art, distinct contribution, empirical hooks

Prior art

OWASP LLM Top 10 (LLM08, LLM06). Willison's lethal trifecta — audit as structural mitigation. 12-Factor Agents "small focused agents" + "structured outputs." HAX Guidelines 13-15 (correction, scope agency, notify changes). CPTO vision: "trusted by default."

Distinct contribution

Audit and observability as usability dimensions for the customer/admin persona, not just operational concerns. Novel claim: a surface where actions aren't legible to the human-in-the-loop is unusable for trust even if every individual call succeeds. Hardest of the 10 to operationalize — trust and observability resist single-metric reduction.

Empirical hooks

  • Per-agent attribution coverage in audit logs (% of actions that record agent identity distinctly from user identity)
  • Customer-facing agent activity feed coverage: % of agent actions visible to portal admins in real time
  • Time-to-detect for anomalous agent behavior (operational metric)

Candidate heuristics for v1

Gaps identified in v0, annotated with prior art and provisional rules. These are under consideration, not yet adopted.

H11 Versioning legibility Candidate

Surface versions and breaking changes are declared in machine-readable form. Agents can detect surface drift before it produces wrong behavior.

Why for agents: Humans pin versions and follow upgrade guides. Agents discover drift through failure. Versioning legibility requires the surface to declare its version and the agent to detect when its expectations no longer match.

Prior art: Semantic Versioning. StableToolBench (shape drift over time). Stripe date-pinned versioning (human-DX gold standard). Connects directly to the Release Management project's open question: "what counts as a breaking change for an agent?"

Empirical hooks: Breaking-change detection rate in agent traces under simulated drift. Static: does the surface expose version + change-history in machine-readable form?

H12 Documentation as agent-readable surface Candidate

Documentation is structured, addressable, and consumable by agents at runtime — not just as a human reference.

Why for agents: H3 covers what the surface itself returns. H12 covers the broader documentation corpus as a separate addressable surface — examples, recipes, conceptual docs. The existence of llms.txt and MCP prompts resource type means docs-for-agents is a distinct design surface, not just human docs the agent reads.

Prior art: Robillard (2009) five factors of API docs. Anthropic "prompt-engineer descriptions." The llms.txt convention emerging in 2024-25. MCP prompts resource type.

Empirical hooks: Agent recovery improvement with indexed docs vs schema-only. Coverage: % of endpoints with worked examples in agent-readable form. Doc-call frequency in real agent traces.

H13 Trust boundaries are legible Candidate

Trust boundaries — between user data, untrusted content, and external comms — are visible to the agent and the human in the loop.

Why for agents: H8 covers scoped authority. H10 covers post-hoc audit. H13 covers the design surface for boundary legibility: does the agent know when it's about to mix tainted content with sensitive data, and does the human in the loop see boundary crossings? Integrating trust as a usability dimension (not just security overlay) is a differentiator vs Anthropic and 12-Factor Agents.

Prior art: Willison's lethal trifecta. OWASP LLM Top 10. Log-To-Leak (OpenReview) on MCP prompt injection. Note: H8 + H13 overlap is a known tension — whether they merge or stay separate is a v1 decision.

Empirical hooks: Prompt-injection resistance under structured red-team scenarios. Static: does the surface tag "untrusted content" zones? Trifecta-detection coverage: can the agent identify trifecta-risky configurations?

— Latency / throughput Likely not v1

Performance as a usability dimension for agents.

Assessment: Likely not a standalone v1 heuristic. Performance is a quality concern that overlaps with existing API-quality guidance. Exception: "predictable latency" could be a sub-aspect of H2 (Predictable shapes) — agents pattern-match on latency too, and a surface that responds in 200ms usually and 30s sometimes confuses agent retry logic. Parked as a sub-aspect of H2.

H14? Stateful interactions are explicit Candidate

Any state held by the surface is declared, queryable, and resettable.

Why for agents: The current heuristics assume request/response. Surfaces with sessions, long-running tool calls, async work, or MCP sampling have different rules for H6 (idempotency on a session vs operation), H4 (errors mid-stream), and H7 (token cost over a session). Likely a v1 addition.

What the annotation pass revealed

Patterns from running every heuristic against the literature.

Open decisions

Five structural questions that determine what v1 looks like. Tentative leanings noted, not decided.

Q1. What shape is the v1 artifact?

Three shapes from prior art, each with different credibility ceilings and authoring costs.

  • Flat heuristics list (Nielsen 1994 shape) — shippable in weeks, low ceremony
  • Heuristics + dimensions (Nielsen + Cognitive Dimensions hybrid) — adds academic depth without full HAX ceremony
  • Full pipeline: heuristics + workbook + playbook + patterns (HAX shape) — highest credibility, months of work

Q2. One rubric or two?

How this composes with the existing human DX quality rubric.

  • Two composable rubrics — human DX + agent DX score on separate axes, overlap where they overlap
  • One unified rubric with persona dimension — each heuristic has "for agent" and "for human" rows

Q3. Audience?

  • Internal first — API designers, platform PMs, API Council
  • External first — vendor-neutral, academically citable
  • Both, sequenced — internal first for credibility-via-use, external second once pressure-tested

Q4. Adopt "ACI" or coin a new term?

  • Adopt SWE-agent's "Agent-Computer Interface" — inherit recognizability, position as "heuristics for designing a good ACI"
  • Extend with a platform-specific term ("Agent-Platform Interface," "Agent Surface")
  • Coin something new — differentiation but starts naming fight from zero

Q5. Derivation method?

  • Opinion + worked examples for v1 — fast, honest framing ("opinion-plus-literature"), with roadmap to factor analysis for v2
  • Nielsen-style factor analysis (250+ agent failures, coded, factor-analyzed) — high credibility, slow, requires data collection

How the heuristics cluster

The core research question splits into two halves: "easier or harder to reason about" and "easier or harder to use effectively." Agent failure modes cluster differently across the two.

Reason-about

Predict the surface before calling it. Failures show up as picking the wrong call or hallucinating shape.

  • H1 Discoverability
  • H2 Predictable shapes
  • H3 Self-describing surface
  • H9 Naming as IA

Use-effectively

Act on the surface and recover well. Failures show up as runtime errors, retry storms, scope mismatches, and trust gaps.

  • H4 Machine-actionable errors
  • H5 Composability
  • H6 Idempotency
  • H7 Token economy
  • H8 Scoped intent
  • H10 Observability

Key concepts

Vocabulary the framework builds on. Each concept either originates in the literature (cited) or is a synthesis the framework owns.

Agent-Computer Interface (ACI)

The designed surface an agent reads, calls, and interprets. Coined by Yang et al. (SWE-agent, 2024). Interface design alone, independent of the model, materially lifts task success.

Schema-as-prompt

Every field name, type, and enum value in a tool schema is consumed by an LLM as natural language. There's no separation between "API contract" and "instructions to the model." The contract is a prompt fragment.

Tool name as primary affordance

Empirically, tool names have stronger influence on agent selection than descriptions (ToolTweak, 2025). Rename-not-redocument is the higher-leverage intervention.

Context economy

Every token in a tool spec or response payload competes for finite context window space. Information density per token is a usability axis, the way visual hierarchy is for human UIs.

Error as second-chance prompt

Errors aren't just diagnostics. They're another input to the agent's next decision. A structured error enables ~100% recovery. A 500 with a string body forces blind retry or hallucination.

pass^k reliability

From τ-bench (Yao et al., 2024): the probability the agent succeeds consistently across k attempts. Frontier agents: pass^1 ≈ 50%, pass^8 < 25%. The gap is the reliability tax, much of it attributable to interface ambiguity.

Reliability tax

Framework-coined term. The amount of agent failure attributable to interface ambiguity rather than model capability. Improving the surface reduces the tax even when the model stays the same.

Surface vs. capability

A platform can fetch contacts (capability). Its particular set of endpoints, parameter names, error shapes, and docs is the surface. Most benchmarks measure both tangled together. This framework separates them.

Position relative to existing work

Where this framework sits in the landscape. Not a replacement for any of these, but a distinct contribution.

Framework Relationship
Nielsen's 10 (1994) Mirror the format and derivation method. Extend from human GUI to agent-facing surfaces.
Cognitive Dimensions Borrow dimensional thinking. Several CDs translate cleanly (viscosity, hidden dependencies, role-expressiveness).
Clarke API-CD (2004) Direct academic ancestor. Clarke did persona-driven API usability for humans. This does it for agents.
Bloch (2006) Bridge insight: "Every API is a little language." Agents read that language as a prompt, not as documentation.
MS HAX (2019) Different subject. HAX guides how agents act on humans. This guides how humans build surfaces for agents.
Anthropic Writing Tools Most direct contemporary. This extends with persona conditioning, cross-surface scope (API+CLI+MCP), and governance as a first-class dimension.
SWE-agent / ACI Adopt the "ACI" term. Provide the heuristics for designing a good one across API, CLI, and MCP.
12-Factor Agents Application-level analogue. This framework is one layer below: the surface, not the agent app.

Top references

The 10 most important sources, in reading order. Full annotated bibliography in the repo.

  1. Yang et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. Coins "ACI" and shows interface design alone lifts task success. arXiv
  2. Anthropic. Writing Effective Tools for AI Agents. Eight principles for tool design. The most direct competing artifact. Link
  3. Yao et al. (2024). τ-bench: Benchmarking Tool-Agent-User Interaction. Introduces pass^k. Frontier agents succeed <50% consistently. arXiv
  4. Nielsen, J. (1994). Enhancing the Explanatory Power of Usability Heuristics. The derivation method (factor analysis of 249 problems) is the credibility template. NN/g
  5. Clarke, S. (2004). Describing and Measuring API Usability with Cognitive Dimensions. The human-API precursor. Persona-driven API usability studies at Microsoft Research. PDF
  6. Bloch, J. (2006). How to Design a Good API and Why It Matters. "Every API is a little language." Agents read that language as a prompt. Link
  7. Anthropic. Code Execution with MCP: Building More Efficient AI Agents. Agent usability as a system architecture choice, not just tool design. Link
  8. Horthy, D. 12-Factor Agents. Application-level reliability principles. This framework is one layer below. GitHub
  9. Luo et al. (2025). MCP-Universe: Benchmarking LLMs with Real-World MCP Servers. Frontier models score 33-44% on MCP tasks. arXiv
  10. Willison, S. The Lethal Trifecta for AI Agents. Private data + untrusted content + external comms = structural exfiltration risk. Link

Parallel workstreams

Others working on adjacent problems. A key next step is syncing across these efforts so the work compounds instead of duplicating.

Edward Langan
API Foundations

Epic Week talk: "Making Our Public APIs Work for Humans AND Agents." Research on where human and agent API needs overlap (mostly) and diverge (sandboxes, idempotency).

Rachel Kellam
API Foundations

Evaluations for how well agents navigate the API. Automations to flag APIs out of compliance with the API Design Guide.

Roger Brown
CLI

Synthetic agent testing with different model/harness combos. Automating Critical User Journeys via CLI for agent-augmented productivity.

SJ Morris
DevRel

Maintains a list of agent builder contacts for research. Connective tissue between internal framework work and external developer community.

Matt Rollender
Breeze Apps

Built an internal agent that debugs how well agents can use developer docs and CLI. Potential test harness for validating heuristics empirically.

Hina Shah
UXR

"Synthetic Users" concept for product feedback within UX Research. Complementary methodology: synthetic users test the product, this framework tests the API/CLI/MCP surface.

Roadmap

Phase 1

Heuristics stabilize In progress

v0 to v1 iteration. Add candidate heuristics (versioning, docs-as-surface, trust boundaries). Pressure-test against real surfaces. Research grounding complete: annotated bibliography, frameworks crosswalk, concepts glossary, and shape-and-positioning decision doc all drafted.

Phase 2

Scorecard / rubric Next

Build a Must / Should / Nice rubric per heuristic. Use it to evaluate a real API, a CLI surface, and an MCP tool.

Phase 3

Worked examples Planned

Apply the rubric to a well-known API surface. Document findings as a case study. Use the case study to refine both rubric and heuristics.

Phase 4

Share Planned

Internal readout to leadership and API Council. Possibly external: short paper or post if the framework lands well internally.