MCP Tool Reliability Auditor

Setup: one sample task — audit a fictional CRM MCP server (acme-crm) whose three tools (manage, get_contact, delete_contact) were seeded with planted anti-patterns: a God-Tool, two duplicate-path collisions, an untyped param, an opaque "error" return, and zero safety annotations on a hard delete. Excerpted from the captured run (gate verdict: PASS).

Phase 1 — Per-tool scorecard (BEFORE)

TOOL 1 — manage

Axis	Finding	Sev	Why the agent mis-selects/mis-calls
F	action:"delete" duplicates the standalone delete_contact tool; action:"search" overlaps get_contact's email-lookup path	highest	Two live paths both hard-delete the same contact; two live paths both "find" a contact — calls will split unpredictably between them
B	Zero when-to-use / boundary text	highest	Nothing tells the agent this is the create/update/delete/search hub vs. the other two tools
A	"Manage CRM stuff." restates the name + filler, no functional signal	high (near auto-fail)	Agent can't tell which of 4 ops exist or map to a given request
C	action:string, no enum, forks 4 branches incl. one destructive	high	One schema/one annotation set can't honestly cover reads through irreversible deletes
D	data:object — no properties at all; schema also has no id property, though update/delete/search plainly need one	calling	Model invents both data's shape and where an identifier goes
I	required:["action"] only — data/id are branch-required but never marked	calling	Model may omit load-bearing fields on create/update/delete
G/H	delete branch destructive; no annotations at all, no dry_run/confirm	AUTO-FAIL	No consent signal, no preview before an irreversible delete
J/K	No examples; search branch unbounded, no outputSchema	polish/planning	Guessed field shapes; can't plan for pagination

TOOL 3 — delete_contact

Axis	Finding	Sev	Why
F	Duplicates manage{action:delete} — other half of the Tool-1 pair	highest	Same destructive op, two entry points
G/H	Hard delete, no confirmation, no annotations	AUTO-FAIL	No destructiveHint; no dry_run preview step before an irreversible delete
E	On failure returns the literal string "error"	AUTO-FAIL-adjacent	Can't distinguish not-found / unauthorized / already-deleted / downstream failure; can't decide retry vs. re-auth vs. give up; a bare "error" string also risks being mistaken for data rather than a failure
J/K	id format unspecified — numeric-only, or does it also accept email like Tool 2's id? No examples, no success-outputSchema	calling/planning	Flagged as a candidate ambiguity — can't assert from static reading alone; ask the buyer to confirm

Server-level top offenders: (1) the delete duplication manage{action:delete} ↔ delete_contact — a genuine architecture bug, not just a selection nuisance; (2) the search/get overlap manage{action:search} ↔ get_contact; (3) Tool 3's undiscriminated "error" string; (4) two AUTO-FAILs (Tool 1's delete branch, Tool 3) that must be fixed before anything else per the skill's own gate.

Phase 2 — Rewrite (the key fix: consolidate to one delete path)

Per axis F's own fix guidance ("merge only if the two are genuinely redundant"), the delete duplication is resolved by removing the delete branch from manage entirely rather than just disambiguating text — the two were never legitimately separate operations. (create/update/search split out into their own tools; get_contact renamed to get_contact_by_id to split its overloaded id.)

// ---- delete_contact (kept; rewritten — now the ONLY delete path) ----
// before: delete_contact(id) — "Deletes a contact by id. Returns an error if it fails." [no annotations, no dry_run, opaque "error" string]
{
  "name": "delete_contact",
  "description": "Permanently deletes one contact by id. Destructive and non-idempotent — cannot be recovered. Call with dry_run:true first to preview, then dry_run:false to apply. This is the only delete path for contacts. Failure modes: no contact with that id (nothing deleted); contact already deleted; unauthorized.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "contact_id": { "type": "string" },
      "dry_run": { "type": "boolean", "default": true, "description": "Preview without applying." }
    },
    "required": ["contact_id"]
  },
  "annotations": { "readOnlyHint": false, "destructiveHint": true, "idempotentHint": false }
}
// handler diff: replace `except Exception: return "error"` with typed messages —
//   NotFound -> "No contact with id 'X'. Nothing deleted."
//   AlreadyDeleted -> "Contact 'X' was already deleted."
//   Unauthorized -> "Unauthorized: re-authenticate."

Token tradeoff (named per the skill's own rule, not reflexed): split = 5 tools vs. the original 3 (net +2 on every tools/list). Constrain alternative (offered): fold create/update/search back into one manage_contact with action: enum[create,update,search] (delete still excluded/merged into standalone delete_contact regardless of this choice) — 3 tools total. Prefer split: create/update/search/get differ enough in destructiveness and in the exact overlap this fixture flagged (F) that honest per-tool annotations and clean selection outweigh the token cost here; constrain is the fallback if the buyer's token budget is tight.

Phase 3 — Verify with a real before/after selection eval

The rewrite's whole claim is "the agent picks the right tool more often" — so it was measured, not asserted. A fresh agent (claude-sonnet-5, tool order randomized per trial) was shown ONLY the tool definitions against a stub executor and asked to pick a single tool for each of 6 gold-labeled tasks (biased to the flagged collision pairs), N=5 trials per task, on the original defs (BEFORE) and the rewritten defs (AFTER).

Arm	Correct-selection rate	Confusion (wrong picks)
BEFORE (original 3 tools)	28/30 = 93.3%	get_contact → manage (1× on "find by email"); manage → NONE (1× on "search by name")
AFTER (rewritten 5 tools)	30/30 = 100%	none

Measured delta: +6.7 pp (93.3% → 100%), N=5/task, model claude-sonnet-5. Both BEFORE misses fell on the find-by-attribute tasks — exactly the get_contact ↔ manage.search collision flagged in Phase 1: with only "Get a contact." (id) and "Manage CRM stuff." to go on, the agent split on whether an email lookup was a get or a manage. The rewritten search_contacts ("…by name or email… not for a known id — use get_contact_by_id") plus the reciprocal boundary clause on get_contact_by_id closed that ambiguity; AFTER selected cleanly on every trial.

Read this honestly, the way the skill instructs. This is a small-N targeted regression check on one 3-tool fixture, not a public benchmark — a capable model already routes even the poor originals ~93% of the time, so the selection headroom here was small by construction. The larger wins the scorecard found are in calling and safety, which a selection eval doesn't score: the God-Tool forcing the model to invent an action string and a data shape, the hard delete with no dry_run, and the opaque "error" return. On a weaker model, a larger tool surface, or tighter collisions, the delta is typically bigger — the point of the harness is that you run it on your model and your tools and read your number, never this one.

(Method note: an earlier arm of this eval labeled "remove Dana Lee" — a name, no id — with delete_contact as the AFTER gold; the agent instead correctly picked search_contacts first, because the rewritten delete_contact refuses a name and demands a resolved contact_id. That "miss" was the eval catching safer behavior, and a mislabeled task on our part — a live example of why the confusion matrix, not just the rate, is the output that matters.)