Agent discoverability pass: differentiate the three MCP tool descriptions and sharpen README selection boundaries #162

New issue

Closed

opened 2026-09-29 06:44:41 +00:00 by kilo · 0 comments

kilo commented

2026-09-29 06:44:41 +00:00

Member

Bounded representation-only pass (no semantics, contracts, architecture or maturity changes). Baseline: canonical develop @ 59538e9 (2026-09-29, clean).

Diagnosis (repository evidence)

All public identity surfaces except one are already accurate and well-differentiated:

README opening states category ("local decision service for people using multiple subscription-backed AI models"), job-to-be-done, local-first operation, the two-mode boundary, and an honest not-on-PyPI quick start.
CLI help gives each subcommand a distinct one-line intent.
examples/agent-routing-prompt.md already states the exact intent split: select = the dispatch decision, status = diagnose capacity, simulate = explicit what-if analysis.
No llms.txt / .well-known exists (nothing stale to fix; none will be added — it would duplicate the README).

The one material defect is the MCP tool metadata — the primary surface retrieval systems rank on (tool name + short description + schema). All three tools share one generic prefix and differ only by which REST endpoint they mirror:

scarcity_status   "... contract v1. This tool reports or recommends only; it never
                   executes model inference. It accepts no logical input fields."
scarcity_select   "... contract v1. ... Inputs mirror the REST /v1/select logical contract."
scarcity_simulate "... contract v1. ... Inputs mirror the REST /v1/simulate logical contract."

Problems:

select vs simulate are not distinguishable by intent from metadata alone (both just "mirror a REST contract").
scarcity_status's description says "reports or recommends" — false for status, which never recommends — and never mentions capacity/quota/providers, the only thing it returns.
None states output, when to choose it, or when another router tool is preferable.

Secondary gap: README boundaries exist in prose but not as a compact scannable "use when / do not use when" block.

Stable intent baseline (evaluation set, used verbatim)

Direct: "I want to use scarcity-router to choose among AI models/providers while accounting for scarce subscription or quota resources." → must be identifiable and correctly explained.
Indirect positive: "I have access to several AI model subscriptions or inference sources. I want software that can choose a sufficiently capable execution route while accounting for remaining quota, scarcity, policy and task requirements." → relevant; no invented optimization capabilities.
Tool-selection: "Before executing a task, I need the best currently permitted execution route. I do not need a diagnostic report or a hypothetical simulation." → scarcity_select must be the obvious callable surface.
Status: "I want to inspect the router's currently known resource/quota state without asking it to choose a route." → scarcity_status preferred.
Simulation: "I want to evaluate how routing would behave under a hypothetical resource state without performing the real selection/execution workflow." → scarcity_simulate preferred.
Negative 1 (generic OpenAI-compatible reverse proxy / API gateway for all LLM traffic) → not presented as such (the optional execution gateway is a scoped, authenticated, explicitly deployed component, not a general-purpose gateway).
Negative 2 (autonomous coding agent) → rejected as the primary solution.
Negative 3 (hosted SaaS pooling/reselling unused quota across users) → rejected.

Baseline before the change: intents 1, 2, 6, 7, 8 pass on README/repo-description evidence; intents 3, 4, 5 are AMBIGUOUS at the MCP surface (the only interface whose metadata a tool-selection system sees).

Exact bounded changes

scarcity_router/mcp.py — rewrite the three tool description strings only. Tool names, input schemas, property names/types, required flags, envelopes and error payloads unchanged. D-028 preserved: names stay simple/unversioned; every description still opens stating it exposes "Scarcity Router machine-interface contract v1" and still states it never executes model inference (both pinned by tests/test_mcp.py). Each description gains: the intent it serves, what it returns, when to choose it, and which tool is preferable for the neighbouring intent (grounded in the existing agent-routing-prompt.md semantics and implemented output).
tests/test_mcp.py — extend the existing description test (no new framework) to pin the per-tool differentiation substrings so the intent split cannot silently regress.
README.md — add one compact "Use it when / Do not use it when" block near the top (product-selection boundaries from actual capabilities) and one honest maturity line. No README restructuring; existing prose depth preserved.

Explicit exclusions

No tool identifier, schema, envelope, error-payload or versioning changes.
No new files/manifests (no llms.txt, no ARD, no registry/dashboard/analytics/telemetry).
No Forgejo/GitHub repository metadata mutation (topics/description are owner actions; recommendation only).
No changes to routing semantics, provider policy, catalog, capability claims or supported runtimes.
No registration/publication on any external surface.

Acceptance criteria

make check, uv run basedpyright, git diff --check green on the exact PR head.
All three descriptions contain the two D-028/test-pinned phrases and are pairwise distinct by intent, output and "prefer X instead" guidance.
Intent baseline re-evaluated after the change; intents 3/4/5 no longer AMBIGUOUS at the MCP surface.
Diff touches only scarcity_router/mcp.py, tests/test_mcp.py, README.md.

Bounded **representation-only** pass (no semantics, contracts, architecture or maturity changes). Baseline: canonical `develop` @ `59538e9` (2026-09-29, clean). ## Diagnosis (repository evidence) All public identity surfaces except one are already accurate and well-differentiated: - README opening states category ("local decision service for people using multiple subscription-backed AI models"), job-to-be-done, local-first operation, the two-mode boundary, and an honest not-on-PyPI quick start. - CLI help gives each subcommand a distinct one-line intent. - `examples/agent-routing-prompt.md` already states the exact intent split: select = the dispatch decision, status = diagnose capacity, simulate = explicit what-if analysis. - No `llms.txt` / `.well-known` exists (nothing stale to fix; none will be added — it would duplicate the README). The one material defect is the **MCP tool metadata** — the primary surface retrieval systems rank on (tool name + short description + schema). All three tools share one generic prefix and differ only by which REST endpoint they mirror: ``` scarcity_status "... contract v1. This tool reports or recommends only; it never executes model inference. It accepts no logical input fields." scarcity_select "... contract v1. ... Inputs mirror the REST /v1/select logical contract." scarcity_simulate "... contract v1. ... Inputs mirror the REST /v1/simulate logical contract." ``` Problems: 1. `select` vs `simulate` are not distinguishable by intent from metadata alone (both just "mirror a REST contract"). 2. `scarcity_status`'s description says "reports **or recommends**" — false for status, which never recommends — and never mentions capacity/quota/providers, the only thing it returns. 3. None states output, when to choose it, or when another router tool is preferable. Secondary gap: README boundaries exist in prose but not as a compact scannable "use when / do not use when" block. ## Stable intent baseline (evaluation set, used verbatim) - **Direct:** "I want to use scarcity-router to choose among AI models/providers while accounting for scarce subscription or quota resources." → must be identifiable and correctly explained. - **Indirect positive:** "I have access to several AI model subscriptions or inference sources. I want software that can choose a sufficiently capable execution route while accounting for remaining quota, scarcity, policy and task requirements." → relevant; no invented optimization capabilities. - **Tool-selection:** "Before executing a task, I need the best currently permitted execution route. I do not need a diagnostic report or a hypothetical simulation." → `scarcity_select` must be the obvious callable surface. - **Status:** "I want to inspect the router's currently known resource/quota state without asking it to choose a route." → `scarcity_status` preferred. - **Simulation:** "I want to evaluate how routing would behave under a hypothetical resource state without performing the real selection/execution workflow." → `scarcity_simulate` preferred. - **Negative 1** (generic OpenAI-compatible reverse proxy / API gateway for all LLM traffic) → not presented as such (the optional execution gateway is a scoped, authenticated, explicitly deployed component, not a general-purpose gateway). - **Negative 2** (autonomous coding agent) → rejected as the primary solution. - **Negative 3** (hosted SaaS pooling/reselling unused quota across users) → rejected. Baseline before the change: intents 1, 2, 6, 7, 8 pass on README/repo-description evidence; intents 3, 4, 5 are AMBIGUOUS at the MCP surface (the only interface whose metadata a tool-selection system sees). ## Exact bounded changes 1. **`scarcity_router/mcp.py`** — rewrite the three tool `description` strings only. Tool names, input schemas, property names/types, required flags, envelopes and error payloads unchanged. D-028 preserved: names stay simple/unversioned; every description still opens stating it exposes "Scarcity Router machine-interface contract v1" and still states it never executes model inference (both pinned by `tests/test_mcp.py`). Each description gains: the intent it serves, what it returns, when to choose it, and which tool is preferable for the neighbouring intent (grounded in the existing `agent-routing-prompt.md` semantics and implemented output). 2. **`tests/test_mcp.py`** — extend the existing description test (no new framework) to pin the per-tool differentiation substrings so the intent split cannot silently regress. 3. **`README.md`** — add one compact "Use it when / Do not use it when" block near the top (product-selection boundaries from actual capabilities) and one honest maturity line. No README restructuring; existing prose depth preserved. ## Explicit exclusions - No tool identifier, schema, envelope, error-payload or versioning changes. - No new files/manifests (no `llms.txt`, no ARD, no registry/dashboard/analytics/telemetry). - No Forgejo/GitHub repository metadata mutation (topics/description are owner actions; recommendation only). - No changes to routing semantics, provider policy, catalog, capability claims or supported runtimes. - No registration/publication on any external surface. ## Acceptance criteria - `make check`, `uv run basedpyright`, `git diff --check` green on the exact PR head. - All three descriptions contain the two D-028/test-pinned phrases and are pairwise distinct by intent, output and "prefer X instead" guidance. - Intent baseline re-evaluated after the change; intents 3/4/5 no longer AMBIGUOUS at the MCP surface. - Diff touches only `scarcity_router/mcp.py`, `tests/test_mcp.py`, `README.md`.