{"criteria_id":"tool-ergonomics/v1","service":"tool-ergonomics-audit","version":"1.0.0","what_is_checked":"Whether a model can tell an MCP tool set's tools apart and call them from what is written: description discrimination (A), parameter specification (B), set economics (C), and naming hygiene (D), measured deterministically over the submitted tools/list document against the rules below. The operator submits a manifest — one server's tools/list, or a whole connected set across servers (the collision checks need the set).","scope":"This audit grades measurable document properties of the submitted manifest only — description discrimination, parameter specification, definition-token economics, and naming hygiene — against the published tool-ergonomics/v1 rules. It never runs the tools, never queries a model, and never measures or promises any model's selection accuracy or any task's success. Properties that require judgement — prose elegance, task-fit, and naming taste — are NOT graded in v1, and no result in this report is a judgement of them.","split_profile_addendum":"Addendum — a separate fleet service, the token-efficiency/v1 `split-profile` op, reports the composition of a submitted catalogue's definition-token count as shares (name / description / schema, catalogue-wide and per tool) and the groups of tools sharing a byte-identical parameter schema under its published serialisation. It publishes no definition-token figure, estimated or exact, for any tool or for a catalogue, and no character count: the per-tool estimated definition tokens in this audit's paid report, and the definition-token total in this audit's free pre-check and paid report, remain the only definition-token figures either service publishes. Its outputs are proportions and schema-identity relations; they state no total, and neither they nor any combination of them with this audit's estimated figures can be used to determine an exact definition-token count — a share multiplied by an estimate is an estimate of a share of the count, not the count.","known_limits":"Known limits of tool-ergonomics/v1: (1) Metric gaming — an author can stuff descriptions with unique filler to raise discriminating-content share while making selection worse; Sections A and C jointly resist this (the boilerplate share and the per-tool definition-token cost punish stuffing), but the residual is real and is named here rather than hidden. (2) Token figures are estimates under the published 4-characters-per-token method, not counts from any specific model's tokenizer. (3) Community-reported reference points are labelled as such and are never pass/fail standards. (4) A manifest that passes every measurement here can still be unclear to a given model on a given task — that is an empirical property of a model-and-task pair, and no document-only audit can measure it.","retention":"Submitted manifest content is processed in memory only and is never stored — not in KV, not in logs, not in the ledger. The payments ledger records only the payer, the time, the price, and the aggregate outcome (verdict band, per-section counted-flag subtotals, and tool/parameter counts). Because nothing is kept, a paid retry re-sends the manifest (or its URL).","fetch_contract":{"applies_to":"URL mode only ({ url } submissions). Manifest mode ({ manifest } submissions) fetches nothing.","method":"Read-only GET requests, the submitted URL fetched at most once per audit, following redirects; the fetched document is graded as the manifest.","user_agent":"ExchangeRebel-ToolErgonomicsAudit/1.0 (tool-ergonomics audit; read-only fetch of a submitted manifest document; contact: toll-api)","urls_fetched":["the submitted manifest URL only"],"never_fetched":"No crawling, no other paths, no form submission, no login, no POSTs to the submitted URL's origin, and the MCP server at any endpoint named in the manifest is never contacted or executed.","caps":{"per_document_bytes":524288,"per_document_timeout_ms":10000,"documents_truncated_at_cap":"a document larger than the cap is NOT graded on a truncated copy — the report is the could-not-grade shape naming the cap"},"url_rules":"http/https only; no embedded credentials; no localhost or .local/.internal/.lan/.home names; IP literals in private, loopback, link-local, CGNAT, benchmarking, or multicast/reserved ranges are rejected; ports 80, 443, 8080, 8443 only. An invalid URL is rejected before any payment is verified and never consumes a payment."},"input_contract":{"format":"JSON POST body","fields":{"manifest":"the tools/list document, in any of these shapes: { tools: [...] }, { result: { tools: [...] } }, or a bare array [...]. Each tool: { name, description?, inputSchema?, annotations? } — annotations are carried in the document but never graded here. Mutually exclusive with url.","url":"the URL of a tools/list JSON document to fetch under the fetch contract and grade as the manifest. Mutually exclusive with manifest.","caller":"Solana base58 wallet of the operator (required for the free pre-check; keys the free allowance). On the paid audit the payer is read from the payment itself."},"caps":{"max_body_bytes":524288,"tool_cap":500},"tool_cap_rule":"The tool cap is 500 tools, stated exactly. A normalized manifest with more than 500 tool entries produces the signed could-not-grade shape (kind too_many_tools); on the paid endpoint it is returned 422 UNPAID before any payment is verified, so it never consumes a payment signature.","zero_tools":"A manifest with zero tools returns the signed could-not-grade shape (verdict 'COULD NOT GRADE — the submitted manifest contains no tools, or contains more than the published 500-tool cap, or (in URL mode) the manifest document could not be fetched and parsed as a tools/list document, so no check could be graded. This is not a pass and not a fail.', no band). On the paid endpoint this is checked BEFORE any payment is verified, so it never consumes a payment."},"definitions":{"D-1_tool_entries":"The graded object is the normalized tool list. A tool entry that is not a JSON object is graded with: name missing, description missing, zero parameters; it appears in per-tool evidence by its array index. A tool's display name is its name if it is a non-empty string, otherwise the string \"#<index>\" with its 0-based array index.","D-2_tokenization":"A description's tokens: if description is not a string, the token list is empty. Otherwise lowercase (String.prototype.toLowerCase), replace every maximal run of characters that are NOT a-z or 0-9 with a single space, split on spaces, drop empty strings. No stemming, no stopword removal.","D-3_sentences":"Split the raw description on runs of one or more of . ! ? followed by whitespace or end-of-string; trim each piece; drop empty pieces. A description with no such delimiter is one sentence. A sentence's identity is its token list joined with single spaces; a sentence with an empty token list has no identity and is ignored by every sentence-based rule.","D-4_boilerplate":"A sentence identity is a boilerplate sentence when the number of distinct tools whose descriptions contain it is at least 2 AND strictly greater than 25% of the number of tools in the set. Tokens belonging to boilerplate sentences are excluded from every discrimination measurement in Section A; Section C measures how much of the set's description text boilerplate is.","D-5_effective_tokens":"A tool's effective token set is the set of tokens occurring in its description's sentences whose identities are NOT boilerplate sentences. If every sentence is boilerplate, or the description is missing, empty, or not a string, the effective token set is empty.","D-6_rounding":"Every similarity, share, and rate reported is rounded to 3 decimal places (Math.round(x*1000)/1000). Every flag decision is made on the unrounded value against the published threshold.","D-7_ordering":"Every list in a report is sorted by its published rule (tool name ascending by code unit, then array index; parameter name likewise; pairs by their two display names). The same manifest always produces byte-identical section results except for timestamps and signatures.","D-8_evidence_classes":"measured — computed from real documents or published measurement of real systems; vendor-guidance — stated in a model vendor's published production guidance; community-reported — reported in community post-mortems, not a controlled measurement and never presented as a standard; fleet convention — a threshold this fleet chose and publishes as its own convention, with no external authority claimed."},"sections":{"A_description_discrimination":{"question":"From what is written, can a reader tell these tools apart? Measured over effective tokens (D-5) only — shared boilerplate never counts for or against discrimination.","a1_pairwise_similarity":{"threshold":0.55,"evidence_class":"fleet convention","method":"For each unordered pair, the Jaccard index of the effective token sets; both empty => 1.000, exactly one empty => 0.000. A pair is flagged when its unrounded similarity is >= 0.55. Flagged pairs are shown with both display names, the rounded similarity, the shared effective tokens (sorted), and the shared raw sentences."},"a2_discriminating_share":{"floor":0.2,"evidence_class":"fleet convention","method":"A tool's nearest sibling is the other tool with the highest A-1 similarity (ties: display name ascending, then index). Discriminating tokens are its effective tokens NOT in the nearest sibling's effective set; the share is |discriminating| / |effective|, or 0 when the effective set is empty. Flagged when the unrounded share is < 0.20. In a one-tool set the share is null and the tool is never flagged under A-2 or A-3."},"a3_lead_discrimination":{"evidence_class":"fleet convention","method":"The opening segment is the description's first sentence (D-3). Passes when (a) the first sentence's identity is NOT a boilerplate sentence, AND (b) at least one of the tool's discriminating tokens (A-2) occurs in the first sentence's token set. Otherwise the tool is flagged (lead)."},"counted_flags":"1 per flagged pair + 1 per A-2-flagged tool + 1 per A-3-flagged tool."},"B_parameter_specification":{"question":"Can a caller fill in each parameter correctly from what is written? Parameters are the entries of inputSchema.properties when inputSchema and properties are JSON objects; otherwise the tool declares zero parameters. A parameter entry that is not a JSON object is treated as an empty schema.","b1_description":"A parameter's description is present when its schema's description field is a string that is non-empty after trimming; otherwise absent.","b2_guidance":"Guidance is present when ANY ONE holds: enum is an array of length >= 1; OR format is a string non-empty after trimming; OR examples is an array of length >= 1; OR an example field is present with a value that is not null and not the empty string. A default alone is NOT guidance and is reported separately where present.","b3_worst_offenders":"A parameter with description absent (B-1) AND guidance absent (B-2) is a worst offender. The fix list has one entry per worst offender (add a description and at least one of enum / format / examples), one per undescribed-but-guided parameter (add a description), and one per B-4 heuristic hit (the fix the heuristic names).","b4_heuristics":{"evidence_class":"fleet convention","rules":[{"id":"id_without_format","rule":"name is id or ends with _id; type is string or unspecified; no format; no enum","fix":"state the identifier system in a description, or add a format / enum"},{"id":"url_without_format","rule":"name is url or uri or ends with _url or _uri; type is string or unspecified; format is not one of uri, uri-reference, url; no examples","fix":"add format: \"uri\" or examples showing the expected URL shape"},{"id":"count_typed_as_string","rule":"name is in {count, limit, offset, page, page_size} or ends with one of _count, _limit, _offset, _quantity; type is string","fix":"type counts as integer (or state the string encoding in a description with examples)"},{"id":"amount_typed_as_string","rule":"name ends with one of _amount, _price, _cost, _fee, _total; type is string","fix":"type amounts as number (or state the string encoding in a description with examples)"},{"id":"boolean_typed_as_nonboolean","rule":"name starts with one of is_, has_, can_, should_, allow_; type is present and is not boolean","fix":"type yes/no parameters as boolean"},{"id":"date_without_format","rule":"name is date or timestamp or ends with one of _at, _date, _time; type is string or unspecified; format is not one of date, date-time, time, duration; no examples","fix":"add a format (date-time / date) or examples"},{"id":"email_without_format","rule":"name is email or ends with _email or starts with email_; type is string or unspecified; format is not email; no enum","fix":"add format: \"email\""},{"id":"closed_set_without_guidance","rule":"name ends with one of _type, _kind, _status, _mode; type is string or unspecified; no enum; no format; no examples and no example","fix":"the name implies a closed value set — publish it as an enum"}]},"counted_flags":"1 per worst-offender parameter + 1 per B-4 heuristic hit. (A parameter missing only a description, or only guidance, is counted and fix-listed but does not increment the banding total.)"},"C_set_economics":{"question":"What does this definition set cost a host before any work starts, and where is that cost concentrated? Every number in this section is an estimate under the published method, labelled an estimate wherever it appears.","c1_estimation":{"chars_per_token":4,"evidence_class":"measured","method":"A tool's definition string is the canonical JSON of the tool entry (object keys sorted recursively by code unit, arrays in input order, no whitespace). Estimated definition tokens = ceil(character_length / 4), character length being the JavaScript string length. This is an estimate: real tokenizers differ, roughly ±20%."},"c2_outliers":{"absolute_tokens":1000,"absolute_evidence_class":"measured","relative_multiple":3,"relative_evidence_class":"fleet convention","method":"A tool is a token-cost outlier when EITHER its estimate is > 1,000 tokens OR its estimate is > 3 x the set median (the relative rule never fires when the median is 0). The report names which sub-rule fired. Mean is total / tool count rounded to 1 decimal; the median is the middle value of the sorted per-tool estimates (for an even count, the mean of the two middle values, rounded to 1 decimal)."},"c3_boilerplate_share":{"flag_threshold":0.25,"evidence_class":"fleet convention","method":"Boilerplate token occurrences (D-2, with multiplicity, in every occurrence of a boilerplate sentence across all descriptions) divided by total description token occurrences; 0 when the denominator is 0. A share >= 0.25 is a set-level flag."},"c4_reference_points":[{"ref":"community_overload_report","value":"~15 tools","evidence_class":"community-reported","status":"reference only — never a flag"},{"ref":"typical_measured_session","value":"20–30 tools / 15–30 KB","evidence_class":"measured","status":"reference only"},{"ref":"community_large_set_figure","value":"~95 definitions ≈ 50k tokens","evidence_class":"community-reported","status":"ALSO a flag rule: definition-token total > 50,000 is 1 set-level counted flag"},{"ref":"heavy_single_tool","value":"~1,000 tokens","evidence_class":"measured","status":"ALSO the C-2 absolute outlier rule"}],"community_figure_note":"A community figure is a report from practitioners, not a standard. This audit never presents a community-reported reference point as a pass/fail standard; the two flag rules in this section that reuse a measured or community figure say so on the row.","counted_flags":"1 per outlier tool + 1 if boilerplate share >= 0.25 + 1 if definition-token total > 50,000."},"D_naming_hygiene":{"question":"Can names collide or be confused inside this set? Exactness only — naming taste is not graded. A tool with a missing, empty, or non-string name is a D-flag (1 per such tool, shown by index) and is excluded from every pair rule.","d1_exact_duplicates":"HARD FAIL. Two or more tool entries with names exactly equal as strings (byte-for-byte; no case folding, no trimming) form a duplicate group, shown with the name, the count, and the array indices. Any duplicate group forces the worst verdict band regardless of every other result.","d2_near_names":{"edit_distance_max":2,"exempt_suffixes":["preflight"],"method":"An unordered pair of distinct names is a near-name pair when ANY relation holds (computed on the names as written, except where a rule says lowercased; a pair is reported once with every relation that applies, and counts once for banding): edit_distance — Levenshtein distance between the lowercased names is 1 or 2; prefix — the lowercased shorter name is a strict prefix of the lowercased longer name, subject to the self-discriminating suffix rule; singular_plural — one lowercased name is exactly the other lowercased name plus a trailing s.","self_discriminating_suffix_rule":"A prefix pair is NOT flagged when the longer name's remainder — the part after the shorter name, with one leading separator (_ or -) removed if present — is exactly preflight or starts with preflight_ or preflight-. Exempt pairs are still listed in a separate self-discriminating prefix pairs (not flagged) list. The exemption applies ONLY to the prefix relation: a pair exempt under prefix but also matching edit_distance or singular_plural is still flagged with those relations shown. The published exempt-suffix list for v1 is exactly [\"preflight\"]."},"d3_namespace_consistency":{"min_named_tools":5,"conformity_floor":0.8,"evidence_class":"fleet convention","method":"Each named tool's pattern is classified in this precedence order: mixed (contains both _ and -); else underscore (contains _); else hyphen (contains -); else dot (contains .); else single_token. The dominant pattern is the most frequent; ties are broken by the order underscore, dot, hyphen, mixed, single_token. The conforming share is (tools with the dominant pattern) / (named tools). Not applicable when the set has fewer than 5 named tools (share null, no flag). When applicable, a conforming share < 0.80 is 1 set-level counted flag and the report lists the non-conforming names. Informational only (never flagged): the most common first segment (substring before the first _, -, or .) and the share of named tools carrying it."},"counted_flags":"1 per near-name pair + 1 per missing-name tool + 1 if the D-3 share flag fires. Duplicate groups are hard fails, not counted flags."}},"banding":{"lines":{"band_1":"TOOL ERGONOMICS: CLEAR ON THE MEASURED PROPERTIES — no hard fails and no counted flags under tool-ergonomics/v1. This grades the submitted document's measured properties only; it is not a judgement that the tools are good, well chosen, or fit for any task.","band_2":"TOOL ERGONOMICS: FLAGS ON THE MEASURED PROPERTIES — the fix list names the measured document properties that need work under tool-ergonomics/v1. This grades the submitted document only; it does not run the tools or judge their quality.","band_3":"TOOL ERGONOMICS: SUBSTANTIAL FLAGS ON THE MEASURED PROPERTIES — hard fails or a substantial count of measured document-property flags under tool-ergonomics/v1. This grades the submitted document only; it does not run the tools or judge their quality."},"rule":"Band 3 if hard fails H >= 1 or counted flags F > S; else Band 2 if F >= 1; else Band 1 — where S = max(20, tool_count).","counted_flags_definition":"F = A (flagged pairs + low-discrimination tools + lead-flagged tools) + B (worst offenders + B-4 hits) + C (outlier tools + boilerplate flag + total-token flag) + D (near-name pairs + missing-name tools + namespace flag). H = the number of D-1 duplicate groups. S = max(20, tool_count). There is no composite score and no percentage in this service.","could_not_grade":"COULD NOT GRADE — the submitted manifest contains no tools, or contains more than the published 500-tool cap, or (in URL mode) the manifest document could not be fetched and parsed as a tools/list document, so no check could be graded. This is not a pass and not a fail."},"master_threshold_table":[{"id":"T1","threshold":"A-1 pairwise similarity flag","value":"Jaccard >= 0.55 (effective tokens)","evidence_class":"fleet convention"},{"id":"T2","threshold":"A-2 discriminating-content floor","value":"share < 0.20 flags","evidence_class":"fleet convention"},{"id":"T3","threshold":"A-3 lead rule","value":"first sentence non-boilerplate + >=1 discriminating token in it","evidence_class":"fleet convention"},{"id":"T4","threshold":"D-4/C-3 boilerplate sentence definition","value":"in >=2 tools AND >25% of descriptions","evidence_class":"fleet convention"},{"id":"T5","threshold":"C-3 boilerplate share flag","value":"share >= 0.25","evidence_class":"fleet convention"},{"id":"T6","threshold":"C-1 token estimation","value":"ceil(chars / 4) on canonical JSON; estimate, ±~20% by tokenizer","evidence_class":"measured"},{"id":"T7","threshold":"C-2 heavy-tool outlier (absolute)","value":"estimate > 1,000 tokens","evidence_class":"measured"},{"id":"T8","threshold":"C-2 outlier (relative)","value":"estimate > 3 x set median","evidence_class":"fleet convention"},{"id":"T9","threshold":"C-4 total-token flag","value":"total > 50,000 estimated tokens","evidence_class":"community-reported"},{"id":"T10","threshold":"C-4 overload reference point","value":"~15 tools","evidence_class":"community-reported"},{"id":"T11","threshold":"C-4 typical-session reference","value":"20–30 tools / 15–30 KB","evidence_class":"measured"},{"id":"T12","threshold":"D-2 edit-distance bound","value":"Levenshtein <= 2 (lowercased)","evidence_class":"fleet convention"},{"id":"T13","threshold":"D-2 exempt suffix list","value":"exactly [\"preflight\"]","evidence_class":"fleet convention"},{"id":"T14","threshold":"D-3 namespace conformity floor","value":"share < 0.80 flags; only when >=5 named tools","evidence_class":"fleet convention"},{"id":"T15","threshold":"banding substantial cutoff","value":"S = max(20, tool_count); F > S => Band 3","evidence_class":"fleet convention"},{"id":"T16","threshold":"B-4 heuristic list","value":"the 8 rules in section B","evidence_class":"fleet convention"},{"id":"T17","threshold":"tool cap","value":"500 tools, stated exactly","evidence_class":"fleet convention"},{"id":"T18","threshold":"fetch caps","value":"512 KB / 10 s, one URL, GET only","evidence_class":"fleet convention"}],"not_graded_in_v1":["annotation presence or truthfulness (the manifest-claim audit's territory)","spec-default traps (the manifest-claim audit's territory)","claim-vs-schema consistency (the manifest-claim audit's territory)","runtime behaviour of any tool","error-message quality","prose elegance","semantic correctness of a description (whether what it says is true of the tool)","task-fit","naming taste","whether the set should contain these tools at all"],"prices":{"precheck":"free — headline counts ONLY, 100/day per caller wallet, rate-limited (minimum interval between free pre-checks)","audit":{"usd":0.05,"includes":"$0.05 buys one full tool-ergonomics audit report for one submitted MCP manifest. — the verdict and banding, the full sections A–D results with per-tool evidence, and the ordered fix list, graded under the published criteria (GET /criteria) and returned as a signed report. This audit grades measurable document properties of the submitted manifest only — description discrimination, parameter specification, definition-token economics, and naming hygiene — against the published tool-ergonomics/v1 rules. It never runs the tools, never queries a model, and never measures or promises any model's selection accuracy or any task's success. Properties that require judgement — prose elegance, task-fit, and naming taste — are NOT graded in v1, and no result in this report is a judgement of them."}}}