{"criteria_id":"memory-probe-grade/v1","version":"1.0.0","corpus":{"id":"probe-grade-corpus/v1","sha256":"33974c67885ec97a9a8991b55ddfb80753e95f6706efaeba7c7a22f2a1f097a5"},"corpus_pin_definition":"The corpus pin is the SHA-256 of the corpus's canonical serialization (the corpus object excluding its sha256 field), computed and verifiable by the published harness (harness.mjs).","price":{"usdc_units_per_call":50000,"label":"$0.05","basis":"per grade","status":"RULED"},"free_tier":{"per_day":100,"min_interval_ms":5000,"on_exhaustion":"402"},"body_cap_bytes":262144,"ops":{"grade":"POST /grade — { family, script_id, issued_nonce, canary_echo, declared_scope?, variant?, claimed_expectations?, probes: { persistence_truth, scope_coherence, replay_duplication, repeatable_read, feature_liveness } }"},"families":{"mem0":{"script_id":"probe-script/mem0/v1","binding_strength":"LOW-TO-MODERATE","binding_wording":"Mem0 — LOW-TO-MODERATE (consistency-bound, not proof-bound): IDs are bare UUIDs (generatable); the OSS hash is MD5 of the text (zero fabrication resistance); Platform created_at is caller-backdatable. What binds is pre-fabrication resistance (the per-run nonce) plus the async event-poll and history read-back chain when the transcript carries them.","expectations":{"persistence_truth":{"grade":"RUNNABLE","entry":"DEEPENING-PASS §5.0 matrix, Mem0, probe (i): RUNNABLE — mandatory condition infer:false (with infer:true the probe grades the extractor, not persistence)"},"scope_coherence":{"grade":"RUNNABLE","entry":"DEEPENING-PASS §5.0 matrix, Mem0, probe (ii): RUNNABLE — the only family with a scoped forget (delete_all) on the coherence surface; single-dimension forget calls; verify via list, never search"},"replay_duplication":{"grade":"RUNNABLE","entry":"DEEPENING-PASS §5.0 matrix, Mem0, probe (iii): RUNNABLE — per-variant expectations (OSS verbatim ⇒ 1 live record; Platform numeric expectation UNVERIFIED, reported not pre-printed)"},"repeatable_read":{"grade":"PARTIAL","entry":"DEEPENING-PASS §5.0 matrix, Mem0, probe (iv): PARTIAL — no version identifier, committed-state token, or as-of primitive exists in any documented Mem0 schema; field-equality in a controlled window is the substitute"},"feature_liveness":{"grade":"PARTIAL","entry":"DEEPENING-PASS §5.0 matrix, Mem0, probe (v): PARTIAL — a configured feature's canary storage and return is probeable; a no-op processing claim is not detectable from the documented surface"}},"replay_expectations":{"oss-verbatim":1,"platform-verbatim":null}},"zep-cloud":{"script_id":"probe-script/zep-cloud/v1","binding_strength":"MODERATE","binding_wording":"Zep Cloud — MODERATE (the strongest in this pass): server-minted UUIDs + task UUIDs + server-clock created_at/expired_at, densely cross-referenced, make a fresh-nonce transcript hard to fabricate cheaply in advance. Caller-supplied event-time fields (valid_at / invalid_at / reference_time) are worthless for binding and are never credited.","expectations":{"persistence_truth":{"grade":"RUNNABLE","entry":"DEEPENING-PASS §5.0 matrix, Zep Cloud, probe (i): RUNNABLE (async-adapted — poll processed; direct graph.edge.add leg carries the strict content check)"},"scope_coherence":{"grade":"PARTIAL","entry":"DEEPENING-PASS §5.0 matrix, Zep Cloud, probe (ii): PARTIAL — no scoped forget exists at all; the probe is re-specified to delete-by-id and the forget leg is recorded NOT RUNNABLE (no such operation), never passed by substitution"},"replay_duplication":{"grade":"RUNNABLE","entry":"DEEPENING-PASS §5.0 matrix, Zep Cloud, probe (iii): RUNNABLE (two-legged; per-mode expectations mandatory — episode replay ⇒ 1 live edge; direct write with deduplicate=false ⇒ 2 edges BY DESIGN)"},"repeatable_read":{"grade":"PARTIAL","entry":"DEEPENING-PASS §5.0 matrix, Zep Cloud, probe (iv): PARTIAL — no version / snapshot / as-of token is documented; committed = after the write's task/processed completes"},"feature_liveness":{"grade":"RUNNABLE","entry":"DEEPENING-PASS §5.0 matrix, Zep Cloud, probe (v): RUNNABLE for the extraction leg (episode ingestion → fact extraction); observations / thread summaries are PARTIAL and excluded from the v1 script"}},"replay_expectations":{"episode":1,"direct-deduplicate-false":2,"direct-deduplicate-true":1}},"graphiti":{"script_id":"probe-script/graphiti/v1","binding_strength":"WEAK","binding_wording":"Graphiti (self-hosted) — WEAK (the weakest shape in this pass): uuid defaults are generated in the client library and are a plain constructor parameter; all timestamps are caller-clock or caller-supplied; there is no server, no task system, no third party in the loop. Within pure Variant B a Graphiti transcript is self-report with a nonce freshness stamp, and this grade says exactly that.","expectations":{"persistence_truth":{"grade":"RUNNABLE","entry":"DEEPENING-PASS §5.0 matrix, Graphiti, probe (i): RUNNABLE (episode leg + direct add_triplet leg)"},"scope_coherence":{"grade":"PARTIAL","entry":"DEEPENING-PASS §5.0 matrix, Graphiti, probe (ii): PARTIAL — no scoped forget on the orchestration surface (the low-level Node.delete_by_group_id is a partial scope-delete primitive, not a forget surface); re-specified to delete-by-id, forget leg NOT RUNNABLE"},"replay_duplication":{"grade":"RUNNABLE","entry":"DEEPENING-PASS §5.0 matrix, Graphiti, probe (iii): RUNNABLE (two-legged; episode replay expects 2 episodes but 1 live edge — live = invalid_at IS NULL AND expired_at IS NULL)"},"repeatable_read":{"grade":"PARTIAL","entry":"DEEPENING-PASS §5.0 matrix, Graphiti, probe (iv): PARTIAL — no version / snapshot / as-of token is documented in either variant"},"feature_liveness":{"grade":"RUNNABLE","entry":"DEEPENING-PASS §5.0 matrix, Graphiti, probe (v): RUNNABLE for the extraction leg; community building / saga summarisation are PARTIAL and excluded from the v1 script"}},"replay_expectations":{"episode":1,"direct":1}},"letta":{"script_id":"probe-script/letta/v1","binding_strength":"LOW-TO-MODERATE","binding_wording":"Letta — LOW-TO-MODERATE: IDs are prefixed UUIDs minted by the ORM — format-checkable but forgeable; blocks carry no timestamps at all in the API shape; PassageCreate.created_at is client-suppliable. The nonce buys replay and lazy-fabrication resistance and consistency cost, not proof; the per-class weakness (blocks: no timestamps, no visible version counter) is named, not averaged away.","expectations":{"persistence_truth":{"grade":"PARTIAL","entry":"DEEPENING-PASS §5.0 matrix, Letta, probe (i): PARTIAL — passages RUNNABLE; blocks RUNNABLE-WITH-CAVEAT (API-write → API-read consistency); the induced-failure half is PARTIAL (expected error shape UNVERIFIED firsthand)"},"scope_coherence":{"grade":"PARTIAL","entry":"DEEPENING-PASS §5.0 matrix, Letta, probe (ii): PARTIAL — runnable only in the Letta-native redefinition (two synthetic agents + archive; absence valid only after the sharing topology is verified disjoint)"},"replay_duplication":{"grade":"PARTIAL","entry":"DEEPENING-PASS §5.0 matrix, Letta, probe (iii): PARTIAL — passages RUNNABLE (numeric expectation UNVERIFIED in the record); blocks NOT-APPLICABLE-BY-DESIGN (API block write is whole-value replace, idempotent by construction)"},"repeatable_read":{"grade":"PARTIAL","entry":"DEEPENING-PASS §5.0 matrix, Letta, probe (iv): PARTIAL — content equality across two reads is runnable; version identity is not (the ORM version counter is absent from the API/export shape)"},"feature_liveness":{"grade":"RUNNABLE","entry":"DEEPENING-PASS §5.0 matrix, Letta, probe (v): RUNNABLE — archival insert → list round-trip and shared-block propagation are directly observable"}},"replay_expectations":{"passages":null,"blocks":null}}},"probes":[{"id":"persistence_truth","label":"(i) persistence truth"},{"id":"scope_coherence","label":"(ii) scope coherence"},{"id":"replay_duplication","label":"(iii) replay duplication"},{"id":"repeatable_read","label":"(iv) repeatable read"},{"id":"feature_liveness","label":"(v) feature liveness"}],"grades":["RUNNABLE — the probe is runnable for this family per the published expectation table AND the transcript carries the full published evidence","PARTIAL — the table caps the probe at PARTIAL for this family, or the transcript's evidence is partial (every cap names its reason)","NOT RUNNABLE — the probe was not run, carries no usable evidence, its evidence is not bound to the issued nonce (canary mismatch), or the table records the leg as having no such operation"],"outcomes":["PASS — the transcript's evidence shows the behavioural property held","FAIL — the transcript's evidence shows the behavioural property did not hold","NOT-OBSERVED — no outcome can be read (probe not run, canary mismatch, or no numeric expectation is established for the family/mode — the value is reported, never graded against an invented number)","NOT-APPLICABLE-BY-DESIGN — the probe leg is idempotent / inapplicable by the family's documented design (Letta blocks replay); recorded as a design fact, never a pass or a failure"],"grading_rule":"Per probe: grade = min(published expectation-table grade, evidence grade) under RUNNABLE > PARTIAL > NOT RUNNABLE, with the table entry cited per probe. The outcome is graded separately from the evidence's behavioural fields. Grading is of behavioural properties — never exact edge-set equality (extraction Caveat B). A transcript's claimed_expectations never move a grade: any claim differing from the published table is named in expectation_mismatches and the published table governs.","canary_rule":"The transcript carries issued_nonce (as issued by the published script for the run) and canary_echo (the nonce as it came back through the store). If canary_echo !== issued_nonce the grade states canary_status MISMATCH — named, never silently accepted — and every probe grades NOT RUNNABLE with the reason printed.","scope_claim_rule":"declared_scope defaults to synthetic. Any other value is graded but flagged on the grade's face (PRODUCTION-SCOPE-CLAIMED / NON-SYNTHETIC-SCOPE-CLAIMED) with the fence asserted — the fence is tripwire-grade: no wording on any surface turns a transcript grade into a production-store certification.","extraction_caveats":["Caveat A — extraction is LLM-mediated and asynchronous (Zep/Graphiti; Mem0 infer path): an episode does not contain its facts, and a nonce in an episode body is not guaranteed to survive into any edge's fact text verbatim. Grading distinguishes extraction variance (fact present, nonce entities present, wording differs — outcome PASS with the variance flagged) from persistence failure (nothing present after processing completed — outcome FAIL). Direct-write legs carry the strict content-equality checks.","Caveat B — no deterministic replay: two runs of the same probe script need not produce identical edge sets. Grading is of behavioural properties (persistence, absence, counts of live records, agreement of operations) — never exact edge-set equality.","Search is never a verification read (Mem0 ranked/thresholded; Letta semantic): absence from search results does not establish absence from the store. Persistence evidence resting on a search read caps that probe's evidence grade at PARTIAL."],"fence":"FENCE: This grade grades the submitted transcript of a challenge run in a synthetic namespace. It is NOT a certification of the operator's production store, and it must never be read or presented as one.","free_refusals":["invalid_request_body","family_missing","family_unknown","script_id_missing","script_id_mismatch","issued_nonce_missing","issued_nonce_invalid","canary_echo_missing","canary_echo_invalid","variant_invalid","declared_scope_invalid","claimed_expectations_invalid","probes_missing","probe_entry_invalid","CAP-EXCEEDED:body_bytes (413)"],"malformed":"A transcript that cannot be assessed at all (missing fields, unknown family, wrong script id, malformed probe entry) is a named FREE refusal, before any payment, at the pre-check and at the paid op alike — distinct from a grade on a well-formed transcript.","price_note":"The $0.05 price was proposed in the source (Eval #2) and is labelled RULED (price ruled 2026-11-03) on every surface.","honesty_boundary":"A grade grades the submitted transcript of a challenge run in a synthetic namespace against the declared family's published expectation table (documentation-verdict matrices, s1-s2-deepening-pass-2026-11-02 §§4–6). It is not a certification of the operator's production store, not a live test of any store (this service executes nothing against any store), and not a claim about any store the transcript did not come from. Binding strength is per family and printed on every grade: Zep Cloud MODERATE (strongest); Mem0 / Letta LOW-TO-MODERATE; Graphiti self-hosted WEAK — a Graphiti transcript is self-report with a nonce freshness stamp.","boundary":"Boundary: this service grades one submitted probe transcript against one published per-family expectation table. It executes nothing against any store, fetches nothing, and never upgrades a transcript grade into a production-store certification.","custody":{"ledger_closed_fields":["payer wallet","timestamp","day","price (USDC units) with the CAD fair-market-value block (rate, rate date, basis, cad_pending, refund — null unless refunded)","payment tx id","Solana fee (lamports)","criteria version","op","verdict line","per-grade status summary (family + binding strength + canary status + grade/outcome counts only — never transcript content, never the nonce)","house flag"],"retention":"Submitted transcripts are processed in memory only, never stored — not in KV, not in logs, not in the payments ledger, not in metrics. The payments ledger records only the closed field list published in the criteria (family, binding strength, canary status, and grade/outcome counts — never transcript content and never the nonce). Because nothing is kept, a paid retry re-sends the transcript."},"payment_includes":"$0.05 per grade — labelled RULED (price ruled 2026-11-03). The payment includes: one deterministic grade of the one probe transcript you submit — five probes (persistence truth; scope coherence; replay duplication; repeatable read; feature liveness) each graded RUNNABLE / PARTIAL / NOT RUNNABLE against the declared family's published expectation table with the table entry cited per probe, a behavioural outcome per probe read from the transcript's evidence, the family's binding-strength wording, the canary status, and the factors. Factors printed; no composite score. The grade grades the submitted transcript of a challenge run only — it is not a certification of your production store. Nothing submitted is stored beyond the payments ledger's closed field list.","kill_line":"Kill line (M-46, under the operator's 2026-11-03 zero-demand doctrine): build, bank, and hold; judged at the fleet's periodic reviews. Tripwires, all stop-and-review: refunds >5% of paid calls in any 30-day window; free use >10x paid use; a wrong grade from undisclosed stale/partial data = stop-serving; a grade circulating as a production-store certification = stop-and-review.","service":"probe-grade","endpoints":{"criteria":"GET /criteria (free)","precheck":"POST /precheck (free, validation + the family's expectation table + binding-strength wording disclosed before payment; no grades)","grade":"POST /grade (paid, $0.05 per grade — labelled RULED (price ruled 2026-11-03): signed probe-transcript grade)","metrics":"GET /metrics"}}