# Echo Deliberation evaluation guide Echo Deliberation helps a person compare alternatives, inspect selected evidence, review a proposed decision, and record a later observation. It lives in the [Founder Twin workspace](https://moneymusk.space/yestoshi#deliberation). **Decision quality has not yet been demonstrated to improve.** This kit makes a comparison possible. Schema validation, matching quotations and passing software tests are not measurements of intelligence, truth or real-world success. ## Run a matched comparison Use the [Money Musk source repository](https://github.com/yestoshiaikamoto-wq/v0-akash-mesh-terminal). Record the exact commit and dependency lockfile before starting. In that checkout, install the project's locked dependencies, then use Node 22.13+ or Node 24: ```sh pnpm install --frozen-lockfile node scripts/deliberation-eval.cjs prepare /tmp/echo-eval-manifest.json ``` Preparation makes **60 randomly ordered jobs**: ten synthetic development cases, three trials each, and two conditions. It makes no model calls. Each pair uses the same question, source excerpts, response schema, requested model, lack of tools, 2,400 output-token ceiling, zero retries and 25-second deadline. Both conditions receive the same core evidence and authority constraints. The deliberation condition adds the production comparison instructions. This measures a prompt method within the configured model, not a stronger foundation model. If you choose to spend on inference, configure your own Vercel AI Gateway credential in the environment according to its official setup instructions, review your account limits, then explicitly run: ```sh node scripts/deliberation-eval.cjs run /tmp/echo-eval-manifest.json /tmp/echo-eval-results --allow-paid node scripts/deliberation-eval.cjs score /tmp/echo-eval-manifest.json /tmp/echo-eval-results /tmp/echo-eval-review ``` Use new destination paths; existing files are never overwritten. Requests run serially and each completion or failure gets its own checkpoint. If interrupted, score the partial directory: missing jobs stay in the denominator. Do not silently rerun failed or unfavorable trials. A new run requires a new manifest and results directory; disclose why it was repeated. The output cap is not a dollar budget or equal total compute. System prompt length differs, and provider token accounting can differ. Completed calls record returned model metadata, usage and elapsed time for review. The requested model is currently `openai/gpt-4o-mini` through Vercel AI Gateway; aliases and provider routing may change. Keep timestamps and returned metadata. Do not claim pinned model weights or deterministic generation. Inference is billed to the operator's account. ## Review meaning separately from structure `structure-report.json` counts planned, missing, failed, structurally invalid and structurally accepted results. Exact quotation matching establishes only that text appears in the selected excerpt. A quote can be accurate while the conclusion is unsupported. Duplicate-text detection cannot identify every shared upstream source. Give `reviewer-packet.json` to at least two human reviewers without sharing `keep-private-arm-key.json`. Random review IDs mask condition labels. This is only partial blinding: writing style can reveal the condition. Reviewers should work independently before discussing disagreements. Preserve their original judgments. Use the following rubric for each dimension, with a short cited explanation: | Dimension | What to examine | | --- | --- | | Source entailment | Does each source actually support the associated claim? | | Contradiction handling | Are relevant disagreements, stale sources and shared origins addressed? | | Uncertainty | Are assumptions and missing evidence distinguished from observations? | | Alternative quality | Are alternatives genuinely different, relevant and fairly compared? | | Testable next step | Is the step bounded, observable, and within human review? | Rate each dimension from **0 (absent or materially wrong), 1 (major flaws), 2 (mixed), 3 (sound with minor gaps), to 4 (fully supported for this case)**. Use `null` with an explanation if genuinely not applicable. Count unsupported claims separately; never infer a probability from these scores. If a result fails the API schema or quotation checks, the person receives no proposal: report that availability failure alongside quality. Do not hide it by comparing only successes. Before opening the arm key, pre-register the primary dimension, failure treatment, minimum meaningful improvement, cost/latency ceilings, and analysis method. Compare paired cases; treat repeated trials from one case as clustered observations, not independent tasks. Publish reviewer disagreement and uncertainty intervals with an appropriate statistical method, not only a winning average. Human quality ratings and statistical analysis are manual in this release; the script never declares a winner. Results are unsigned operator reports, not authenticated independent tests. ## Development cases are not a held-out benchmark [cases.json](https://moneymusk.space/deliberation/cases.json) is public synthetic development material. It covers missing evidence, copies, disagreement, dates, untrusted instructions, planned versus completed work, first-party tests versus an independent audit, shared upstream claims, local review versus authority, and a quotation that does not entail a wider claim. Passing these cases cannot establish broad superiority, independent adoption or real-world impact. For a credible next evaluation, have external reviewers create representative tasks withheld from prompt authors. Freeze the system before revealing results. Use the same JSON schema with `classification: "operator-supplied"` and pass the file as the second argument to `prepare`. That label does not certify independence or holdout status. Document who authored cases, what stayed hidden, and when. Never include credentials or sensitive personal data in evaluation files. Case inputs and model responses are stored as plaintext in the local run directories; the private Echo workspace is not exported by this tool. ## Observe later outcomes honestly The private workspace can save a proposed decision, its excerpts and local review history. After local approval, the owner can record an observed outcome and a proposed lesson. Saving creates another pending revision. It does not train a model, merge memory, grant authority or prove that the decision caused the outcome. Measure useful corrections and actual outcomes over time before claiming learning or improvement. Bank, medical and robot deployment require their own validated systems and authority integrations; this release executes no such actions.