> ## Documentation Index > Fetch the complete documentation index at: https://docs.moonbounce.ai/llms.txt > Use this file to discover all available pages before exploring further. # Preview a structure change against past feedback > Replays a structure change over past cases with feedback: it spends model calls and stores a run. Not a dry run of evaluate. Replays a local structure and its base over the decision's reviewed and imported cases in the key's environment (a test key previews against test cases only), and returns fixes, breaks, the action mix, the `loss_report` (base and candidate judged by the decision's objective, or balanced losses when none is set, with loss per case, macro-F1, where the cost sits, the costliest mistakes, and whether the candidate is `accepted`) and an evidence label: `training_cases` when a scored case is a training case (its deciding feedback is not an import), else `imported_examples` (every scored case is an imported example); either way the cases are ones a suggestion may have learned from. The result must decide outcomes (bands, rules or match): a value result is 400 OUTCOMES_INVALID. Each case reuses the answers its evaluation stored for questions whose hash is unchanged; changed and new questions are asked the way evaluate asks them (each question's `reads`, grouped into System One calls), so a preview costs what evaluating the changed questions would. Writes an eval run only: never an evaluation, never traffic, never acceptance. A preview does not register the candidate: before evaluating by its structure hash, register it with POST /v1/decisions (or send the structure inline on evaluate once). The base, in order: the base of `suggestion_id` when it is sent (then `base` is ignored), else `base`, else the version serving primary traffic in the key's environment, else the newest one examples were imported under there, else none. With `suggestion_id`, `hashes_match` says whether the structure is that suggestion's candidate; another environment's suggestion is 404. The evidence replays only cases with an outcome from the owner, reviewers or imported examples; the action mix replays a sample of the environment's recent primary traffic. For questions the candidate changes, `before` is a fresh re-run of the base question (the fair baseline), so model nondeterminism can move it between previews. `noise_rate` measures outcome flips from that re-run against the stored answers, not value drift. A test key's preview neither reads nor fills the shared cache of model answers. A decision nothing was registered, evaluated or imported under is 404. ## OpenAPI ````yaml /openapi.json post /v1/decisions/{decision_id}/previews openapi: 3.1.0 info: title: Moonbounce Decisions API version: 0.0.0 description: >- Moonbounce decisions: evaluate an item's state against a decision structure (System One questions, attributes you supply and aggregations over both), record feedback and import labeled examples, review a ranked queue, and improve the structure through suggestions that ship as code diffs, each with a preview (backtest) and an evidence label. **Authentication.** Send an API key as `Authorization: Bearer ` on every operation except `GET /health` and `POST /v1/validate`. Call the API from a server, never a browser: a key must not reach one, and the API sends no CORS headers. A key belongs to one organization and one environment: production, or test for a key with the `env:test` permission. Test traffic never counts as production traffic. **Requests.** JSON bodies (`Content-Type: application/json`) of at most 1 MB, parsed strictly: duplicate keys, lone surrogates, integers outside ±(2^53 - 1), `__proto__` keys, U+0000 and nesting deeper than 64 are a 400. **Retries.** Send an `Idempotency-Key` on every POST under /v1 that takes one (all but `POST /v1/validate` and `POST /v1/review/items/fill`, whose retry is another fill within the day's budget). Keep it for the request's retries. The same key with the same request returns the first stored result, marked `Idempotent-Replayed: true`, for 24 hours (410 if that result was since deleted); with another request it is 422 IDEMPOTENCY_KEY_REUSED. Keys are scoped to the organization and environment across every operation, so use a fresh key per request. A request that failed before its result was stored runs again when retried. 429 and 503 under /v1 may carry `Retry-After` (seconds); a daily QUOTA_EXCEEDED cap instead names its reset time. **Errors.** Every error under /v1 is `{"error": {"type", "message", "reason"?, "param"?, "details"?}}` (ErrorBody); `GET /health` answers its own shape. Every response carries a `Request-Id` header. Rate limits apply per key, and per client IP for requests without a valid key; a 429 RATE_LIMITED carries `Retry-After`. servers: - url: https://api.moonbounce.ai description: Production security: [] tags: - name: Evaluate description: Run one item through a decision and read evaluations back. Start here. - name: Decisions description: >- Validate a decision file, register its versions, and read decisions and versions. - name: Feedback description: >- Say what the right answer was, import labeled examples, and read feedback and cases. - name: Suggestions description: Ask for a better version of a decision, and follow the improvement loop. - name: Previews & tests description: >- Replay a structure change over past cases, and score a version over many items. - name: Models description: List the models a decision can run on and choose one. - name: Objective description: >- Set what a decision optimizes for, then judge runs and tune cuts against it. - name: Usage description: What a decision's model calls cost. - name: Account description: Identify your key, and check that the API is up. paths: /v1/decisions/{decision_id}/previews: post: tags: - Previews & tests summary: Preview a structure change against past feedback description: >- Replays a structure change over past cases with feedback: it spends model calls and stores a run. Not a dry run of evaluate. Replays a local structure and its base over the decision's reviewed and imported cases in the key's environment (a test key previews against test cases only), and returns fixes, breaks, the action mix, the `loss_report` (base and candidate judged by the decision's objective, or balanced losses when none is set, with loss per case, macro-F1, where the cost sits, the costliest mistakes, and whether the candidate is `accepted`) and an evidence label: `training_cases` when a scored case is a training case (its deciding feedback is not an import), else `imported_examples` (every scored case is an imported example); either way the cases are ones a suggestion may have learned from. The result must decide outcomes (bands, rules or match): a value result is 400 OUTCOMES_INVALID. Each case reuses the answers its evaluation stored for questions whose hash is unchanged; changed and new questions are asked the way evaluate asks them (each question's `reads`, grouped into System One calls), so a preview costs what evaluating the changed questions would. Writes an eval run only: never an evaluation, never traffic, never acceptance. A preview does not register the candidate: before evaluating by its structure hash, register it with POST /v1/decisions (or send the structure inline on evaluate once). The base, in order: the base of `suggestion_id` when it is sent (then `base` is ignored), else `base`, else the version serving primary traffic in the key's environment, else the newest one examples were imported under there, else none. With `suggestion_id`, `hashes_match` says whether the structure is that suggestion's candidate; another environment's suggestion is 404. The evidence replays only cases with an outcome from the owner, reviewers or imported examples; the action mix replays a sample of the environment's recent primary traffic. For questions the candidate changes, `before` is a fresh re-run of the base question (the fair baseline), so model nondeterminism can move it between previews. `noise_rate` measures outcome flips from that re-run against the stored answers, not value drift. A test key's preview neither reads nor fills the shared cache of model answers. A decision nothing was registered, evaluated or imported under is 404. operationId: preview parameters: - schema: $ref: '#/components/schemas/DecisionId' required: true description: >- A name you choose for this decision, such as `refund-exception`: a lowercase letter, then up to 127 of `a-z 0-9 _ . -`. Unique in your organization and shared by its test and production keys. The first registration or evaluation under a new one creates the decision. name: decision_id in: path - schema: type: string minLength: 1 maxLength: 255 pattern: ^[\x20-\x7e]+$ description: >- Makes a retry safe. For 24 hours, the same key with the same request returns the first stored result (with Idempotent-Replayed: true); the same key with a different request is a 422 IDEMPOTENCY_KEY_REUSED. Keys are scoped to the organization and environment across every operation, so use a fresh key per request. A request that failed before its result was stored runs again when retried with its key. required: false description: >- Makes a retry safe. For 24 hours, the same key with the same request returns the first stored result (with Idempotent-Replayed: true); the same key with a different request is a 422 IDEMPOTENCY_KEY_REUSED. Keys are scoped to the organization and environment across every operation, so use a fresh key per request. A request that failed before its result was stored runs again when retried with its key. name: idempotency-key in: header requestBody: required: true content: application/json: examples: new_question: summary: >- Add a question about late or lost shipping, which the decision counts as a reason to grant value: structure: questions: item_used: type: noul instructions: >- Has the customer used the item beyond trying it on or setting it up once? reads: - state.message defective: type: noul instructions: >- Does the customer describe a defect, damage on arrival, or the item failing under normal use? reads: - state.message shipping_failure: type: noul instructions: >- Did the order arrive later than promised, or get lost in transit? reads: - state.message attributes: days_since_delivery: type: number minimum: 0 aggregations: grant_score: type: weighted_mean inputs: - name: questions.item_used invert: true - name: questions.defective - name: questions.shipping_failure handling: type: rules rules: - when: name: attributes.days_since_delivery gt: 30 outcome: escalate - when: name: aggregations.grant_score gte: 0.72 outcome: grant - when: name: aggregations.grant_score lt: 0.38 outcome: deny else: escalate result: aggregations.handling schema: $ref: '#/components/schemas/PreviewRequest' responses: '200': description: The preview headers: Idempotent-Replayed: schema: type: string enum: - 'true' description: >- Present and `true` when this is the stored result of an earlier request with the same Idempotency-Key. required: false description: >- Present and `true` when this is the stored result of an earlier request with the same Idempotency-Key. content: application/json: schema: $ref: '#/components/schemas/Preview' '400': description: 'Invalid request: `param` names the field at fault when one applies' content: application/json: schema: $ref: '#/components/schemas/ErrorBody' '401': description: Missing or invalid API key (reason KEY_INVALID) content: application/json: schema: $ref: '#/components/schemas/ErrorBody' '403': description: The key may not do this content: application/json: schema: $ref: '#/components/schemas/ErrorBody' '404': description: Not found, or not in this key's organization and environment content: application/json: schema: $ref: '#/components/schemas/ErrorBody' '409': description: Conflict with the current state content: application/json: schema: $ref: '#/components/schemas/ErrorBody' '410': description: >- The Idempotency-Key's first result was deleted, so it cannot be replayed content: application/json: schema: $ref: '#/components/schemas/ErrorBody' '413': description: The HTTP body exceeds 1 MB (reason BODY_TOO_LARGE) content: application/json: schema: $ref: '#/components/schemas/ErrorBody' '422': description: >- The request is well formed but cannot be processed (IDEMPOTENCY_KEY_REUSED requires a fresh key; REQUEST_TOO_LARGE requires a smaller request) content: application/json: schema: $ref: '#/components/schemas/ErrorBody' '429': description: >- RATE_LIMITED: retry after Retry-After seconds; QUOTA_EXCEEDED: details.reset_at gives the daily reset time (no Retry-After) headers: Retry-After: schema: type: string description: Seconds to wait before retrying, when waiting helps. required: false description: Seconds to wait before retrying, when waiting helps. content: application/json: schema: $ref: '#/components/schemas/ErrorBody' '502': description: >- A model failed to answer a question the result needs (type engine_failed, reason ENGINE_FAILED) content: application/json: schema: $ref: '#/components/schemas/ErrorBody' '503': description: >- Temporarily unavailable, with Retry-After (for example reason AUTH_UNAVAILABLE, ENGINE_UNAVAILABLE or STORE_UNAVAILABLE): retry with the same Idempotency-Key headers: Retry-After: schema: type: string description: Seconds to wait before retrying, when waiting helps. required: true description: Seconds to wait before retrying, when waiting helps. content: application/json: schema: $ref: '#/components/schemas/ErrorBody' security: - bearerAuth: - decisions:develop components: schemas: DecisionId: type: string pattern: ^[a-z][a-z0-9_.-]{0,127}$ description: >- A name you choose for this decision, such as `refund-exception`: a lowercase letter, then up to 127 of `a-z 0-9 _ . -`. Unique in your organization and shared by its test and production keys. The first registration or evaluation under a new one creates the decision. PreviewRequest: type: object properties: structure: $ref: '#/components/schemas/DecisionStructure' suggestion_id: $ref: '#/components/schemas/SugId' base: type: object properties: structure_hash: $ref: '#/components/schemas/StructureHash' required: - structure_hash additionalProperties: false required: - structure additionalProperties: false Preview: type: object properties: preview_id: $ref: '#/components/schemas/RunId' decision_id: $ref: '#/components/schemas/DecisionIdOutput' environment: $ref: '#/components/schemas/Environment' base: anyOf: - $ref: '#/components/schemas/VersionRef' - type: 'null' candidate: $ref: '#/components/schemas/VersionRef' hashes_match: anyOf: - type: object properties: structure: type: boolean required: - structure additionalProperties: true x-moonbounce-sdk-known-fields: true - type: 'null' evidence: $ref: '#/components/schemas/Evidence' cost: $ref: '#/components/schemas/Cost' objective: anyOf: - $ref: '#/components/schemas/ObjectiveRef' - type: 'null' description: >- The decision's stored objective revision this preview was scored with, or null when none is set (balanced losses). Absent only on a replay of a response stored before it was recorded. loss_report: type: object properties: objective: type: object properties: source: type: string enum: - stored - balanced description: >- `stored`: the decision's stored objective. `balanced`: none is set, so each outcome's mistakes weigh the same in total. revision: type: - integer - 'null' minimum: 1 objective_hash: type: - string - 'null' description: The stored objective as written (null when balanced). effective_hash: type: string description: >- The hash of the severities scored by (each run weighs outcomes by its own cases; the target is left out, since it changes only `target_met`). Never compare reports under different ones. unknown_outcomes: type: array items: type: string required: - source - revision - objective_hash - effective_hash - unknown_outcomes additionalProperties: true description: What this run scored with. x-moonbounce-sdk-known-fields: true base: anyOf: - $ref: '#/components/schemas/LossReport' - type: 'null' description: The base's report; null when there is no base. candidate: $ref: '#/components/schemas/LossReport' accepted: type: boolean description: >- The candidate lowers loss per case and adds no new severe mistake on a human-labeled case. False without a base. reasons: type: array items: type: string description: Why it is not accepted; empty when it is. new_severe_human: type: array items: type: string description: >- Human-labeled cases the candidate gets severely wrong that the base did not. required: - objective - base - candidate - accepted - reasons - new_severe_human additionalProperties: true description: >- Base and candidate judged by the decision's objective, on the same scored cases (each mistake's loss is its severity, every mistake `normal` without an objective, weighted so each expected outcome counts equally), and whether the candidate is accepted: lower loss per case and no new severe mistake on a human-labeled case. Absent only on a replay of a response stored before previews were judged. x-moonbounce-sdk-known-fields: true required: - preview_id - decision_id - environment - base - candidate - hashes_match - evidence - cost additionalProperties: true x-moonbounce-sdk-known-fields: true ErrorBody: type: object properties: error: type: object properties: type: $ref: '#/components/schemas/ErrorType' reason: type: string pattern: ^[A-Z][A-Z0-9_]+$ description: >- Set where a caller can act on it, one of the reasons in the ErrorReason schema. New reasons may be added: treat an unknown one by the status and `type`. message: type: string description: For people; don't parse it. param: type: string description: >- The request field at fault, as a dotted path (`attribute_values.region`, `claims.1.value`). details: type: object properties: issues: type: array items: $ref: '#/components/schemas/ErrorIssue' additionalProperties: {} required: - type - message additionalProperties: true x-moonbounce-sdk-known-fields: true required: - error additionalProperties: true x-moonbounce-sdk-known-fields: true DecisionStructure: type: object properties: questions: type: object additionalProperties: $ref: '#/components/schemas/Question' description: System One questions, keyed by name. attributes: type: object additionalProperties: $ref: '#/components/schemas/Attribute' description: >- Values your app supplies with each call in `attribute_values`. The model never sees them. aggregations: type: object additionalProperties: $ref: '#/components/schemas/Aggregation' result: type: string minLength: 1 maxLength: 80 description: The final feature's path, such as `aggregations.band`. examples: - questions.asks_for_refund - aggregations.band required: - questions - result additionalProperties: false description: >- Everything that defines a decision: questions, attribute declarations, aggregations and the result. Names are unique across all three maps. SugId: type: string pattern: >- ^sug_[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$ StructureHash: type: string pattern: ^sha256:[0-9a-f]{64}$ RunId: type: string pattern: >- ^run_[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$ DecisionIdOutput: type: string pattern: ^[a-z][a-z0-9_.-]{0,127}$ description: >- A name you choose for this decision, such as `refund-exception`: a lowercase letter, then up to 127 of `a-z 0-9 _ . -`. Unique in your organization and shared by its test and production keys. The first registration or evaluation under a new one creates the decision. Environment: type: string enum: - production - test description: >- The environment of the key that made the call: `production` or `test`. Evaluations, feedback, reviews and suggestions are separate per environment; decisions and their versions belong to the whole organization. VersionRef: type: object properties: version: $ref: '#/components/schemas/EvaluatedVersion' model: $ref: '#/components/schemas/ModelProfile' required: - version - model additionalProperties: true x-moonbounce-sdk-known-fields: true Evidence: type: object properties: label: $ref: '#/components/schemas/EvidenceLabel' frozen_at: $ref: '#/components/schemas/TimestampOutput' by_split: type: object properties: example: $ref: '#/components/schemas/SplitStats' train: $ref: '#/components/schemas/SplitStats' additionalProperties: true x-moonbounce-sdk-known-fields: true noise_rate: type: - number - 'null' minimum: 0 maximum: 1 by_source: type: object properties: app_native: $ref: '#/components/schemas/SplitStats' app_resolution: $ref: '#/components/schemas/SplitStats' reviewer: $ref: '#/components/schemas/SplitStats' owner: $ref: '#/components/schemas/SplitStats' import: $ref: '#/components/schemas/SplitStats' synthetic_review: $ref: '#/components/schemas/SplitStats' action_mix: anyOf: - $ref: '#/components/schemas/ActionMix' - type: 'null' changed_items: type: array items: $ref: '#/components/schemas/ChangedItem' credited_feedback: type: array items: $ref: '#/components/schemas/FbkIdOutput' required: - label - frozen_at - by_split - noise_rate - action_mix - changed_items - credited_feedback additionalProperties: true x-moonbounce-sdk-known-fields: true Cost: type: object properties: model_calls: type: integer minimum: 0 description: >- Upstream model calls, retries included. One call can answer several questions (Jev compiles them into one). questions: type: integer minimum: 0 description: >- Questions sent to a model. An answer from the cache, or one a stored case kept, is not counted. cache_hits: type: integer minimum: 0 description: >- Questions answered from the answer cache, when any were: previews and case runs always report it, evaluate and imports only when the cache served an answer. input_tokens: type: integer minimum: 0 output_tokens: type: integer minimum: 0 usd_micros: type: integer minimum: 0 description: What the model calls cost, in millionths of a US dollar. required: - model_calls - questions - input_tokens - output_tokens - usd_micros additionalProperties: true x-moonbounce-sdk-known-fields: true ObjectiveRef: type: object properties: revision: type: integer minimum: 1 objective_hash: type: string required: - revision - objective_hash additionalProperties: true description: The stored objective revision in effect, and its hash. x-moonbounce-sdk-known-fields: true LossReport: type: object properties: 'n': type: integer minimum: 0 description: Cases scored. loss: type: number minimum: 0 description: Total loss of the mistakes. loss_per_case: type: number minimum: 0 target_met: type: - boolean - 'null' description: >- Whether loss per case is at or below the objective's target; null without one. macro_f1: type: - number - 'null' minimum: 0 maximum: 1 description: >- Mean F1 over the outcomes expected or given: the headline. Not the loss. accuracy: type: - number - 'null' minimum: 0 maximum: 1 description: >- Share of cases given their expected outcome; null with none. Unweighted, so the commonest outcome dominates it: the loss and `macro_f1` weigh each outcome equally. Absent only on a replay of a response stored before it was added. outcomes: type: array items: type: object properties: outcome: $ref: '#/components/schemas/OutcomeOutput' expected: type: integer minimum: 0 given: type: integer minimum: 0 precision: type: - number - 'null' minimum: 0 maximum: 1 recall: type: - number - 'null' minimum: 0 maximum: 1 f1: type: - number - 'null' minimum: 0 maximum: 1 required: - outcome - expected - given - precision - recall - f1 additionalProperties: true x-moonbounce-sdk-known-fields: true by_mistake: type: array items: type: object properties: expected: $ref: '#/components/schemas/OutcomeOutput' got: anyOf: - $ref: '#/components/schemas/OutcomeOutput' - type: 'null' 'n': type: integer minimum: 1 loss: type: number minimum: 0 examples: type: array items: type: string description: A few of its cases, worst first, by the id each case carries. required: - expected - got - 'n' - loss - examples additionalProperties: true x-moonbounce-sdk-known-fields: true description: >- Loss by mistake cell, worst first: the loss-weighted confusion matrix. A response lists the worst 50. by_mistake_total: type: integer minimum: 0 description: How many mistake cells there are. confusion: type: array items: type: object properties: expected: $ref: '#/components/schemas/OutcomeOutput' got: type: - string - 'null' minLength: 1 maxLength: 200 description: >- The outcome given; null when there was none (a failure or an abstention with no abstain outcome). 'n': type: integer minimum: 1 required: - expected - got - 'n' additionalProperties: true x-moonbounce-sdk-known-fields: true description: >- The confusion matrix: every cell with a case, right answers (`expected` equals `got`) included, by expected outcome then `got` (null last). Unweighted counts; `by_mistake` weighs the wrong cells by the objective. A response lists the 400 largest cells. Absent only on a replay of a response stored before it was added. confusion_total: type: integer minimum: 0 description: >- How many confusion-matrix cells there are. Absent only on a replay of a response stored before it was added. by_question: type: array items: type: object properties: question: type: string loss: type: number minimum: 0 share: type: number minimum: 0 maximum: 1 examples: type: array items: type: string description: A few of its cases, worst first, by the id each case carries. required: - question - loss - share - examples additionalProperties: true x-moonbounce-sdk-known-fields: true description: >- Loss by suspect question, worst first: each wrong case's loss spread over the questions a reviewer corrected, else those whose flip alone fixes it. A large share points at rewriting that question. missing_criterion: type: object properties: 'n': type: integer minimum: 0 loss: type: number minimum: 0 examples: type: array items: type: string description: A few of its cases, worst first, by the id each case carries. required: - 'n' - loss - examples additionalProperties: true description: >- Wrong cases with no suspect: no reviewer correction, and no single unconfirmed answer whose flip fixes it. A criterion the structure lacks; clustered on one theme, it points at adding a question. x-moonbounce-sdk-known-fields: true failed: type: object properties: 'n': type: integer minimum: 0 loss: type: number minimum: 0 required: - 'n' - loss additionalProperties: true description: >- Cases the version failed permanently, counted as its dearest mistake: an input or model problem. x-moonbounce-sdk-known-fields: true near_cuts: type: array items: type: object properties: aggregation: type: string at: type: number lower_it: type: object properties: 'n': type: integer minimum: 0 loss: type: number minimum: 0 required: - 'n' - loss additionalProperties: true description: >- Wrong cases just below the cut that belong in the band above it. x-moonbounce-sdk-known-fields: true raise_it: type: object properties: 'n': type: integer minimum: 0 loss: type: number minimum: 0 required: - 'n' - loss additionalProperties: true description: >- Wrong cases just above the cut that belong in the band below it. x-moonbounce-sdk-known-fields: true examples: type: array items: type: string description: A few of its cases, worst first, by the id each case carries. required: - aggregation - at - lower_it - raise_it - examples additionalProperties: true x-moonbounce-sdk-known-fields: true description: >- Wrong cases within 0.1 of a cut of the result's bands whose expected band is the adjacent one, by the direction that would fix them. severe: type: object properties: 'n': type: integer minimum: 0 human: type: integer minimum: 0 required: - 'n' - human additionalProperties: true x-moonbounce-sdk-known-fields: true cases: type: array items: $ref: '#/components/schemas/LossCase' description: >- The worst mistakes, at most 25, with the answers behind them. The roll-ups cover every case. cases_total: type: integer minimum: 0 description: How many mistakes there are; `cases` lists the worst. attributed: type: integer minimum: 0 description: >- How many mistakes, worst first and permanent failures aside, `by_question`, `missing_criterion` and `fixing_flips` cover (at most 100). The other roll-ups cover every case. required: - 'n' - loss - loss_per_case - target_met - macro_f1 - outcomes - by_mistake - by_mistake_total - by_question - missing_criterion - failed - near_cuts - severe - cases - cases_total - attributed additionalProperties: true x-moonbounce-sdk-known-fields: true ErrorType: type: string enum: - invalid_request - authentication - permission_denied - not_found - conflict - rate_limited - engine_failed - unavailable - internal ErrorIssue: type: object properties: param: type: string code: type: string pattern: ^[A-Z][A-Z0-9_]+$ description: >- Stable, extensible issue code; unknown codes still have a parameter and human-readable message. message: type: string required: - param - code - message additionalProperties: true x-moonbounce-sdk-known-fields: true Question: oneOf: - $ref: '#/components/schemas/NoulQuestion' discriminator: propertyName: type mapping: noul: '#/components/schemas/NoulQuestion' Attribute: oneOf: - $ref: '#/components/schemas/StringAttribute' - $ref: '#/components/schemas/NumberAttribute' - $ref: '#/components/schemas/BooleanAttribute' discriminator: propertyName: type mapping: string: '#/components/schemas/StringAttribute' number: '#/components/schemas/NumberAttribute' boolean: '#/components/schemas/BooleanAttribute' Aggregation: oneOf: - $ref: '#/components/schemas/WeightedMean' - $ref: '#/components/schemas/Logistic' - $ref: '#/components/schemas/Min' - $ref: '#/components/schemas/Max' - $ref: '#/components/schemas/Product' - $ref: '#/components/schemas/NoisyOr' - $ref: '#/components/schemas/All' - $ref: '#/components/schemas/Any' - $ref: '#/components/schemas/Bands' - $ref: '#/components/schemas/Rules' - $ref: '#/components/schemas/Match' discriminator: propertyName: type mapping: weighted_mean: '#/components/schemas/WeightedMean' logistic: '#/components/schemas/Logistic' min: '#/components/schemas/Min' max: '#/components/schemas/Max' product: '#/components/schemas/Product' noisy_or: '#/components/schemas/NoisyOr' all: '#/components/schemas/All' any: '#/components/schemas/Any' bands: '#/components/schemas/Bands' rules: '#/components/schemas/Rules' match: '#/components/schemas/Match' EvaluatedVersion: type: object properties: structure_hash: $ref: '#/components/schemas/StructureHashOutput' number: type: - integer - 'null' minimum: 1 description: >- The version's number under the decision (1, 2, 3...). Null when the structure isn't registered: a dry run or a preview of a structure nobody registered. The hash is the identity. required: - structure_hash - number additionalProperties: true description: The version that ran. x-moonbounce-sdk-known-fields: true ModelProfile: type: object properties: profile: type: string description: >- Every question's model and connector, as one opaque string. Compare it for equality (two runs are comparable when their profiles match); don't parse it. required: - profile additionalProperties: true description: The model that answered, as its opaque profile. x-moonbounce-sdk-known-fields: true EvidenceLabel: type: string enum: - imported_examples - training_cases TimestampOutput: type: string format: date-time SplitStats: type: object properties: 'n': type: integer minimum: 0 fixes: type: integer minimum: 0 breaks: type: integer minimum: 0 base_correct: type: integer minimum: 0 candidate_correct: type: integer minimum: 0 required: - 'n' - fixes - breaks - base_correct - candidate_correct additionalProperties: true x-moonbounce-sdk-known-fields: true ActionMix: type: object properties: before: type: object additionalProperties: type: number minimum: 0 maximum: 1 after: type: object additionalProperties: type: number minimum: 0 maximum: 1 'on': type: string 'n': type: integer minimum: 0 required: - before - after - 'on' - 'n' additionalProperties: true x-moonbounce-sdk-known-fields: true ChangedItem: type: object properties: evaluation_id: $ref: '#/components/schemas/EvlIdOutput' feedback_id: $ref: '#/components/schemas/FbkIdOutput' split: $ref: '#/components/schemas/Split' change: type: string enum: - fix - break before: type: object properties: outcome: $ref: '#/components/schemas/OutcomeOutput' value: type: number answers: type: object additionalProperties: type: - number - 'null' minimum: 0 maximum: 1 required: - answers additionalProperties: true x-moonbounce-sdk-known-fields: true after: type: object properties: outcome: $ref: '#/components/schemas/OutcomeOutput' value: type: number answers: type: object additionalProperties: type: - number - 'null' minimum: 0 maximum: 1 required: - answers additionalProperties: true x-moonbounce-sdk-known-fields: true required: - evaluation_id - feedback_id - split - change - before - after additionalProperties: true x-moonbounce-sdk-known-fields: true FbkIdOutput: type: string pattern: >- ^fbk_[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$ OutcomeOutput: type: string minLength: 1 maxLength: 200 description: >- An outcome name, or the exact enum label when a string attribute is the result. LossCase: type: object properties: evaluation_id: type: string pattern: >- ^evl_[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$ description: A stored case's evaluation; absent for a dataset item. item_id: type: string description: >- A dataset item's id, as an evaluate run was given it; absent for a stored case. expected: $ref: '#/components/schemas/OutcomeOutput' got: type: - string - 'null' minLength: 1 maxLength: 200 description: >- The outcome given; null when there is none: a permanent failure (`failed`), or an abstention whose result names no abstain outcome. failed: type: boolean description: >- The version failed this case permanently: an input or model problem, not a missing criterion. loss: type: number minimum: 0 severe: type: boolean description: >- The objective's severity for this mistake, before weighting by outcome, is at least `severe` (10). Always false without a stored objective. human: type: boolean description: >- Labeled by a person, so it counts toward the no-new-severe-mistake rule. margin: anyOf: - type: object properties: value: type: number cut: type: number description: The cut to cross to reach the expected outcome's band. distance: type: number minimum: 0 adjacent: type: boolean description: >- The expected band is next to the value's, so moving this one cut alone fixes it. required: - value - cut - distance - adjacent additionalProperties: true x-moonbounce-sdk-known-fields: true - type: 'null' description: >- For a bands result: the boundary of the nearest band holding the expected outcome, and how far the value is from it. Null otherwise. attributed: type: boolean description: >- Its fixing flips and suspects were worked out: one of the worst mistakes (see `attributed` on the report). fixing_flips: type: array items: type: string description: >- Questions whose opposite answer (0 or 1) alone gives the expected outcome. Empty: a missing criterion. flips_total: type: integer minimum: 0 description: How many fixing flips there are; a response lists at most 10. flip_effects: type: array items: $ref: '#/components/schemas/FlipEffect' description: >- One per listed fixing flip, in the same order, on the worst 25 mistakes. A flip whose effect wasn't worked out (past the report's budget) is left out. answer_conflicts: type: array items: type: string description: >- Questions whose answer disagrees with a reviewer's `expected_answer` for this case. suspects: type: array items: type: string description: >- The questions blamed, as `by_question` counts them: the conflicts, else the fixing flips no reviewer confirmed. Empty on an attributed mistake: a missing criterion. answers: type: object additionalProperties: type: - number - 'null' description: >- The answers of its fixing flips and conflicting questions (a stored run keeps every answer). required: - expected - got - failed - loss - severe - human - margin - attributed - fixing_flips - flips_total - answer_conflicts - suspects - answers additionalProperties: true x-moonbounce-sdk-known-fields: true NoulQuestion: type: object properties: type: type: string enum: - noul instructions: $ref: '#/components/schemas/Instructions' criteria: type: object properties: 'true': type: string minLength: 1 maxLength: 4000 'false': type: string minLength: 1 maxLength: 4000 additionalProperties: false reads: type: array items: $ref: '#/components/schemas/StatePath' minItems: 1 maxItems: 64 description: >- Paths into an object `state` this question reads, such as `state.message` or `state.order.customer.tier`. The question sees just those values, nested as in the state. Without it the question sees the whole state, as in System One. Every path must exist in the call's `state`; no entry may repeat or contain another. required: - type - instructions additionalProperties: false description: P(true) for a statement about the state. StringAttribute: type: object properties: type: type: string enum: - string enum: type: array items: type: string minLength: 1 maxLength: 200 minItems: 1 maxItems: 255 default: type: string source: type: string minLength: 1 maxLength: 200 description: >- The model and version behind this attribute, such as `toxicity-classifier-v3`. Hashed. from: type: string minLength: 1 maxLength: 256 description: >- A state path (as in `reads`) evaluate fills this attribute from, in place of `attribute_values`: the value at that path, validated by this declaration with no coercion. A missing or null path takes `default`, else it is a 400. A value for it in `attribute_values` is a 400. Hashed as written. examples: - state.message - state.order.order_total_usd required: - type - enum additionalProperties: false NumberAttribute: type: object properties: type: type: string enum: - number minimum: type: number maximum: type: number default: type: number source: type: string minLength: 1 maxLength: 200 description: >- The model and version behind this attribute, such as `toxicity-classifier-v3`. Hashed. from: type: string minLength: 1 maxLength: 256 description: >- A state path (as in `reads`) evaluate fills this attribute from, in place of `attribute_values`: the value at that path, validated by this declaration with no coercion. A missing or null path takes `default`, else it is a 400. A value for it in `attribute_values` is a 400. Hashed as written. examples: - state.message - state.order.order_total_usd required: - type additionalProperties: false BooleanAttribute: type: object properties: type: type: string enum: - boolean default: type: boolean source: type: string minLength: 1 maxLength: 200 description: >- The model and version behind this attribute, such as `toxicity-classifier-v3`. Hashed. from: type: string minLength: 1 maxLength: 256 description: >- A state path (as in `reads`) evaluate fills this attribute from, in place of `attribute_values`: the value at that path, validated by this declaration with no coercion. A missing or null path takes `default`, else it is a 400. A value for it in `attribute_values` is a 400. Hashed as written. examples: - state.message - state.order.order_total_usd required: - type additionalProperties: false WeightedMean: type: object properties: type: type: string enum: - weighted_mean inputs: type: array items: type: object properties: name: $ref: '#/components/schemas/FeatureRef' weight: type: number exclusiveMinimum: 0 invert: type: boolean required: - name additionalProperties: false minItems: 1 maxItems: 64 required: - type - inputs additionalProperties: false description: 'Σ wᵢ·xᵢ / Σ wᵢ: a weighted average from 0 to 1, not a probability.' Logistic: type: object properties: type: type: string enum: - logistic bias: type: number inputs: type: array items: type: object properties: name: $ref: '#/components/schemas/FeatureRef' weight: type: number required: - name - weight additionalProperties: false minItems: 1 maxItems: 64 required: - type - bias - inputs additionalProperties: false description: σ(bias + Σ wᵢ·xᵢ), with signed weights on raw input values. Min: type: object properties: type: type: string enum: - min inputs: type: array items: type: object properties: name: $ref: '#/components/schemas/FeatureRef' invert: type: boolean required: - name additionalProperties: false minItems: 1 maxItems: 64 required: - type - inputs additionalProperties: false description: 'min xᵢ: all of these, when they overlap.' Max: type: object properties: type: type: string enum: - max inputs: type: array items: type: object properties: name: $ref: '#/components/schemas/FeatureRef' invert: type: boolean required: - name additionalProperties: false minItems: 1 maxItems: 64 required: - type - inputs additionalProperties: false description: 'max xᵢ: any of these, when they overlap.' Product: type: object properties: type: type: string enum: - product inputs: type: array items: type: object properties: name: $ref: '#/components/schemas/FeatureRef' invert: type: boolean required: - name additionalProperties: false minItems: 1 maxItems: 64 required: - type - inputs additionalProperties: false description: 'Π xᵢ: all of these, when they are independent.' NoisyOr: type: object properties: type: type: string enum: - noisy_or inputs: type: array items: type: object properties: name: $ref: '#/components/schemas/FeatureRef' invert: type: boolean required: - name additionalProperties: false minItems: 1 maxItems: 64 required: - type - inputs additionalProperties: false description: '1 − Π(1 − xᵢ): any of these, when they are independent.' All: type: object properties: type: type: string enum: - all inputs: type: array items: type: object properties: name: $ref: '#/components/schemas/FeatureRef' gte: type: number gt: type: number lte: type: number lt: type: number required: - name additionalProperties: false minItems: 1 maxItems: 64 required: - type - inputs additionalProperties: false description: 1 when every input passes its threshold (default ≥ 0.5), else 0. Any: type: object properties: type: type: string enum: - any inputs: type: array items: type: object properties: name: $ref: '#/components/schemas/FeatureRef' gte: type: number gt: type: number lte: type: number lt: type: number required: - name additionalProperties: false minItems: 1 maxItems: 64 required: - type - inputs additionalProperties: false description: 1 when any input passes its threshold (default ≥ 0.5), else 0. Bands: type: object properties: type: type: string enum: - bands input: $ref: '#/components/schemas/FeatureRef' below: $ref: '#/components/schemas/Name' cuts: type: array items: type: object properties: at: type: number outcome: $ref: '#/components/schemas/Name' required: - at - outcome additionalProperties: false minItems: 1 maxItems: 9 abstain: $ref: '#/components/schemas/Name' required: - type - input - below - cuts additionalProperties: false description: >- Names an outcome for each range of one feature's value. `at` means ≥; `below` is under the first cut. Rules: type: object properties: type: type: string enum: - rules rules: type: array items: type: object properties: when: $ref: '#/components/schemas/Condition' outcome: $ref: '#/components/schemas/Name' required: - when - outcome additionalProperties: false minItems: 1 maxItems: 64 else: $ref: '#/components/schemas/Name' required: - type - rules - else additionalProperties: false description: >- The first rule whose condition is true names its `outcome`; otherwise `else`. Match: type: object properties: type: type: string enum: - match input: $ref: '#/components/schemas/FeatureRef' cases: type: object additionalProperties: $ref: '#/components/schemas/Name' description: >- Maps labels to outcomes. "__proto__" is not supported as a case key; use a rules is-condition for that label. else: $ref: '#/components/schemas/Name' abstain: $ref: '#/components/schemas/Name' required: - type - input - cases additionalProperties: false description: Maps a label straight to an outcome. StructureHashOutput: type: string pattern: ^sha256:[0-9a-f]{64}$ EvlIdOutput: type: string pattern: >- ^evl_[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$ Split: type: string enum: - example - train FlipEffect: type: object properties: question: type: string from: anyOf: - type: number enum: - 0 - type: number enum: - 1 description: >- The answer side flipped: 1 when this case's answer is at least 0.5 (turned to 0), else 0 (turned to 1). fixes: type: integer minimum: 0 description: >- Wrong cases with an answer on that side that the flip alone gets right, this one included. breaks: type: integer minimum: 0 description: Right cases with an answer on that side that the flip gets wrong. loss_change: type: number description: >- Total loss after the flip minus before, over every case with an answer on that side: negative is better. broken: type: array items: type: string description: A few of the cases it breaks, costliest first. required: - question - from - fixes - breaks - loss_change - broken additionalProperties: true description: >- The same flip on every case: what rewording the question so it answers the other way, wherever it now answers as it does here, would fix and break. x-moonbounce-sdk-known-fields: true Instructions: anyOf: - type: string minLength: 1 maxLength: 4000 - type: object additionalProperties: {} - type: array items: {} minItems: 1 StatePath: type: string minLength: 1 maxLength: 256 description: >- `state.` or a nested object path such as `state.order.customer.tier`: keys of an object `state`. A key has no `.`, `[` or `]`. Array indexes are not supported yet. examples: - state.message - state.order.order_total_usd FeatureRef: type: string minLength: 1 maxLength: 80 description: >- A feature's path: `questions.`, `attributes.` or `aggregations.`. examples: - questions.asks_for_refund - aggregations.band Name: type: string pattern: ^[a-z][a-z0-9_]{0,63}$ Condition: anyOf: - type: object properties: name: $ref: '#/components/schemas/FeatureRef' gte: type: number gt: type: number lte: type: number lt: type: number is: type: string minLength: 1 maxLength: 200 in: type: array items: type: string minLength: 1 maxLength: 200 minItems: 1 maxItems: 255 abstained: type: boolean enum: - true required: - name additionalProperties: false - type: object properties: all: type: array items: $ref: '#/components/schemas/Condition' minItems: 1 maxItems: 64 required: - all additionalProperties: false - type: object properties: any: type: array items: $ref: '#/components/schemas/Condition' minItems: 1 maxItems: 64 required: - any additionalProperties: false - type: object properties: not: $ref: '#/components/schemas/Condition' required: - not additionalProperties: false securitySchemes: bearerAuth: type: http scheme: bearer description: >- An API key, sent as `Authorization: Bearer `. Each operation lists the scope its key needs: `decisions:evaluate` (evaluate and read evaluations), `decisions:feedback` (your app's feedback) or `decisions:develop` (everything else, including owner feedback and the review queue). An operation listing no scope takes any scoped key. A key with no scope, or without the operation's, gets 403 INSUFFICIENT_SCOPE. ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.