Structured output

By default a completed Format run hands you media and a paragraph of text. That is fine for a human and awkward for a database. Bind a schema and you get a typed object instead — the same run, shaped so you can write it straight into your own records.

Exact request and response schemas come from live OpenAPI (https://api.sume.com/reference/json). The tables here are a readable summary, not a second schema.

`input`

A run request carries two JSON-shaped fields and they behave nothing alike. Getting them confused is the most common way a first integration goes wrong.

	`input`	`output_schema`
What it is	Caller data	A contract for the receipt
Direction	You → the run	The run → you
Shape	Any JSON object that suits your backend	JSON Schema, inside the supported subset
Checked for	Object type, key count, byte size	Every rule in the subset
A shape Sume does not expect	Runs anyway; unknown keys are just more data	`400 output_schema_invalid` — nothing runs, nothing is charged
Where it lands	The agent's prompt, as a fenced data block	The post-run projection

So input is a flexible concatenation and output_schema is a strict typed receipt. Loosen your input freely; you cannot loosen your output_schema at all — strict: false does not do it, and there is no other escape hatch.

The two do not meet. The projection never sees your input (see below), so a value you sent cannot come back in output unless the run itself repeats it in the text it finishes with.

Where your object comes from

This is the part that differs from a chat completion, and it is worth understanding before you design a schema. There are two paths, and the receipt tells you which one you got.

filled_by: "agent" — the run answered. Your schema is handed to the run as a tool it must call before finishing, with your schema as the tool's own argument shape. The model doing the work is the model filling your object, while it still remembers what it made and why. It can see your input and your instruction, because they are simply part of the run.

filled_by: "projection" — the fallback. If the run finishes without submitting a valid object, a separate constrained pass builds one afterwards from what the run left behind. That pass is an OpenAI-strict json_schema completion at temperature 0, and it is given exactly two facts:

Fact	Detail
The run's generated media	Every artifact the run produced, with its durable URL and metadata.
The run's closing text	The final assistant message, truncated to the first 8000 characters.

Not your input, not your instruction, not the Format body, not the run's intermediate steps. On this path, anything you want in output has to be present in one of those two facts.

So filled_by is worth reading. It is the difference between an object written by the run and an object reconstructed from its leftovers, and it explains most surprises:

On the projection path, your own identifiers do not round-trip. An order_id you sent in input is invisible there. Keep your identifiers on your side, keyed by run.id or by the Idempotency-Key you sent, and let output carry only what the run made.
Nothing in output is invented, on either path. Both are gated (below) before they reach you, by the same checks. An object the run wrote earns no extra trust.
On the projection path, an empty text field is a signal. The projection never sees your input, so a schema whose titles, descriptions or ids come from the brief comes back null in exactly those places while the media fields are full. filled_by: "projection" with null prose is the shape of a run that stopped early, not of a Format that forgot to write copy.

If you score runs automatically — a smoke matrix, a partner integration, a dashboard — read filled_by before you count a run as delivered, and for video read the file rather than the number beside it. ffprobe on primary_output_url is two seconds and it is the only thing that distinguishes an assembled cut from a clip that is shaped like one.

The practical rule that falls out is unchanged: require only what the Format actually makes. A schema demanding a field the recipe never produces will fail on the projection path every time, and the failure is silent until you read output_error.

Bind a schema

Two spellings, one behaviour. Use whichever fits your client.

Native — output_schema:

OpenAI-shaped alias — response_format:

response_format is normalized into output_schema on the receipt, so what you read back is always the native spelling. Sending both is 400 invalid_request.

The alias mirrors the Chat Completions spelling — type at the top, the binding nested under json_schema. The OpenAI Responses API flattens the same fields into text.format: { type, name, strict, schema }; that flattened shape is not accepted here. If your client builds Responses-style bodies, either lift the fields into output_schema or re-nest them under json_schema.

Field	Rules
`name`	Required. 1–64 characters, `^[A-Za-z0-9._/-]+$`. Namespace it — it shows up on every receipt.
`strict`	Defaults `true`. See the note under Supported schemas.
`schema`	Required. A JSON Schema object inside the supported subset.

A Format can also carry its own default schema, bound in the dashboard. A per-request output_schema overrides it for that run. Which one applied is on the receipt as output_schema.source:

`source`	Meaning
`default`	Nothing was bound. `output` is the built-in schema.
`action_default`	The Format's own bound schema. (`action_` is the wire spelling; it is shared with Scheduled.)
`request_override`	The `output_schema` sent on this run request.

Supported schemas

Schemas must satisfy the OpenAI strict-mode subset. This is a real subset, not a recommendation — a schema outside it is rejected at submit with 400 output_schema_invalid and a details.violations[] array naming each problem. Nothing runs, so nothing is charged.

The supported keywords, exactly

The subset works from an allowlist. A keyword that is not on this list is a violation, not a keyword that gets quietly ignored — an ignored constraint would be a schema Sume could not promise to meet.

Group	Accepted
Structure	`type`, `properties`, `required`, `additionalProperties`, `items`, `$defs`, `$ref`, `anyOf`
Values	`enum`, `const`
Strings	`format`, `pattern`, `minLength`, `maxLength`
Numbers	`minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`, `multipleOf`
Arrays	`minItems`, `maxItems`
Annotation	`title`, `description`, `default`, `examples`, `$schema`, `$id`

Types are string, number, integer, boolean, object, array, null.

Everything else is rejected. The ones that catch real integrations:

Rejected	Instead
`oneOf`	`anyOf`. Only `anyOf` is on the list, and schemas ported from OpenAPI reach for `oneOf` by reflex.
`allOf`	Flatten the branches into one object.
`not`, `if` / `then` / `else`, `dependentRequired`, `dependentSchemas`	Not expressible. Model the alternatives as `anyOf`, or validate on your side after reading `output`.
`nullable: true`	A nullable union: `"type": ["string", "null"]`.
`patternProperties`, `propertyNames`, `unevaluatedProperties`, `additionalItems`	Declare the properties you want; `additionalProperties: false` covers the rest.

`type`

Or a $ref, or an anyOf. A bare { "description": "…" } — legal JSON Schema, meaning "anything" — is a missing_type violation. An array must also declare items.

$ref and anyOf each short-circuit the node they sit on: sibling keywords next to them are checked against the allowlist but otherwise carry no meaning. Put constraints inside the anyOf branches or inside the $defs entry, not beside the $ref.

The root must be an object

The root's type must be exactly "object" — a single type, so even { "type": ["object", "null"] } is rejected. A top-level array, string, or union is rejected too. Wrap it:

`additionalProperties: false`

Every object in the schema, not only the root — including objects nested inside array items and inside $defs.

`required`

There is no optional property. A property you declare is a property that must be present.

Express optionality as a nullable union instead:

This is the rule that trips up the most schemas ported from elsewhere. Read null as "the run had nothing to put here", which is exactly the case you wanted optional for.

Size and nesting limits

Limit	Value	Violation
Nesting depth	10 levels	`max_depth`
Total properties	5000, counted across the whole document	`max_properties`
Enum values	1000 per enum	`max_enum_values`
Total string length	120,000 characters, summed over every property name, key, and string value in the document	`max_string_length`

The last one is a document-wide budget rather than a per-field cap, so long description annotations on a large schema can exhaust it even when no single string is remarkable.

is limited

Only two targets resolve:

Target	Use
`#/$defs/*`	Your own definitions, declared at the root of the schema document.
`SumeMediaFile#`	Sume's media shape. See below.

An external $ref — a URL, a sibling document, #/components/... — is rejected. So is $ref: "#": OpenAI's strict mode permits root recursion that way, and Sume does not. A #/$defs/* target that has no matching root $defs entry is rejected as well, which is what catches a $defs block nested inside a sub-schema rather than declared at the root.

Recursion through a named definition is fine: a $defs entry may $ref itself. The depth limit counts literal nesting in the document, so a self-referential definition does not consume it.

does not relax any of this

It is accepted and stored, and it changes nothing about the subset above. A schema outside the subset is rejected whether strict is true or false. Do not reach for it as an escape hatch — there isn't one.

There is also no equivalent of OpenAI's JSON mode ({"type": "json_object"}), the loose "valid JSON, any shape" setting. Bind a schema or take the built-in one; those are the two options.

`details.violations[]`

Each entry is { path, rule, message }. path is a JSON-Pointer-style location such as #/properties/scenes/items/properties/clip, message is written for a human and may change, and rule is a stable lowercase token that is safe to switch on:

`rule`	Meaning
`root_must_be_object`	The root is missing, is not an object, or its `type` is not exactly `"object"`.
`not_an_object`	A schema node is not a JSON object.
`missing_type`	A node has no `type`, `$ref`, or `anyOf`.
`unsupported_type`	A `type` outside the seven listed above.
`unsupported_keyword`	A keyword off the allowlist.
`additional_properties_false`	An object node without `additionalProperties: false`.
`required_completeness`	A declared property missing from `required`, or a `required` entry with no matching property.
`missing_items`	An `array` node with no `items`.
`unsupported_ref`	A `$ref` that is neither `#/$defs/<name>` nor `SumeMediaFile#`, or one naming a definition that does not exist.
`invalid_defs`	`$defs` is present but is not an object of named schemas.
`max_depth`, `max_properties`, `max_enum_values`, `max_string_length`	The limits above.

Every problem is reported, not just the first, so one 400 is enough to fix the schema.

Coming from OpenAI structured outputs

If you have used response_format: { type: "json_schema", … } or the Responses API's text.format, most of what you know transfers. The subset rules are the same ones, quoted from the same guide. What is different is where the schema is applied.

OpenAI	Sume	Note
`response_format.json_schema`	`output_schema`, or `response_format` verbatim	Chat Completions spelling only; `text.format` is not accepted.
`json_schema.name`	`output_schema.name`	Required both places. Namespace it; it lands on every receipt.
`json_schema.strict`	`output_schema.strict`	Accepted, defaults `true`, and changes nothing — the subset is always enforced.
`{"type": "json_object"}` (JSON mode)	(no equivalent)	Bind a schema, or take the built-in one.
The model emits the JSON	A post-run projection emits it	The schema constrains the projection, never the run.
`refusal` on the message	`output_error` on the receipt	Different mechanism: not a safety refusal but a failed projection.
`incomplete_details.reason: "max_output_tokens"`	(not applicable)	The projection is small and bounded; there is no truncated-JSON case to handle.
Streamed partial JSON	(not applicable)	`output` appears once, on the terminal receipt.
`$ref: "#"` root recursion	Rejected	Recurse through a named `#/$defs/*` entry instead.
Nothing comparable	The URL gate	Every URL in `output` is checked against the media the run really produced.
Nothing comparable	`SumeMediaFile#`	A built-in `$ref` target for the run's media.

The mental shift is one sentence: with OpenAI you constrain what the model says; here you constrain how a finished run is read back. Everything else follows from that — why the schema cannot make the Format produce a video, why a URL cannot be hallucinated into output, and why a run can hand you output: null.

`SumeMediaFile`

Reference it with { "$ref": "SumeMediaFile#" } anywhere you want a piece of the run's media in your output. Every field is required and every field but type and url is nullable.

Field	Type	Notes
`type`	`"image" \| "video" \| "audio" \| "file"`
`url`	string (uri)	Must be a URL this run actually produced — see the URL gate.
`content_type`	string \| null	e.g. `video/mp4`.
`file_name`	string \| null
`size_bytes`	integer \| null
`width`, `height`	integer \| null	Images and video.
`duration_ms`	integer \| null	Video and audio.
`expires_at`	string (date-time) \| null	`null` for durable `media.sume.com` URLs, which is the normal case. Populated only when a signed URL is returned.

Sume-hosted media is served from media.sume.com as public, max-age=31536000, immutable and does not expire. Store the URL against your own record and render it later — no refresh dance. Note that a durable URL is also a public URL; see Map artifacts into your UI if that matters to your product.

The built-in schema

Bind nothing and output is projected onto sume/action-run-output/v1:

text is nullable; the four arrays are always present and may be empty.

The built-in schema is filled deterministically from the run's generated media and its final text. No model is involved, so it cannot fail the way a custom schema can. If you want the media without designing a schema, this is already enough to ship on.

The URL gate

Before a custom output reaches you, every URL in it is checked against the set of media this run actually produced. The comparison is exact string equality.

Two passes catch it. The first collects every http(s):// string anywhere in the object, at any depth, whether or not you declared the field as media. The second collects the url of every SumeMediaFile-shaped value whatever it contains — so a placeholder like "none" or "" sitting in a url cannot slip past by not looking like a URL.

A URL that was not produced by this run — even a well-formed, plausible-looking media.sume.com one — fails the projection rather than being returned. Then the object is validated against your schema.

So a completed run either returns output matching your schema with real media URLs, or it returns output: null and tells you why. It never returns a schema-shaped guess.

The deliverable is not one of its parts

A schema that describes a whole made of parts — scenes, shots, segments — usually also has a field for the finished, assembled file. Those are different files, and a receipt that fills the whole with one of its own parts is rejected.

The check is narrow on purpose. It needs two or more parts that report "status": "succeeded" with their own video, and a video outside every part reusing one of those files. A Format whose single clip really is the deliverable, and a poster or thumbnail deliberately reused from a part, are both untouched.

A run that could not assemble has an honest answer available to it, and it is not "here is a scene". Report the parts you made and leave the assembled field null, or report the assembly's real status.

Durations are checked against the file

A duration_ms inside a SumeMediaFile comes off the same ledger that fills artifacts[]. Where that ledger recorded a length, the value in output has to agree with it within 10% — a claim that does not is describing a different file, and fails the projection rather than reaching you.

Where the ledger did not record one, nothing is checked: null there means "not measured", not "zero", and a run is not failed over a fact nobody measured. So a duration_ms you read back is either the artifact's own or unverified — never a number computed from something else.

What the gate does not check

Beyond the above: URLs, durations, and the shape. Every other value — ids, labels, captions, counts — is read out of the run's media metadata and closing text by the projection pass. It is grounded in what the run reported, not verified against it, so treat those fields as the run's own account of its work rather than as measurements. A duration_seconds number you declared yourself is written by the projection; the duration_ms on the media file is the checked one.

The gate also only admits media the run generated. A file the run merely uploaded is not in that set, so a schema field holding an upload's URL fails the projection and takes the whole output with it. Keep uploads out of your schema.

and

Most integrations have one thing to show. Name it and the receipt resolves it for you:

primary_output_url on the receipt is then the URL at that key. Resolution order:

The primary_output_key on the run request.
The Format's own primary_output_key.
The first top-level key holding a media object, or an array whose first element is one.

For the built-in schema it falls back through videos → images → audio → files.

A key you named in step 1 or 2 is echoed back whenever output carries a value under it — including a value that is not media. Point at a headline string and you get primary_output_key: "headline" with primary_output_url: null, because there is no URL to resolve. Only a key that is missing from output, or empty there, falls through to step 3.

primary_output_key is capped at 64 characters. Both fields are null on any non-terminal status, and null when output_error is set.

When output cannot be produced

Check output_error before reading output.

On a run over the API, a projection failure is a run failure. Those runs are unattended — the same qualification Call a Format states — so a completed receipt whose output is null reads as a success and is not one. status is failed and error carries the same reason as output_error. In the Agents UI, where a person is reading the thread, the same shape stays completed: there it is a draft, not a receipt.

A failed run still publishes what it produced. output is null when nothing satisfied your schema — not merely because the run failed. A live-commerce show that rendered 20 of 40 scenes and then stopped reports those 20 on output, with the rest carrying whatever "failed" value your own schema defines, next to the output_error explaining why it stopped. What a failure never gets is a pointer: primary_output_key and primary_output_url are null on every non-completed run, so if (run.primary_output_url) remains a safe test for "the deliverable exists" and cannot be fooled by a partial.

For that to work your schema has to permit the partial in the first place — see Make a partial result legal.

artifacts[] is populated either way, with everything the run made. You always have the media, even when the shape did not work out.

The exception is a projection that never got to look at the run. output_extraction_failed records a transport failure rather than a verdict, so it stays completed and re-projects by itself on your next read.

`output_error.code`	Meaning	`details`
`output_schema_unsatisfied`	The projection did not match your schema, or it referenced media this run did not produce.	`rejected_urls[]` (first 10 only) or `violations[]`, plus a `harvested` count by media type.
`output_extraction_failed`	The projection could not run. `status` stays `completed`; a `reason` of `harvest_unavailable` means the run's media could not be read while it finalized, and the receipt fills in on the next read.	`reason`
`unattended_blocked`	The run stopped rather than claim a deliverable it did not produce. `message` is the reason in the run's own words.	A `harvested` count by media type, plus `assembled_deliverable: false` — the harvested media are intermediates, not the finished cut.
`deliverable_missing`	The Format produces media (`io.output_kind`) the run never made, so no structured output was allowed to claim it.	`declared_output_kind`, plus a `harvested` count by media type.
`primary_output_missing`	The run satisfied your schema but left the `primary_output_key` you declared without a value. `output` still carries the partial result; the run is `failed` because the thing you named is not in it.	`primary_output_key`

Treat this set as open: new codes may appear, so branch on the ones you handle and fall through on the rest rather than switching exhaustively.

Handle it like this:

output_schema_unsatisfied, repeatedly, on the same Format. Almost always a schema requiring media the recipe does not generate. Compare details.harvested against your required fields — the example above requires an image and got one, but wanted a second. Loosen the field to a nullable union, or change the instruction so the run makes it.
output_schema_unsatisfied with violations[]. The shape, not the media. The violations name the offending paths.
output_extraction_failed. Transient. Read the run once more before doing anything else — that alone clears a harvest_unavailable. If it persists, retry the run with a new idempotency key; the old key is bound to the receipt you already have.
Any of them, in your UI. You still have artifacts[]. Showing the media and logging the shape failure beats showing an error to a customer whose video exists.

Make a partial result legal

A run that produces some of its output and then fails can only report that if your schema says a partial is a legal shape. The platform adds no floor of its own — Ajv enforces exactly the keywords you wrote — so a schema that demands every scene turns a 20-of-40 show back into output: null.

Two rules do all the work, and both are counter-intuitive under the strict subset:

Optional means nullable, not absent. Every property must still be listed in required (see Every property must be listed in required). You express "may not exist" as "type": ["string", "null"].
Leave minItems off the arrays you want to receive partially. It is genuinely enforced, so minItems: 1 on a scene list will reject the very ledger you are trying to read.

Pair it with "primary_output_key": "full_video". That is what keeps the loosened schema honest: a run that fills scenes but leaves full_video null has satisfied the schema and still not produced the deliverable, so it terminalizes failed with primary_output_missing rather than reporting a success. The looseness buys you visibility into the partial; it does not buy the run a pass.

Reading one of these, branch on full_video for "did I get a show", and on each scenes[].status for "what do I need to retry". To retry, send the failed run's id as previous_run_id on a new run — see Continue a run — and the next turn picks up the same conversation with those clips already on it.

Failures at submit

Schema problems caught before anything runs. Nothing is charged.

Code	Status	What to do
`output_schema_invalid`	400	Your schema is outside the supported subset. `details.violations[]` names each problem.
`invalid_request`	400	Includes sending both `output_schema` and `response_format`.

Checklist

input — caller data — the other half of the run body, and the one with no schema
Runs and results — the receipt that carries all of this
Calling a Format — the invoke contract
Embed a Format in your product — mapping output into your own records