# Widgets

A widget is a small native interface inside an answer: a chart, a table, a
comparison, a diagram or a small form that helps explain something. Agents write
it as a `widget` block inside their answer text ([reply-parts.md](reply-parts.md),
Widgets), so text and widgets read as one answer: text, a widget, more text,
another widget, a conclusion. Rebyte Agents
generate the content. The API does not call a model or render a page. iOS and
Android render the same A2UI v0.9 document using the pinned Impo patch of AGenUI 1.7.0 (`1.7.0-impo.1`).
Web and Desktop run the Impo AGenUI fork's C++/Yoga core through WebAssembly
(`@impoai/agenui-web` 0.1.1, based on upstream 1.7.0), with React and Astryx
components. New widgets use `impo.native.v4`; all three clients retain
`impo.native.v3`, `impo.native.v2` and `impo.mobile.v1` for saved widgets.

## Agent contract

- **Spec:** `widgetSchema` in `server/src/widgets.ts` (`nativeUIAuthoringParameters`
  from `contracts/generated-ui/catalog-schema.ts`) lists the native components available
  to new widgets.
- **Prompt:** the `widgets` part in `server/src/prompts/app.ts` says when to write a
  widget (prefer widgets when explaining; one idea each; prose stays in the text), how
  to write the block, data binding and action semantics. Main Chat,
  task and scheduled-task agents receive it; gadget conversations are spoken and do not.
  Requested standalone webpages can still be delivered through the ordinary file path.
- **Component lookup:** `impo_get_widget_components` reads documentation for the named
  components per call, plus only their referenced shared definitions. Optional local
  functions are requested by name, including nested functions. The complete schema is
  never a default tool parameter or a fixed prompt. The response identifies the catalog
  version and does not create or publish a widget. Definitions can be reused in the Session.
- **Compiler:** `inlineWidgets` in the same file validates each block and compiles it into
  the A2UI document, inside `normalizeAnswerText`. An invalid block is left out of the
  answer. There is no widget creation tool or widget link: the model cannot name a widget it did not write.
- **Renderer:** native AGenUI surfaces and the compatible Web renderer display
  the document.

The block holds `summary` (one line, at most 200 characters: the accessibility
label, and the text that copy, share, speech and history use),
`components` and an optional `dataModel` (omitted means `{}`). A widget has no title and no declared height: it is
as tall as its laid-out content, up to 640 points/dp, and scrolls inside beyond that.
The native catalog contains all 18 standard components, the three native SDK
extensions and the integrated Chart extension. The schema derives from the pinned, unmodified upstream
[AGenUI 1.7.0 catalog](generated-ui/agenui-1.7.0.json), with its
[Apache-2.0 license](generated-ui/AGenUI-LICENSE).

| Group | Components |
| --- | --- |
| Content | Text, Image, Icon, Divider, Video, AudioPlayer |
| Layout | Row, Column, List, Card, Tabs, Modal |
| Input and actions | Button, TextField, CheckBox, Slider, ChoicePicker, DateTimeInput |
| Native SDK extensions | Table, Carousel, RichText |
| Integrated native extension | Chart (line, grouped bar, pie, donut) |

The SDK's Web component is deliberately absent from native authoring. Markdown
and Lottie remain unregistered playground plugins. Chart uses the native engines
from the upstream playground: DGCharts 5.1.0 on iOS and MPAndroidChart v3.1.0 on
Android. Web/Desktop use an inline SVG adapter and Astryx theme tokens. Existing
ImpoHTML cards retain their sandbox; neither HTML component can be authored. Standalone interactive webpages can still be delivered as files.

Chart uses `chartType: "line" | "bar" | "pie" | "donut"` and `data`, either a
literal object or a JSON Pointer binding to one. The object contains optional
`xAxis` labels and `series: [{name, data: [{value, label}]}]`. There are at most
eight series and 64 points per series. All series and supplied X labels have
matching lengths. Values are finite numbers with absolute magnitude at most
1e12. Pie/donut accept a single nonnegative series. Empty series and all-zero
circular data have explicit empty states; missing or invalid data never becomes
zero. Names and labels are at most 120 UTF-16 code units.

The app owns series colors, axes, legends and light/dark appearance. Charts have
no agent action property: local point inspection sends no messages. A binding to
a chart object updates when its numeric data changes through existing local
controls or `setValue`. Read values through screen-reader descriptions or local
point inspection. Line/bar axes include zero. When circular charts have more
than eight slices, omit the crowded legend and inspect individual slices.
The shared [chart fixture](fixtures/generated-ui-chart-input.json) exercises all
four types, grouped and negative values, a local slider/reset and empty data.

Components form one tree rooted at `root`, with unique IDs, no shared children
or cycles, at most 128 definitions, 512 expanded template instances and depth
12. Nested JSON data and arrays support JSON Pointer bindings and list templates.
Relative paths resolve within each template item. Editable values remain local
until a button submits them. Action context accepts literals, bindings and local
functions, with at most 16 keys and 6 KB after resolution. Input is limited to
96 KB; published documents may include up to 192 KB of app-supplied styles.

All 15 upstream catalog functions remain available. The Impo fork adds `setValue` and `calculate` for local interactions. Formatting, validation, tokens and
logical functions run in the shared core. `openUrl` is allowed only as a button
action with an HTTPS URL; the publisher converts it to a reserved tap event.
The client opens that URL through its existing OS/browser boundary, without
sending an agent message. Media and rich-text links require HTTPS. RichText
accepts a bounded set of formatting tags, with no scripts or event attributes.

Native layout styles, alignment, weights, input variants, checks, tabs and
multi-selection are exposed. The server supplies readable defaults; style
values remain bounded. iOS uses a UIKit calendar adapter to preserve pure dates
in local time. Both mobile clients use the shared native Lucide assets for all
preset icons, filling gaps in the SDK's icon map. `Icon.name.svgPath` is not
advertised because neither released native renderer implements it; preset names
and bindings are supported.

The browser uses a generated, static schema validator, with no runtime code
evaluation. Run `npm run generated-ui:sync` after catalog changes and
`npm run generated-ui:check` to verify the committed validator. The compact
model schema shares definitions instead of repeating styles per component.

A compiled widget, the one line of JSON inside an `impo-widget` block, is:

```json
{
  "kind": "widget",
  "schemaVersion": 4,
  "catalog": "impo.native.v4",
  "surfaceId": "uuid-shaped hash of the block and its position",
  "summary": "Choose a city and ask Impo to plan a trip.",
  "document": "A2UI v0.9 JSONL: createSurface, updateDataModel, updateComponents"
}
```

`document` uses catalog ID `urn:impo:catalog:native:v4`. The complete catalog
fixture is [generated-ui-v4.json](fixtures/generated-ui-v4.json), authored from
[generated-ui-native-input.json](fixtures/generated-ui-native-input.json).
`schemaVersion` and `catalog` name the document format. Each block is one immutable
widget, shown where it is written. A follow-up reply can write a new widget; patching
an old one and persisting unsent form state are outside this immutable contract.
Widgets made by the earlier `impo_widget` tool (2026-10-28) are not shown any more.

## Host appearance and local interaction

The app selects light/dark appearance. `generated-ui/appearance.json` is the shared
palette and component style-role map; sync generates the native theme data. New
authoring exposes layout and typography, not paint colors. Clients strip legacy
paint overrides and RichText color attributes before rendering, apply semantic
tokens, and update existing surfaces on theme changes without clearing form data.
Images and video retain their original colors. Legacy HTML keeps its sandbox.
The Impo SDK patch makes native controls, table cells and rich text repaint when
only their styles change. Scroll indicators sit near the card edge while content
keeps its inset.

An interactive card has initial state, bound controls, local behavior and an
optional submission. Inputs edit their bound state locally. Local buttons use
`action.functionCall`; `setValue` replaces the existing scalar at
`args.targetPath` with `args.value` of the same type. Only user activation can
write state. Bindings and pure catalog functions provide derived values and
validation. Local state lasts while the card is mounted, does not save account
data and does not require a new reply or card.

Only an explicit request for agent work uses `action.event`. The button label
and event name describe that request; its context binds the relevant current
values. Submission is optional. Ordinary edits, selections and calculations
must not send messages. Unsupported local behavior must be explained, never
silently replaced with an agent request. Clients execute declared actions;
they do not infer intent from labels.

The bounded `calculate` function accepts arithmetic text and returns result/error
text. It supports decimal numbers, +, -, *, / and parentheses, with limits of
512 characters, 128 terms, 16 nesting levels and magnitude 1e12. It evaluates no
script and makes no network requests. The
[calculator fixture](fixtures/generated-ui-calculator-input.json) is a regression
case for these generic local actions, not a special card type or authoring rule.
Clients that do not know `impo-widget` blocks show them as code. Saved widgets with unwanted event actions need
to be regenerated; clients do not guess or rewrite their intent.

The patched SDK source is the pinned `vendor/agenui` submodule of the private
workspace, from `impoai/AGenUI`. Run `git submodule update --init vendor/agenui` to
edit it. Client builds download checksum-pinned public SDK artifacts and need no
submodule checkout. The public export therefore needs no private build source.

## Persistence, replay and interaction

Widgets travel inside the answer text: the same text deltas, stored answer and
history read as any other reply, so no new SSE chunk, history part or table is
involved. Replaying or opening history recompiles the same blocks to the same text
and never sends an action.

An explicit agent submission uses `action.event.name` and optional literal/bound `context`.
Its event must identify this surface and fit in 8 KB. A tap formats the action
name and form values as **ordinary user input**, then uses the existing main
Chat/task message path, authentication, idempotency and subscription. It does
not call an arbitrary server endpoint or native tool. UI callbacks are disabled
while the conversation is busy; stale disposed surfaces cannot submit input.
Form values do not grant authority to access another user's resources.

Widgets have no title bar and no Full screen view; a widget is part of the
answer's flow, at the text's width.

Web validates and re-encodes the complete document before passing it to the
shared core. C++ resolves bindings, action context and Yoga layout. A single
runtime serves isolated per-card sessions; Astryx views provide synchronous
intrinsic measurements and render the resolved snapshots. Only bounded numeric
layout coordinates reach the host; Astryx owns control styling and appearance.
Labelled TextFields use measured height instead of the mobile fixed height.
The Web adapter renders all 22 native catalog components and retains legacy
ImpoHTML. Tabs use scrollable panels, modal bodies use Astryx dialogs, and media
uses browser decoding with Astryx transport controls. Template actions retain
the tapped instance scope even when the core reports a definition ID.
Unsupported versions or
malformed documents display the card's summary. Actions use the account-scoped
outbox and preserve the composer's draft and attachments. Reloading, leaving a
conversation, virtualization or changing the HTML theme can reset unsent local
form/game state; submitted messages remain in history. The Wasm binary is fetched
only when a valid card mounts. Loading failures have an explicit retry action.
The pinned public GitHub release URL and npm integrity hash make clean installs
reproducible without Emscripten. The Web distribution includes the Wasm binary
and its licenses; Desktop bundles the same files and serves `.wasm` with
`application/wasm`. A Web deployment does not update an installed Desktop app.

## Legacy HTML games

The retained ImpoHTML implementation is an Impo-owned AGenUI custom component, not the SDK's unrestricted
Web component. Its HTML/CSS/JavaScript runs in an opaque-origin iframe with
`sandbox="allow-scripts"`. CSP denies networking, external assets, forms and
base URLs. Native navigation policies deny external navigation. Android also
blocks network requests, file/content access and downloads. iOS uses an
ephemeral WKWebsiteDataStore. Web nests the game inside a trusted sandboxed frame
whose CSP restricts child navigation to `about:` documents. Both browser frames
have opaque origins and deny device permissions. No client registers a game-to-host
bridge or injects credentials. Local scripts, inline styles, canvas and inline
data images work; external libraries, account access and durable browser
storage do not. Games must include touch controls and responsive layouts.

## Setup and validation

Run root workspace commands:

```sh
npm run setup:agenui
npm run db:push
npm run build:ios
npm run build:android
npm run test:server
npm run test:rebyte
npm run test:swift
npm run test:android:client
npm run typecheck:web
npm run test:web
npm run build:web
```

`setup:agenui` verifies pinned release SHA-256 digests; native build helpers run
it automatically. Archives live in `.local/vendor/agenui-sdk`; extracted iOS
binaries live in ignored `ios/Vendor/AGenUI`. No binary or signing output is
committed. Regenerate Xcode after adding native sources with `npm run generate:ios`.
The upstream Android release only contains arm64-v8a; unsupported native
libraries produce the card summary instead of crashing the chat.

Focused native tests are `InstantNativeTests/GeneratedUIRenderingTests` and
`ai.impo.ui.GeneratedUIInstrumentedTest`. Run them with:

```sh
IMPO_IOS_TEST_CLASS=InstantNativeTests/GeneratedUIRenderingTests npm run test:ios:app -- --simulator <UDID>
IMPO_ANDROID_TEST_CLASS=ai.impo.ui.GeneratedUIInstrumentedTest npm run test:android:ui
```

They exercise the actual SDK, changed
form values returning through a button, local JavaScript, blocked network
access and blocked origin storage. The Rebyte integration uses isolated
PostgreSQL, real API/worker processes and a provider protocol double. It checks
streaming output, durable replay after cleanup and account isolation. These
checks do not establish physical-device, production-account or remote-model
authoring acceptance. Apply the schema before starting an updated server.

Web tests execute the installed Wasm against the shared native fixture, checking
resolved edits/actions, layout resizing, isolated sessions, busy/disposed callbacks
and admission limits. Parser tests reject invalid trees/bindings/actions and styles.
HTTP/SSE fixture tests cover streamed cards,
main/task follow-ups, history replay, idempotency and account isolation. For local
browser acceptance, start `npm run dev:web:fixture -- --generated-ui`, then use
the fixture-mode Web command in [the Web guide](../web/README.md). Verify actual
game interaction and network/storage/navigation isolation in the browser as well
as the generated sandbox markup.
