Ostler design review (October 2026)
The October 2026 design review of Ostler: five design flaws and their status.
On this page
- Status
- The five design flaws
- 1. "Never guess" is not actually enforced: wrong fills can be reported as Verified (critical)
- 2. The safety model is checked after the browser has acted (high)
- 3. Secrets leak through side channels (medium-high)
- 4. The browser layer has no recovery and almost no tests (high)
- 5. The evidence does not support the headline yet (critical for credibility)
- Other notable findings
- Recommended order of work
Five independent read-only reviews: form engine, Qdrant memory, MCP/security, CDP/lifecycle, benchmarks/claims. 74 findings in total; this page keeps the ones that matter, grouped by the design flaw they share. "C" = confirmed from the code path, "S" = suspected. No reviewer reproduced issues live; all 73 tests pass, which mostly says the tests miss these cases.
Status#
Updated as work lands. Fixed means there is a regression test that fails without the change.
| finding | status |
|---|---|
| Ambiguous key wins by elimination after rivals are filled | fixed (keys resolve against all visible fields) |
Form re-mounted mid-fill abandons keys; ok() ignored unmatched keys | fixed; also follows stale refs by label for read-back |
| Memory: playbook key cut at first space | fixed (structured provenance) |
| Memory: one accepted semantic guess becomes a permanent 0.95 rule | fixed (only playbook/autocomplete matches persist) |
Memory: --learn overwrites user facts | fixed (Memory::learn never replaces a non-learned fact) |
| Memory: semantic top-1 with no margin; "spouse/emergency" labels | fixed (margin 0.05; other-person labels never use semantic) |
| Secrets in notes, snapshots, episodes | fixed (notes hidden, snapshot masks, episodes scrubbed of typed values and long digit runs) |
| Untrusted wrapper escapable; reports/titles unwrapped | fixed (defanged; reports wrapped; titles, URLs, dialog text and the look legend are now quarantined too, found by a prompt-injection page in the 2026-10-31 soak) |
| Allowlist checked after the fact | fixed for every request type and WebSockets (catch-all interception plus a browser-wide resolver rule; verified with a canary server). Ports are still ignored |
| Tools skip the policy check | fixed for snapshot, inspect, fill, submit, screenshot |
--require-confirm-submit only guards form_submit | fixed (all non-GET requests blocked outside a confirmed submit) |
| Dead Chrome never relaunched; first call waits a timeout | fixed |
panic = "abort" defeats Chrome cleanup | fixed (removed) |
| WebSocket frame limit kills the connection | fixed (limits raised) |
| Popups unseen and unpoliced | fixed (popups become tabs under the same policy) |
| Tool output bloat | fixed: terse by default, --verbose for detail |
| Upload of any local path | fixed (refused unless --upload-dir; paths resolved so .. and symlinks cannot escape) |
DevTools TCP port reachable by local users (use --remote-debugging-pipe) | fixed (pipe transport by default; verified no listening socket; OSTLER_CDP_TRANSPORT=ws is the old behaviour) |
| Default policy permits loopback/private/metadata addresses; ports ignored | partly fixed: --block-private-networks (default on for a2a) and host:port allowlist entries. serve with no allowlist is still open by default; DNS rebinding can outwit the name check |
confirm() auto-accepted | fixed (confirm/prompt dismissed unless --accept-dialogs) |
| Chrome orphaned on SIGKILL/SIGTERM of the server | fixed on the pipe transport (Chrome exits when the pipe closes; the temp profile directory of a SIGKILLed server is still left behind) |
| Submit classification, error attribution, settle on busy pages, O(n^2) re-scan, isolated world, refs reused across documents | open |
| Combobox async options; some widgets (MUI select, read-only Ant select, cross-origin iframes) | open |
Qdrant: short deadline + circuit breaker, forget cascade, episode TTL, tenant auth, hnsw m:0, model-name check | open |
| Benchmarks: no LLM in the loop, strawman baseline, no batched-tool baselines | labels corrected; the real evaluation is open |
The five design flaws#
1. "Never guess" is not actually enforced: wrong fills can be reported as Verified (critical)#
The project's central promise is that ambiguity is reported, not guessed. Several paths break it.
| where | flaw | |
|---|---|---|
fill.rs:227-236 | Keys are re-resolved every round against the remaining fields, so a key reported ambiguous in round 1 wins by elimination in round 2 ("phone" lands in Shipping phone). | C |
fill.rs:217-223, 292-295 | If the framework re-mounts the <form> mid-fill, pick_form returns None, the report block is skipped, and pending keys vanish with ok() == true. | C |
memory/lib.rs:479-492, app.rs:151 | Semantic recall takes top-1 over an absolute threshold, with no margin, no type check, no "someone else" guard (spouse/reference/emergency). Auto-fill of required fields is the default when memory is on. | C (scores S) |
app.rs:662-666 | Any memory-sourced fill on an accepted submit becomes a 0.95-confidence playbook mapping, so one wrong guess is replayed forever. | C |
app.rs:665 | Playbook key parsed with split_whitespace().next(): "Email address" is stored as "Email", "work email" as "work". | C (bug) |
app.rs:748-752 | --learn upserts page-labelled facts over user-entered facts; a "refer a friend: Email" form overwrites the user's email. | C |
fill.rs:536-551, page.rs:500-512 | Native-setter fallback verifies immediately with no settle; frameworks revert later. type_into ignores focused:false, so text can land in another field. No final re-verify pass. | C |
runtime.js:690-704, fill.rs:771-779 | Combobox picks the first visible options (stale async results), falls back to any [role=option] on the page, accepts one-way substring hits; verification treats "United States Minor Outlying Islands" ≈ "United States". | C |
fill.rs:799, runtime.js:430 | datetime-local always verifies; value-less radios all read "on". | C |
Design fix: resolve all keys once against the full field list and freeze the result; make memory suggestions advisory unless top-1 has a margin and a type match; learn mappings only from user/agent-provided values; a final settle + re-verify pass over every result.
2. The safety model is checked after the browser has acted (high)#
| where | flaw | |
|---|---|---|
page.rs:278-378, policy.rs | The allowlist is checked at navigate start and after actions. No Fetch interception, so cross-origin form actions, redirects, page fetch/XHR, beacons and iframes reach forbidden origins first. | C |
app.rs:343-536 | snapshot, form_inspect, form_fill, screenshot, wait_for skip the policy check; a delayed redirect gets the agent's (and memory's) values typed into a forbidden page. | C |
app.rs:705 | --require-confirm-submit is only on form_submit. click on the submit button, type submit:true, press Enter, or page JS on change/blur all submit. | C |
fill.rs:483, page.rs:598 | File fields accept any local path (~/.ssh/id_rsa); errors reveal which files exist. | C |
app.rs:168, runtime.js:233 | The <page_content untrusted> wrapper is not escaped, and fill/submit reports, titles and dialog text are not wrapped at all. | C |
launch.rs:117 | The DevTools TCP port is reachable by any local user; with --profile-dir that is the user's logged-in browser. SECURITY.md understates this. | C |
policy.rs:27-30 | An empty allowlist permits loopback, LAN and 169.254.169.254; entries ignore ports. | C |
launch.rs:131 | --disable-popup-blocking with no target auto-attach: popups load unpoliced and unseen. | C |
page.rs:154-168 | Every confirm() is auto-accepted, including a site's own "really pay?" check. | C |
Design fix: enforce in the browser, not the tool layer: Fetch.enable to block disallowed document/subframe requests and to hold submit-triggered navigations for confirmation; Target.setAutoAttach; --remote-debugging-pipe; private ranges blocked by default; uploads off unless --upload-dir.
3. Secrets leak through side channels (medium-high)#
Masking covers requested/observed only. Leaks: the note field (Mismatch and NoMatch embed raw values, fill.rs:715, values.rs:81); snapshots show value= for sensitive non-password inputs (runtime.js:255); episodes store page validation text that echoes values, never filtered and not removed by forget (app.rs:758-766). looks_sensitive is substring-based: it blocks shipping_address ("pin") and misses passports and card numbers inside text. forget does not cascade; Qdrant WAL and snapshots keep data.
4. The browser layer has no recovery and almost no tests (high)#
- A dead Chrome is never relaunched: every later call fails until the MCP process restarts (
app.rs:247-256,cdp/lib.rs:85-119). C - A single oversized frame (a tall full-page screenshot over tungstenite's limits) ends the reader loop permanently (
cdp/lib.rs:85). S - Chrome and the temp profile are left behind on SIGTERM or panic. v0.2 had added
panic = "abort", which stoppedkill_on_dropfrom running (launch.rs:150). C. Reverted after this review; signal handling is still missing. - Navigation detection uses a 180 ms window plus a bare load counter: late navigations are missed and stale load events satisfy later waits; the
navigatetimeout is discarded (page.rs:290-345). C - Eval retries on context loss can run side effects twice (
page.rs:180-230). S - No tests for the CDP client, lifecycle or navigation logic. C
5. The evidence does not support the headline yet (critical for credibility)#
- No LLM in the loop: both modes are a fixed script with hard-coded labels (
forms.rs:28-178). C - The per-field baseline is weakened:
observe:"full"plus an extra full snapshot after every field, while Ostler's own tools default to delta (forms.rs:315, 329). C - The batched-fill competitors (Playwright MCP
browser_fill_form, chrome-devtools-mcpfill_form) are not run, and the README misdescribes chrome-devtools-mcp as debugging-only. C - "5.4x faster" is
turns x 3 s: the call-count ratio restated, presented as "measured". C - A 1.5k-token base context is unrealistic for coding agents; at 20k the 20-field cost ratio drops from ~5x to ~2.7x. C
- The headline is the best case (20 fields); the 9-scenario aggregate is 1.7x, and combobox is 1.0x. "Median of 3" applies to tool time only. C
- The memory benchmark uses hash vectors, 20 identical points per tenant, no wait for indexing, a single RSS sample: "flat at 200k" is true by construction. The Qdrant multitenancy setup is missing
hnsw_config {m:0, payload_m:16}. C - Stagehand's action caching is a shipped equivalent of playbooks; the moat section ignores it. C
Other notable findings#
- Form-less pages: all controls become one "page" form whose submit candidates are document-wide;
SUBMIT_REmatches "search"/"go", so the header search button can be clicked (runtime.js:507-552). Any type-less<button>counts as submit (runtime.js:468). C - Submit classification: a confirmation page that has any input is "Advanced"; XHR success with the form still present is "Unchanged";
hiddenis never set, so its check is dead code (fill.rs:648-663). C - Error attribution searches the grandparent, so one invalid field's message is attached to its neighbours;
.help-blockcounts as an error (runtime.js:370-376). C - Settle never settles on pages with polling, streaming or animation, and the result is ignored; each round re-walks the whole DOM, making fills O(n²) (
runtime.js:32-44, 764-779). C - Runtime lives in the page world (
window.__wm, wrappedfetch/setTimeoutvisible to bot detection); should use an isolated world. S - Refs restart at
e1per document and the refs Map never frees (runtime.js:51-58). C - Qdrant: 10 s timeout × up to 4 sequential requests per field, errors swallowed, with no "memory degraded" signal (
qdrant.rs:56,app.rs:486); tenant is an unauthenticated string defaulting to "default"; model changes with the same dimension go undetected. C - JSON-RPC: the empty batch gets no error, batch replies are not an array, requests are strictly sequential, so a 60 s
wait_forblocksping; cancellation is ignored. C - Blocking
ps/footprintcalls inside async code while holding the browser mutex (launch.rs:220, 277). C
Recommended order of work#
- Stop wrong fills (flaw 1): freeze key resolution, gate memory suggestions, fix the playbook key bug, never overwrite user facts, final re-verify. Each comes with a fixture that currently passes wrongly.
- Add browser relaunch (
panic = "abort"already reverted), signal handling and--remote-debugging-pipe(flaw 4, plus part of flaw 2). - Browser-level enforcement of allowlist, confirm gate and uploads; escape and extend the untrusted wrapper; mask notes and snapshots (flaws 2, 3).
- Honest benchmarks: add batched/delta baselines, a realistic base context, and "modelled" labels now; then an LLM-in-the-loop eval against real Playwright MCP / chrome-devtools-mcp / agent-browser, plus a held-out real-site set (flaw 5). Update the README and site until then.