Skip to main content

Converter feedback: carry the agent's got-vs-expected, and how often may it ask to send?

· 4 min read
Jon-Michael Lastimosa
Senior Web Developer

The questions: the opt-in --share flow sanitizes each conversion into a structural-only JSON and POSTs it to a Google Form. (1) Will an external developer's AI agent send us something we can actually use to improve the converter? (2) What's the right consent cadence — does it send cumulatively or per bug; does one "yes" cover the rest of the site; and is it OK to keep asking after a "no"?

Context

The share report is auto-generated from the converter's own trace — the agent's only real decision is whether to run --share, not what to put in it. The payload carries per-element role / detected / mapped-shortcode / fallback+opportunity flags / the converter's own "why" / tag / class tokens, section decisions + styling-drop property names, aggregate stats, and a salted host hash. Anonymization is solid (no URL, content, images, links, PII).

Two gaps surfaced in the audit:

  1. Got, but not expected. The build protocol tells the agent to flag misses as "element → got vs. expected," but the payload had no expected field and no free-text — so the agent's diagnosis (the single most actionable signal) was dropped, and a confident-but-wrong mapping (no fallback flag, so the row looks clean — exactly the class of bug we fixed all week) was invisible.
  2. Consent cadence was unspecified. The protocol said only "get consent," leaving room for an over-eager agent to fire per bug, re-ask after a yes, or nag after a no.

Options considered

  • Add many Google Form fields per finding. Rejected — the payload field is an opaque JSON blob; more fields = field-id churn and a brittle schema. A single JSON blob + one human summary is the right shape.
  • Leave the payload as-is; rely on aggregate mining. Rejected — good for ranking recurring misses, but never recovers "what it should have been" for a specific/novel miss, and can't see silent-wrong maps.
  • Add an agent-supplied findings[] inside the existing payload JSON (no Form change), enrich the summary with a findings count, and codify the consent cadence in the protocol. Chosen.

Decision

  • findings[] rides inside the payload JSON ({ ref, got, expected, note, systematic }), written by the agent to share-findings.json and merged by --share. note is capped at 120 chars and auto-redacted of URLs/emails/quoted content, so it stays structural. The summary field gains an N agent-findings count for at-a-glance Sheet triage. No new Google Form field is required — the data is a new key in the opaque blob; an optional human-readable "findings" field is a Tier-2 ergonomics nice-to-have, not needed for the mechanism.
  • Consent cadence (now explicit in site-build-protocol.md):
    • Cumulative, once per site — collect the whole conversion's systematic misses into one share-findings.json; ask + send once, at the end (Phase 5). One capture run = one submission (the salted host-hash dedupes). Never a separate --share per bug.
    • One "yes" covers this site's whole report — don't re-ask for each subsequent miss. Consent is the site owner's, so it does not carry to a different site; a new site = ask again.
    • A "no" is final for that site — never re-prompt. At most one ask per site.

Why

The auto report is a good aggregate signal and a safe pipe, but it isn't a precise bug channel until the agent's got-vs-expected can travel — so we gave it a sanitized field inside the existing blob rather than restructuring the transport. On cadence: consent belongs to the site/owner, not the bug, so scoping one ask (and one send) per site is both less annoying and more correct; and re-asking after a "no" is a coercive dark pattern that would erode trust in an opt-in feature. Batching also keeps the Sheet clean (one contextful row per site) and lets us see a site's misses together.