Browsing That Survives UI Changes

Every website your Agents browse will eventually change — a redesign, an A/B test, a moved button. Whether that costs you an afternoon or nothing at all depends on decisions you make when building: how elements are identified, what you record, and what you tell the Agent to verify. All of it follows from one principle: depend on what things are, not where they are.

​

How the Agent Actually Sees a Page

The Agent’s primary view of a page is not a screenshot. It reads the page’s structure — the accessibility tree of interactive elements with their roles and labels (“the button named Submit order”) — and acts on elements by identity. Screenshots and pixel coordinates exist as a documented last resort for canvas-heavy or unlabeled interfaces, not as the default. This is why browser Agents tolerate restyles far better than pixel-based automation: a button that moves, changes color, or gets re-nested is still the button named Submit order. What actually breaks identification is structure without names — which is the next section.

​

Give Your Controls Stable Anchors

If your Agents browse an application your own company controls (an internal tool, your admin, your storefront), the single highest-leverage act is in that application’s codebase, not in any AOP: give the controls the Agent uses stable, human-meaningful identifiers — an aria-label, a name, a test id. That’s precisely what recorded Skills anchor to. What makes an anchor stable vs. brittle: A recorded step doesn’t rely on a single selector either — it stores a ladder of them, from the most meaningful down to the positional fallback, and at replay picks the first one that matches exactly one element on the live page. That’s the tolerance model in one sentence: a moved or restyled control still resolves; a duplicated or renamed one deliberately does not, because guessing between two matches is how automations act on the wrong record.

​

Record the Routine, Let the Model Judge

Recording a browsing Skill is the fastest path to reliable browser work: you perform the flow once, and it replays deterministically — same steps, no improvisation, a fraction of the time and cost. To get the most from it, understand what recording deliberately does not make deterministic:

• Judgment steps hand off to the Agent. A click that selects which entity to act on (which product, which order), and any step the recorder can’t anchor stably, replays as a handoff: the Agent performs that one step with the task’s actual context, then the deterministic replay resumes. This is correct behavior — a recording of you picking this product must not hard-code that product forever.
• One-time codes are never replayed. An OTP is stale the moment recording ends; at replay the Agent generates a fresh one from the Login’s authenticator. Relatedly, passwords and OTP values are never stored in the recording — sign-in during recording resolves credentials server-side, and the Skill file contains no secrets.
• Read the Execution outlook. Every recorded Skill states how many of its steps are expected to replay deterministically (e.g. “12/21 steps expected deterministic; 9 hand off to the agent”). A low share is a design signal: the flow leans on unlabeled controls or judgment mid-stream. Restructure — do the judgment first, then record the mechanical tail — or add anchors to the app.

When a step does fail at replay — the element is gone, renamed, or ambiguous — recovery is per-step: the Agent is handed exactly that step to perform with judgment, then returns to the deterministic path for the rest. One broken step costs one step, not the recording.

​

Verify After Actions That Change Data

The platform is deliberately conservative around mutations: if a browser session dies mid-Run, it is replaced and the Agent is told to check whether its last action already took effect before repeating it — never to blindly retry a click that may have submitted an order. Extend the same discipline into your AOPs:

• After every mutating step, verify the outcome on the page. Wait for the confirmation or the error banner — checking for both outcomes beats waiting for success and timing out on failure — and treat “neither appeared” as a failure to report, not a green light.
• Make success observable. “Submit the form and confirm the order number appears, then record that order number” is verifiable and gives you the stable identifier for the audit trail. “Submit the form” alone is a coin flip that always reports heads.
• Fail loudly on access. A missing Login fails soft (see One Builder, Many Users) — tell the Agent to stop and fail with the reason if it cannot sign in, rather than browsing around the login wall.

​

Related