Playwright vs Bot Arena

Verdict: technically patchable, but it's an arms race that the page always wins eventually.

Each of the five remaining signals can in principle be spoofed from Playwright:

• navigator.webdriver can be hidden via --disable-blink-features=AutomationControlled plus an addInitScript that redefines the property.
• The User-Agent can be spoofed with --user-agent="..." to strip the HeadlessChrome token.
• navigator.plugins, navigator.languages, and the Notification.permission / permissions.query pair can all be patched via Object.defineProperty in an init script.

Off-the-shelf stealth bundles (playwright-extra + puppeteer-extra-plugin-stealth) ship most of these patches already. The catch: every Chrome release introduces new tells, and commercial bot-detection vendors (Cloudflare, DataDome, PerimeterX, Imperva) maintain fingerprint databases of every known stealth-plugin signature. You spend more time updating your evasions than writing tests, and you only ever win temporarily.

Verdict: partially patchable in script; the pixel measurements require effectively rebuilding what VNC-AIVA already is.

The JavaScript-level signals (chrome.app, chrome.csi, driver shims, toString integrity) can be polyfilled with an addInitScript at page load. Easy.

The window/screen pixel measurements are different. outerHeight - innerHeight = 0 is true because the headless browser literally has no toolbars. Two ways out, neither great:

1. Run headed (headless: false) on a server with Xvfb/Xvnc. But then you need a real desktop environment with a window manager and a panel to populate screen.availHeight < screen.height, plus you need Chrome to actually display its chrome (not --kiosk). At that point, you have rebuilt the AIVA architecture from scratch.
2. Spoof the values from JS — override window.outerHeight, screen.height, etc. via addInitScript. But the spoofs need to be internally consistent across signals: if you claim a 1080-pixel screen with a 40-pixel taskbar, the browser viewport's actual height needs to plausibly fit inside that. Cross-signal correlation catches these mismatches.

In practice: an automation team trying to fix L2 with Playwright ends up reinventing AIVA badly.

Verdict: bypassable for the basic checks Bot Arena does, but only by hand-rolling humanized interactions everywhere — and any sophisticated behavioural model still wins.

Playwright does expose lower-level mouse APIs that can generate intermediate moves:

• page.mouse.move(x, y, { steps: 30 }) emits 30 intermediate mousemove events along a straight line.
• Wrap that in a Bezier-curve helper with randomized jitter and you produce trajectories with the right shape and curvature.
• page.keyboard.type(text, { delay: rand(80, 200) }) dispatches one key at a time with randomized inter-key delays.

The catch: every interaction in the test suite needs this treatment. A one-line page.click() becomes a thirty-line "humanize" helper. And advanced behavioural fingerprinting (used by serious bot-detection vendors) trains ML models on real human mouse telemetry — they pick up on acceleration curves, overshoot-and-correct patterns, pause-before-click latency, and dozens of other features that synthetic Bezier curves don't replicate. So: bypassable here, in this demo. Increasingly hard against production-grade defenders.

Verdict: effectively impossible to fix from inside Playwright without a real GPU and real fonts.

Every individual fingerprint can be spoofed with an addInitScript hook:

• Override WebGLRenderingContext.prototype.getParameter to return a fake renderer string like "Intel Iris Xe Graphics".
• Patch HTMLCanvasElement.prototype.toDataURL and getImageData to return a pre-computed "real GPU" hash.
• Replace OfflineAudioContext.prototype.startRendering to return a pre-recorded waveform.
• Spoof document.fonts.check and the font-width measurement trick to claim the right font set.

The trap: internal consistency. If you claim "Intel Iris Xe Graphics" for WebGL, your canvas pixel hash needs to match what an actual Intel iGPU produces — and that hash depends on subtle floating-point rounding, anti-aliasing kernels, and driver-specific quirks. Without the actual hardware you cannot reproduce it. Detection services maintain databases of valid combinations: GPU X must produce canvas hash within set Y for fonts Z. Spoofing one signal in isolation creates a contradiction with the others, which is itself a stronger signal than the original tell.

This is where Playwright fundamentally loses against any site doing real fingerprint-based bot detection.

Verdict: functionally impossible to bypass from inside Playwright. The only working "fix" is to outsource the problem.

Turnstile's logic is intentionally closed-source. It runs every kind of signal the previous four levels illustrate, plus a stack of private checks Cloudflare keeps to itself, plus IP reputation, plus behavioural analysis trained on the firehose of real human traffic across the Cloudflare network. Even a Playwright author who perfectly fixed levels 1-4, ran from a residential IP, and hand-rolled humanized interactions would still be classified as automated with high confidence — Cloudflare's behavioural model is too good.

The "solution" used in the wild is paid CAPTCHA-solver services (2Captcha, anti-captcha, CapMonster, etc.). They route the challenge through real-browser farms — either real humans or sophisticated stealth setups — and return a valid token in a few seconds, for a few cents each. Wire one of those into your test:

const token = await solver.solveTurnstile({
  sitekey: '0x4AAAAAADOBZMoei4aG9CNO',
  url: 'https://bot-arena.jhero.app/bot-detection/level-5/',
});
await page.evaluate((t) => {
  document.querySelector('input[name="cf-turnstile-response"]').value = t;
}, token);

This works — but it has defeated the original purpose of using Playwright. You have paid a third-party service to act as the human in front of the human-detector. Your "automated" tests now have a per-run cost and a human-in-the-loop dependency. This is exactly the kind of corner that VNC-AIVA, by being a real browser session at the OS level, avoids without any third-party dependency.

Verdict: impossible with selector-based Playwright. The DOM is empty of anything to query.

Playwright's locator APIs all resolve to nothing here:

• page.getByLabel('Email') — no <label> element exists.
• page.getByRole('textbox') — no <input> element exists.
• page.getByText('Sign in') — the text "Sign in" is painted pixels, not a text node.

The only theoretical path is to take a screenshot from Playwright, pass it to an external OCR / template-matching pipeline to find UI elements visually, then use page.mouse.click(x, y) at the resolved coordinates and page.keyboard.type(...) to fill them. At that point you have built a worse version of the classic AIVA — and you've moved the actual automation outside Playwright entirely.

There is no Playwright-native way to interact with canvas-rendered UIs. This is a structural mismatch between the tool and the target, not an arms race over signals.

Verdict: possible with brittle fallback selectors, but the entire promise of accessibility-based testing is gone.

Playwright's normal idioms break:

• page.getByLabel('Email') — no <label> element exists, so this returns nothing.
• page.getByRole('textbox', { name: 'Email' }) — no accessible name, so this returns nothing.
• page.getByPlaceholder('Email') — no placeholder, so this returns nothing.

Possible fallbacks, in increasing brittleness:

1. page.locator('input[type="email"]') — works this run; breaks if the input type is also randomised, or another email input is added.
2. page.locator('input').nth(0) — works this run; breaks the moment the form reorders or grows.
3. page.locator('div:has-text("Email") + input') — works for this layout; breaks if the DOM structure is rewritten.

For a truly hostile site, every fallback is one revision away from breaking. The maintenance burden grows linearly with the number of forms; the test suite becomes the single largest source of flakiness in the project.

Verdict: impossible from inside Playwright. Closed shadow roots are an explicit privacy boundary that Playwright respects.

Playwright's locator engine can pierce open shadow roots automatically (and the >>> combinator works there too). For a closed shadow root there is no path at all — not from page.locator, not from page.evaluate (the component author may not expose any reference), not from any selector trick.

The only theoretical paths are:

• Convince the component author to expose mode: 'open' — rarely possible for third-party code.
• Use a Chrome DevTools Protocol command that exposes the closed shadow — Playwright does not surface this in its public API.
• Take a screenshot and use external OCR + page.mouse.click(x, y) — at which point you have rebuilt vision-based automation outside Playwright.

For real apps built on closed Web Components, selector-based testing is structurally a dead end.

Verdict: solvable but every test that touches frame content has to be rewritten with explicit frameLocator calls.

The frame-aware version of this test would look like:

const frame = page.frameLocator('iframe[title="login-frame"]');
await frame.getByLabel('Email').fill('[email protected]');
await frame.getByLabel('Password').fill('hunter2');
await frame.getByRole('button', { name: 'Sign in' }).click();
await expect(frame.getByText('Access granted')).toBeVisible();

For a single iframe this is annoying but tractable. In real applications the cost compounds: every Stripe payment field, every Auth0 step, every embedded widget is a separate frame. Some frames are cross-origin (Stripe Elements, for example), at which point Playwright cannot reach in at all — you would need a separate test running against the iframe URL directly, with no shared session.

Net effect: frame-heavy SaaS produces brittle, fragmented test suites where one frame change cascades into many test rewrites.

Verdict: impossible from inside Playwright. Playwright can drag (mouse.down/move/up at coordinates) but it cannot SEE the target zone.

The only paths a Playwright author has:

• Take a screenshot via Playwright, pass the image to an external OCR / template-matching service, extract the target X coordinate, dispatch page.mouse.down/move/up. At that point you have built half of AIVA inside your test runner.
• Use a paid CAPTCHA-solving service (similar to the Turnstile case). The service routes the challenge through real browsers, returns the solved token. Per-run cost + third-party dependency.

For real production slider CAPTCHAs (GeeTest, Alibaba, AWS WAF), the gap position is also rotated, scaled, and obfuscated with noise — generic OCR fails. Vendor-specific solver services are the only working option, and they cost ~$1-3 per 1000 solves.

Verdict: only brittle structural fallbacks remain. The promise of accessibility-driven testing is gone here.

Every label is an SVG/PNG image with empty alt. Playwright's accessibility-based locators return empty:

• page.getByLabel('Email') — no <label> element exists.
• page.getByRole('textbox', { name: 'Email' }) — no accessible name on the input.
• page.getByPlaceholder('Email') — no placeholder.
• page.getByText('Email') — text is in image pixels, not a DOM text node.

Possible fallbacks:

1. page.locator('input').nth(0) — works this layout; breaks on the slightest reorder.
2. Click at hard-coded pixel coordinates via page.mouse.click(x, y) — exactly the kind of brittle, screen-resolution-dependent code that motivates moving away from selector tests in the first place.
3. Integrate an OCR library, OCR the screenshot, find the label position, derive coordinates — at which point your test suite has reimplemented visual automation badly.

In real production deployments (bank login keypads with shuffled-position digit images), each session also changes the layout — so even nth-child fallbacks decay across runs.

Verdict: this is a hard ceiling enforced by the browser security model, not by Playwright.

Cross-origin iframes are sandboxed by the same-origin policy. The browser does not let any JavaScript in the parent page (including Playwright's injected scripts) read or write content in a cross-origin frame. page.frameLocator can list cross-origin iframes but cannot script into them.

For real third-party widgets (Stripe Elements, Auth0, Turnstile) the only supported automation paths are:

• Use the vendor's test mode + their own SDK. Stripe ships a cardElement.update() helper in test mode; Auth0 has Lock mock APIs.
• Run a parallel test against the iframe URL directly, with no shared session state with the host page. This breaks any flow that needs the host's auth state.
• Pay a third-party solver to operate the widget in a real browser and inject the resulting token.

None of these is "your test calling page.getByLabel('Email').fill(...)". The Playwright-native solution does not exist by design.

Verdict: solvable with significant per-list bespoke code; the standard test idioms do not work.

For each virtualised list a test interacts with, the test author has to:

1. Know that the list is virtualised (it might not be obvious from the rendered HTML).
2. Know the total number of rows and the row height to compute where to scroll.
3. Dispatch a programmatic scroll on the list container, wait for the new rows to mount, query, repeat.
4. Alternatively, scroll incrementally and probe for the target row after each step.

All of this is custom code that lives inside the test suite. Different virtualisation libraries (react-window, TanStack Virtual, AG Grid) expose different APIs — there is no portable solution. And many sites virtualise BOTH rows AND columns, multiplying the complexity.

For real apps with very long lists (Gmail, Slack, AG Grid dashboards), the per-test cost compounds: every test that needs to click a non-visible item has its own scroll helper, its own retry logic, its own flake mode.