SI or AI Powered Selenium Testing with Playwright
Using Claude and OpenAI to develop maintain and investigate browser tests
SI or AI can help automation teams turn requirements into test drafts, improve reusable browser actions, and investigate failures. The strongest integration gives the model useful evidence and a limited set of capabilities, while the framework remains responsible for execution and verification.
For teams with an established Selenium framework, Playwright can become a second browser engine. Claude or OpenAI can sit above that engine layer as an optional assistant. This approach preserves the value of existing test data, keywords, reports, and CI pipelines while introducing new ways to create and maintain tests.
This article presents an implementation approach, not a claim that Selenium or Playwright includes a built in AI engine. The prompt and JSON examples are illustrative. The screenshots show real Playwright tools from its official documentation, rather than a live Claude or OpenAI test run.
Three technologies with different responsibilities
Selenium runs the WebDriver based tests already supported by the framework. Playwright runs selected browser flows through its own APIs. The AI service proposes plans, code, or diagnostic explanations based on the information it receives. A model’s statement that a test passed is not execution evidence.
In Java, Playwright can use familiar runners such as JUnit and TestNG. Browser instances may be reused within an appropriate lifecycle, with a new BrowserContext for each test. The Java library does not automatically provide the Node.js Playwright Test runner’s fixtures and reporting. [1]
Start with assistance during development
The easiest first step is to ask Claude or an OpenAI model to review a recorded workflow, generate test variations, or explain a failure. Review the result, commit the approved code, and execute it through the normal runner. The regression suite can then run without calling a model on every step.
Live AI execution is a separate capability. It requires a tool connection, a controlled action loop, and limits on what the model can do. Add it when it solves a specific need, such as supervised exploration or proposing a missing generic keyword.
Fit AI into the existing framework
Keep the test controller, environment configuration, data readers, and reporting services. Add an engine selection per suite or scenario and a provider selection for AI assistance. Choose one browser engine for a scenario; WebDriver elements and Playwright locators are different objects.
A small architecture with clear boundaries
The runner loads the scenario and selects Selenium or Playwright. An AI adapter sends approved context to Claude or OpenAI. A validator checks the returned plan against the keyword registry, locator policy, and required assertions. The browser executor performs approved actions and sends actual outcomes to the report.
Use a common internal plan format so provider response structures do not leak into every test. Give the two adapters the same task contract. Handle provider errors, timeouts, and incomplete responses explicitly. An AI outage should not turn a failed planning request into a passing test.
Three execution policies
Existing Keywords Only lets the model choose from approved operations without inventing code. Generate and Review lets it propose a missing operation for a tester to inspect. Auto Validated permits a proposal only after the configured checks pass. These are framework policies to implement, not built in guarantees from either provider.
For generated keywords, prefer reusable names such as CLICK_LINK_TEXT, FILL_FIELD, or VERIFY_TEXT. Parameters should describe the link, field, container, and expected outcome. Avoid names tied to one site or screen. Existing method names and data formats can stay stable when an adapter supplies the browser specific behavior.
Make validation concrete
Check that the plan is well formed, uses supported operations, and supplies required arguments. Confirm that locators identify the intended target and that assertions preserve the expected business result. Compile generated code, review changes, and run a representative test before registration. Passing one test does not prove a generic keyword is correct in every situation.
Do not silently rewrite an expected result to make a failing test pass. If a plan is ambiguous, report the missing information. The framework should be able to stop with a clear error rather than asking the model to keep clicking until something looks successful.
Using Claude and OpenAI
Both services can support an application controlled tool loop. The provider formats differ, so keep their request and response handling inside separate adapters. API access also requires valid credentials and an account permitted to use the selected model. Verify current access and limits during setup.
Claude usage
With Claude’s Messages API, define custom tools with an input_schema. Claude can return tool_use blocks requesting those tools. Your application checks and executes an approved client tool, then returns a matching tool_result. This round trip supplies the model with the actual outcome of the operation. [2]
For example, expose propose_test_plan and inspect_page before exposing browser actions. Give Claude the approved keyword list, relevant page structure, and expected result. Ask it to identify missing information and propose a reusable plan. Review that plan before execution.
OpenAI usage
With OpenAI’s Responses API, define function tools and their JSON schemas. The model can request a function call; your application validates the arguments, performs the operation, and returns a function_call_output associated with its call_id. Continue the conversation when another step is needed. [3]
A useful first workflow is generating a plan from a requirement and a keyword catalog. Another is analyzing a failed step with its exception, redacted page evidence, and relevant logs. Record the model configuration with the result so the proposal can be traced back to the request.
Keep the same contract across providers
Concern | Claude adapter | OpenAI adapter |
API entry point | Messages API | Responses API |
Tool definition | name and input_schema | Function name and parameters schema |
Requested action | tool_use block | Function call item |
Returned evidence | tool_result linked to tool use | function_call_output linked by call_id |
The mappings above summarize custom tool use. They are not interchangeable wire formats; use the current SDK and API documentation for implementation. [2, 3]
Choose between providers using a small evaluation set of your own requirements and failures. Compare valid plans, unsupported actions, quality of assertions, latency, and cost. Avoid assuming one provider will always be better for every automation task.
From a recording to a reviewed test
Playwright Codegen records browser interactions and proposes locators. It can also record visibility, text, and value assertions. The browser and Inspector windows give testers a concrete starting point to share with an AI assistant. [4]
Screenshot 1 Official Playwright Codegen and Inspector example at a desktop viewport. Source [4].
A productive workflow is to record a short journey, review its selectors, remove secrets, and supply the draft with a business requirement. Ask the model to parameterize data, extract repeated operations, and add assertions for the intended result. Inspect its changes before registering or committing the test.
A prompt you can use with either provider
Task: turn this recorded flow into a reusable test plan.
Use only the attached approved keyword catalog.
Keep the supplied business assertions unchanged.
Use parameters for environment URLs and test data.
Prefer roles, labels, or approved test IDs for locators.
Do not invent credentials or missing application facts.
Return steps, assertions, assumptions, and unresolved questions.
Propose any missing generic keyword separately for review.
Attach the relevant evidence to the request. If the model has not received a recording, DOM excerpt, or browser tool result, it cannot reliably know which controls are on the current page.
Turn AI output into deterministic browser actions
A reviewed plan should be simple enough for the framework to validate and replay. The following JSON is an example internal format, not an OpenAI or Claude request body. The framework must implement the keyword semantics and locator translation.
{
"engine": "playwright",
"steps": [
{"keyword": "FILL_FIELD", "label": "Search",
"valueRef": "searchTerm"},
{"keyword": "CLICK_BUTTON", "name": "Search",
"exact": true}
],
"assertions": [
{"keyword": "VERIFY_TEXT", "testId": "results",
"expectedRef": "searchTerm", "match": "contains"}
]
}
Resolve valueRef and expectedRef through the existing data layer. Reject unknown keywords and missing references. A browser executor can map the approved Search button action to a Playwright role locator and map the assertion to a retrying text assertion.
Use Playwright waits deliberately
Playwright’s actionability checks can wait for conditions such as visibility, stability, and whether an element receives events before clicking. They do not establish that a transaction succeeded. Assert the expected application state after the action. Avoid routine forced clicks or fixed sleeps as a repair strategy. [5]
Handle browser events explicitly
Register popup and download waits around the triggering action. Switch to the returned tab and verify its expected content. Keep frame boundaries, URL checks, download paths, and screenshot timing in the executor rather than letting the model invent a different convention for each test.
What MCP adds
Microsoft’s Playwright MCP server exposes browser capabilities to compatible AI clients and uses structured accessibility snapshots for many interactions. It is an optional tool connection, not a replacement for a runner, test suite, or assertion policy. Its own documentation cautions that it is not a security boundary. [6]
Use a compatible host and a reviewed configuration if you choose MCP. A Java framework can instead expose controlled functions through its own adapter. Connecting an AI client to a browser does not by itself integrate that session with Selenium’s reports or share WebDriver objects.
For live agent execution, cap action counts and elapsed time, restrict navigation to approved environments, and validate each action. Treat text from web pages as application data, including text that tries to instruct the model to ignore the task.
Use evidence for AI failure analysis
A screenshot shows one moment. A Playwright trace helps inspect actions, DOM snapshots, source locations, logs, and network activity. Start tracing before the flow and finalize the artifact before closing its context. Keep evidence linked to the specific test and run. [7]
Screenshot 2 Official Playwright Trace Viewer example showing recorded actions and the timeline. Source [7].
Provide Claude or OpenAI with a focused failure package: the failed step, original exception, expected outcome, redacted screenshot, and relevant extracted trace details. Do not assume the model can read a trace ZIP directly; extract usable evidence or provide an appropriate tool to inspect it.
Ask for a diagnosis that can be checked
Separate observed facts from possible explanations.Identify which evidence supports each explanation.Do not change the expected result or declare a pass.Propose the smallest diagnostic action or repair.State what cannot be established from this evidence.
Store AI analysis beside the original failure. A timeout may reflect an ambiguous locator, hidden control, expired session, slow response, or real application defect. The explanation remains a hypothesis until a targeted check supports it.
Retain screenshots and traces according to the team’s policy. Remove passwords, tokens, personal data, and sensitive page content before sending evidence to an external service. Use unique artifact names for each test, browser, and attempt.
Practical habits for a useful rollout
Begin with reviewed test generation and failure analysis. These uses can improve the development workflow without making every regression step depend on a model call. Add supervised browser exploration after the team has a validated action contract and reliable evidence collection.
Keep prompts and approved generated keywords under version control. Measure whether AI reduces maintenance time and produces meaningful assertions. Track rejected plans as well as accepted ones. Apply time and spend limits, redact logs, and keep API keys in environment variables or an appropriate secret store.
Exercise the pilot at more than one viewport. A smaller layout can change menus, control visibility, and navigation. Ask AI to propose responsive scenarios, then verify them with real browser runs.
Screenshot 3 Official Playwright Codegen example using mobile web emulation. Source [4].
The long term benefit is a framework that turns approved intent into repeatable tests and turns failures into useful evidence. Selenium and Playwright execute the tests; Claude and OpenAI help the team create, understand, and maintain them.
References and screenshot credits
Sources checked October 3, 2026. Screenshots come from official Playwright documentation and may show older UI versions. The article does not present them as an AI execution demonstration.
[1] Java test runners
[2] Claude tool use
[3] OpenAI function calling
[4] Playwright test generator and screenshots
[5] Playwright auto waiting
[6] Microsoft Playwright MCP
[7] Playwright Trace Viewer and screenshot
Comments