How Behavior-Driven Development Works When Agents Write the Tests

When AI agents write Gherkin scenarios and step definitions, Behavior-Driven Development does not become simpler. Human responsibility moves from authoring examples to governing the specification behind them. For BDD to keep testing the behavior the business needs, that behavior must be set out apart from the code, rather than in tests that confirm whatever code an agent writes.

What BDD is actually for, and what breaks when agents write the tests

BDD revolves around discovery, collaboration, and examples. Its purpose is to make intended behavior legible before implementation begins, through a structured conversation between the people who understand the business and those who build the software. Gherkin records the outcome of that conversation in a form that can become executable. The artifact is not the deliberation itself, and producing it does not establish that the deliberation happened. Test coverage is therefore an incomplete measure of whether BDD is doing its job.

That distinction matters when an agent receives an instruction to make CI go green. The agent can write implementation code and then generate tests that verify the choices it just made. Those tests may be syntactically valid and pass without establishing that the software meets the actual requirements. The executable specification has stopped specifying and started describing. A passing suite then gives confidence in the agreement between two generated artifacts.

Automation can replace some typing and remove the point at which people resolve ambiguity and decide which examples express the domain correctly. Without that decision, the scenario set can reflect familiar patterns from the agent’s training data rather than the failures that matter in a particular business. The agent has taken the pen, and that inversion changes the responsibility attached to every artifact downstream.

How agent authorship of BDD scenarios actually works, tier by tier

Agent involvement progresses through three tiers, each transferring a different part of the work. At tier one, a human writes the Gherkin scenario and AI auto-completes the step definitions. The behavioral example remains a human-authored specification, while the agent supplies the implementation plumbing that connects it to automated execution. Discovery can still take place because automation starts after people have agreed on the example.

At tier two, AI generates full Gherkin scenarios from user stories. The human still supplies the story, but the agent now drafts the examples through which that story becomes testable. This introduces a new interpretive step between the requirement and its executable form. A reviewer can refine the generated language with the business side before the automation layer connects the scenarios to step definitions.

At tier three, the agent generates scenarios, step definitions, and implementation from domain descriptions. Human work moves toward judging scenario completeness. The generation process can now proceed without a human discovery conversation, because the same automated workflow supplies both the behavioral account and the code intended to satisfy it. A domain description remains an input, but its presence does not establish that every important ambiguity has been resolved. This is the tier where the distinction between a specified requirement and a generated assumption becomes hardest to see.

BDDTestAIGen, presented at ICAART 2025, demonstrates this broader automation of BDD test creation through LLMs, NLP, human-in-the-loop, and agentic AI methods. Its design retains user control over generation and treats AI as a helper. The BDDTestAIGen work found that small open-weight models such as Llama 3.1 strike a balance between how much computing power they need and how easy they are to use. It also identifies fine-tuning an internal LLM as a way to improve domain-specific customization, data privacy, and cost efficiency compared with relying on external models. These choices affect how generation operates, but they do not by themselves settle who owns the behavioral specification.

Structural correctness is another distinct concern at this tier. Earlier work discussed in the notes found that common syntax errors included missing keywords and keywords in the wrong order. Those errors need structural guardrails as well as instructions in the prompt. Yet valid Gherkin establishes only that the artifact has the right form. It does not establish that the agent selected the right behavior to put inside it.

Why agents writing their own tests is structurally different from humans writing tests after the fact

An autonomous agent plans, implements, runs tests, reads failures, and corrects its work until CI passes. In that loop, passing tests is the oracle, the signal that shows whether the work has succeeded. If the agent also defines the tests, the loop has no independent behavioral target unless one is supplied from outside it. Better prompting can influence generation, but it cannot turn agreement between generated code and generated tests into independent evidence.

Test-conditioned generation and execution-guided refinement therefore serve different purposes. In the former, tests exist before the code and constrain what the agent implements. In the latter, tests and implementation are generated together and revised through execution feedback. BDD requires the prior behavioral constraint, even when an agent handles much of the subsequent work. Execution-guided refinement can improve an implementation against its tests while leaving the choice of tested behavior unexamined.

GitLab’s Duo Workflow illustrates the operational loop. It “works through a generated plan, generating or rewriting code to meet requirements, ensuring the code passes all CI pipeline tests, including security scans,” and, when tests fail, “automatically addresses issues and troubleshoots as needed”. Whether passing the gate also demonstrates behavioral correctness depends on the requirements already encoded there. The loop cannot recover an omitted business constraint merely by succeeding against the tests it has.

The Cornell/UCSD experienced-developer study from 2025 offers a useful contrast to complete delegation. It found that experienced developers plan before implementation and validate agentic outputs, retaining agency over software design and implementation because of fundamental software quality attributes. Expert control preserves a source of judgment outside the agent’s success signal. BDD governance makes that separation an institutional responsibility, because co-generated tests cannot falsify the assumptions shared with their own implementation.

What governance actually means when the human is no longer the author

Governance begins with ownership of the domain model from which scenarios are generated. Reviewing Gherkin for correct syntax and plausible wording is not the same as establishing that it covers the business’s actual failure distribution. A reviewer can approve every example they see while missing a whole class of behavior that has been left out. That is a specification-source problem. The human responsibility is to determine what must constrain generation before the output arrives.

The input must include edge cases from production incidents, boundary conditions from customer support, and known regulatory constraints. These are structured evidence about the domain, rather than corrections appended after an agent has drafted a plausible suite. Supplying them changes the basis on which the agent chooses examples. Review tests correspondence with an owned specification.

BDDTestAIGen’s human-in-the-loop design provides a concrete model for that correspondence. It puts the human between user specification, steps, and source code implementation, with control over what gets generated. The important role is maintaining the connection between domain intent and executable scenarios as automation moves across those layers. Human involvement is meaningful when it governs that connection.

Living specifications must stay synchronized with production behavior. Permissions can depend on how risky a change in behavior is, so agents can auto-generate and auto-merge scenario updates for low-risk surfaces. Logic changes, schema migrations, and changes touching authentication or payments require human scenario review in addition to CI approval.

How BDD is being turned around to test the agents themselves

The strongest expression of this governance model is to make agent behavior itself the subject of the specification. Instead of asking only whether an agent can write BDD tests, the framework asks whether the agent follows a defined behavioral contract. ABTest takes this approach through behavior-driven fuzzing. It turns failures that users report from the real world into test cases grounded in repositories for coding agents. The source of expected behavior is thus separate from the agent’s account of what it successfully completed.

ABTest begins by mining reported anomalies for reusable Interaction Patterns and Action Types. It composes those elements into fuzzing templates and then instantiates executable cases in real repositories. Next, it runs the cases with coding agents while recording traces and artifacts. Automated oracles use that evidence to flag behavioral anomalies. The process carries a behavioral expectation from a reported failure through to an executable test and an inspectable result.

ABTest used 400 agent failures that users reported and developers confirmed to extract 47 Interaction Patterns and 128 Action Types. It generated 647 repository-grounded fuzzing cases and applied them to Claude Code, OpenAI Codex CLI, and Gemini CLI. Across those agents, it flagged 1,573 behavioral anomalies, including command-execution failures and boundary overreach. A completed task is not sufficient evidence that the agent followed the required interaction pattern.

ABTest reported detection precision of 40.8%, which limits how its flags should be interpreted. A flagged anomaly is not automatically an established behavioral failure. Nevertheless, the framework makes multi-step deviations available for systematic evaluation rather than reducing success to a final output or passing command. This is the kind of contextual behavior that Gherkin’s Given/When/Then structure is designed to express.

A framework capable of testing agent behavior has to define expectations that the agent does not get to rewrite through successful execution. That is also what an agent-authored BDD suite needs if it is to constrain application behavior. In both cases, the domain model supplies the contract and execution supplies evidence about compliance. The discipline that prevents self-confirming tests is the same discipline that makes BDD a valid way to test agents.

What the software factory infrastructure layer needs to make this work at scale

This separation of specification and execution creates an infrastructure requirement as well as a governance role. If every generated scenario set waits for human comparison against a domain model, specification review becomes another serialized gate. Teams with high AI adoption merge nearly twice as many pull requests as before, but a corresponding 91% increase in review time leaves delivery metrics barely changed. More generated work still has to pass through a review pipeline and merge queue sized for human typing speed. Governance cannot scale through manual inspection alone.

Flaky tests add another source of congestion. An agent instructed to get a pull request merged can repeatedly re-queue the same change, consuming a full CI run with each attempt. At agent frequency, that retry pattern turns wasted compute into a denial-of-service on the merge queue. The number of scenarios needing review compounds the interaction between generated arrivals, unreliable failure signals, and repeated execution.

The response is infrastructure that can handle agent-frequency arrivals without discarding behavioral control. Merge queues need to classify changes by behavioral risk, while CI gates need to distinguish scenario-level failures from implementation failures. Signal routing must also bring evidence of behavior in production back to the specification layer. These capabilities make different kinds of failure visible instead of collapsing them into a single red or green result. That distinction lets governance address the contract when the contract is wrong, rather than sending every failure back into implementation repair.

Agent transcripts provide the evidence needed to close that loop. Analysis of invoked tools, skipped steps, and departures from specified interaction patterns supplies an operational counterpart to the discovery conversation that automated generation can bypass. ABTest records prompts, command traces, file diffs, and generated outputs for behavioral validation. At scale, programs must be able to analyze that artifact trail so people do not have to inspect it for every pull request.