AI Can Generate Tests. But Who Tests the Tests?

We spent years trying to make test automation easier.

Now AI can generate the tests for us.

Sounds like a good problem to have.

Maybe.

Because I think we are about to inherit another problem:

Who is checking whether all those AI-generated tests are actually any good?

A test can compile.

It can run.

It can pass.

It can even increase coverage.

And it can still be a poor test.

That distinction is going to matter a lot more as AI-generated automation becomes normal.

More tests does not always mean better testing

There is a simple assumption we need to be careful with.

If AI can generate more tests, then surely quality must improve.

I don't think it is that simple.

Imagine this:

Requirement
   ↓
AI generates 42 tests
   ↓
42/42 PASS
   ↓
Coverage increases
   ↓
Pipeline GREEN

Looks great.

But I would still ask:

Did we improve quality, or did we simply increase the number of tests?

Those are not always the same thing.

A large automated suite can look impressive on a dashboard.

Hundreds of scenarios.

Thousands of executions.

Everything green.

But if the tests are repetitive, weak, poorly designed or checking the wrong behaviour, then the numbers don't really tell us much.

We should not confuse volume with confidence.

A test can be technically correct and still be wrong

This is probably the part I find most interesting.

An AI-generated test might be perfectly valid from a technical point of view.

The code is clean.

The selectors work.

The test executes.

The assertions pass.

No errors.

But what if the original understanding of the requirement was wrong?

Then we may have created excellent automation around the wrong expectation.

For example:

Business rule:
Premium customers should receive free delivery.

AI interpretation:
Premium customers receive discounted delivery.

Generated tests:
All validating the discount rule.

Result:
100% PASS

The automation is working exactly as designed.

The problem is that it is validating the wrong thing.

That is not really a tooling problem.

It is a quality problem.

And it is one reason why I don't think AI-generated tests should simply be accepted because they run successfully.

The assertion is where the truth should be

A test is only as useful as what it actually proves.

Consider something like this:

await expect(page).toHaveURL('/success')

That may prove that the user reached a success page.

But did the payment actually complete?

Was the correct amount charged?

Was the order created?

Was inventory updated?

Was the confirmation sent?

Did the downstream system receive the transaction?

The browser reaching /success may be one useful check.

But it might not tell us whether the business process actually succeeded.

That is where test quality becomes important.

AI can generate the journey.

It can click the buttons.

It can fill the fields.

It can even create assertions.

But somebody still needs to ask:

Does this test prove something that actually matters?

The cost of generating tests is dropping

This is where things become interesting from a QA practice point of view.

Historically, automated tests were expensive to create.

Someone had to understand the scenario.

Design the test.

Write the code.

Debug it.

Maintain it.

Because of that effort, teams usually had some natural limit on how many tests they created.

AI changes that.

Generating another ten, fifty or hundred automated tests can become extremely cheap.

That sounds fantastic.

But the cost of owning poor tests has not disappeared.

Every unnecessary test can still create:

• maintenance overhead

• longer execution time

• flaky failures

• duplicate coverage

• noisy pipelines

• weak debugging signals

• false confidence

So the economics are changing.

Creating tests is becoming cheaper. Maintaining bad tests is still expensive.

That means QA may need to become more selective, not less.

We may need to test the testing system itself

For me, this is one of the bigger changes coming into Quality Engineering.

If AI starts generating a significant part of our automation, we cannot only review the application.

We also need ways to evaluate the quality of the generated tests.

That might mean asking questions like:

• Does this test trace back to a real requirement or risk?

• Is it testing something meaningful?

• Is the assertion strong enough?

• Is this scenario already covered elsewhere?

• Is the test stable?

• Does it add useful confidence?

• Would the test detect an actual defect?

• Is this scenario worth maintaining?

That last question is important.

Just because a test can be automated does not mean it should be.

And just because AI generated it in five seconds does not mean we should keep it forever.

Mutation testing becomes much more interesting

One technique I think becomes especially useful in this world is mutation testing.

The idea is simple.

Instead of only asking whether the tests pass against the correct code, deliberately introduce small defects into the code and see whether the tests detect them.

For example:

Expected rule:
Discount = 10%

Mutation:
Discount = 5%

If all the tests still pass, that tells us something important.

The suite may be green, but it is not strong enough to detect that change.

That is useful information.

It changes the question from:

Did the tests pass?

to:

Would these tests actually catch something if the system was wrong?

That is a much stronger measure of test quality.

Flaky tests will become an even bigger problem if we are careless

Another risk is flakiness.

Anyone who has worked with automation for long enough knows the pain.

The test fails.

Someone investigates.

Nothing is actually wrong.

Run it again.

Green.

AI can generate tests quickly, but if those tests rely on poor waits, unstable selectors, unpredictable data or timing assumptions, we can generate flaky automation just as quickly.

Then we end up with a pipeline full of noise.

And once people stop trusting the pipeline, we have a much bigger problem.

A test suite that nobody trusts is almost worse than having no automation at all.

Because now we have the appearance of control without the confidence.

AI should help QA create better tests, not just more tests

I am very positive about AI-assisted testing.

I think it can remove a huge amount of repetitive work.

It can help generate scenarios.

It can suggest edge cases.

It can create test data.

It can help with Playwright or other automation code.

It can analyse failures.

It can suggest missing coverage.

All of that is useful.

But I don't think the target should be:

AI generated 500 tests.

That is not a quality outcome.

A better target is something like:

High-risk areas identified
        ↓
Meaningful scenarios generated
        ↓
Weak and duplicate tests removed
        ↓
Assertions reviewed
        ↓
Critical flows validated
        ↓
Useful evidence produced

That is much closer to what QA is actually trying to achieve.

The human review still matters

There will be a temptation to automate the whole chain.

AI reads the requirement.

AI writes the application.

AI creates the tests.

AI runs the tests.

AI fixes the failures.

AI decides everything is fine.

Very efficient.

But there is a danger if the same assumption flows through the entire process.

If the understanding is wrong at the beginning, every layer after that may reinforce the same mistake.

That is why human review still matters, especially around critical workflows and business rules.

Not because humans are always right.

We aren't.

But a human can bring different context.

Business understanding.

Operational experience.

Customer impact.

Historical knowledge.

Risk awareness.

Sometimes the most valuable question a tester asks is not technical at all.

It is simply:

Why are we expecting this behaviour?

That question can uncover more than another fifty automated tests.

QA should not become a test-generation factory

This is also why I think QA teams need to be careful about how they measure AI adoption.

If the success measure becomes:

“Look how many tests we generated with AI”

we may be solving the wrong problem.

A QA practice should not become a factory producing thousands of tests because they are now cheap to create.

The objective should still be confidence.

The objective should still be risk reduction.

The objective should still be finding meaningful problems before customers do.

The tools are changing.

The goal has not.

My view

I think AI-generated test automation is going to become normal.

And that is a good thing.

It will make automation more accessible.

It will reduce repetitive coding.

It will help teams move faster.

But we should not automatically assume that faster test generation means better quality.

Soon, generating another hundred automated tests may be almost free.

Knowing whether those hundred tests are worth keeping will still require Quality Engineering.

That is the part I find most interesting.

Maybe the next maturity level in QA will not be about how many tests we can automate.

Maybe it will be about how well we can judge the quality of the automation itself.

Because ultimately, a test does not become valuable just because it passes.

It becomes valuable when it gives us confidence about something that actually matters.