Improve E2E Test Stability Strategies

1

Hello E2E test ninjas :slight_smile:

What are your approaches to make your e2e tests more stable? With stable I mean that the test pass when the application under test is working as expected but the test fails for “unknown” (not known yet) reason.

I tried to rerun failed test jobs automatically in GitLab CI, but I removed it again because it hides also the “flakiness” of the tests a bit.

Another approach I took was to start a “failure log” table where I write down every failure, the reason I suspect the test to tail and how a fix could look like. That helped a lot to make the “unstability” / “flakiness” more explicit and to find real solutions for some problems.

I also try to automate only use cases with high value, not every feature.

I am very interested in your strategies and approaches to make your e2e tests more stable.

2

What works for me is I.retry().someAction().

I put it only for steps that I expect to be flaky (after some action on a page). This helps.
Also I use autoDelay plugin for cases where page updates doesn’t get as fast as they are expected to.
And retryFailedStep plugin is also nice so you don’t have to guess what step is stable and what is not.

So far I’m pretty comfortable with those tools and simple I.wait* functions.
Anyway, flakiness requires some investigation before implementing some solution.

1 Like