
## The Problem with Flaky Tests

We've all been there - you have a test suite that works perfectly in isolation, but occasionally fails due to external factors beyond your control (like a third-party service being down). Maybe it's a network hiccup, a temporary service unavailability, or leftover test data from previous runs. While Playwright offers built-in retry mechanisms, sometimes you need more control over what happens between retry attempts.

In this post, I'll show you how to leverage Playwright's retry system with custom fixtures to perform cleanup operations between retries, ensuring each retry attempt starts with a clean slate.

## Understanding Playwright's Built-in Retry Mechanism

Playwright provides a simple retry configuration that you can set in your `playwright.config.ts`:

```typescript
// playwright.config.ts
export default defineConfig({
  // Retry failed tests up to 2 times
  retries: process.env.CI ? 2 : 0,

  // Or configure per project
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
      retries: 2,
    },
  ],
})
```

While this works great for simple flaky tests, it doesn't help when your test failures are caused by leftover data or state from previous attempts. This is where custom retry handling becomes invaluable.

## The Solution: Custom Retry Fixture

The key insight is to create a custom fixture that can detect when a test is being retried and perform necessary cleanup operations. Here's how I implemented this pattern in one of my projects:

### Step 1: Create a RetryHandler Class

First, let's create a `RetryHandler` class that manages the cleanup logic.
In this example, we are cleaning up test data that could be stored in various systems like MongoDB, Redis, files, or other external services.

```typescript
// utils/retry-handler.ts
import { logger } from '@moonactive/admin-common-logger'
import { Fixtures } from 'fixture/base.fixture'
import { TestInfo } from 'playwright-decorators'

export class RetryHandler {
  async retryHandler(
    testInfo: TestInfo,
    context: Fixtures, // Where you pass other fixtures you want to clean up
    testData: any[] // Where you pass the test data you want to clean up
  ): Promise<void> {
    // Only perform cleanup if this is a retry attempt (retry > 0)
    if (testInfo.retry > 0) {
      await RetryHandler.retryCleaner(context, testData)
    }
  }

  static async retryCleaner(
    context: Partial<Fixtures>,
    testData: any[]
  ): Promise<void> {
    try {
      const { mongo, redis } = context
      const dataIds = testData.map(
        item => item.id || item._id || item.identifier
      )

      const contextDetails = {
        mongo: !!mongo,
        redis: !!redis,
      }

      logger.info(
        `Triggering retry cleanup for data: ${dataIds}, context: ${JSON.stringify(
          contextDetails
        )}`
      )

      // Clean up database records if MongoDB is used
      if (mongo) {
        await mongo.deleteTestDataFromDB(testData)
      }

      // Clean up Redis records if Redis is used
      if (redis) {
        await redis.deleteTestDataFromRedis(testData)
      }

      if (!mongo && !redis) {
        logger.error('Missing context for retryCleaner')
        throw new Error('Missing context for retryCleaner')
      }
    } catch (err) {
      logger.error(`Failed to clean up after retry: ${err}`)
      throw err
    }
  }
}
```

### Step 2: Integrate with Playwright Fixtures

Next, integrate the RetryHandler into your Playwright fixture system:

```typescript
// fixture/base.fixture.ts
import { RetryHandler } from '../utils/retry-handler'
// ... other imports

export type Fixtures = {
  logger: Types.SomeLogger
  app: App
  expect: typeof expect
  mongo: DatabaseHelper
  redis: RedisHelper
  bq: BQHelper
  api: APIHelper
  sqs: SqsHelper
  retry: RetryHandler // Our custom retry handler
}

export const test = base.extend<Fixtures>({
  // ... other fixtures

  retry: async ({}, use) => {
    const retryHandler = new RetryHandler()
    await use(retryHandler)
  },

  // ... other fixture implementations
})
```

### Step 3: Use the Retry Handler in Tests

Now you can use the retry handler in your tests:

```typescript
// tests/example.spec.ts

test('Create and validate test data', async ({
  app,
  api,
  mongo,
  redis,
  retry,
}, testInfo) => {
  // Define test data that might need cleanup
  const testData = [
    { id: 'test-item-1', name: 'Test Item 1', type: 'example' },
    { id: 'test-item-2', name: 'Test Item 2', type: 'example' },
  ]

  // Call retry handler at the beginning of the test
  // This will clean up any leftover data if this is a retry attempt
  await retry.retryHandler(testInfo, { mongo, redis }, testData)

  // Your actual test logic
  await app.createTestItems(testData)

  // Validate the created items
  await api.validateTestItems(testData)

  // Perform additional test actions
  await app.performActionsOnItems(testData)
})
```

## How Does It Work?

The magic happens in the `testInfo.retry` property. Playwright automatically increments this value for each retry attempt:

- **First attempt**: `testInfo.retry = 0` (no cleanup performed)
- **First retry**: `testInfo.retry = 1` (cleanup is performed)
- **Second retry**: `testInfo.retry = 2` (cleanup is performed again)

This ensures that cleanup only happens when needed, avoiding unnecessary operations on the first test run.

## Key Benefits

### 1. **Intelligent Cleanup**

Only performs cleanup operations when actually retrying, not on the initial test run.

### 2. **Flexible Context**

You can pass different fixtures and resources based on what your specific test needs to clean up.

### 3. **Comprehensive Coverage**

The example shows cleanup across multiple systems (database, GitHub, configuration), but you can adapt it to your needs.

### 4. **Proper Error Handling**

Includes logging and error handling to help debug cleanup issues.

### 5. **Type Safety**

Leverages TypeScript for better IDE support and catch errors at compile time.

## Advanced Usage Patterns

### Multi-Service Cleanup

You can extend this pattern to handle different types of cleanup based on the test context:

```typescript
await retry.retryHandler(
  testInfo,
  {
    mongo,
    redis,
    elasticsearch,
    fileSystem,
    api,
  },
  testData
)
```

### Conditional Cleanup

Add logic to perform different cleanup based on test metadata:

```typescript
async retryHandler(
  testInfo: TestInfo,
  context: Fixtures,
  data: any[],
  options?: { skipRedis?: boolean; skipDB?: boolean; skipFiles?: boolean }
): Promise<void> {
  if (testInfo.retry > 0) {
    await RetryHandler.retryCleaner(context, data, options)
  }
}
```

### Custom Retry Limits

You can even implement custom retry logic based on specific conditions:

```typescript
async retryHandler(
  testInfo: TestInfo,
  context: Fixtures,
  data: any[],
  maxRetries: number = 2
): Promise<void> {
  if (testInfo.retry > 0 && testInfo.retry <= maxRetries) {
    await RetryHandler.retryCleaner(context, data)
  } else if (testInfo.retry > maxRetries) {
    logger.warn(`Exceeded max retries (${maxRetries}), skipping cleanup`)
  }
}
```

## Best Practices

1. **Call Early**: Always call the retry handler at the very beginning of your test, before any test actions.

2. **Be Specific**: Only clean up the specific data your test creates to avoid affecting other tests.

3. **Log Everything**: Include comprehensive logging to help debug retry scenarios.

4. **Handle Failures**: Make sure cleanup failures don't prevent the retry from proceeding.

5. **Test Your Cleanup**: Consider writing separate tests to verify your cleanup logic works correctly.

## Conclusion

This pattern transforms Playwright's basic retry mechanism into a powerful tool for handling complex test scenarios. By combining custom fixtures with intelligent cleanup logic, you can ensure that flaky tests get a truly fresh start on each retry attempt.

The approach is particularly valuable for integration tests that interact with external services, databases, or file systems where leftover state can cause cascading failures.

Have you implemented similar retry patterns in your test suites? I'd love to hear about your experiences and any creative solutions you've developed!


Do you have a flaky test that is out of your control? Do you have some functionality you need to apply between retries? This is the tip for you.

Playwright Tips: Leverage Playwright's Retry Mechanism as a Custom Fixture


So a little bit context first - As our test suites grow in complexity and execution parallelism, pinpointing the origin of specific network requests and responses becomes increasingly crucial dude to some random flakiness.

In our Playwright automation framework, we recently encountered a scenario where we needed to intercept a response from a particular worker instance in our parallel test execution. This led us to devise a solution leveraging Playwright's request interception capabilities and custom headers.

When running Playwright tests in parallel using multiple workers, identifying which worker initiated a specific network request can be challenging. This is especially true when dealing with APIs that might behave slightly differently based on the context of the worker or the test being executed.
In our case, we needed to verify the response of a specific API call triggered by an action within a particular test running on a specific worker. Using the builtin `page.waitForResponse()` with a URL matcher wasn't sufficient because multiple workers could potentially trigger the same API call around the same time. We needed a more precise way to target the response we were interested in.

Therefore, our approach was to inject a custom header into the outgoing request from a specific worker instance. This unique header would then allow us to identify the corresponding response. Playwright's `page.route()` API provides a powerful mechanism to intercept and modify requests before they are sent.

Here's an example of how we implemented this solution:

```typescript
import { Page, Response } from '@playwright/test'

async function waitForResponseOfSpecificWorkerInstance(
  page: Page,
  partialRoute: string,
  uniqueIdentifier: string,
  timeout = 21000
): Promise<Response> {
  const regexRoute = new RegExp(partialRoute)

  await page.route('**/*', route => {
    const url = route.request().url()
    const urlObj = new URL(url)
    const pathname = urlObj.pathname

    if (regexRoute.test(pathname)) {
      const newHeaders = { ...route.request().headers() }
      newHeaders['X-Unique-Identifier'] = uniqueIdentifier
      route.continue({ headers: newHeaders })
    } else {
      route.continue()
    }
  })

  const response = await page.waitForResponse(
    response => {
      const requestHeaders = response.request().headers()
      return (
        regexRoute.test(response.url()) &&
        requestHeaders['x-unique-identifier'] === uniqueIdentifier
      )
    },
    { timeout }
  )
  return response
}
```

Let's break down what this simple function does:

```typescript
waitForResponseOfSpecificWorkerInstance(page, partialRoute, uniqueIdentifier, timeout):
```

- Takes the Playwright Page object, a partialRoute (a string or regular expression to match the URL), a uniqueIdentifier (a string to identify the worker instance), and an optional timeout as arguments.
- Returns a Promise that resolves with the intercepted Response object.

```typescript
const regexRoute = new RegExp(partialRoute)
```

Creates a regex from the partialRoute variable to allow for flexible URL matching.

```typescript
await page.route('**/*', route => { ... });
```

This is the heart of the request interception. `page.route('**/*')` intercepts all network requests made by the page.
The provided callback function is executed for each intercepted request.

```typescript
const url = route.request().url()
const pathname = new URL(url).pathname

if (regexRoute.test(pathname)) { ... }
```

Extracts the URL and then specifically the pathname from the intercepted request. We often focus on the pathname for route matching.
Then we check if the pathname of the current request matches the provided partialRoute.

```typescript
const newHeaders = { ...route.request().headers() }

newHeaders['X-Unique-Identifier'] = uniqueIdentifier
```

Creates a copy of the existing request headers. It's important to copy the headers to avoid modifying the original object directly. Then we inject our custom header. We add a new header named `X-Unique-Identifier` with the value of the uniqueIdentifier variable that was passed to the function as an argument.

```typescript
route.continue({ headers: newHeaders })
else { route.continue() }

```

Allows the intercepted request to continue to the server, but now with our newly added custom header.
If the request URL doesn't match our target partialRoute, we simply allow the request to continue without modification.

```typescript
const response = await page.waitForResponse(response => { ... }, { timeout })

```

This is where we wait for the specific response we are interested in. The callback function within `waitForResponse` checks two conditions:

- `regexRoute.test(response.url())` Ensures the response URL matches our target route.
  `requestHeaders['x-unique-identifier'] === uniqueIdentifier` Checks if the request that led to this response had our custom X-Unique-Identifier header with the expected value.

And finally, once a matching response is found (both URL and custom header match), the promise resolves with the Response object and returns it.

Now, lets take a look at how we can use this function in a test:

```typescript
import { test, expect } from '@playwright/test'
import { waitForResponseOfSpecificWorkerInstance } from './utils' // Assuming the function is in a utils.ts file

test('Edit refresh lock', async ({ app, api }, testInfo) => {
  // Assuming 'app' and 'api' is a fixture object containing your page objects and api helper function like waitForResponseOfSpecificWorkerInstance

  const responsePromise = api.waitForResponseOfSpecificWorkerInstance(
    app.myPage,
    '/api/v1/example/',
    `worker_${testInfo.workerIndex}`
  )

  // Perform some action that will trigger the API call

  const response = await responsePromise
  expect(response.status()).toBe(200)
})
```

In this example:

- We use `testInfo.workerIndex` (testInfo is a built-in fixture of Playwright) to get a unique identifier for the current worker running the test. This ensures that each parallel test execution will have a distinct X-Unique-Identifier header.
- We call our `waitForResponseOfSpecificWorkerInstance` function before performing the action that triggers the API call.
- We pass the page object, the partial route of the API endpoint we're interested in (/api/v1/example/), and the unique worker identifier as a string.
- The test then proceeds to perform the action on the browser.
- We await the `responsePromise`, which will resolve only when a response from the /api/v1/example/ endpoint is received and its corresponding request had the X-Unique-Identifier header matching the current worker's index.

By doing that simple trick, we can now precisely target and handle responses from specific worker instances in our parallel test execution, reduce flakiness (even in parallel execution) and enhanced debugging abilities.


Target specific API responses with Playwright parallel execution using custom headers for reliable automation.

Playwright Articles

All Playwright Articles

Playwright Tips: Leverage Playwright's Retry Mechanism as a Custom Fixture

Playwright Tips: Using Custom Headers in Playwright for Worker-Specific Response Handling