
## The Problem with Flaky Tests

We've all been there - you have a test suite that works perfectly in isolation, but occasionally fails due to external factors beyond your control (like a third-party service being down). Maybe it's a network hiccup, a temporary service unavailability, or leftover test data from previous runs. While Playwright offers built-in retry mechanisms, sometimes you need more control over what happens between retry attempts.

In this post, I'll show you how to leverage Playwright's retry system with custom fixtures to perform cleanup operations between retries, ensuring each retry attempt starts with a clean slate.

## Understanding Playwright's Built-in Retry Mechanism

Playwright provides a simple retry configuration that you can set in your `playwright.config.ts`:

```typescript
// playwright.config.ts
export default defineConfig({
  // Retry failed tests up to 2 times
  retries: process.env.CI ? 2 : 0,

  // Or configure per project
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
      retries: 2,
    },
  ],
})
```

While this works great for simple flaky tests, it doesn't help when your test failures are caused by leftover data or state from previous attempts. This is where custom retry handling becomes invaluable.

## The Solution: Custom Retry Fixture

The key insight is to create a custom fixture that can detect when a test is being retried and perform necessary cleanup operations. Here's how I implemented this pattern in one of my projects:

### Step 1: Create a RetryHandler Class

First, let's create a `RetryHandler` class that manages the cleanup logic.
In this example, we are cleaning up test data that could be stored in various systems like MongoDB, Redis, files, or other external services.

```typescript
// utils/retry-handler.ts
import { logger } from '@moonactive/admin-common-logger'
import { Fixtures } from 'fixture/base.fixture'
import { TestInfo } from 'playwright-decorators'

export class RetryHandler {
  async retryHandler(
    testInfo: TestInfo,
    context: Fixtures, // Where you pass other fixtures you want to clean up
    testData: any[] // Where you pass the test data you want to clean up
  ): Promise<void> {
    // Only perform cleanup if this is a retry attempt (retry > 0)
    if (testInfo.retry > 0) {
      await RetryHandler.retryCleaner(context, testData)
    }
  }

  static async retryCleaner(
    context: Partial<Fixtures>,
    testData: any[]
  ): Promise<void> {
    try {
      const { mongo, redis } = context
      const dataIds = testData.map(
        item => item.id || item._id || item.identifier
      )

      const contextDetails = {
        mongo: !!mongo,
        redis: !!redis,
      }

      logger.info(
        `Triggering retry cleanup for data: ${dataIds}, context: ${JSON.stringify(
          contextDetails
        )}`
      )

      // Clean up database records if MongoDB is used
      if (mongo) {
        await mongo.deleteTestDataFromDB(testData)
      }

      // Clean up Redis records if Redis is used
      if (redis) {
        await redis.deleteTestDataFromRedis(testData)
      }

      if (!mongo && !redis) {
        logger.error('Missing context for retryCleaner')
        throw new Error('Missing context for retryCleaner')
      }
    } catch (err) {
      logger.error(`Failed to clean up after retry: ${err}`)
      throw err
    }
  }
}
```

### Step 2: Integrate with Playwright Fixtures

Next, integrate the RetryHandler into your Playwright fixture system:

```typescript
// fixture/base.fixture.ts
import { RetryHandler } from '../utils/retry-handler'
// ... other imports

export type Fixtures = {
  logger: Types.SomeLogger
  app: App
  expect: typeof expect
  mongo: DatabaseHelper
  redis: RedisHelper
  bq: BQHelper
  api: APIHelper
  sqs: SqsHelper
  retry: RetryHandler // Our custom retry handler
}

export const test = base.extend<Fixtures>({
  // ... other fixtures

  retry: async ({}, use) => {
    const retryHandler = new RetryHandler()
    await use(retryHandler)
  },

  // ... other fixture implementations
})
```

### Step 3: Use the Retry Handler in Tests

Now you can use the retry handler in your tests:

```typescript
// tests/example.spec.ts

test('Create and validate test data', async ({
  app,
  api,
  mongo,
  redis,
  retry,
}, testInfo) => {
  // Define test data that might need cleanup
  const testData = [
    { id: 'test-item-1', name: 'Test Item 1', type: 'example' },
    { id: 'test-item-2', name: 'Test Item 2', type: 'example' },
  ]

  // Call retry handler at the beginning of the test
  // This will clean up any leftover data if this is a retry attempt
  await retry.retryHandler(testInfo, { mongo, redis }, testData)

  // Your actual test logic
  await app.createTestItems(testData)

  // Validate the created items
  await api.validateTestItems(testData)

  // Perform additional test actions
  await app.performActionsOnItems(testData)
})
```

## How Does It Work?

The magic happens in the `testInfo.retry` property. Playwright automatically increments this value for each retry attempt:

- **First attempt**: `testInfo.retry = 0` (no cleanup performed)
- **First retry**: `testInfo.retry = 1` (cleanup is performed)
- **Second retry**: `testInfo.retry = 2` (cleanup is performed again)

This ensures that cleanup only happens when needed, avoiding unnecessary operations on the first test run.

## Key Benefits

### 1. **Intelligent Cleanup**

Only performs cleanup operations when actually retrying, not on the initial test run.

### 2. **Flexible Context**

You can pass different fixtures and resources based on what your specific test needs to clean up.

### 3. **Comprehensive Coverage**

The example shows cleanup across multiple systems (database, GitHub, configuration), but you can adapt it to your needs.

### 4. **Proper Error Handling**

Includes logging and error handling to help debug cleanup issues.

### 5. **Type Safety**

Leverages TypeScript for better IDE support and catch errors at compile time.

## Advanced Usage Patterns

### Multi-Service Cleanup

You can extend this pattern to handle different types of cleanup based on the test context:

```typescript
await retry.retryHandler(
  testInfo,
  {
    mongo,
    redis,
    elasticsearch,
    fileSystem,
    api,
  },
  testData
)
```

### Conditional Cleanup

Add logic to perform different cleanup based on test metadata:

```typescript
async retryHandler(
  testInfo: TestInfo,
  context: Fixtures,
  data: any[],
  options?: { skipRedis?: boolean; skipDB?: boolean; skipFiles?: boolean }
): Promise<void> {
  if (testInfo.retry > 0) {
    await RetryHandler.retryCleaner(context, data, options)
  }
}
```

### Custom Retry Limits

You can even implement custom retry logic based on specific conditions:

```typescript
async retryHandler(
  testInfo: TestInfo,
  context: Fixtures,
  data: any[],
  maxRetries: number = 2
): Promise<void> {
  if (testInfo.retry > 0 && testInfo.retry <= maxRetries) {
    await RetryHandler.retryCleaner(context, data)
  } else if (testInfo.retry > maxRetries) {
    logger.warn(`Exceeded max retries (${maxRetries}), skipping cleanup`)
  }
}
```

## Best Practices

1. **Call Early**: Always call the retry handler at the very beginning of your test, before any test actions.

2. **Be Specific**: Only clean up the specific data your test creates to avoid affecting other tests.

3. **Log Everything**: Include comprehensive logging to help debug retry scenarios.

4. **Handle Failures**: Make sure cleanup failures don't prevent the retry from proceeding.

5. **Test Your Cleanup**: Consider writing separate tests to verify your cleanup logic works correctly.

## Conclusion

This pattern transforms Playwright's basic retry mechanism into a powerful tool for handling complex test scenarios. By combining custom fixtures with intelligent cleanup logic, you can ensure that flaky tests get a truly fresh start on each retry attempt.

The approach is particularly valuable for integration tests that interact with external services, databases, or file systems where leftover state can cause cascading failures.

Have you implemented similar retry patterns in your test suites? I'd love to hear about your experiences and any creative solutions you've developed!


Do you have a flaky test that is out of your control? Do you have some functionality you need to apply between retries? This is the tip for you.

Playwright Tips: Leverage Playwright's Retry Mechanism as a Custom Fixture


So a little bit context first - As our test suites grow in complexity and execution parallelism, pinpointing the origin of specific network requests and responses becomes increasingly crucial dude to some random flakiness.

In our Playwright automation framework, we recently encountered a scenario where we needed to intercept a response from a particular worker instance in our parallel test execution. This led us to devise a solution leveraging Playwright's request interception capabilities and custom headers.

When running Playwright tests in parallel using multiple workers, identifying which worker initiated a specific network request can be challenging. This is especially true when dealing with APIs that might behave slightly differently based on the context of the worker or the test being executed.
In our case, we needed to verify the response of a specific API call triggered by an action within a particular test running on a specific worker. Using the builtin `page.waitForResponse()` with a URL matcher wasn't sufficient because multiple workers could potentially trigger the same API call around the same time. We needed a more precise way to target the response we were interested in.

Therefore, our approach was to inject a custom header into the outgoing request from a specific worker instance. This unique header would then allow us to identify the corresponding response. Playwright's `page.route()` API provides a powerful mechanism to intercept and modify requests before they are sent.

Here's an example of how we implemented this solution:

```typescript
import { Page, Response } from '@playwright/test'

async function waitForResponseOfSpecificWorkerInstance(
  page: Page,
  partialRoute: string,
  uniqueIdentifier: string,
  timeout = 21000
): Promise<Response> {
  const regexRoute = new RegExp(partialRoute)

  await page.route('**/*', route => {
    const url = route.request().url()
    const urlObj = new URL(url)
    const pathname = urlObj.pathname

    if (regexRoute.test(pathname)) {
      const newHeaders = { ...route.request().headers() }
      newHeaders['X-Unique-Identifier'] = uniqueIdentifier
      route.continue({ headers: newHeaders })
    } else {
      route.continue()
    }
  })

  const response = await page.waitForResponse(
    response => {
      const requestHeaders = response.request().headers()
      return (
        regexRoute.test(response.url()) &&
        requestHeaders['x-unique-identifier'] === uniqueIdentifier
      )
    },
    { timeout }
  )
  return response
}
```

Let's break down what this simple function does:

```typescript
waitForResponseOfSpecificWorkerInstance(page, partialRoute, uniqueIdentifier, timeout):
```

- Takes the Playwright Page object, a partialRoute (a string or regular expression to match the URL), a uniqueIdentifier (a string to identify the worker instance), and an optional timeout as arguments.
- Returns a Promise that resolves with the intercepted Response object.

```typescript
const regexRoute = new RegExp(partialRoute)
```

Creates a regex from the partialRoute variable to allow for flexible URL matching.

```typescript
await page.route('**/*', route => { ... });
```

This is the heart of the request interception. `page.route('**/*')` intercepts all network requests made by the page.
The provided callback function is executed for each intercepted request.

```typescript
const url = route.request().url()
const pathname = new URL(url).pathname

if (regexRoute.test(pathname)) { ... }
```

Extracts the URL and then specifically the pathname from the intercepted request. We often focus on the pathname for route matching.
Then we check if the pathname of the current request matches the provided partialRoute.

```typescript
const newHeaders = { ...route.request().headers() }

newHeaders['X-Unique-Identifier'] = uniqueIdentifier
```

Creates a copy of the existing request headers. It's important to copy the headers to avoid modifying the original object directly. Then we inject our custom header. We add a new header named `X-Unique-Identifier` with the value of the uniqueIdentifier variable that was passed to the function as an argument.

```typescript
route.continue({ headers: newHeaders })
else { route.continue() }

```

Allows the intercepted request to continue to the server, but now with our newly added custom header.
If the request URL doesn't match our target partialRoute, we simply allow the request to continue without modification.

```typescript
const response = await page.waitForResponse(response => { ... }, { timeout })

```

This is where we wait for the specific response we are interested in. The callback function within `waitForResponse` checks two conditions:

- `regexRoute.test(response.url())` Ensures the response URL matches our target route.
  `requestHeaders['x-unique-identifier'] === uniqueIdentifier` Checks if the request that led to this response had our custom X-Unique-Identifier header with the expected value.

And finally, once a matching response is found (both URL and custom header match), the promise resolves with the Response object and returns it.

Now, lets take a look at how we can use this function in a test:

```typescript
import { test, expect } from '@playwright/test'
import { waitForResponseOfSpecificWorkerInstance } from './utils' // Assuming the function is in a utils.ts file

test('Edit refresh lock', async ({ app, api }, testInfo) => {
  // Assuming 'app' and 'api' is a fixture object containing your page objects and api helper function like waitForResponseOfSpecificWorkerInstance

  const responsePromise = api.waitForResponseOfSpecificWorkerInstance(
    app.myPage,
    '/api/v1/example/',
    `worker_${testInfo.workerIndex}`
  )

  // Perform some action that will trigger the API call

  const response = await responsePromise
  expect(response.status()).toBe(200)
})
```

In this example:

- We use `testInfo.workerIndex` (testInfo is a built-in fixture of Playwright) to get a unique identifier for the current worker running the test. This ensures that each parallel test execution will have a distinct X-Unique-Identifier header.
- We call our `waitForResponseOfSpecificWorkerInstance` function before performing the action that triggers the API call.
- We pass the page object, the partial route of the API endpoint we're interested in (/api/v1/example/), and the unique worker identifier as a string.
- The test then proceeds to perform the action on the browser.
- We await the `responsePromise`, which will resolve only when a response from the /api/v1/example/ endpoint is received and its corresponding request had the X-Unique-Identifier header matching the current worker's index.

By doing that simple trick, we can now precisely target and handle responses from specific worker instances in our parallel test execution, reduce flakiness (even in parallel execution) and enhanced debugging abilities.


Target specific API responses with Playwright parallel execution using custom headers for reliable automation.

Playwright Tips: Using Custom Headers in Playwright for Worker-Specific Response Handling


## The data-test-id Debate: To Use or Not to Use?

For those of us who have a passion for code testing, automated testing plays a crucial role in web development. When talking about end-to-end or functional testing of web applications, one technique that often sparks a debate is the use of data-test-id attributes in HTML elements. This article delves into the pros and cons of using these attributes.

### So What Are data-test-id Attributes?

data-test-id is a custom HTML attribute used to uniquely identify elements for automated testing. It serves as a stable hook for testing tools, allowing them to locate and interact easily with specific elements regardless of changes to styling, layout, structure, or any other non-functional aspects of the application.

So let's start by examining the arguments for and against using data-test-id attributes in web applications.

### The Case For data-test-id

1. **Stability and Resilience:**

Using data-test-id provides a stable selector, mitigating the risk of tests breaking or flakiness due to changes in CSS classes or DOM structure, which usually happens quite frequently on modern web applications being constantly developed.
It is also especially vital in complex dynamic applications with frequent updates and state changes.

2. **Insulation from Style Changes:**

Due to the fact that almost any new web application is being developed using some kind of a framework or a library (for example, tailwind), CSS classes are often subject to change for styling purposes in a more frequent way then it used to be changed in the past. Relying on them for testing often creates flaky tests.

3. **Improved Testing Efficiency:**

Whatever you think on data-test-id usage, you can't deny its the most efficient method of building an automation testing framework from scratch.
Due to the fact that it is a custom attribute, it is not used for any other purpose other than testing, which makes it a perfect candidate for testing and of course it makes our locators more quicker to find with less of an effort and time to figure out which selectors we should use.

4. **Improved Debugging:**

During test failures, data-test-id attributes make it easier to pinpoint the problematic element, streamlining the debugging process which saves us time and effort.

5. **Dynamic Addition and Removal:**

data-test-id attributes can be dynamically added or removed based on the environment using build tools, server-side rendering, or client-side JavaScript. This allows for their removal in production builds, addressing potential performance and security concerns (spoiler for some of the cons).

### The Case Against data-test-id

1. **Potential Clutter and Performance Impact:**

Leaving data-test-id attributes in production can increase the size of the DOM, potentially leading to slight performance degradation.

2. **Maintenance Overhead:**

Maintaining unique and meaningful data-test-id values can become challenging in large applications, requiring consistent naming conventions and careful management.

3. **Security Concerns:**

On some cases, if left in production, data-test-id attributes might inadvertently expose sensitive information about the application's structure or functionality for potential attackers.

4. **Scraping:**

Simply put, these attributes can make web scraping much easier. While true, it is not the only way to scrape a web application.

5. **Discourage of User Facing Locators:**

Using data-test-id encourages testing implementation details rather than user behavior.
For example, if you look at [Playwright's documentation, under best practices,](https://playwright.dev/docs/best-practices#prefer-user-facing-attributes-to-xpath-or-css-selectors) they encourage using selectors that are visible to the user, such as text content, and other user-visible attributes. Although we need to be fair and say, they were mainly aiming at the usage of CSS selectors, and XPath which these days are considered as unreliable.
You can find similar recommendations in the React Testing Library.

---

To conclude, let's go over of some of the **best practices** and considerations when using data-test-id attributes.

**Naming Conventions:** Establish clear and consistent naming conventions for data-test-id values (e.g., component-element-action).

**Environment-Specific Handling:** Use build tools or server-side rendering to remove data-test-id attributes in production.

**Balanced Approach:** Use data-test-id judiciously, complementing it with user-centric testing approaches that target user-visible attributes.

**Avoid Over-Reliance:** Do not rely solely on data-test-id. Test user flows and user visible information.


In this article, I will discuss the pros and cons of using test-ids in your web applications projects.

The Case For and Against test-id Usage


We've implemented extensive improvements, including expanding device capacity, enhancing monitoring systems, introducing features like automated recovery mechanisms, chaging regions using internal network VPN and more.
These advancements will streamline testing processes, optimize resource utilization, and provide a seamless experience for developers and testers across global locations.
With ~250 users, supporting 25 different platforms and 150 testing devices, we are well-positioned for continued success.

For those of you who are not familiar with DAZN, here's a brief overview:

> _DAZN is the world’s leading live sports BTC streaming service. With DAZN, fans in any country can watch sport in any language, on any device, live and on-demand. Purpose built to stream live sports globally, DAZN is the market leader in Italy, Spain, Germany, and Japan, where it holds the top-tier domestic football rights — Serie A, La Liga, Bundesliga and J League, respectively — as well as the richest catalogue of live and on-demand content. DAZN is also available in over 200 other territories, where it boasts the most extensive library of elite and emerging sports content._

DAZN develops both HTML-based and native applications designed to run on a variety of platforms, including TVs, set-top boxes, gaming consoles, and native devices. Ensuring high quality across these diverse platforms necessitates thorough testing at every stage of development, ranging from unit tests and component tests to emulator tests and real-device testing.

The global pandemic in 2020 prompted a paradigm shift in work practices, underscoring the importance of enabling engineers to access diverse resources and conduct testing remotely. Quality remains paramount, necessitating comprehensive testing methodologies despite remote work environments.

To address these challenges, we adopted a proactive “Lab as a Service” approach, culminating in the establishment of an in-house lab facility in Katowice, Poland. This strategic decision aimed to centralize resources and provide engineers with the necessary tools and infrastructure to conduct testing efficiently.

However, establishing and operating an in-house lab posed unique challenges that required careful consideration and resolution. These challenges encompassed logistical, technical, and organizational aspects, underscoring the need for adaptive solutions and collaborative efforts.

In response to these challenges, ongoing initiatives are focused on optimizing the efficiency and effectiveness of the in-house lab, streamlining workflows, enhancing collaboration among teams, and leveraging technology to overcome geographical barriers. By embracing innovation and agility, DAZN aims to continuously improve its testing capabilities and deliver exceptional quality across its diverse range of applications and platforms.

## Challenges

DAZN employs a diverse team of developers and test engineers spread across different global locations such as Poland, India, Israel, and the UK. Each team comes with its unique set of needs and requirements, necessitating a robust, highly available, and cost-effective platform to accommodate them effectively.

Key points include:

- Diverse Platform Support:
  The platform must ensure comprehensive testing across all platforms supported by DAZN, emphasizing thoroughness and quality assurance in our testing procedures.
- Tailored Solutions:
  Teams may require different tools and resources for their testing processes; for instance, some teams may rely on manual UI/UX testing devices, while others prioritize automation solutions.
- Integration Challenges:
  Incorporating in-house tools and services into our platform is essential to ensure seamless continuity for engineers, requiring careful integration efforts.
- Scalability and Maintenance:
  The platform needs to support a large number of engineers on a daily basis across multiple global sites, balancing scalability with efficient maintenance to avoid operational bottlenecks.

- Cost Considerations:
  Maintaining devices, infrastructure, and manpower across various locations can incur significant expenses, necessitating a cost-effective approach to resource allocation and management.

- Ownership and Accountability:
  Clear ownership and accountability structures are crucial to prevent issues such as maintenance gaps or device mismanagement.

To address these challenges, ongoing efforts are focused on streamlining processes, enhancing collaboration between teams, and leveraging automation wherever possible to optimize resource utilization and improve efficiency in our testing and development workflows.

## Solution — Lab as a Service

Our team, composed of seasoned experts in DevOps, Automation Infrastructure, and IT, recognized the need for a novel solution to address the challenges faced in managing Device Under Test (DUT) usage at DAZN. In the first quarter of 2023, we conducted a comprehensive assessment to identify and map out the major pain points associated with DUT utilization within the organization. This assessment culminated in the development of a pioneering approach coined “Lab as a Service,” aimed at alleviating the burden on developers and testers by handling all DUT-related tasks and physical infrastructure management.

During the second quarter of 2023, we introduced the Minimum Viable Product (MVP) setup, leveraging the Suitest framework — a vendor offering remote control and automation capabilities tailored for both HTML-based and native platforms. Our team meticulously orchestrated the deployment of all requisite physical infrastructure, assembling a dedicated lab housing 50 test devices within a spacious room at the DAZN Katowice office.

![DUT lab v1.0 (QA room — May 2023)](https://miro.medium.com/v2/resize:fit:640/format:webp/1*MDbJo7nq77qwWrRCpNvsrA.jpeg)_DUT lab v1.0 (QA room — May 2023)_

To bolster reliability and resilience, we implemented a comprehensive monitoring and alerting system, with Grafana serving as the frontend interface. Additionally, we integrated DUT recovery mechanisms to enhance the robustness of our testing environment.

![Grafana monitoring dashboard](https://miro.medium.com/v2/resize:fit:640/format:webp/1*8a9dpHjMLH1FZCQptCNplw.png)_Grafana monitoring dashboard_

The efficacy of these enhancements became apparent swiftly, as an increasing number of engineers gravitated towards utilizing the centralized lab infrastructure instead of configuring local devices. Leveraging the valuable feedback provided by users, we iteratively refined and enhanced the lab environment, ensuring that it continues to meet the evolving needs of our development and testing teams.

## Lab as a service 2.0

As the user base and the number of devices in the DAZN lab continued to surge, spurred by the establishment of a new DAZN site in India and the expansion of supported platforms throughout the latter half of 2023, we undertook further enhancements to the Katowice lab. These adjustments were aimed at accommodating the organization’s growth and fortifying its capacity to scale even further. Throughout the first quarter, we initiated a series of improvements targeting both the physical infrastructure and supporting features:

### Infrastructure enhancements

- **Relocation and Expansion:**

The DUT lab underwent relocation to a larger space, facilitating increased scalability and the seamless onboarding of additional platforms and devices. Leveraging existing racks more efficiently, we enabled the mounting of devices on both sides of each rack, resulting in a 50% boost in the number of TVs and a 40% increase in the number of set-top boxes (STBs), totaling 120 DUTs overall, with ample room for further expansion.

![DUT lab v2.0 (open space — January 2024)](https://miro.medium.com/v2/resize:fit:640/format:webp/1*IKUuw6aU2wQMzTBKCKS_bA.jpeg)_DUT lab v2.0 (open space — January 2024)_

- **Streaming Solutions:**

Collaborating with Suitest, we identified a cost-effective solution enabling the streaming of device HDMI output for STBs and gaming consoles. Additionally, we enhanced the TV streaming setup by refining the phone streaming app.

![Live DUT Streaming Phone camera setup](https://miro.medium.com/v2/resize:fit:720/format:webp/1*K460jjXRPqXUVQf8PVg0ng.jpeg)_Live DUT streaming mobile phone camera setup_

![HDMI grabber/splitter combo](https://miro.medium.com/v2/resize:fit:640/format:webp/1*s-cKLE8Oh1h5mliovKc91Q.jpeg)_HDMI grabber/splitter combo_

### Features enhancements

- **Collaboration with Suitest:**

Over the course of the year, we actively collaborated with Suitest to influence their roadmap, contributing to improvements in user experience and interface. A significant milestone was the introduction of a Unified UI integrating video streaming, remote control, and logs within the same browser window, enhancing productivity for lab users.

![Suitest Unified UI](https://miro.medium.com/v2/resize:fit:640/format:webp/1*Wg96k859tBYZwz13vVXG3Q.png)_Suitest Unified UI_

- **LR (Living Room) Launcher**:

In collaboration with the DAZN LivingRoom development team, we introduced LR Launcher, simplifying the workflow for manual testers and streamlining the onboarding process for partner platforms such as Sky and Movistar for the sake of ease of use when testing different development environments.

![LR Launcher](https://miro.medium.com/v2/resize:fit:640/format:webp/1*5AObQs1_Vdyd9cXhf5qF3w.png)_LR Launcher_

- **Enhanced Monitoring**:

With the growing user base and device count, robust monitoring became paramount. We revamped our monitoring service, streamlining processes and transitioning from an API-based solution to WSS notifications. Furthermore, we expanded monitoring coverage and implemented alerting for control units and streaming devices.

- **Automated Recovery**:

Leveraging smart plugs across all TV platforms, we implemented remote manual and automated recovery mechanisms for malfunctioning DUTs. Efforts are underway to develop similar mechanisms for cameras and control units, bolstering the resilience and efficiency of our lab environment.

- **Local Tunnel**:

A local tunnel solution providing exposure of a local development server allows developers and testers to seamlessly execute their code on a real physical lab device. This capability not only streamlines the testing process but also provides a practical and authentic environment for validating local code during development.

- **Lab Insights Report (BI)**:

An automated report detailing lab usage and testing trends. This report will provide valuable insights into usage patterns, enabling us to further optimize our services based on real data.

![BI Report](https://res.cloudinary.com/dtt2usukl/image/upload/v1716458627/Image_27-08-2023_at_11.58_sas9dh.jpg)_BI Report_

**VPN Change Region & VPN Console**:

A VPN solution that will empower lab users to seamlessly transfer DUTs between regions remotely. Using firewalls controlled by OPNSense, a lambda backend and a UI portal written in React (NextJS) users are able to testing of region-dependent content across all lab devices by switching regions as needed.

![VPN Console](https://res.cloudinary.com/dtt2usukl/image/upload/v1716458833/Image_23-05-2024_at_13.06_oxbech.jpg)_VPN Console_

## Service & Support

- **Continuous Improvement:**

Since the launch of the DUT lab in May 2023, we have continually bolstered our service and support infrastructure. This includes expanding our support engineering team, enhancing documentation and guides, and maintaining accessible support channels during working hours. Our commitment to swift issue resolution is underscored by our expanded remote resolution capabilities, facilitated by proactive monitoring and device error-based recovery processes.

- **Onboarding Optimization:**

Through the introduction of “Lab as a Service 2.0” and tailored training sessions, we have streamlined the onboarding process for new users, ensuring a seamless transition and maximizing productivity from the outset using official docs portal (using github pages and backstage).

These enhancements collectively represent our ongoing commitment to optimizing the DAZN lab environment, fostering innovation, and empowering our engineering teams to deliver exceptional quality across all platforms and devices.

## Acknowledgements

Special thanks to the amazing AP Nexus team for their dedication and hard work, it was a pleasure working with each one of you!

[Dennis Groisman](https://www.linkedin.com/in/dennis-groisman-9811b335/), [Daniel-Shahar Cohen](https://www.linkedin.com/in/danielshaharcohen/), [Rafal Muszalik](https://www.linkedin.com/in/rafa%C5%82muszalik/), [Blazej Milewski](https://www.linkedin.com/in/blazej-milewski/), [Alex Rivkin](https://www.linkedin.com/in/alex-rivkin-b037a6a1/), [Yoav Brown](https://www.linkedin.com/in/yoav-brown-970b55265/) and the one and only [Gal Moran](https://www.linkedin.com/in/gal-moran-a3454095/).

[![DAZN Lab as a Service](https://res.cloudinary.com/dtt2usukl/image/upload/v1716460903/play_video_lab_fnkajn.jpg)](https://res.cloudinary.com/dtt2usukl/video/upload/v1716460249/tvlab_v2_withmusic_twifar.mp4 'DAZN Lab as a Service - Nexus Team')_My team - Lab as a Service 2.0_

Article on [Medium (DAZN Engineering Blog)](https://medium.com/dazn-tech/lab-as-a-service-in-dazn-9bcc9d549970)


I'm incredibly proud to share the outstanding work our team has accomplished recently in upgrading our testing lab infrastructure.

Lab as a Service in DAZN


Ensuring consistent application performance across multiple platforms is a cornerstone of quality software delivery. In this post, I'll share my approach for using GitHub Actions to automate testing across a variety of platforms, drawing from a real use case at my workplace.

Our solution utilizes GitHub Actions, a CI/CD service that enables automation of build, test, and deployment workflows directly within GitHub. The highlight of the approach is the dynamic matrix strategy, which handles both scheduled runs and on-demand tests across specified platforms.

**Scheduled and On-demand Testing**

In this example, the workflow is being triggered automatically at 22:00 every day for nightly tests. Additionally, we enabled manual triggers for specific platforms, allowing for targeted testing as needed. This flexibility is crucial for maintaining a fast-paced development cycle without compromising on quality.

```yaml
on:
  schedule:
    - cron: '0 22 * * *'
  workflow_dispatch:
    inputs:
      platform:
        description: 'Choose platform to run the test on'
        required: true
        type: choice
        options:
          - webos
          - tizen
          - playstation4
          - playstation5
          - xbox
```

This configuration leverages the `schedule` and `workflow_dispatch` events in GitHub Actions. The `cron` syntax schedules automated test runs, while `workflow_dispatch` allows for manual execution with platform-specific parameters.

**Dynamic Test Matrix Configuration**

The core of our workflow dynamically adjusts the test matrix based on the context of the trigger—whether it's a scheduled job covering all platforms or a manual job focusing on a specific platform.

```yaml
jobs:
  setup-test:
    runs-on: ubuntu-latest
    outputs:
      platform: ${{ steps.set-matrix.outputs.platform }}
    steps:
      - id: set-matrix
        run: |
          if [ "${{ github.event_name }}" = "schedule" ]; then
            echo 'platform=["webos", "tizen", "playstation4", "playstation5", "xbox"]' >> $GITHUB_ENV
          else
            echo 'platform=["${{ github.event.inputs.platform }}"]' >> $GITHUB_ENV
```

In this snippet, we employ conditional logic to populate the platform variable differently based on the trigger type. For scheduled runs, we test across all platforms, while manual triggers allow for focused testing on a selected platform. This strategy ensures that resources are allocated efficiently, focusing our testing efforts where they are most needed.
Of course, you could always do that with an external script or with a composite action, but this is a simple and effective way to achieve the same result.

**Executing the Tests with Dynamic Configuration**

Our approach to executing tests in this workflow benefits significantly from the dynamic nature of GitHub Actions, especially when it comes to handling multiple platforms which can be quite different from each other. This flexibility is further extended into our application's `package.json` file, where we define specific scripts for our testing needs. The `package.json` plays a crucial role in integrating the GitHub environment variables directly into our test execution process.

Without getting into the tests code, you can imagine that each platform has its own specific tests and configurations, and that's why we need to run the tests separately for each platform. This is where the dynamic matrix comes into play.

```yaml
- name: Run tests
  working-directory: ./src/lr-launcher-e2e
  env:
    npm_config_env: ${{ matrix.platform }}
  run: npm test
```

Here, the `npm test` command is executed within the context of each platform specified by the dynamic matrix. This ensures that our E2E tests are relevant and efficient, automatically adapting to the complexities of multi-platform support.

Here's a look at the relevant part of our `package.json` file:

```json
{
  "name": "lr-launcher",
  "version": "1.0.0",
  "scripts": {
    "pre-test": "node ./utils/setupTest.js --",
    "test": "mocha \"tests/*.test.js\" \"tests/$npm_config_env/*.test.js\" --no-timeouts --retries 1 --exit"
  },
  "dependencies": {
    "axios": "^1.6.2",
    "mocha": "^10.2.0"
  }
}
```

This script utilizes the `npm_config_env` environment variable, which we set dynamically based on the selected platform in our GitHub Actions workflow. By incorporating `$npm_config_env` into the mocha command, we ensure that the test runner picks up not only the general tests from tests/_.test.js but also platform-specific tests located in directories named after each platform (e.g., tests/webos/_.test.js).

This setup allows us to maintain a clear and organized test structure where platform-agnostic tests are separated from those that are platform-specific. It ensures that when our GitHub Actions workflow runs, it only executes the relevant tests for the specified platform, making our testing process both efficient and comprehensive.


Automate cross-platform testing with GitHub Actions, using dynamic matrices for efficient, targeted tests.

GitHub Actions for Dynamic Cross-Platform Testing


I absolutely love using pytest as my go-to testing tool whenever I'm working on a Python project. Among its many features, pytest's parametrization is a standout and has been crucial in simplifying my test cases and maximizing code reusability instead of duplicating tests.
What's more, if it's not clear, parameterized tests are being presented as different tests on each test execution and report.

I have been using it ever since I started using pytest for many different test types, such as unit tests, end-to-end tests, and integration tests.

Let's take a look at some real examples (although I had to make them a little bit more generic) of how to use pytest parametrization.

**Basic Parametrization**

The most straightforward use case is to feed multiple inputs to a single test function. This is useful for testing different edge cases or multiple data points without writing separate test cases.

In the example below we wrote a test function that asserts on the length of a string, while with parametrization we can test multiple strings with a single test function.
We provide test input (for example "hello") and the expected result (in this case 5).

```python
import pytest

@pytest.mark.parametrize("input_string, expected_length", [
    ("hello", 5),
    ("world", 5),
    ("pytest", 6)
])
def test_string_length(input_string, expected_length):
    assert len(input_string) == expected_length
```

**Indirect Parametrization**

With indirect, rather than passing the value directly to the test function, pytest will treat the given input as a fixture name. This allows us to use the fixture's logic to transform or process the input value before passing it to the test. Here, each number is doubled using a fixture before being tested.

```python
import pytest

@pytest.mark.parametrize("input, expected", [
    (2, 4),
    (3, 6),
    (4, 8)
], indirect=["input"], ids=["double-2", "double-3", "double-4"])
def test_double_number(input, expected):
    assert double(input) == expected

@pytest.fixture
def double(input):
    return input * 2
```

**Custom ID for Test Cases**

Sometimes it's beneficial to have custom names for our tests, especially when the default names become unclear or too lengthy. The ids parameter lets us define custom names for our tests. In this example, we use custom IDs to quickly identify tests for even and odd numbers using the `ids` argument that expects a list.

```python
import pytest


@pytest.mark.parametrize("number", [2, 3, 4, 5, 6],
                         ids=["ev-2", "odd-3", "ev-4", "odd-5", "ev-6"])
def test_even_numbers(number):
    is_even = number % 2 == 0
    if "ev" in pytest.current_test().name:
        assert is_even
    else:
        assert not is_even
```

**Parametrization with Multiple Arguments**

Here we demonstrate how to parameterize a test with multiple input arguments. We're testing the addition of two numbers. This is a powerful feature when we want to test a function with various combinations of inputs.

```python
import pytest


@pytest.mark.parametrize("a, b, expected_sum", [
    (1, 2, 3),
    (3, 3, 6),
    (5, 5, 10)
])
def test_addition(a, b, expected_sum):
    assert a + b == expected_sum
```

**Utilizing Parametrization in Asynchronous Tests**

With pytest-asyncio plugin (which is kind of standard these days), we can also parameterize asynchronous tests. In this example, we test an asynchronous function that returns a string after a built-in delay just for the sake of the example.

```python
import pytest
import asyncio

@pytest.mark.parametrize("input_str", ["hello", "world", "pytest"],
                         ids=["hello-async", "world-async", "pytest-async"])
@pytest.mark.asyncio
async def test_string_return_async(input_str):
    returned_str = await async_string_return(input_str)
    assert returned_str == input_str

async def async_string_return(s):
    await asyncio.sleep(0.1)
    return s
```

**Parametrization with Dynamic Fixtures**

The following test demonstrates a more complex usage of pytest's parametrization. This is especially the case when you have a dynamic fixture (like `dynamic_execution_payload`) which depends on the parameter passed (`path_arg` in this example).

```python
import pytest


BASE_URL = "http://example.com/api"


@pytest.fixture(scope="function")
def payload_for_execution(file_path):
    return {
        'repo': "demo-repo",
        'ref': "demo-branch",
        "paths": [file_path]
    }


@pytest.mark.parametrize('file_path', [
    'mock/testDir/file4.js',
    'mock/testDir/file3.js',
    'mock/testDir/file1.js',
    'mock/testDir/file2.js'
])
def test_file_processing_with_dynamic_payload(file_path, payload_for_execution):
    execution = requests.post(url=f"{BASE_URL}/startExecution",
                              headers={"Content-Type": "application/json"},
                              json=payload_for_execution)
    execution_id = execution.json()['executionId']

    response = requests.get(url=f"{BASE_URL}/getExecution/{execution_id}",
                            headers={"Content-Type": "application/json"})
    response_json = response.json()

    assert 'result' in response_json, "Response JSON does not have the 'result' key"
```

First we specified different file paths as inputs for the test function. For each of these paths.
With it we utilize the fixture `payload_for_execution` that generates a payload dynamically based on the `file_path` given and this payload is then used in our test. This demonstrates how you can use the parameters of your test in your fixtures, making your tests more dynamic and flexible.

Within the test, an HTTP POST request is made to presumably start some kind of processing for the file specified in the payload. This request returns an execution ID. A subsequent GET request fetches the result of this execution. Finally, assertions are performed on this result.

**Parametrization with Using Dates**

Dates in every language can be a true pain, therefore its important we tests them thoroughly.

```python
import pytest

from datetime import datetime, timedelta


@pytest.mark.parametrize(
    "input_date, expected_query",
    [
        (datetime(2023, 3, 15), "?start=2023-03-01&end=2023-04-01&category=premium"),
        (datetime(2023, 12, 31), "?start=2023-12-01&end=2024-01-01&category=premium"),
        (datetime(2023, 1, 1), "?start=2023-01-01&end=2023-02-01&category=premium"),
        (datetime(2023, 2, 28), "?start=2023-02-01&end=2023-03-01&category=premium"),
    ],
    ids=["mid-March", "end-of-year", "start-of-year", "end-of-February"]
)
def test_generate_query_based_on_date(mocker, input_date, expected_query):
    mocker.patch("module_under_test.datetime", MockDateTime)
    MockDateTime.now = mocker.Mock(return_value=input_date)
    assert generate_query_for_service(category='premium') == expected_query

def generate_query_for_service(category):
    now = datetime.now()
    start_of_month = now.replace(day=1)
    end_of_month = (start_of_month + timedelta(days=32)).replace(day=1)
    return f"?start={start_of_month.strftime('%Y-%m-%d')}&end={end_of_month.strftime('%Y-%m-%d')}&category={category}"
```

First we provide different datetime objects as inputs. For each datetime, there's a corresponding expected output. This is a classic example of table-driven testing. But in many scenarios, functions might fetch the current date and time using something like `datetime.now()` which can become tricky for testing, thats why you want to control what "now" means so you can predict the output. The `mocker.patch` (using pytest-mock plugin) line mocks the return value of `datetime.now()` to be the input date, allowing for predictable and testable behavior.

The hypothetical function `generate_query_for_service` is responsible for generating a query based on the current date and a given category. It's logic is simplified for this demonstration.

With this structure, you're dynamically testing the function with multiple dates, ensuring it works across various boundary conditions like month ends, year starts, etc.

**Parametrization for Testing Common Web Components**

There are some cases where you have components that are used in multiple places in your application. For example, you might have a navigation bar component that displays on all of your website pages. So you might want to test that component in isolation, but also test it in the context of different pages, this can be done easily with parametrization.

```python
import pytest


test_data = [
    ('https://www.kazis.dev/', "homepage"),
    ('https://www.kazis.dev/about', "about page"),
    ('https://www.kazis.dev/blogs', "blogs page"),
    ('https://www.kazis.dev/blogs/python-type-checking', "blog page")
]

@pytest.mark.parametrize("test_url, test_id", test_data, ids=[item[1] for item in test_data])
def test_logo_element_exists(sync_page, test_url, test_id):
    home_page = NavBar(sync_page)
    home_page.open(test_url)
    link = sync_page.locator('role=link >> text=Kazis Dev Blog Kazi\'s Dev Blog')
    assert link, "Link not found."
```

So in this example we're using Playwright test framework, but the same concept can be applied to any other framework. We have a list of tuples, each tuple contains a URL and a test ID. The test ID is used to identify the test in the report and display it in every test execution.
We then use the `parametrize` decorator to pass the URL and test ID to the test function. We also use the `ids` argument to specify the test ID for each test case.

And in the test function itself we assert that the logo element exists on each page that we're providing as an input. This test will run for each URL in the list and it's a classic example how we can test simple functionality of a web component that doesn't change on different pages.

To conclude,
Pytest parametrization is a powerful feature that reduces redundancy, improves test clarity, and enhances the reusability of test code. These are just a few examples showcasing its capabilities. Using parametrization, you can write more concise and robust tests, and that's without mentioning other out of the box plugins which support pytest.


I absolutely love using pytest as my go-to testing tool whenever i'm working on a Python project. Pytest Parametrization is one of reasons why.

Powerful Pytest Parametrization


While working on a project at work, I delved into unit tests with Sinon v8. I discovered that unit tests using Sinon can sometimes pose challenges, so here's a glimpse of what I learned.
For this project I used Mocha as a test runner, Chai for assertions, Sinon for mocking and Proxyquire for sandboxing.

This is our demo app code that we will be using for this article. It is a simple Node.js application that manipulates files on the file system.

```js
const {
  readdir,
  readdirSync,
  readFileSync,
  unlinkSync,
  writeFileSync,
} = require('fs')
const util = require('util')

module.exports = {
  createFile: filename => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    return writeFileSync(`./data/${filename}`, '', { flag: 'wx' })
  },
  createFileInjected: (filename, fs) => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    return fs.writeFileSync(`./data/${filename}`, '', { flag: 'wx' })
  },
  createFileSafe: filename => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    try {
      const result = writeFileSync(`./data/${filename}`, '', { flag: 'wx' })
    } catch (error) {
      const files = readdirSync('./data')
      const [name, extension] = filename.split('.')

      let max =
        files
          .filter(f => f.match(/test[1-9]/))
          .map(f => Number(f.replace(name, '').replace(`.${extension}`, '')))
          .sort()
          .pop() || 0

      const newName = `${name}${++max}.${extension}`
      writeFileSync(`./data/${newName}`, '', { flag: 'wx' })
    }
  },
  deleteFile: filename => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    return unlinkSync(`./data/${filename}`)
  },
  getFile: filename => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    return readFileSync(`./data/${filename}`)
  },
  getAllFiles: cb => {
    readdir('./data', cb)
  },
  getAllFilesPromise: () => {
    const readPromise = util.promisify(readdir)
    return readPromise('./data')
  },
  saveFile: (filename, contents) => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    return writeFileSync(`./data/${filename}`, contents)
  },
}
```

### Spies

Spies are a good choice for monitoring functions - such as tracking logs, check if a function is being called, etc. You can think about it as Spies are good at answering questions about your methods.

Consider this potentially misleading use case for a spy:

```js
describe('File Management', () => {
  afterEach(() => {
    sinon.restore()
  })

  it('Should create a new file', () => {
    const writeSpy = sinon.spy(fs, 'writeFileSync')
    const fileManagement = proxyquire('./file.management', { fs })

    fileManagement.createFile('test.txt')
    expect(writeSpy.calledWith('./data/test.txt', '')).to.be.true
  })
})
```

In the given example, the test might pass once but fail on subsequent runs due to an existing `test.txt` file, causing an exception. Spies don't always prevent such side effects. In situations like these, Stubs can be a better choice.

### Stubs

Stubs ensure safety during testing. They can manipulate functions and handle side effects like calling APIs or interacting with third-party libraries.
The common stub approaches are side effects like returning a specific values (basically override the original function return value), throw exceptions, resolve or reject promises to understand the function behavior according to real use cases, or call a separate fake function in order to keep the function under test isolated.
There are also other common use cases such as saving time in your tests if you have function with longer execution time.

Using our previous example but with a stub:

```js
describe('File Management', () => {
  afterEach(() => {
    sinon.restore()
  })

  it('Should create a new file', () => {
    const writeStub = sinon.stub(fs, 'writeFileSync')
    const fileManagement = proxyquire('./file.management', { fs })

    fileManagement.createFile('test.txt')

    expect(writeStub.callCount).to.equal(1)
  })
})
```

Here, no actual file is created since we override the `writeFileSync` function, making repeated test runs consistent. It's due to the fact that we override the `writeFileSync` function using a stub. Meaning that unlike the spy use case, I can now run the test multiple times and it won’t fail after the first execution.

But what if we actually want to check that in the use case of file already exists an exception is raised?

We can modify our stub for the dependent `writeFileSync` function to throw an exception and then check that createFile function is behaving as expected.

such as follows:

```js
it('Should throw an exception if file already exists', () => {
  const writeStub = sinon.stub(fs, 'writeFileSync')
  writeStub.throws(new Error())
  const fileManagement = proxyquire('./file.management', { fs })

  expect(() => fileManagement.createFile('test.txt')).to.throw()
})
```

But in most production cases, you’d like to throw an exception based on the actual use case, not every time.
So we can configure our stub to throw exception only in a specific case (which is our test case) using the `withArgs` Sinon function.

```js
it('createFileSafe should create a file named test1 when test file already exists', () => {
  const writeStub = sinon.stub(fs, 'writeFileSync')
  const readStub = sinon.stub(fs, 'readdirSync')

  const fileManagement = proxyquire('./file.management', { fs })

  writeStub.withArgs('./data/test.txt').throws(new Error())
  writeStub.returns(undefined)
  readStub.returns(['test.txt'])
})
```

You can see that in this case we throw an exception only when the function is being called with this specific argument of test file ("./data/test.txt").

Another common example, is to check what happens if `createFile` function is being called without required argument.

```js
it('Should not create a new file if no name is specified', () => {
  const writeStub = sinon.stub(fs, 'writeFileSync')
  const fileManagement = proxyquire('./file.management', { fs })

  try {
    fileManagement.createFile()
  } catch (err) {}
  expect(writeStub.notCalled).to.be.true
})
```

Thing we need to take into account is if the function throws an exception, we have to wrap the call in the function of our test with a try-catch block and then make the assertion.

Now let's explore an Async use case.

Async functions bring a bit of complexity, so understanding JS callbacks helps. In Node, callbacks typically have an error as their first parameter and the return value as the second.

```js
;(Err, data) => {}
```

So in such cases you can emulate a returned callback using Sinon's `yields` function.

```js
it('getAllFiles should return a list of files', () => {
  const readStub = sinon.stub(fs, 'readdir')
  const fileManagement = proxyquire('./file.management', { fs })
  readStub.yields(null, ['test.txt'])

  fileManagement.getAllFiles((err, data) => {
    expect(data).to.eql(['test.txt'])
  })
})
```

Couple of things to note;

In the test we should make the assertion inside a callback.

The stub yields null as the err parameter, and the array with the file name as the data of the callback.

While yield is nice to emulate callbacks, a lot of applications have moved to promises instead, so let’s see how we can deal with promises as well.

```js
it('getAllFilesPromise should return a list of files', () => {
  const readStub = sinon.stub(fs, 'readdir')

  const fileManagement = proxyquire('./file.management', { fs })
  readStub.resolves(['test.txt'])

  return fileManagement
    .getAllFilesPromise()
    .then(files => expect(files).to.eql(['test.txt']))
})
```

As you can see, you can just tell the stub to resolve or reject the promise using the data you’d expect.

There is another important use case with Sinon, which is using `callsFake`.
callsFake function usually return a single argument, but unlike `returns` and `yields`, it is used to return a function.

Why would I want to return a function? One example, is if I have some logic I want to execute.

For example, here I’ll use the spread operator to gain access in the stub to the arguments:

```js
const writeStub = sinon.stub(fs, “writeFileSync”).callsFake((...args) => {});
```

The receiving arguments in this case are `“./data/test.txt”`, `""`, `{ flag: “wx” }`.

You can also add behavior to a function (although I think it is less common) using the `addBehavior` function.

```js
sinon.addBehavior(“logsCall”, (fake, msg) => {
	console.log(msg)’
});
```

Then you can use this new defined behavior as a method on your stub.

But two important notes on `addBehavior`; Behaviors must be defined before the stubs, but they can be also chained.

So you want, you can do something like this:

```js
myStub.logsCall(“called”).returns(“hi”);
```

### Fakes

Unlike Stubs, Fakes are immutable and don't replace the default function behavior automatically.

So let’s see the differences.

Replacing a Spy with a Fake:

```js
it('Should create a new file', () => {
  const writeFake = sinon.fake(fs.writeFileSync)
  sinon.replace(fs, 'writeFileSync', writeFake)
  const fileManagement = proxyquire('./file.management', { fs })

  fileManagement.createFile('test.txt')

  expect(writeFake.calledWith('./data/test.txt', '')).to.be.true
})
```

Note that unlike stub, fake receives one argument (which in our case it is the `module.function`).
Also, note that I had to specify (using Sinon’s replace function) what will be the behavior or else it will act as the default behavior.
The `replace` function takes 3 arguments (module, function and the return fake).

In the case above, the test will pass but will fail after another execution due to the same issue we had with our Spy (text.txt) file is created.

Replacing a Stub with a Fake:

One important concept, is instead of faking the function (as you do with Stubs), you use the property, not the function, then using Sinon’s replace function, override the default behavior.

```js
it('Should throw an exception if file already exists', () => {
  const writeFake = sinon.fake.throws(new Error())
  sinon.replace(fs, 'writeFileSync', writeFake)
  const fileManagement = proxyquire('./file.management', { fs })

  expect(() => fileManagement.createFile('test.txt')).to.throw()
})
```

Now let’s do the same for an async function:

```js
it('getAllFiles should return a list of files', () => {
  const readFake = sinon.fake.yields(null, ['test.txt'])
  sinon.replace(fs, 'readdir', readFake)
  const fileManagement = proxyquire('./file.management', { fs })

  fileManagement.getAllFiles((err, data) => {
    expect(data).to.eql(['test.txt'])
  })
})
```

I had to chain the behavior or else the test will fail.

Most of the use cases which using Fakes are usually when you only need one behavior (not based on things like different arguments, etc).

### Mocks

Basically, we do not have re-define what a mock is because it’s pretty strait forward, although it’s a matter of practice. With mocks you first define what to expect and then assert on it.

If we compare it to Stubbing, you’ll need to do something like this:

```js
// stub
myStub.returns(1);
Const result = myStub();
expect(result).to.eql(1);
```

With mocks however it’s almost inverted, for example:

```js
// mock
expect(myStub).to.return(1) // setting expectation first
myStub() // then calls the function
```

Let’s use one of our “real” example as above:

```js
it('Should call writeFileSync when creating a file', () => {
  const writeMock = sinon.mock(fs)
  writeMock.expects('writeFileSync').once().withArgs('./data/test.txt', '')
  const fileManagement = proxyquire('./file.management', { fs })

  fileManagement.createFile('test.txt', '')

  writeMock.verify()
})
```

You can note that I’m mocking an entire module (`fs` module in this case).
I’m setting expectation first using the `expects` function and then asserts using the `verify` function.
Same as stubs, the test will pass but the test.txt file won’t be created and that’s because the mock is intercepting the function under test just like a stub does.

Let's take a look at another test case:

```js
it('createFileSafe should created a new file with a number appended', () => {
  const writeMock = sinon.mock(fs)
  writeMock.expects('writeFileSync').withArgs('./data/test.txt').throws()
  writeMock.expects('writeFileSync').withArgs('./data/test1.txt').once()
  writeMock.expects('readdirSync').returns(['test.txt']).once()
  const fileManagement = proxyquire('./file.management', { fs })

  fileManagement.createFileSafe('test.txt')

  writeMock.verify()
})
```

What if we want to check a function was NOT called?

```js
it('createFile should never call writeFileSync when the file is empty', () => {
  const writeMock = sinon.mock(fs)
  writeMock.expects('writeFileSync').never()

  const fileManagement = proxyquire('./file.management', { fs })

  try {
    fileManagement.createFile()
  } catch (err) {}

  writeMock.verify()
})
```

As you can see, it’s pretty strait forward, you can just utilize the `never()` function.

### To conclude

**Spies** are great for answering questions such as how many times is this function called, with which argument, etc.
It’s considered as bad practice to try and control a function.

**Stubs** is like a stunt double - It keeps us safe from all the side effects of the functions under test like exceptions, calls for APIs or a third party application/libraries, and in any case of forcing a behavior like throwing exception, it’s the stub that does it, not the actual function under test.

**Fake** is a method to combine the two together, they ask questions AND defines behavior.
It has the ability to set a behavior, record arguments, return values, emulates exceptions and more.

Unlike Stubs, Fakes are immutable, and do not replace automatically the original functions behavior.
Unlike stubs, you cannot modify its behavior based on received args or the number of times it’s called, and there are couple of missing functions that are commonly being used with stubs such as `callsFake()`, `callsThrough()` and `addBehavior()`.

**Mocks** - With mocks you first define what to expect and then assert on it and usually you mock an entire module instead of a specific function.

Unlike Fakes, Mocks do contains all of the Spies and Stubs function, with some additional function on top of it (like `never()`, `atLeast()` AND `atMost()`).

As you can see there’s a lot of commonality in all those test doubles, but also some uniqueness in each one of them.

_Code examples were taken from [Nate Taylor PluralSight Course on Mocking Node JS with Sinon](https://app.pluralsight.com/library/courses/mocking-nodejs-sinon). Highly recommended!_


Unit testing and mocking in Node.js with Sinon, get familiar with basic Spies, Stubs, Fakes, and Mocks, concepts.

Explore Mocking In Node.js Using Sinon


When it comes to testing in Python, pytest stands out as an excellent framework with its rich feature-set that includes detailed info on failing assert statements, auto-discovery of test modules, and advanced fixture management. While working on unit testing for one of our python services at work I wanted to test some scenarios that involves some more critical logging for debugging at real time.

Taking a look at [pytest official docs](https://docs.pytest.org/en/7.1.x/how-to/logging.html) I got to know the `caplog` fixture. I had many cases before where I actually found better solutions from built-in fixtures, but this time I was amazed by how easy it was to use and how much it can do. And it's safe to say I never looked back when it's comes to logging capturing in pytest.

**So what is caplog?**

In pytest, caplog is a fixture provided by pytest, which captures log messages emitted by the logging module. This allows you to validate that your application is logging the right messages or to debug a test by capturing and examining the log output and the log level.

Let's start with a concrete test case to illustrate caplog's usage:

```python
import pytest
import logging

def test_logging(caplog):
    logging.getLogger().info("Test logging")
    assert "Test logging" in caplog.text
```

In the test above, we import the caplog fixture from pytest. We then log an info message and assert that the message is present in the captured log text.

While caplog is great for checking that a specific log message was output, it can do a lot more. You can use it to validate the level of logging messages, check the logger that issued them, and even change the log level for the duration of the test.

Here's an example where we're doing more advanced checks:

```python
def test_logging_advanced(caplog):
    caplog.set_level(logging.INFO)
    logger = logging.getLogger("TestLogger")
    logger.warning("This is a warning")

    assert "This is a warning" in caplog.text
    assert caplog.records[0].levelname == "WARNING"
    assert caplog.records[0].name == "TestLogger"
```

In the example above, we first set the logging level to INFO, then log a warning message. Afterwards, we assert that the message is in the captured log, but also that the level of the message is WARNING and that it came from the logger "TestLogger".

You can also assert messages from specific loggers:

```python
def test_logging_from_specific_logger(caplog):
    logger = logging.getLogger('MyLogger')
    logger.error("An error message")
    assert "An error message" in caplog.text
    assert caplog.records[0].name == "MyLogger"
```

In this test, we create a logger with the name 'MyLogger', emit an error message, and then confirm that the message was logged by that specific logger.

In some cases, you may want to clear the messages captured by caplog. This is particularly useful in parameterized tests or when a single test function is checking multiple scenarios.

```python
def test_clearing_log_messages(caplog):
    logger = logging.getLogger()
    logger.info("First message")
    caplog.clear()

    logger.info("Second message")
    assert "First message" not in caplog.text
    assert "Second message" in caplog.text
```

Here, after logging the "First message", we clear the caplog. Then we log the "Second message" and check that the first message is not in the captured log, while the second message is.

There are some cases where the tested function or method may produce multiple logs. In such cases, you can use the caplog.records list to check the log messages. This list contains all the log records captured by caplog, and you can use it to check the log messages, levels, and loggers.

```python
import pytest
import logging

def function_that_logs():
    logger = logging.getLogger()
    logger.warning("This is a warning message")
    logger.error("An error has occurred!")
    logger.info("Completed operation successfully.")

def test_log_capturing(caplog):
    function_that_logs()
    log_messages = [record.message for record in caplog.records]
    assert "This is a warning message" in log_messages
    assert "An error has occurred!" in log_messages
    assert "Completed operation successfully." in log_messages
```

In this example, the function_that_logs() generates multiple log messages of different levels: WARNING, ERROR, and INFO.

Our test test_log_capturing() invokes function_that_logs(), captures the log messages into a list, and then verifies each of the expected log messages is present in the captured logs.

Note that caplog by default captures log messages of WARNING level or higher. So if you need to capture lower-level messages (e.g. INFO or DEBUG), you should set the logging level in caplog to a lower value before the function call.

It's important to note, that such tests as the example above, you are asserting that the messages are present somewhere in the log, regardless of their position. This is done by checking that the expected message is in the list of log messages, which will return True if the expected message is found anywhere in the list.

If you want to assert on the order of the log messages, then you would need to check the messages based on their index in the log_messages list. For example, if you want to check that the first logged message is "This is a warning message", you could do:

```python
assert log_messages[0] == "This is a warning message"
```

This will only pass if "This is a warning message" is the first message in the log. However, remember that relying on the order of log messages can make your test more brittle, as changes to the order of logging in the code would cause the test to fail. In general, it's better to check that the expected messages are present somewhere in the log, unless the order is particularly important.

That's about it, easy, right?
By using pytest's caplog fixture, you can create robust tests that not only verify your application's behavior, but also confirm it's producing the right log messages. This can be invaluable when diagnosing issues in production, as it ensures your logging is accurate.


Pytest's built-in caplog fixture to capture log messages is nothing but amazing. Capture logs for unit testing can't get easier

Capturing Logs with Pytest Can't Be Easier


I'll start with a confession - I'm fairly new to Typescript. Still learning how to work with it. I'm loving it so far, but, I have struggled so badly when it came to testing while using the notorious combination of Node, TypeScript & ES Module.
Maybe it's because i'm still learning TS, but I really appreciate tools which requires minimal to zero configuration. I know that's how it works with most tools in the JS/TS echo-system, but that's exactly what Vitest is.

So far, I could work my way with unit testing mostly using Jest, but it took me some time to make things up and running.

So to configure Jest correctly on my project, I had to add and install the following packages:

- jest
- ts-jest
- @types/jest

Then I had to setup a `jest.config.js` file with the following content:

```js
import type { Config } from '@jest/types'

const baseDir = 'some/path'

const config: Config.InitialOptions = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  verbose: true,
}

export default config;
```

If we are going with this minimal example, let's take some Utils class example:

```typescript
export class StringUtils {
  public toUpperCase(arg: string) {
    if (!arg) {
      throw new Error('Invalid Argument')
    }
    return toUpperCase(arg)
  }
}

export function toUpperCase(arg: string) {
  return arg.toUpperCase()
}

export type stringInfo = {
  lowerCase: string
  upperCase: string
  characters: string[]
  length: number
  extraInfo: Object | undefined
}

export function getStringInfo(arg: string): stringInfo {
  return {
    lowerCase: arg.toLowerCase(),
    upperCase: arg.toUpperCase(),
    characters: Array.from(arg),
    length: arg.length,
    extraInfo: {},
  }
}
```

This TypeScript code defines a class StringUtils with a method toUpperCase, which converts a string to uppercase, throwing an error for invalid arguments. The standalone function toUpperCase is also doing the same thing, converting the passed string to uppercase. The getStringInfo function takes a string as input and returns an object containing information about that string, such as its lowercase and uppercase forms, an array of its characters, its length, and an extra information object.

Let's take a look at some unit tests written with Jest:

```typescript
import { StringUtils, getStringInfo, toUpperCase } from './Utils'

describe('utils test suite', () => {
  test.each([
    { input: 'test', expected: 'TEST' },
    { input: 'TEST', expected: 'TEST' },
    { input: 'test1$&*#()^41', expected: 'TEST1$&*#()^41' },
  ])('$input to UpperCase should return $expected', ({ input, expected }) => {
    const result = toUpperCase(input)

    expect(result).toBe(expected)
  })

  test('should return info for valid string', () => {
    const result = getStringInfo('My-String')

    expect(result.extraInfo).toEqual({})
    expect(result.lowerCase).toBe('my-string')
    expect(result.length).toBe(9)

    expect(result.characters).toStrictEqual([
      'M',
      'y',
      '-',
      'S',
      't',
      'r',
      'i',
      'n',
      'g',
    ])
    expect(result.characters).toHaveLength(9)
    expect(result.characters).toContain<string>('-')
  })
})

describe('StringUtils class', () => {
  let sut: StringUtils

  beforeEach(() => {
    sut = new StringUtils()
  })

  test('Sanity - valid string', () => {
    const result = sut.toUpperCase('abc')

    expect(result).toBe('ABC')
  })

  test('Should throw error on invalid argument (empty string) - function', () => {
    function expectError() {
      const result = sut.toUpperCase('')
    }

    expect(expectError).toThrow()
    expect(expectError).toThrowError('Invalid Argument')
  })

  test('Should throw error on invalid argument (empty string) - arrow function', () => {
    expect(() => {
      sut.toUpperCase('')
    }).toThrowError('Invalid Argument')
  })

  test('Should throw error on invalid argument (empty string) - try catch block (workaround)', done => {
    try {
      sut.toUpperCase('')
      done('GetStringInfo should throw error for invalid arg')
    } catch (error) {
      expect(error).toBeInstanceOf(Error)
      expect(error).toHaveProperty('message', 'Invalid Argument')
      done()
    }
  })
})
```

What if I want to switch to Vitest Now? Well, I'll have to install the following package:

- vitest

Yep, that's it.

A project using TS & ES Module will work out of the box with Vitest. No need to configure anything.

Let's take a look at the same tests written with Vitest:

```typescript
import { test, describe, expect, beforeEach } from 'vitest'

import { StringUtils, getStringInfo, toUpperCase } from './Utils'

describe('utils test suite', () => {
  test.each([
    { input: 'test', expected: 'TEST' },
    { input: 'TEST', expected: 'TEST' },
    { input: 'test1$&*#()^41', expected: 'TEST1$&*#()^41' },
  ])('$input to UpperCase should return $expected', ({ input, expected }) => {
    const result = toUpperCase(input)

    expect(result).toBe(expected)
  })

  test('should return info for valid string', () => {
    const result = getStringInfo('My-String')

    expect(result.extraInfo).toEqual({})
    expect(result.lowerCase).toBe('my-string')
    expect(result.length).toBe(9)

    expect(result.characters).toStrictEqual([
      'M',
      'y',
      '-',
      'S',
      't',
      'r',
      'i',
      'n',
      'g',
    ])
    expect(result.characters).toHaveLength(9)
    expect(result.characters).toContain<string>('-')
  })
})

describe('StringUtils class', () => {
  let sut: StringUtils

  beforeEach(() => {
    sut = new StringUtils()
  })

  test('Sanity - valid string', () => {
    const result = sut.toUpperCase('abc')

    expect(result).toBe('ABC')
  })

  test('Should throw error on invalid argument (empty string) - function', () => {
    function expectError() {
      const result = sut.toUpperCase('')
    }

    expect(expectError).toThrow()
    expect(expectError).toThrowError('Invalid Argument')
  })

  test('Should throw error on invalid argument (empty string) - arrow function', () => {
    expect(() => {
      sut.toUpperCase('')
    }).toThrowError('Invalid Argument')
  })

  test('Should throw error on invalid argument (empty string)', async () => {
    let error
    try {
      sut.toUpperCase('')
    } catch (e) {
      error = e
    }
    expect(error).toBeInstanceOf(Error)
    expect(error).toHaveProperty('message', 'Invalid Argument')
  })
})
```

The code may look quite similar to Jest because Vitest intelligently adopts the Jest APIs, thereby reducing the learning curve for those transitioning from Jest. This is one of the many reasons why it feels so familiar and yet so refreshingly simple.

To wrap things up, transitioning from Jest to Vitest for a TypeScript project was a refreshing experience for me. With its familiar API and almost non-existent setup, Vitest feels like a game changer, especially for a newcomer to TypeScript like me. And of course, this is without mentioning other features a abilities of Vitest like:
Executing tests in parallel, running tests in watch mode, running tests in debug mode, it's much faster than Jest, and more.

If you're in a similar boat, I encourage you to give Vitest a shot.


I have struggled so badly when it came to testing while using the notorious combination of Node, TypeScript & ES Module. And then I found Vitest

I Wish I Knew Vitest When I Started My TypeScript Journey


Cypress.io offers an intuitive approach to REST API testing, primarily due to its user-friendly design and well-structured documentation. The platform supports JavaScript, which simplifies the learning curve for developers already familiar with it. Additionally, Cypress.io's real-time reloading feature allows for efficient test development, as developers can observe test results instantaneously without manually refreshing the test runner.

Furthermore, the platform's robust API enables seamless handling of network requests and responses, providing developers with comprehensive tools to test and validate various REST API endpoints. By incorporating Cypress.io's built-in assertions and custom commands, developers can create test suites with enhanced readability and maintainability.

But, Let's take it the practical way - It's better to start with some examples.

So for the sake of the example, let's create a small express application with some register and login endpoints, install a data validation library (Joi), and create some basic user model.

Source code can be found in this [repository](https://github.com/Kazaz-Or/automation-projects/tree/develop/APITestingUsingCypress).

So i'll start with the data validation Schemas, which are quite straightforward. First, create a `dataValidation.js` file in your root directory of your project. As you can tell, i used `Joi` library (`npm install joi`).

```javascript
const Joi = require('@hapi/joi')

class DataValidation {
  static registerValidation(request) {
    const validationSchema = Joi.object({
      name: Joi.string().min(2).required(),
      email: Joi.string().min(6).required().email(),
      password: Joi.string().min(8).required(),
    })
    return validationSchema.validate(request)
  }

  static loginValidation(request) {
    const validationSchema = Joi.object({
      email: Joi.string().min(6).required().email(),
      password: Joi.string().min(8).required(),
    })
    return validationSchema.validate(request)
  }
}

module.exports = { DataValidation }
```

Next, Let's create a user model Scheme using `mongoose` (can be installed with `npm install mongoose`). Create a file called Users.js under model directory.

```javascript
const mongoose = require('mongoose')

const userSchema = new mongoose.Schema({
  name: {
    type: String,
    required: true,
    min: 6,
    max: 255,
  },
  email: {
    type: String,
    required: true,
    min: 6,
    max: 255,
  },
  password: {
    type: String,
    required: true,
    min: 6,
    max: 1000,
  },
  date: {
    type: Date,
    default: Date.now,
  },
})

module.exports = mongoose.model('User', userSchema)
```

Finally, we can create are endpoints under routes directory. Note, that I used `express` and `bcryptjs` here.
Now, create a file named auth.js under routes directory, and let's create two routes - once for registration, and one for login.

```javascript
const router = require('express').Router()
const User = require('../model/Users')
const { DataValidation } = require('../dataValidation')
const bcrypt = require('bcryptjs')
const Users = require('../model/Users')

router.post('/register', async (req, res) => {
  const { error } = DataValidation.registerValidation(req.body)
  if (error) return res.status(400).send(error.details[0].message)

  const emailExists = await User.findOne({ email: req.body.email })
  if (emailExists) return res.status(400).send('Email already exists')

  const encryptedPassword = await bcrypt.hash(req.body.password, 10)

  const user = new User({
    name: req.body.name,
    email: req.body.email,
    password: encryptedPassword,
  })
  try {
    const savedUser = await user.save()
    res.send(savedUser)
  } catch (err) {
    res.status(400).send(err)
  }
})

router.post('/login', async (req, res) => {
  const { error } = DataValidation.loginValidation(req.body)
  if (error) return res.status(400).send(error.details[0].message)

  const userExists = await Users.findOne({ email: req.body.email })
  if (!userExists) return res.status(400).send('Email not found')

  const validPassword = await bcrypt.compare(
    req.body.password,
    userExists.password
  )
  if (!validPassword) return res.status(401).send('Invalid password')

  res.status(200).send('Logged in successfully')
})

module.exports = router
```

If you want to make sure everything works correctly, simply start the application `npm start`, and you can just send requests to both endpoints by using cURL.

```bash
register endpoint:

curl -d '{"name": "Kazi Tester", "email": "kazi@dev.io", "password": "simplepass"}' -H "Content-Type: application/json" http://localhost:3000/api/user/register

login endpoint:

curl -d '{"email": "kazi@dev.io", "password": "simplepass"}' -H "Content-Type: application/json" http://localhost:3000/api/user/login
```

## And now let's get to testing!

First, Let's install Cypress `npm install cypress`.

Once done, under e2e directory, let create two files, one for each route - `login.spec.cy.js` and `register.spec.cy.js`.

Starting with the register endpoint:

I installed the `guid` library just for the sake of easy randomization of users creation (of course, this method shouldn't be used in a real production testing environment, this is just for the sake of example).

So I guess this route is quite basic, therefore, let's write some sanity tests first which will assert (or more precise expect), on status code and correct response.

```javascript
const registerEndpoint = 'http://localhost:3000/api/user/register'
const Guid = require('guid')

describe('User Registration - Sanity', () => {
  it('User Registration - Status Code 200', () => {
    let dynamicEmail = Guid.raw() + '@example.com'
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      body: {
        name: 'tester',
        email: dynamicEmail,
        password: '12345678',
      },
    }).then(response => {
      expect(response.status).to.equal(200)
    })
  })

  it('User Registration - Response', () => {
    let dynamicEmail = Guid.raw() + '@example.com'
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      body: {
        name: 'tester',
        email: dynamicEmail,
        password: '12345678',
      },
    }).then(response => {
      expect(response.body.name).to.equal('tester')
      expect(response.body.email).to.equal(dynamicEmail)
      expect(response.body.password).to.not.equal('123456')
    })
  })
})
```

The first test checks if the response status code is 200 (OK) when a user registration request is made with valid data. A unique email address is generated using the Guid library, and a POST request is made to the registration endpoint with a predefined name, the dynamic email, and a password.

The second test, verifies if the response contains the correct name, email, and password fields. The test checks if the name matches the provided input, the email address matches the generated dynamic email, and the password does not match the plain-text password provided in the request. This test ensures that the registration API is functioning correctly and returning the expected results.

Let's move on to some schema validation tests.

```javascript
describe('User Registration - Schema Fields Validation', () => {
  it('User Registration missing name field', () => {
    let dynamicEmail = Guid.raw() + '@example.com'
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
      body: {
        email: dynamicEmail,
        password: '12345678',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('"name" is required')
    })
  })

  it('User Registration missing email field', () => {
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
      body: {
        name: 'tester',
        password: '12345678',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('"email" is required')
    })
  })

  it('User Registration missing password field', () => {
    let dynamicEmail = Guid.raw() + '@example.com'
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
      body: {
        name: 'tester',
        email: dynamicEmail,
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('"password" is required')
    })
  })

  it('User Registration name field under minimum chars', () => {
    let dynamicEmail = Guid.raw() + '@example.com'
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
      body: {
        name: 't',
        email: dynamicEmail,
        password: '12345678',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal(
        '"name" length must be at least 2 characters long'
      )
    })
  })

  it('User Registration email field invalid email', () => {
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
      body: {
        name: 'test-user',
        email: 'tester.example.com@',
        password: '12345678',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('"email" must be a valid email')
    })
  })

  it('User Registration password field under minimum chars', () => {
    let dynamicEmail = Guid.raw() + '@example.com'
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
      body: {
        name: 'test-user',
        email: dynamicEmail,
        password: '1234567',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal(
        '"password" length must be at least 8 characters long'
      )
    })
  })

  it('User Registration name field wrong type', () => {
    let dynamicEmail = Guid.raw() + '@example.com'
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
      body: {
        name: 5352,
        email: dynamicEmail,
        password: '1234235967',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('"name" must be a string')
    })
  })

  it('User Registration password field wrong type', () => {
    let dynamicEmail = Guid.raw() + '@example.com'
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
      body: {
        name: '5352user',
        email: dynamicEmail,
        password: 1235674533,
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('"password" must be a string')
    })
  })

  it('User Registration email field wrong type', () => {
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
      body: {
        name: '5352user',
        email: true,
        password: '1235674pass',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('"email" must be a string')
    })
  })
})
```

These tests are a series of schema validation tests. They are focusing on verifying that the API correctly handles and responds to requests with missing or invalid data in the request body.

Each test within the suite targets a specific scenario where a required field is either missing, has an invalid value, or has a value that does not meet the required constraints.

The tests cover the following scenarios:

Missing name field in the request body
Missing email field in the request body
Missing password field in the request body
Name field with fewer than the minimum required characters
Email field with an invalid email address
Password field with fewer than the minimum required characters
Name field with an incorrect data type (number instead of a string)
Password field with an incorrect data type (number instead of a string)
Email field with an incorrect data type (boolean instead of a string)
For each test, a POST request is made to the registration endpoint with the specific invalid data, and the response is checked for the expected status code (400) and an appropriate error message. These tests ensure that the registration API correctly validates the request body schema and provides meaningful error messages for invalid data.

So far so good - Why not create some negative tests cases as well?

```javascript
describe('User Registration - Negative Cases', () => {
  it('User Registration without payload - Response 400', () => {
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('"name" is required')
    })
  })

  it('User Registration - Special chars in name field', () => {
    let dynamicEmail = Guid.raw() + '@example.com'
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      body: {
        name: 'tester!@#$%^&*()-_=+/.><~`  ',
        email: dynamicEmail,
        password: '12345678',
      },
    }).then(response => {
      expect(response.status).to.equal(200)
    })
  })

  it('User Registration - Special chars in password field', () => {
    let dynamicEmail = Guid.raw() + '@example.com'
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      body: {
        name: 'testerspecialchars',
        email: dynamicEmail,
        password: '123456!@#$%^&*()-_=+/.><~`  ',
      },
    }).then(response => {
      expect(response.status).to.equal(200)
    })
  })

  it('User Registration - Registration with an already exists email', () => {
    cy.request({
      method: 'POST',
      url: registerEndpoint,
      failOnStatusCode: false,
      body: {
        name: 'tester',
        email: 'tester@example.com',
        password: '12345678',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('Email already exists')
    })
  })
})
```

This test suite includes a set of negative test cases for the user registration route using Cypress.io. These tests focus on verifying that the API can handle and respond to various unfavorable or edge-case scenarios during user registration.

These negative test cases help ensure that the registration API can properly handle different edge cases and return the appropriate responses for various input scenarios.

![register-spec](https://user-images.githubusercontent.com/83350680/191088312-76a48269-703e-41cf-aee7-5be1963670bb.jpeg)

Let's finish with the login endpoint!

Same steps, starting with some sanity tests first.

```javascript
const loginEndpoint = 'http://localhost:3000/api/user/login'

describe('User Login - Sanity', () => {
  it('User Login - Status Code 200', () => {
    cy.request({
      method: 'POST',
      url: loginEndpoint,
      body: {
        email: 'static@test.io',
        password: 'simplepass',
      },
    }).then(response => {
      expect(response.status).to.equal(200)
    })
  })

  it('User Login - Response', () => {
    cy.request({
      method: 'POST',
      url: loginEndpoint,
      body: {
        email: 'static@test.io',
        password: 'simplepass',
      },
    }).then(response => {
      expect(response.body).to.equal('Logged in successfully')
    })
  })

  it('User Login - Wrong email', () => {
    cy.request({
      method: 'POST',
      url: loginEndpoint,
      failOnStatusCode: false,
      body: {
        email: 'static1@test.io',
        password: 'simplepass',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('Email not found')
    })
  })

  it('User Login - Wrong password', () => {
    cy.request({
      method: 'POST',
      url: loginEndpoint,
      failOnStatusCode: false,
      body: {
        email: 'static@test.io',
        password: 'simplepass1',
      },
    }).then(response => {
      expect(response.status).to.equal(401)
      expect(response.body).to.equal('Invalid password')
    })
  })
})
```

These tests focusing on validating the status code and response content of the API for various login scenarios.
These sanity tests help ensure that the login API can properly handle different scenarios and return the appropriate responses for valid and invalid login attempts.

Then moving on to some edge cases and negative cases.

```javascript
describe('User Login - Error Cases', () => {
  it('User Login - Missing email field', () => {
    cy.request({
      method: 'POST',
      url: loginEndpoint,
      failOnStatusCode: false,
      body: {
        password: 'simplepass',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('"email" is required')
    })
  })

  it('User Login - Missing password field', () => {
    cy.request({
      method: 'POST',
      url: loginEndpoint,
      failOnStatusCode: false,
      body: {
        email: 'static@test.io',
      },
    }).then(response => {
      expect(response.status).to.equal(400)
      expect(response.body).to.equal('"password" is required')
    })
  })
})
```

These tests focusing on verifying that the API can handle and respond to various unfavorable or edge-case scenarios during user login.
These error and edge case tests help ensure that the login API can properly handle different scenarios and return the appropriate responses for various input situations.

![login-spec](https://user-images.githubusercontent.com/83350680/191088279-6bb19b69-6bf0-4354-99c4-08ece316fa43.jpeg)

Not too bad, right?

Don't forget to run the test by running `npm test`!

![test-summary](https://user-images.githubusercontent.com/83350680/191088369-19a49e8a-a24b-4fc0-8142-6b7560f2601e.jpeg)

The source code of the simple express application and the tests, can be found here - [https://github.com/Kazaz-Or/APITestingUsingCypress](https://github.com/Kazaz-Or/automation-projects/tree/develop/APITestingUsingCypress)


Most common use cases of Cypress.io are UI & Visual testing. But, employing Cypress for testing REST APIs can also be quite efficient.

Testing REST API with Cypress is easy


Pytest is a powerful and flexible testing framework for Python that allows developers to easily write and execute test cases for their code. One of the key features of pytest is the ability to use custom markers to organize and categorize tests in efficient way.

Custom markers in pytest are user-defined labels that can be attached to individual tests, test classes, or entire test suites. These markers can be used to provide additional context and information about the purpose and intent of each test, which can help developers better understand the test results and identify issues more quickly.
For those of us who had the chance to write down manual tests - You should be familiar with the above.

There are several advantages of using pytest custom markers in your testing workflow:

**Improved organization and readability:** Custom markers can help organize tests into logical categories, making it easier to navigate and understand complex test suites. For example, you might use markers to group tests by feature, priority, or level of complexity.

**Flexible test selection:** With custom markers, you can selectively run subsets of tests based on their markers. This can be especially useful when you have a large test suite and want to focus on a specific set of tests, such as those related to a particular feature or module.

In a small Test Automation project I had a chance to take part in, at some point we wanted to differentiate between tests that will be executed on each uploaded Pull request, and for heavier test to run in a nightly scheduled job.
Pytest custom markers was such an easy solution.

In your pytest.ini file just use the markers tag and create as many markers as you like.

```python
[pytest]
markers =
    sanity:Run only tests marked as sanity
```

From there you can mark the tests you'd like to tag with the pytest mark decorator.
This is a small test from my project as an example:

```python
@pytest.mark.sanity
def test_leave_room_sanity(self, get_default_parameters, room_id_private_after_join):
    ec_api.leave_room(room_id=room_id_private_after_join, **get_default_parameters)
```

Now if you want to run only the tests with the custom marker you created, simply run pytest command with -m flag and the marker name.

```bash
pytest -m sanity
```

You can also run all the tests that aren't marked with the marker you created by running a reverse command.

```bash
pytest -m "not sanity"
```

There are many more advantages we can leverage using custom markers, such as:

**Customizable reporting**: Pytest's reporting system can be customized to display information based on the markers attached to each test. This allows you to generate more detailed reports that provide additional context and insights into the test results.

**Integration with other tools**: Custom markers can be used to integrate pytest with other tools and systems in your development workflow. For example, you might use markers to trigger automated deployments or trigger additional testing based on the results of certain tests.

Overall, custom markers are a powerful feature of pytest that can help developers write more organized, readable, and flexible tests. By using markers to categorize and label tests, developers can better understand their test results and identify issues more quickly, leading to a more efficient and effective testing process.


This is the easiest way to categorize and maximize the flexibility and readability of your testing workflow and results

Organizing Your Tests with Custom Markers in Pytest


I'll start with a full disclosure - Up until now, I have only written unit tests using Jest. However, at my current job, I had the opportunity to work on a small project that allowed me to choose a new testing library.

After some research, I decided to go with Mocha as a test runner and Chai for assertions. Then I reached the fun part - Mocking and Spying.
The most obvious choice was Sinon, which works great as a spy (for example, if you want to know whether a certain log was printed). However, when it came to Mocking (or Stubbing), it wasn't easy at all.

If you're interested - I found this article back then and I found it to be quite true ([why is it so hard to test in NodeJS](https://www.vinnie.work/blog/2021-09-18-why-so-hard-testing-with-es6-imports))

After struggling with Sinon, I turned to the Discord Node.js community with a specific question:

![discordquestion](/images/discord-node-test.png)

The first response was a link to the [rewire](https://github.com/jhnns/rewire) repository. I must say, once I started using it, I couldn't stop. It made mocking incredibly easy.

To install it simply run:

```
npm install rewire
```

import it:

```js
const rewire = require('rewire')
```

To initialize it, you need to "rewire" the module that contains the functions, methods, or variables you'd like to mock.

```js
const mockedIndex = rewire('../../index')
```

So let's look at some examples:

I'll start with the original question I raised in the Node Discord server.
Here's my original function (just for the sake of the example):

```js
async function updatePolicy(project, policy, file) {
  return Promise.allSettled(Object.entries(file).map(([status]) => {
    try {
      if (!countryRules[country]) {
        return Promise.resolve();
      }
      if (statusDisable(status))
        console.log('disable');
      else if (statusEnable(status))
        console.log('enable');
        [...]
}
```

I wanted a simple test case to check that the correct function is called given the right arguments.
Since I had many test cases like these, I created a more "generic" function that receives the function I'd like to test as an argument.

```js
// rewire the module
const lib = rewire('../../index')

// generic function

const checkFunctionCall = (functionName, done, file) => {
  lib.__set__(`${functionName}IsCalled`, false) // setting the function under test to isCalled false
  lib.__set__(functionName, () => {
    // function under test is called
    lib.__set__(`${functionName}IsCalled`, true) // changing the isCalled to true in this case.
  })
  lib.updatePolicy(project, policy, file) // Calling the function that eventually calls the function i'd like to assert on
  assert.equal(lib.__get__(`${functionName}IsCalled`), true) // asserting that isCalled is true
  done()
}

// test case

it('statusEnable function called', done => {
  let file = 'example argument'

  checkFunctionCall('statusEnable', done, file)
})
```

Doesn't seem so complicated, right?
Moving on to mocking of an async function that should return resolved promise. Let's say this is the original function:

```js
async function persistCatalogs(bucketName, catalogs) {
  const catalogBucket = storageClient.bucket(bucketName)
  return new Promise(async (resolve, reject) => {
    for (const chunk of keyChunks) {
      await Promise.all(
        chunk.map(async catalog =>
          uploadCatalog(catalogBucket, cacheTtl, catalog, catalogs[catalog])
        )
      )
    }
    resolve()
  }).then(() => logger.info('Catalog uploaded!'))
}
```

I'd like to test that given a specific argument, the `uploadCatalog` function was called 12 times.

```js
// stubbed function that will be called when mocking the uploadCatalog function
function uploadCatalogStub() {
  return Promise.resolve('Mocked result')
}
//test case
it('should upload each catalog in chunks', async () => {
  lib.__set__('uploadCatalog', uploadCatalogStub) // rewire will set uploadCatalogStub once uploadCatalog is called
  await lib.persistCatalogs(bucketName, catalogs) // Original function called
  expect(uploadCatalogStub.callCount).to.equal(12) // assertion
})
```

Moving on the mocking a global variable that eventually will be used in a function.

This is the function i'd like to test:

```js
function addEventToCatalog(rawEvent, catalog) {
  rawEvent.countryAvailability.forEach(c => {
    if (COUNTRIES_LIST.includes(c))
      catalog[c].push(event);
    else
      logger.warn(`Unsupported country ${c} on event ${event.assetId}, skipping for catalog`);
  });
```

Now `COUNTRIES_LIST` variable is a global variable, which it's data in it might effect the result of the test. So let's mock it:

```js
lib.__set__('COUNTRIES_LIST', structuredClone(countryCodesData)) //COUNTRIES_LIST is a global variable, here it'll return a clone of another data i chose.
```

and call the tested function is test as usual:

```js
it('should add event to US catalog if countryAvailability includes US', () => {
  lib.addEventToCatalog(rawEvent, catalog)

  expect(catalog.US).to.have.lengthOf(1) // assertion
})
```

In conclusion, `rewire` proved to be the simplest mocking library I've had the opportunity to use in Node.js projects.
Ultimately, pairing it with a dynamic test runner like Mocha made for an easy choice compared to other libraries.

As a result, I ended up replacing all other mocks, such as Sinon Stubbing, with Rewire and haven't looked back since.


Unit testing in Node.js can be challenging. After trying various libraries, I found Rewire to be the most practical for testing purposes

Leverage Rewire for Unit Testing in Node.js


Explaining this article without divulging specific product details may be a challenge, but I'll attempt to simplify it.
The product is an embedded device that streams video packets over IP via multiple network interfaces, such as Ethernet, WiFi, or modems, and then bonds (aka, [bonding](https://en.wikipedia.org/wiki/Bonding_protocol)) them on the server to produce clear video without artifacts.

The company's hardware team developed a new device with improved hardware peripherals to achieve certain objectives that the company's existing devices couldn't. The new device was highly anticipated, and time was of the essence.

The new hardware architecture was based on something called PHY to PHY, which means the NIC of the board and the NIC of the video encoder were directly connected to each other, while the network interface was defined as VLAN to run a "keep-alive" traffic between the two.
This architecture was entirely different than what out other products were designed.

![PHY-TO-PHY](/images/phy-system-diagram.jpg)

This required some testing of course.

The test involved physically connecting the two [PCB's evaluation boards](https://www.tempoautomation.com/blog/pcb-evaluation-vs-development-board-what-is-the-difference/#:~:text=The%20evaluation%20board%20is%20used,pertinent%20components%20and%20connectors%20included) with Ethernet cable (RJ45) and generating UDP/TCP traffic between them while measuring the board's CPU usage, network capability and packet losses.

The most common use case for the product was transmitting 100 Mbps of UDP packets from the encoder card to the main system board (IMX6 at the time). The IMX board was based on ARM architecture, and testing tools such as [iperf](https://iperf.fr/) and [mpstat](https://linux.die.net/man/1/mpstat) were added using the [Buildroot](https://buildroot.org/) toolchain to generate network traffic and monitor CPU usage.

Since this was an internal network connected physically, packet loss was unexpected.

So I started testing using Iperf

From the encoder:

```
iperf –c 0.0.0.1 –I 1 -u –b 100m –t 500 -p 9000
```

On the IMX board (receiving board):

```
iperf -s -l 9000
```

![iperf-test](/images/iperf-test.png)

However, during a five-minute test, there were packet losses. In a video use case, video with losses results in artifacts, as shown in the example below.

![videoartifacts](/images/videoartifacts.png)

Kinda bad right?

The company saved a significant amount of time by letting the QA team test the products on PCB evaluation boards before they were actually ready for testing, resulting in a new hardware revision.
I don't know if your'e aware of the heavy cycle in terms on time and research it takes to fix an issue and make it ready for production in the hardware development world, but i'll give you a hint: It's far more than a software cycle.

The issue was eventually determined to be a hardware problem with the [clock signal](https://en.wikipedia.org/wiki/Clock_signal) on the board.


The test that saved my company a significant amount of time and effort - Using two PCB's, RJ45 cable, and the Iperf tool

Testing Articles

All Testing Articles