
When your Node.js or Electron application needs to interact with the OS, it uses the `child_process` module. While `spawn` is better for large data streams, `exec` is the go-to for command-line utilities because it buffers the output and provides a simple interface.

However, how you consume that interface impacts your application's memory usage, error handling, and maintainability.

Let's look at the three primary ways to handle asynchronous execution.

**The Low-Level Foundation: The Callback Pattern**

The exec function spawns a shell and runs the command within that shell. Internally, Node.js allocates a buffer (defaulting to 1MB) to store the stdout and stderr results.

```js
import { exec } from 'child_process'

const options = {
  cwd: '/path/to/repo', // Current Working Directory
  env: { ...process.env, GIT_TERMINAL_PROMPT: '0' }, // custom env vars
  timeout: 10000, // Kill process if it exceeds 10s
  maxBuffer: 1024 * 1024 * 10, // Extend buffer to 10MB
}

exec('git status --porcelain', options, (error, stdout, stderr) => {
  if (error) {
    // error.code: The exit code of the process (e.g., 128 for Git errors)
    // error.signal: The signal that terminated the process (e.g., SIGTERM)
    console.error(`Process exited with code ${error.code}`)
    return
  }
  // stdout and stderr are strings containing the full output
  console.log(stdout)
})
```

Memory Management: The callback only fires once the process is closed and the entire output is buffered. If a command outputs 2MB and your maxBuffer is 1MB, the callback fires with an ERR_CHILD_PROCESS_STDIO_MAXBUFFER error, and the process is killed.

Execution Context: This pattern is strictly "Push" based. You are handing control to the OS and waiting for a signal to return.

The Problem (Sequential Dependency): If you need to check git status, then git fetch, then git pull, you create a recursive execution context. Each nested exec keeps the previous one in memory (closure), leading to potential memory leaks in long-running processes if not handled carefully.

**Using a Promise Wrapper**

By wrapping `exec` in a new `Promise`, you transition the function from a Push-based callback to a Pull-based awaitable.

```js
function execAsyncManual(command, options = {}) {
  return new Promise((resolve, reject) => {
    // Logic executed immediately upon Promise instantiation
    const child = exec(command, options, (error, stdout, stderr) => {
      if (error) {
        // Enriched Error Object
        reject({
          message: error.message,
          exitCode: error.code,
          command: command,
          stderr: stderr,
        })
      } else {
        // Resolve only on successful exit (code 0)
        resolve({ stdout, stderr })
      }
    })

    // We can even attach listeners to the child process object if needed
    child.on('spawn', () => console.log(`PID ${child.pid} started.`))
  })
}
```

Control Flow: This allows for Try/Catch blocks. In the compiled JavaScript, this transforms your logic into a state machine. Instead of nesting closures, the state machine pauses execution, allowing the Event Loop to process other tasks while waiting for the OS kernel to signal the process completion.

Why Manual? Some CLI tools (like curl or ffmpeg) are "noisy." They might write progress logs to stderr. A manual wrapper lets you write logic like: if (stderr && !stderr.includes('Warning')) reject().

The Trap: If you forget to handle the reject, you will trigger an unhandledRejection event, which in modern Node.js versions, will crash your entire process.

**The Abstraction: Promisify**

Node's `util.promisify` uses a specific internal Symbol (`util.promisify.custom`) or a standard argument-guessing logic to wrap the function for you.

```js
import { exec } from 'child_process'
import { promisify } from 'util'

const execAsync = promisify(exec)

async function gitWorkflow() {
  try {
    // Destructuring the returned object
    const { stdout, stderr } = await execAsync('git pull origin main')

    // Technical Note: If stderr is not empty, it doesn't mean the command failed.
    // promisify only rejects if the error argument in the callback is truthy.
  } catch (err) {
    // err is an instance of Error but with extra properties:
    // err.stdout, err.stderr, err.code, err.signal, err.cmd
    console.error(`Command [${err.cmd}] failed with: ${err.stderr}`)
  }
}
```

Standardization: promisify ensures that the Promise always resolves to an object with { stdout, stderr } keys. This is critical for team-based projects where custom wrappers might return different formats.

Microtask Queue: Using await places the continuation of your function into the Microtask Queue. This is higher priority than the Macrotask Queue (where setTimeout lives), meaning your code resumes almost instantly after the OS process finishes.

Limitations: You lose access to the ChildProcess object (the PID, the .stdin stream, etc.) unless you use the advanced promisify features. You only get the final result.

Important thing to take into account is a common pitfall called Shell Injection.

When you use `exec`, Node.js calls `/bin/sh -c [command]` (on Unix) or `cmd.exe /d /s /c [command]` (on Windows). The shell is an interpreter.

If your code looks like this:

```js
execAsync(`cat ${filename}`)
```

And a user provides:

```bash
"important.txt; rm -rf /"
```

The shell interprets the `;` as a command separator. It finishes the `cat` and then immediately starts the `rm`.

Never pass raw input to `exec`. Use a strict Whitelisting Regex. If you need to handle complex arguments with spaces and special characters, consider using `child_process.spawn` instead, which passes arguments as an array and does not spawn a shell by default, bypassing the injection risk entirely.

---

For 90% of use cases in Electron or Backend services, promisify(exec) is the superior choice for maintainability. However, if you are building a tool that requires real-time monitoring of a process (like a progress bar based on PID stats), the Manual Wrapper or raw Callbacks are necessary to maintain a handle on the ChildProcess instance.


Handling Node shell commands can be tricky. Should you use Callbacks, manual Promises, or an abstraction layer like Promisify?

Executing Shell Commands in Node.js: Callbacks vs Promises vs Promisify


Ensuring consistent application performance across multiple platforms is a cornerstone of quality software delivery. In this post, I'll share my approach for using GitHub Actions to automate testing across a variety of platforms, drawing from a real use case at my workplace.

Our solution utilizes GitHub Actions, a CI/CD service that enables automation of build, test, and deployment workflows directly within GitHub. The highlight of the approach is the dynamic matrix strategy, which handles both scheduled runs and on-demand tests across specified platforms.

**Scheduled and On-demand Testing**

In this example, the workflow is being triggered automatically at 22:00 every day for nightly tests. Additionally, we enabled manual triggers for specific platforms, allowing for targeted testing as needed. This flexibility is crucial for maintaining a fast-paced development cycle without compromising on quality.

```yaml
on:
  schedule:
    - cron: '0 22 * * *'
  workflow_dispatch:
    inputs:
      platform:
        description: 'Choose platform to run the test on'
        required: true
        type: choice
        options:
          - webos
          - tizen
          - playstation4
          - playstation5
          - xbox
```

This configuration leverages the `schedule` and `workflow_dispatch` events in GitHub Actions. The `cron` syntax schedules automated test runs, while `workflow_dispatch` allows for manual execution with platform-specific parameters.

**Dynamic Test Matrix Configuration**

The core of our workflow dynamically adjusts the test matrix based on the context of the trigger—whether it's a scheduled job covering all platforms or a manual job focusing on a specific platform.

```yaml
jobs:
  setup-test:
    runs-on: ubuntu-latest
    outputs:
      platform: ${{ steps.set-matrix.outputs.platform }}
    steps:
      - id: set-matrix
        run: |
          if [ "${{ github.event_name }}" = "schedule" ]; then
            echo 'platform=["webos", "tizen", "playstation4", "playstation5", "xbox"]' >> $GITHUB_ENV
          else
            echo 'platform=["${{ github.event.inputs.platform }}"]' >> $GITHUB_ENV
```

In this snippet, we employ conditional logic to populate the platform variable differently based on the trigger type. For scheduled runs, we test across all platforms, while manual triggers allow for focused testing on a selected platform. This strategy ensures that resources are allocated efficiently, focusing our testing efforts where they are most needed.
Of course, you could always do that with an external script or with a composite action, but this is a simple and effective way to achieve the same result.

**Executing the Tests with Dynamic Configuration**

Our approach to executing tests in this workflow benefits significantly from the dynamic nature of GitHub Actions, especially when it comes to handling multiple platforms which can be quite different from each other. This flexibility is further extended into our application's `package.json` file, where we define specific scripts for our testing needs. The `package.json` plays a crucial role in integrating the GitHub environment variables directly into our test execution process.

Without getting into the tests code, you can imagine that each platform has its own specific tests and configurations, and that's why we need to run the tests separately for each platform. This is where the dynamic matrix comes into play.

```yaml
- name: Run tests
  working-directory: ./src/lr-launcher-e2e
  env:
    npm_config_env: ${{ matrix.platform }}
  run: npm test
```

Here, the `npm test` command is executed within the context of each platform specified by the dynamic matrix. This ensures that our E2E tests are relevant and efficient, automatically adapting to the complexities of multi-platform support.

Here's a look at the relevant part of our `package.json` file:

```json
{
  "name": "lr-launcher",
  "version": "1.0.0",
  "scripts": {
    "pre-test": "node ./utils/setupTest.js --",
    "test": "mocha \"tests/*.test.js\" \"tests/$npm_config_env/*.test.js\" --no-timeouts --retries 1 --exit"
  },
  "dependencies": {
    "axios": "^1.6.2",
    "mocha": "^10.2.0"
  }
}
```

This script utilizes the `npm_config_env` environment variable, which we set dynamically based on the selected platform in our GitHub Actions workflow. By incorporating `$npm_config_env` into the mocha command, we ensure that the test runner picks up not only the general tests from tests/_.test.js but also platform-specific tests located in directories named after each platform (e.g., tests/webos/_.test.js).

This setup allows us to maintain a clear and organized test structure where platform-agnostic tests are separated from those that are platform-specific. It ensures that when our GitHub Actions workflow runs, it only executes the relevant tests for the specified platform, making our testing process both efficient and comprehensive.


Automate cross-platform testing with GitHub Actions, using dynamic matrices for efficient, targeted tests.

GitHub Actions for Dynamic Cross-Platform Testing


While working on a project at work, I delved into unit tests with Sinon v8. I discovered that unit tests using Sinon can sometimes pose challenges, so here's a glimpse of what I learned.
For this project I used Mocha as a test runner, Chai for assertions, Sinon for mocking and Proxyquire for sandboxing.

This is our demo app code that we will be using for this article. It is a simple Node.js application that manipulates files on the file system.

```js
const {
  readdir,
  readdirSync,
  readFileSync,
  unlinkSync,
  writeFileSync,
} = require('fs')
const util = require('util')

module.exports = {
  createFile: filename => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    return writeFileSync(`./data/${filename}`, '', { flag: 'wx' })
  },
  createFileInjected: (filename, fs) => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    return fs.writeFileSync(`./data/${filename}`, '', { flag: 'wx' })
  },
  createFileSafe: filename => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    try {
      const result = writeFileSync(`./data/${filename}`, '', { flag: 'wx' })
    } catch (error) {
      const files = readdirSync('./data')
      const [name, extension] = filename.split('.')

      let max =
        files
          .filter(f => f.match(/test[1-9]/))
          .map(f => Number(f.replace(name, '').replace(`.${extension}`, '')))
          .sort()
          .pop() || 0

      const newName = `${name}${++max}.${extension}`
      writeFileSync(`./data/${newName}`, '', { flag: 'wx' })
    }
  },
  deleteFile: filename => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    return unlinkSync(`./data/${filename}`)
  },
  getFile: filename => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    return readFileSync(`./data/${filename}`)
  },
  getAllFiles: cb => {
    readdir('./data', cb)
  },
  getAllFilesPromise: () => {
    const readPromise = util.promisify(readdir)
    return readPromise('./data')
  },
  saveFile: (filename, contents) => {
    if (!filename) {
      throw new Error('Must supply file name')
    }

    return writeFileSync(`./data/${filename}`, contents)
  },
}
```

### Spies

Spies are a good choice for monitoring functions - such as tracking logs, check if a function is being called, etc. You can think about it as Spies are good at answering questions about your methods.

Consider this potentially misleading use case for a spy:

```js
describe('File Management', () => {
  afterEach(() => {
    sinon.restore()
  })

  it('Should create a new file', () => {
    const writeSpy = sinon.spy(fs, 'writeFileSync')
    const fileManagement = proxyquire('./file.management', { fs })

    fileManagement.createFile('test.txt')
    expect(writeSpy.calledWith('./data/test.txt', '')).to.be.true
  })
})
```

In the given example, the test might pass once but fail on subsequent runs due to an existing `test.txt` file, causing an exception. Spies don't always prevent such side effects. In situations like these, Stubs can be a better choice.

### Stubs

Stubs ensure safety during testing. They can manipulate functions and handle side effects like calling APIs or interacting with third-party libraries.
The common stub approaches are side effects like returning a specific values (basically override the original function return value), throw exceptions, resolve or reject promises to understand the function behavior according to real use cases, or call a separate fake function in order to keep the function under test isolated.
There are also other common use cases such as saving time in your tests if you have function with longer execution time.

Using our previous example but with a stub:

```js
describe('File Management', () => {
  afterEach(() => {
    sinon.restore()
  })

  it('Should create a new file', () => {
    const writeStub = sinon.stub(fs, 'writeFileSync')
    const fileManagement = proxyquire('./file.management', { fs })

    fileManagement.createFile('test.txt')

    expect(writeStub.callCount).to.equal(1)
  })
})
```

Here, no actual file is created since we override the `writeFileSync` function, making repeated test runs consistent. It's due to the fact that we override the `writeFileSync` function using a stub. Meaning that unlike the spy use case, I can now run the test multiple times and it won’t fail after the first execution.

But what if we actually want to check that in the use case of file already exists an exception is raised?

We can modify our stub for the dependent `writeFileSync` function to throw an exception and then check that createFile function is behaving as expected.

such as follows:

```js
it('Should throw an exception if file already exists', () => {
  const writeStub = sinon.stub(fs, 'writeFileSync')
  writeStub.throws(new Error())
  const fileManagement = proxyquire('./file.management', { fs })

  expect(() => fileManagement.createFile('test.txt')).to.throw()
})
```

But in most production cases, you’d like to throw an exception based on the actual use case, not every time.
So we can configure our stub to throw exception only in a specific case (which is our test case) using the `withArgs` Sinon function.

```js
it('createFileSafe should create a file named test1 when test file already exists', () => {
  const writeStub = sinon.stub(fs, 'writeFileSync')
  const readStub = sinon.stub(fs, 'readdirSync')

  const fileManagement = proxyquire('./file.management', { fs })

  writeStub.withArgs('./data/test.txt').throws(new Error())
  writeStub.returns(undefined)
  readStub.returns(['test.txt'])
})
```

You can see that in this case we throw an exception only when the function is being called with this specific argument of test file ("./data/test.txt").

Another common example, is to check what happens if `createFile` function is being called without required argument.

```js
it('Should not create a new file if no name is specified', () => {
  const writeStub = sinon.stub(fs, 'writeFileSync')
  const fileManagement = proxyquire('./file.management', { fs })

  try {
    fileManagement.createFile()
  } catch (err) {}
  expect(writeStub.notCalled).to.be.true
})
```

Thing we need to take into account is if the function throws an exception, we have to wrap the call in the function of our test with a try-catch block and then make the assertion.

Now let's explore an Async use case.

Async functions bring a bit of complexity, so understanding JS callbacks helps. In Node, callbacks typically have an error as their first parameter and the return value as the second.

```js
;(Err, data) => {}
```

So in such cases you can emulate a returned callback using Sinon's `yields` function.

```js
it('getAllFiles should return a list of files', () => {
  const readStub = sinon.stub(fs, 'readdir')
  const fileManagement = proxyquire('./file.management', { fs })
  readStub.yields(null, ['test.txt'])

  fileManagement.getAllFiles((err, data) => {
    expect(data).to.eql(['test.txt'])
  })
})
```

Couple of things to note;

In the test we should make the assertion inside a callback.

The stub yields null as the err parameter, and the array with the file name as the data of the callback.

While yield is nice to emulate callbacks, a lot of applications have moved to promises instead, so let’s see how we can deal with promises as well.

```js
it('getAllFilesPromise should return a list of files', () => {
  const readStub = sinon.stub(fs, 'readdir')

  const fileManagement = proxyquire('./file.management', { fs })
  readStub.resolves(['test.txt'])

  return fileManagement
    .getAllFilesPromise()
    .then(files => expect(files).to.eql(['test.txt']))
})
```

As you can see, you can just tell the stub to resolve or reject the promise using the data you’d expect.

There is another important use case with Sinon, which is using `callsFake`.
callsFake function usually return a single argument, but unlike `returns` and `yields`, it is used to return a function.

Why would I want to return a function? One example, is if I have some logic I want to execute.

For example, here I’ll use the spread operator to gain access in the stub to the arguments:

```js
const writeStub = sinon.stub(fs, “writeFileSync”).callsFake((...args) => {});
```

The receiving arguments in this case are `“./data/test.txt”`, `""`, `{ flag: “wx” }`.

You can also add behavior to a function (although I think it is less common) using the `addBehavior` function.

```js
sinon.addBehavior(“logsCall”, (fake, msg) => {
	console.log(msg)’
});
```

Then you can use this new defined behavior as a method on your stub.

But two important notes on `addBehavior`; Behaviors must be defined before the stubs, but they can be also chained.

So you want, you can do something like this:

```js
myStub.logsCall(“called”).returns(“hi”);
```

### Fakes

Unlike Stubs, Fakes are immutable and don't replace the default function behavior automatically.

So let’s see the differences.

Replacing a Spy with a Fake:

```js
it('Should create a new file', () => {
  const writeFake = sinon.fake(fs.writeFileSync)
  sinon.replace(fs, 'writeFileSync', writeFake)
  const fileManagement = proxyquire('./file.management', { fs })

  fileManagement.createFile('test.txt')

  expect(writeFake.calledWith('./data/test.txt', '')).to.be.true
})
```

Note that unlike stub, fake receives one argument (which in our case it is the `module.function`).
Also, note that I had to specify (using Sinon’s replace function) what will be the behavior or else it will act as the default behavior.
The `replace` function takes 3 arguments (module, function and the return fake).

In the case above, the test will pass but will fail after another execution due to the same issue we had with our Spy (text.txt) file is created.

Replacing a Stub with a Fake:

One important concept, is instead of faking the function (as you do with Stubs), you use the property, not the function, then using Sinon’s replace function, override the default behavior.

```js
it('Should throw an exception if file already exists', () => {
  const writeFake = sinon.fake.throws(new Error())
  sinon.replace(fs, 'writeFileSync', writeFake)
  const fileManagement = proxyquire('./file.management', { fs })

  expect(() => fileManagement.createFile('test.txt')).to.throw()
})
```

Now let’s do the same for an async function:

```js
it('getAllFiles should return a list of files', () => {
  const readFake = sinon.fake.yields(null, ['test.txt'])
  sinon.replace(fs, 'readdir', readFake)
  const fileManagement = proxyquire('./file.management', { fs })

  fileManagement.getAllFiles((err, data) => {
    expect(data).to.eql(['test.txt'])
  })
})
```

I had to chain the behavior or else the test will fail.

Most of the use cases which using Fakes are usually when you only need one behavior (not based on things like different arguments, etc).

### Mocks

Basically, we do not have re-define what a mock is because it’s pretty strait forward, although it’s a matter of practice. With mocks you first define what to expect and then assert on it.

If we compare it to Stubbing, you’ll need to do something like this:

```js
// stub
myStub.returns(1);
Const result = myStub();
expect(result).to.eql(1);
```

With mocks however it’s almost inverted, for example:

```js
// mock
expect(myStub).to.return(1) // setting expectation first
myStub() // then calls the function
```

Let’s use one of our “real” example as above:

```js
it('Should call writeFileSync when creating a file', () => {
  const writeMock = sinon.mock(fs)
  writeMock.expects('writeFileSync').once().withArgs('./data/test.txt', '')
  const fileManagement = proxyquire('./file.management', { fs })

  fileManagement.createFile('test.txt', '')

  writeMock.verify()
})
```

You can note that I’m mocking an entire module (`fs` module in this case).
I’m setting expectation first using the `expects` function and then asserts using the `verify` function.
Same as stubs, the test will pass but the test.txt file won’t be created and that’s because the mock is intercepting the function under test just like a stub does.

Let's take a look at another test case:

```js
it('createFileSafe should created a new file with a number appended', () => {
  const writeMock = sinon.mock(fs)
  writeMock.expects('writeFileSync').withArgs('./data/test.txt').throws()
  writeMock.expects('writeFileSync').withArgs('./data/test1.txt').once()
  writeMock.expects('readdirSync').returns(['test.txt']).once()
  const fileManagement = proxyquire('./file.management', { fs })

  fileManagement.createFileSafe('test.txt')

  writeMock.verify()
})
```

What if we want to check a function was NOT called?

```js
it('createFile should never call writeFileSync when the file is empty', () => {
  const writeMock = sinon.mock(fs)
  writeMock.expects('writeFileSync').never()

  const fileManagement = proxyquire('./file.management', { fs })

  try {
    fileManagement.createFile()
  } catch (err) {}

  writeMock.verify()
})
```

As you can see, it’s pretty strait forward, you can just utilize the `never()` function.

### To conclude

**Spies** are great for answering questions such as how many times is this function called, with which argument, etc.
It’s considered as bad practice to try and control a function.

**Stubs** is like a stunt double - It keeps us safe from all the side effects of the functions under test like exceptions, calls for APIs or a third party application/libraries, and in any case of forcing a behavior like throwing exception, it’s the stub that does it, not the actual function under test.

**Fake** is a method to combine the two together, they ask questions AND defines behavior.
It has the ability to set a behavior, record arguments, return values, emulates exceptions and more.

Unlike Stubs, Fakes are immutable, and do not replace automatically the original functions behavior.
Unlike stubs, you cannot modify its behavior based on received args or the number of times it’s called, and there are couple of missing functions that are commonly being used with stubs such as `callsFake()`, `callsThrough()` and `addBehavior()`.

**Mocks** - With mocks you first define what to expect and then assert on it and usually you mock an entire module instead of a specific function.

Unlike Fakes, Mocks do contains all of the Spies and Stubs function, with some additional function on top of it (like `never()`, `atLeast()` AND `atMost()`).

As you can see there’s a lot of commonality in all those test doubles, but also some uniqueness in each one of them.

_Code examples were taken from [Nate Taylor PluralSight Course on Mocking Node JS with Sinon](https://app.pluralsight.com/library/courses/mocking-nodejs-sinon). Highly recommended!_


Unit testing and mocking in Node.js with Sinon, get familiar with basic Spies, Stubs, Fakes, and Mocks, concepts.

Explore Mocking In Node.js Using Sinon


Before we dive into hands on examples of such validation, let's first understand what is AJV library and see what other options I explored at work when I was in charge of a task to implement such validator in one of our JS services.

The more "Generic" way - [AJV](https://www.npmjs.com/package/ajv)

The "JS syntax" way - [Joi](https://www.npmjs.com/package/joi)

## AJV

The big win you get from JSON Schema is that it's cross platform. JSON Schema validator implementations exist in every major programming language. No matter what language you choose, you can use the same JSON Schema on the frontend and the backend and get consistent validation results. Write once, validate anywhere.

The downside is that because it's cross platform, it's also somewhat limited in what it can do. It's purposely kept simple enough so that it's not too difficult to implement in any programming language. Because JSON Schema isn't a full fledged programming language, it's limited in what it can do. Although I admit, I still haven't encountered with a scenario where I couldn't find a way to validate something.

## Joi

Joi's biggest advantage is its usability. It's easy to use, easy to extend, and it has the full power of JavaScript.

The downside is that if you want to reuse your validation logic on the frontend and the backend, your only choice of language on the backend is node.

You might choose Joi if you use JavaScript for the frontend and the backend and don't anticipate non-JavaScript clients that you need to support. This is a fairly narrow situation, but if it is your situation, you will likely get more out or Joi than AJV/JSON Schema.

## So let's move on to some comparison i've made

Two things really stood out to me and decided were in terms of performance and popularity.

**Performance**

There are many resources out there that compares both libraries performance, but I it's not even close (and to be fair, it's like comparing oranges to apples). AVJ is way faster.
Here are a few resources I found back then -

- [https://github.com/hapijs/joi/issues/1601](https://github.com/hapijs/joi/issues/1601)
- [https://github.com/icebob/validator-benchmark](https://github.com/icebob/validator-benchmark)

As I said - Not even close.

**Popularity**

Github stars aren't a great measure of popularity. If you look at npm daily downloads you see a very different story (AJV: 18.9M, Joi: 2.2M). JSON Schema is by far more widely used because it is cross platform. AJV is just one implementation in one language. But, if your development stack fits Joi's particular niche, it can be just as good a choice or a better choice than JSON Schema.

![npm-ajv-vs-joi](/images/npm-ajv-vs-joi.jpeg)

Here's a nice [NPMCompare](https://npmcompare.com/compare/ajv,joi,schema-inspector) resource I used.

## That's pretty much it. Let's move on to some AJV and Json Schema examples

For the sake of the example, we'll start with an express route which its post method has an api validator middleware where we're passing the schema and validation type (body, params, query, etc..). In this case we'll use validator for the body of the request.

**Route:**

```js
const router = express.Router()

// Define a route to '/devices', with GET and POST methods.
// For POST requests, an API validation middleware is applied to check the request body against a schema.
router
  .route('/devices')
  .get(
    getDevices() // business logic
  )
  .post(
    apiValidator(postDeviceSchema, validationType.BODY), // validation middleware
    postDevices() // business logic
  )
```

**Generic validation middleware:**

```js
const Ajv = require('ajv')
const addFormats = require('ajv-formats')

// Create an instance of AJV, adding all formats for validation.
const ajvInstance = new Ajv({ allErrors: true })
addFormats(ajvInstance)

// Define the types of validations that can be performed.
const validationType = {
  PARAMS: 'params',
  BODY: 'body',
}

// Create an API validator middleware function. This function uses AJV to validate the data
// of the request (either body or params) against a provided schema.
// If the data is not valid, an error is returned.
function apiValidator(schema, type) {
  return (req, res, next) => {
    try {
      let data = null
      if (type === validationType.BODY) {
        data = req.body
      } else if (type === validationType.PARAMS) {
        data = req.params
      }
      const ajvValidate = ajvInstance.compile(schema)
      const valid = ajvValidate(data)
      if (!valid) {
        const errors = ajvValidate.errors
        res.status(400).json(errors)
        return
      }
      next()
    } catch (err) {
      res
        .status(500)
        .json({ message: 'Internal Server Error, Failed to validate request' })
    }
  }
}
```

The `apiValidator` middleware takes a schema and the type of validation to be performed (e.g., BODY or PARAMS). The middleware compiles the schema using AJV and validates the request data against it. If the validation fails, it sends a 400 status response detailing the errors.

**Post device schema:**

```js
// Base schema for device data. This includes various properties that the device should have,
// along with their expected types and constraints.

const deviceProperties = {
  deviceName: { type: 'string' },
  manufacturer: { type: 'string', enum: config.get('allowedManufacturers') },
  model: { type: 'string', minLength: 3, maxLength: 30 },
  os: { type: 'string', enum: config.get('allowedOs') },
  osVersion: { type: 'string', minLength: 1, maxLength: 30 },
  serialNumber: { type: 'string', minLength: 5, maxLength: 30 },
  site: { type: 'string', enum: config.get('allowedSite') },
  ipAddress: {
    type: 'string',
    anyOf: [{ format: 'ipv4' }, { format: 'ipv6' }],
  },
}

// Extended schema for POST requests. This extends the base device schema,
// adding a requirement for each property and disallowing additional properties.

const postDeviceSchema = {
  type: 'object',
  properties: deviceProperties,
  required: [
    'deviceName',
    'manufacturer',
    'model',
    'os',
    'osVersion',
    'serialNumber',
    'site',
    'ipAddress',
  ],
  additionalProperties: false,
}
```

So what do we have in the schemas that isn't obvious like expecting a string and min/max length?

`manufacturer`: A string that must be one of the allowed manufacturers. The list of allowed manufacturers is retrieved from the application's configuration (using config.get('allowedManufacturers')). The use of the enum keyword ensures that the manufacturer value provided must match one of the ones in this list.

`ipAddress`: A string that must be either an IPv4 or IPv6 address. This is accomplished by providing an array of schema objects to the `anyOf` keyword. Each schema object has a format keyword, which AJV uses to match the data to a specific data format. In this case, the ipAddress property value will be checked against both the IPv4 and IPv6 formats, and if it matches either, it will be considered valid.

For the `postDeviceSchema`, all these properties are required, and no additional properties are allowed (enforced by additionalProperties: false). This means a valid device object must have all of these properties and cannot have any other properties.

There are many other options available in JSON Schema, such as regex (which is used as `pattern`, and many different formats such as data, uuid, etc).

Now let's see some practical use cases?

In the following example, we'll demonstrate a scenario in which a POST request fails validation because the request body doesn't match our defined postDeviceSchema.

Here's the example payload for the request:

```json
{
  "deviceName": "TestDevice",
  "manufacturer": "UnknownManufacturer", // this manufacturer is not in the allowed list
  "model": "Model1",
  "os": "OS1",
  "osVersion": "1.0",
  "serialNumber": "12345",
  "site": "Site1",
  "ipAddress": "192.168.1.1"
}
```

In this example, the manufacturer is "UnknownManufacturer", which is not in the allowed list defined in postDeviceSchema. When the request is sent, the apiValidator middleware will compare the request body with postDeviceSchema. Since "UnknownManufacturer" is not an allowed value, ajvValidate(data) will return false, indicating the validation failed.

The middleware will then gather the error information from ajvValidate.errors and send a 400 status response with the error details. This allows the client to understand what part of the request was invalid and take corrective action.

Here's an example of what the error response could look like:

```json
[
  {
    "keyword": "enum",
    "dataPath": ".manufacturer",
    "schemaPath": "#/properties/manufacturer/enum",
    "params": {
      "allowedValues": ["Manufacturer1", "Manufacturer2", "Manufacturer3"]
    },
    "message": "should be equal to one of the allowed values"
  }
]
```

The keyword tells us that the error relates to an 'enum' check. dataPath shows where in the data the error occurred, and schemaPath shows where in the schema the error was defined. params gives us additional context, such as what the allowed values for this field are. Finally, message provides a human-readable description of the error.

This kind of detailed error response is beneficial for debugging and allows clients to easily understand and correct their mistakes.

Adding to this, there are many other features which weren't covered in this blog post but AJV can handle without any issues, such as `Nested schema validation`, `Async validation` and `Custom validation keywords`.


AJV & Json Schema, a generic yet effective method for JavaScript API request validation

API Validation with AJV and Json Schema


In the realm of JavaScript, developers have a many different sets of data structures at their disposal. One such data structure, is the Map object, which offers significant advantages over other commonly used data structures, such as Arrays and Objects. In this article, I will share the benefits I discovered while using Map in JavaScript and provide comparisons to other data types like arrays and objects.

P.S:
I'm far from an expert so if you're interested in an article that cover this subject much more thoroughly, I recommend this amazing builder.io blog post - [Use Maps More and Objects Less](https://www.builder.io/blog/maps).

So let's begin.

**_Key-Value Pairs and Unique Keys_**
One of the most significant advantages of using Map is its ability to store key-value pairs. Unlike arrays, which only use numeric indices (indexes), Map objects allows to use any data type as a key, including objects and functions. Additionally, Map ensures that each key is unique, simplifying data organization and retrieval.

**Arrays:** Numeric indices only, leading to limitations in organizing data.

**Objects:** Can use any data type as keys but do not guarantee unique keys.

**Maps:** Any data type as keys, offering flexibility and unique keys for better organization.

```javascript
// Arrays
const array = ['value1', 'value2', 'value3']
// Access element using numeric index
console.log(array[0]) // Output: 'value1'

// Objects
const object = {
  key1: 'value1',
  key2: 'value2',
  key3: 'value3',
}
// Access element using any data type as key (but not guaranteed unique)
console.log(object['key1']) // Output: 'value1'
object['key1'] = 'newValue1'
console.log(object['key1']) // Output: 'newValue1'

// Maps
const map = new Map()
const objKey = { id: 1 }
const funcKey = function () {
  return 'hello'
}
map.set(objKey, 'value1')
map.set(funcKey, 'value2')
map.set('key3', 'value3')

// Access element using any data type as key (unique keys)
console.log(map.get(objKey)) // Output: 'value1'
console.log(map.get(funcKey)) // Output: 'value2'
```

**_Iteration and Order Preservation_**
Another notable advantage of using Map is its built-in iteration capability, which preserves the insertion order of elements. This feature is particularly useful when developers need to maintain the order of elements while performing operations or iterating over the data.

**Arrays:** Preserves order, but lacks built-in iteration methods for key-value pairs.

**Objects:** Order not guaranteed, and lacks built-in iteration methods for key-value pairs.

**Maps:** Preserves order and provides built-in iteration methods for key-value pairs.

```javascript
// Arrays
const array = ['value1', 'value2', 'value3']
array.forEach((value, index) => {
  console.log(index, value)
})

// Objects
const object = {
  key1: 'value1',
  key2: 'value2',
  key3: 'value3',
}
for (const key in object) {
  if (object.hasOwnProperty(key)) {
    console.log(key, object[key])
  }
}

// Maps
const map = new Map()
map.set('key1', 'value1')
map.set('key2', 'value2')
map.set('key3', 'value3')
map.forEach((value, key) => {
  console.log(key, value)
})
```

**_Efficient Additions and Deletions_**
When comparing arrays, objects, and maps, it is important to consider the differences in how they handle additions and deletions. Maps are specifically designed for efficient manipulation of key-value pairs, making them more suitable for situations where frequent updates are required.

**Arrays:** When adding or deleting elements in arrays, the length of the array needs to be adjusted. Deleting an element from an array using `splice()` leaves no gaps, but it requires re-indexing the remaining elements, leading to performance issues in large data sets. Additionally, the length property of an array will update accordingly when elements are added or removed.

Keep in mind that if your'e using the `delete` keyword (common root cause for bugs) - it removes the element, but the length of the array remains the same. The deleted element's position will be `undefined`.

**Objects:** Deleting properties from objects does not require re-indexing or updating the length, but it can be slower with large objects. Objects also do not have a built-in property for tracking the number of properties, making it harder to manage the size of the object.

**Maps:** Maps are designed for efficient addition and deletion of key-value pairs. They have a built-in size property for tracking the number of elements, making it easier to manage the size of the Map. The performance of maps is optimized for frequent additions and deletions, providing better performance in scenarios that require frequent updates.

```javascript
// Arrays
const array = ['value1', 'value2', 'value3']

// Adding an element
array.push('value4')

// Deleting an element
const indexToDelete = 1
array.splice(indexToDelete, 1)

// Deleting an element using the delete keyword
delete array[1] // Removes the element at index 1

console.log(array) // Output: ["value1", undefined, "value3"]
console.log(array.length) // Output: 3

// Objects
const object = {
  key1: 'value1',
  key2: 'value2',
  key3: 'value3',
}

// Adding a property
object.key4 = 'value4'

// Deleting a property
delete object.key2

// Maps
const map = new Map()
map.set('key1', 'value1')
map.set('key2', 'value2')
map.set('key3', 'value3')

// Adding a key-value pair
map.set('key4', 'value4')

// Deleting a key-value pair
map.delete('key2')
```

**_Enhanced Performance_**
When it comes to performance, Map outshines both arrays and objects in specific use cases, such as frequent additions and deletions of key-value pairs. Maps are optimized for these operations, providing faster performance than arrays and objects under such circumstances.

**Arrays:** Slower performance when adding or deleting elements, especially in large data sets.

**Objects:** Slower performance when adding or deleting properties, particularly with large objects.

**Maps:** Optimized for faster performance during frequent additions and deletions.

As for the examples - Using `console.time()` and `console.timeEnd()` to measure the time taken for performing the same set of actions on Arrays, Objects, and Maps may not show significant differences in small examples. However, I'll provide an example with these methods for a general idea. Note that the difference in performance will be more evident with larger data sets or more complex operations.

```javascript
// Arrays
const array = ['value1', 'value2', 'value3']
console.time('ArrayOperations')
array.push('value4')
array.splice(1, 1)
console.timeEnd('ArrayOperations') // Output: ArrayOperations: 808.588ms

// Objects
const object = {
  key1: 'value1',
  key2: 'value2',
  key3: 'value3',
}
console.time('ObjectOperations')
object.key4 = 'value4'
delete object.key2
console.timeEnd('ObjectOperations') // Output: ObjectOperations: 799.399ms

// Maps
const map = new Map()
map.set('key1', 'value1')
map.set('key2', 'value2')
map.set('key3', 'value3')
console.time('MapOperations')
map.set('key4', 'value4')
map.delete('key2')
console.timeEnd('MapOperations') // Output: MapOperations: 608.085ms
```

**_Efficient Size Retrieval_**
When working with different data structures, it's important to know how easily and efficiently you can retrieve their sizes. Maps offer a simple and efficient way to do this compared to arrays and objects.

**Arrays:** Arrays have a built-in length property that provides the count of elements. While this is convenient, it doesn't offer the key-value pair structure that some use cases may require.

**Objects:** Objects do not have a built-in property to count their elements. To retrieve the count, you must loop through properties or use methods like Object.keys() or Object.getOwnPropertyNames(), which can be cumbersome and less efficient.

**Maps:** Maps provide a built-in size property, making it easy and efficient to determine the number of key-value pairs in the Map. This offers an advantage in use cases that require quick access to the element count.

```javascript
// Arrays
const array = ['value1', 'value2', 'value3']
console.log(array.length) // Output: 3

//Objects
const object = {
  key1: 'value1',
  key2: 'value2',
  key3: 'value3',
}
console.log(Object.keys(object).length) // Output: 3

//Maps
const map = new Map()
map.set('key1', 'value1')
map.set('key2', 'value2')
map.set('key3', 'value3')
console.log(map.size) // Output: 3
```

There are many more advantages which i won't get into such as memory management and etc.

In conclusion, Map objects provide several advantages over arrays and objects in JavaScript.
They offer flexibility with key-value pairs, preserve order during iteration, optimize performance for frequent additions and deletions, simplify element counting, and support garbage collection using WeakMaps. By understanding the differences and benefits of each data structure, developers can make informed decisions when choosing the best option for their specific use cases. Maps are a powerful addition to the JavaScript toolkit, offering enhanced functionality and performance in various scenarios where arrays and objects may fall short.


The advantages of using Map in JavaScript over other Data Types

You Should Really Use Map When Possible


How can we hide API keys, secrets and passwords in our code, especially if pushing our code to an external repository? How can we run an application locally using these env vars?

Preventing exposure of sensitive information in our code is essential and necessary - We don't want external parties to have unauthorized access to our databases, passwords, and services we use. Therefore, we want to secure this information as best as possible.

So how exactly do we do that?

The solution I used is quite simple - Wwe need to create a local file in the project root directory that will store and centralize all the API keys and sensitive information we want to secure.

Additionally, we will ensure that this file is added to the gitignore file, of course, as it is intended to be stored only in the local environment.

This file called .env, which is short for environment variables. Its purpose is to manage and allow runtime access to variables containing sensitive information, without using hardcoding in the text for these values.

The file will contain only variables in a key-value pair format.

For example, in Node.js, to access the values ​​instead of writing in our code like this - const API_KEY = "123", we will write like this - const API_KEY = process.env.API_KEY (using the global Node object -> process).

Not only does the env file improve the modularity and maintenance of the code, but it also allows flexibility and separation between the values ​​of each key in different environments - for example, for the development stage, we will provide a separate env file with different passwords than in the testing or production stages.

In Node.js project I use the **dotenv** package to make life easier.

The dotenv package is a popular tool that simplifies the process of swapping various env file parameters in the system according to the work environment.

You can install it with npm (saved under dev dependencies)

```js
npm install dotenv --save-dev
```

And just for the sake of the example, here's how I run my unit tests using dotenv & Mocha.
From the package.json:

```js
"test": "mocha --inline-diffs tests/unit-tests/ -r dotenv/config"
```

In conclusion, you should definitely consider using env files to secure your code, especially in personal projects and during the development stage.


How can we hide API keys, secrets and passwords in our code, especially if pushing our code to an external repository? How can we run an application locally using these env vars?

Using Secrets or Private Environment Variables? .env is the Answer


I'll start with a full disclosure - Up until now, I have only written unit tests using Jest. However, at my current job, I had the opportunity to work on a small project that allowed me to choose a new testing library.

After some research, I decided to go with Mocha as a test runner and Chai for assertions. Then I reached the fun part - Mocking and Spying.
The most obvious choice was Sinon, which works great as a spy (for example, if you want to know whether a certain log was printed). However, when it came to Mocking (or Stubbing), it wasn't easy at all.

If you're interested - I found this article back then and I found it to be quite true ([why is it so hard to test in NodeJS](https://www.vinnie.work/blog/2021-09-18-why-so-hard-testing-with-es6-imports))

After struggling with Sinon, I turned to the Discord Node.js community with a specific question:

![discordquestion](/images/discord-node-test.png)

The first response was a link to the [rewire](https://github.com/jhnns/rewire) repository. I must say, once I started using it, I couldn't stop. It made mocking incredibly easy.

To install it simply run:

```
npm install rewire
```

import it:

```js
const rewire = require('rewire')
```

To initialize it, you need to "rewire" the module that contains the functions, methods, or variables you'd like to mock.

```js
const mockedIndex = rewire('../../index')
```

So let's look at some examples:

I'll start with the original question I raised in the Node Discord server.
Here's my original function (just for the sake of the example):

```js
async function updatePolicy(project, policy, file) {
  return Promise.allSettled(Object.entries(file).map(([status]) => {
    try {
      if (!countryRules[country]) {
        return Promise.resolve();
      }
      if (statusDisable(status))
        console.log('disable');
      else if (statusEnable(status))
        console.log('enable');
        [...]
}
```

I wanted a simple test case to check that the correct function is called given the right arguments.
Since I had many test cases like these, I created a more "generic" function that receives the function I'd like to test as an argument.

```js
// rewire the module
const lib = rewire('../../index')

// generic function

const checkFunctionCall = (functionName, done, file) => {
  lib.__set__(`${functionName}IsCalled`, false) // setting the function under test to isCalled false
  lib.__set__(functionName, () => {
    // function under test is called
    lib.__set__(`${functionName}IsCalled`, true) // changing the isCalled to true in this case.
  })
  lib.updatePolicy(project, policy, file) // Calling the function that eventually calls the function i'd like to assert on
  assert.equal(lib.__get__(`${functionName}IsCalled`), true) // asserting that isCalled is true
  done()
}

// test case

it('statusEnable function called', done => {
  let file = 'example argument'

  checkFunctionCall('statusEnable', done, file)
})
```

Doesn't seem so complicated, right?
Moving on to mocking of an async function that should return resolved promise. Let's say this is the original function:

```js
async function persistCatalogs(bucketName, catalogs) {
  const catalogBucket = storageClient.bucket(bucketName)
  return new Promise(async (resolve, reject) => {
    for (const chunk of keyChunks) {
      await Promise.all(
        chunk.map(async catalog =>
          uploadCatalog(catalogBucket, cacheTtl, catalog, catalogs[catalog])
        )
      )
    }
    resolve()
  }).then(() => logger.info('Catalog uploaded!'))
}
```

I'd like to test that given a specific argument, the `uploadCatalog` function was called 12 times.

```js
// stubbed function that will be called when mocking the uploadCatalog function
function uploadCatalogStub() {
  return Promise.resolve('Mocked result')
}
//test case
it('should upload each catalog in chunks', async () => {
  lib.__set__('uploadCatalog', uploadCatalogStub) // rewire will set uploadCatalogStub once uploadCatalog is called
  await lib.persistCatalogs(bucketName, catalogs) // Original function called
  expect(uploadCatalogStub.callCount).to.equal(12) // assertion
})
```

Moving on the mocking a global variable that eventually will be used in a function.

This is the function i'd like to test:

```js
function addEventToCatalog(rawEvent, catalog) {
  rawEvent.countryAvailability.forEach(c => {
    if (COUNTRIES_LIST.includes(c))
      catalog[c].push(event);
    else
      logger.warn(`Unsupported country ${c} on event ${event.assetId}, skipping for catalog`);
  });
```

Now `COUNTRIES_LIST` variable is a global variable, which it's data in it might effect the result of the test. So let's mock it:

```js
lib.__set__('COUNTRIES_LIST', structuredClone(countryCodesData)) //COUNTRIES_LIST is a global variable, here it'll return a clone of another data i chose.
```

and call the tested function is test as usual:

```js
it('should add event to US catalog if countryAvailability includes US', () => {
  lib.addEventToCatalog(rawEvent, catalog)

  expect(catalog.US).to.have.lengthOf(1) // assertion
})
```

In conclusion, `rewire` proved to be the simplest mocking library I've had the opportunity to use in Node.js projects.
Ultimately, pairing it with a dynamic test runner like Mocha made for an easy choice compared to other libraries.

As a result, I ended up replacing all other mocks, such as Sinon Stubbing, with Rewire and haven't looked back since.


Unit testing in Node.js can be challenging. After trying various libraries, I found Rewire to be the most practical for testing purposes

NodeJS Articles

All NodeJS Articles