
Managing a monorepo with multiple Python packages presents unique challenges, particularly in ensuring efficient CI/CD. In this article, I will share how I created CI/CD workflows for our monorepo of Python packages using GitHub Actions.

We will delve into the workflows, including matrix complexity, and scripts for checking changed paths, alongside other custom scripts and composite actions that facilitate this process.

Our primary CI workflow (`on-pr-build-push.yaml`) triggers on pull requests, ensuring that all changes are validated before merging. This workflow comprises several jobs: `ChangedPaths`, `Lint`, `Typecheck`, `Test`, and `Build-Push`.

**Job: ChangedPaths**

This job determines which packages have been modified and outputs a matrix that drives the subsequent jobs. So for example, if a PR modifies `package1` and `package2`, the matrix will contain two rows: `package1` and `package2`.

```yaml
name: Build & Push Python Package

on:
  pull_request:

jobs:
  ChangedPaths:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.get_changed_paths.outputs.matrix }}
    timeout-minutes: 10
    permissions:
      id-token: write
      contents: read

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GLOBAL_GITHUB_TOKEN }}
          fetch-depth: 0

      - name: Run init
        uses: ./.github/actions/init # Custom composite action for initializing the environment

      - name: Get Changed Paths
        id: get_changed_paths
        shell: bash
        run: |
          python $GITHUB_WORKSPACE/.github/scripts/internal/get_changed_paths.py
```

The `get_changed_paths.py` python script identifies the changed paths and filters them to include only those within the packages directory.
This script is quite straightforward and uses the `git diff` command by using the built-in `subprocess` module in order to identify the changed paths.

```python
import subprocess
import os

output_file = os.getenv('GITHUB_OUTPUT')


def get_changed_paths():
    cmd = 'git diff --name-only $(git rev-parse HEAD) HEAD~1'
    result = subprocess.run(cmd, shell=True, capture_output=True)
    if result.returncode != 0:
        raise Exception("Error getting changed paths")

    changed_paths = result.stdout.decode("utf-8").strip().split("\n")
    print(f"Changed paths: {changed_paths}")
    return changed_paths


def get_filtered_paths(changed_paths):
    filtered_paths = set()
    for path in changed_paths:
        if path.startswith("packages/"):
            filtered_paths.add(path.split("/")[1])
    print(f"Filtered paths: {filtered_paths}")
    return list(filtered_paths)


if __name__ == "__main__":
    changed_paths = get_changed_paths()
    filtered_paths = get_filtered_paths(changed_paths)
    print(filtered_paths)
    with open(output_file, "a") as file:
        file.write(f"matrix={filtered_paths}")
```

Note the `capture_output=True` argument allows us to capture the command's output. We then decode and split the output to get a list of changed paths. The `get_filtered_paths` function filters these paths to include only those that start with `packages/` which is my path in my repos case.

Also note that GitHub Actions uses environment files to pass data between steps and jobs. By writing the output matrix to a file (GITHUB_OUTPUT), we can ensure that the list of changed paths is available as an output in the ChangedPaths job. This allows subsequent jobs to access and iterate over the paths efficiently.

**Job: Lint, Typecheck, and Test**

These jobs runs composite actions for linting (flake8), type checking (mypy), and tests (pytest) on the changed packages using a matrix strategy. They depend on the ChangedPaths job.

```yaml
Lint:
  needs: ChangedPaths
  if: ${{ needs.ChangedPaths.outputs.matrix != '[]' }}
  strategy:
    matrix:
      path: ${{ fromJson(needs.ChangedPaths.outputs.matrix) }}
  runs-on: ubuntu-latest
  timeout-minutes: 10
  permissions:
    id-token: write
    contents: read

  steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Get path
      run: echo ${{ matrix.path }}

    - name: Run init
      uses: ./.github/actions/init
      with:
        change_path: ${{ matrix.path }}

    - name: Run Lint action
      uses: ./.github/actions/lint
      env:
        paths: ${{ matrix.path }}
```

**Job: Build-Push**

The final job in the CI pipeline builds and pushes the packages if all previous jobs succeed.

```yaml
Build-Push:
  needs: [ChangedPaths, Lint, Typecheck, Test]
  if: ${{ needs.ChangedPaths.outputs.matrix != '[]' }}
  strategy:
    matrix:
      path: ${{ fromJson(needs.ChangedPaths.outputs.matrix) }}
  runs-on: ubuntu-latest
  timeout-minutes: 15
  permissions:
    id-token: write
    contents: read

  steps:
    - name: Checkout
      uses: actions/checkout@v3
      id: checkout

    - name: Run init
      id: run-init
      uses: ./.github/actions/init
      with:
        change_path: ${{ matrix.path }}

    - name: Build & Publish Python Packages
      uses: ./.github/actions/build-push-package
      env:
        paths: ${{ matrix.path }}
```

This job calls a custom composite action (`build-push-package`) that builds and upload the package to GitHub cache, so the user can install it from there.

```yaml
name: Build & Release Package
description: 'Build & Push Python Package'
author: 'Or Kazaz'

runs:
  using: composite

  steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Install Python Dependencies
      shell: bash
      run: |
        pip install --upgrade pip
        pip install poetry twine check-wheel-contents

    - name: Run Poetry Check for pyproject.toml
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: poetry check

    - name: Build Package Distribution
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: poetry build

    - name: Check Distribution Descriptions
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: |
        twine check dist/*

    - name: Check Wheel Contents
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: |
        check-wheel-contents dist/*.whl

    - name: Cache Package Distribution
      uses: actions/upload-artifact@v3
      with:
        name: ${{ env.paths }}
        path: ./packages/${{ env.paths }}/dist/
```

The secondary workflow (`tag-and-release.yaml`) is triggered on a manually on demand by using `workflow_dispatch`. It tags the package, updates its version is pyproject.toml.toml, and publish the packages into our internal PyPI repository (on Jfrog Artifactory). It consists of multiple jobs to ensure a smooth release process.

**Job: Tag and Release**

This job tags the repository and creates a release on GitHub by zipping it and using an action to create a release to be available for download.

```yaml
jobs:
  tag-and-release:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    outputs:
      package_version: ${{ steps.dynamic_version.outputs.package_version }}

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          ref: ${{ github.event.inputs.commit_hash }}

      - name: Run Dynamic-Versioning
        id: dynamic_version
        uses: ./.github/actions/dynamic-versioning
        with:
          package_name: ${{ github.event.inputs.package_name }}

      - name: Archive package directory
        working-directory: ./packages
        run: |
          zip -r ${{ github.event.inputs.package_name }}.zip ./${{ github.event.inputs.package_name }}
          echo "ARTIFACT_FILE_PATH=./packages/${{ github.event.inputs.package_name }}.zip" >> $GITHUB_ENV
        env:
          ARTIFACT_FILE_PATH: ${{ env.ARTIFACT_FILE_PATH }}

      - name: Create Release
        uses: ncipollo/release-action@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          artifacts: '${{ env.ARTIFACT_FILE_PATH }}'
          commit: ${{ github.event.inputs.commit_hash }}
          tag: ${{ github.event.inputs.package_name }}-v${{ steps.dynamic_version.outputs.package_version }}
          name: ${{ github.event.inputs.package_name }}-v${{ steps.dynamic_version.outputs.package_version }}
          token: ${{ secrets.GLOBAL_GITHUB_TOKEN }}
          skipIfReleaseExists: true
```

Note that the `dynamic-versioning` action is a custom action that updates the version in the `pyproject.toml` file.

```yaml
name: 'Dynamic Versioning'
description: 'Bump version in python package'
author: 'Or Kazaz'

inputs:
  package_name:
    description: 'Package name'
    required: true

outputs:
  package_version:
    description: 'bumped package version'
    value: ${{ steps.bump_version.outputs.package_version }}

runs:
  using: composite

  steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: Bump version
      id: bump_version
      shell: bash
      run: |
        PACKAGE_VERSION=$($GITHUB_WORKSPACE/.github/scripts/internal/bump_version.sh ${{ inputs.package_name }})
        echo "${{ inputs.package_name }} package will be bumped to version: ${PACKAGE_VERSION}"
        echo "package_version=$PACKAGE_VERSION" >> $GITHUB_OUTPUT
```

Bump version is a very simple bash script the increments the package version in the `pyproject.toml` file.

```bash
#!/bin/bash

PACKAGE_NAME=$1

VERSION_LINE=$(grep version ./packages/$PACKAGE_NAME/pyproject.toml)
OLD_VERSION=$(echo "$VERSION_LINE" | awk -F'"' '{print $2}')
NEW_VERSION=$(echo "$OLD_VERSION" | awk -F. -v OFS=. '{$3=$3+1}1')

sed -i "s/$VERSION_LINE/version = \"$NEW_VERSION\"/g" ./packages/$PACKAGE_NAME/pyproject.toml

VERSION=$(grep version ./packages/$PACKAGE_NAME/pyproject.toml | awk -F'\"' '{print $2}')
echo $VERSION
```

**Job: Build**

This job builds the package distribution using Poetry.

```yaml
build:
  needs: tag-and-release
  name: Build Python Package
  runs-on: ubuntu-latest
  timeout-minutes: 30

  steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        ref: ${{ github.event.inputs.commit_hash }}

    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Install Python Dependencies
      working-directory: ./packages/${{ github.event.inputs.package_name }}
      run: |
        pip install --upgrade pip
        pip install --upgrade poetry twine check-wheel-contents

    - name: Run Dynamic-Versioning
      id: dynamic_version
      uses: ./.github/actions/dynamic-versioning
      with:
        package_name: ${{ github.event.inputs.package_name }}

    - name: Build Package Distribution
      working-directory: ./packages/${{ github.event.inputs.package_name }}
      run: |
        poetry build
```

**Job: Publish to JFrog**

This job publishes the built package to JFrog by retrieving the cached package distribution and using the official `gh-action-pypi-publish` action.

```yaml
publish-jfrog:
  needs: build
  name: Publish Package to JFrog
  runs-on: ubuntu-latest
  timeout-minutes: 15

  steps:
    - name: Retrieve Cached Package Distribution
      uses: actions/download-artifact@v3
      with:
        name: ${{ github.event.inputs.package_name }}
        path: ./packages/${{ github.event.inputs.package_name }}/dist

    - name: Publish to JFrog
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        packages-dir: ./packages/${{ github.event.inputs.package_name }}/dist
        user: ${{ secret.secret_user }}
        password: ${{ secret.secret_password }}
        repository-url: https://the.internal.repo.url/artifactory/api/pypi/pypi
```

**Job: Commit Updated Version**

Finally, this job commits the updated version to the repository while skipping CI.

```yaml
commit-version:
  needs: [tag-and-release, build, publish-jfrog]
  name: Commit updated version
  runs-on: ubuntu-latest
  timeout-minutes: 15

  steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Commit updated version
      shell: bash
      run: |
        git config --global user.name $GITHUB_USER
        git config --global user.email $GITHUB_EMAIL
        git pull
        bash $GITHUB_WORKSPACE/.github/scripts/internal/bump_version.sh ${{ github.event.inputs.package_name }}
        git add ./packages/${{ github.event.inputs.package_name }}/pyproject.toml
        git commit -m "Bump ${{ github.event.inputs.package_name }} to version ${{ needs.tag-and-release.outputs.package_version }} [skip ci]"
        git push origin ${GITHUB_REF#refs/heads/}
```

![CI-ON-PR](https://res.cloudinary.com/dtt2usukl/image/upload/v1718703406/Image_18-06-2024_at_12.36_ge6oku.jpg)_On Pull request CI of 3 packages modified_

This setup ensures that only the affected packages are processed, reducing build times and resource usage. The use of matrix strategies, custom scripts, and composite actions provides a flexible, generic and scalable approach to managing CI/CD workflows of such use case.


In this article i'll share how I handheld the CI/CD in a mono repo with multiple python packages using GitHub Actions.

Handling CI/CD in a Mono Repo With Multiple Python Packages


We've implemented extensive improvements, including expanding device capacity, enhancing monitoring systems, introducing features like automated recovery mechanisms, chaging regions using internal network VPN and more.
These advancements will streamline testing processes, optimize resource utilization, and provide a seamless experience for developers and testers across global locations.
With ~250 users, supporting 25 different platforms and 150 testing devices, we are well-positioned for continued success.

For those of you who are not familiar with DAZN, here's a brief overview:

> _DAZN is the world’s leading live sports BTC streaming service. With DAZN, fans in any country can watch sport in any language, on any device, live and on-demand. Purpose built to stream live sports globally, DAZN is the market leader in Italy, Spain, Germany, and Japan, where it holds the top-tier domestic football rights — Serie A, La Liga, Bundesliga and J League, respectively — as well as the richest catalogue of live and on-demand content. DAZN is also available in over 200 other territories, where it boasts the most extensive library of elite and emerging sports content._

DAZN develops both HTML-based and native applications designed to run on a variety of platforms, including TVs, set-top boxes, gaming consoles, and native devices. Ensuring high quality across these diverse platforms necessitates thorough testing at every stage of development, ranging from unit tests and component tests to emulator tests and real-device testing.

The global pandemic in 2020 prompted a paradigm shift in work practices, underscoring the importance of enabling engineers to access diverse resources and conduct testing remotely. Quality remains paramount, necessitating comprehensive testing methodologies despite remote work environments.

To address these challenges, we adopted a proactive “Lab as a Service” approach, culminating in the establishment of an in-house lab facility in Katowice, Poland. This strategic decision aimed to centralize resources and provide engineers with the necessary tools and infrastructure to conduct testing efficiently.

However, establishing and operating an in-house lab posed unique challenges that required careful consideration and resolution. These challenges encompassed logistical, technical, and organizational aspects, underscoring the need for adaptive solutions and collaborative efforts.

In response to these challenges, ongoing initiatives are focused on optimizing the efficiency and effectiveness of the in-house lab, streamlining workflows, enhancing collaboration among teams, and leveraging technology to overcome geographical barriers. By embracing innovation and agility, DAZN aims to continuously improve its testing capabilities and deliver exceptional quality across its diverse range of applications and platforms.

## Challenges

DAZN employs a diverse team of developers and test engineers spread across different global locations such as Poland, India, Israel, and the UK. Each team comes with its unique set of needs and requirements, necessitating a robust, highly available, and cost-effective platform to accommodate them effectively.

Key points include:

- Diverse Platform Support:
  The platform must ensure comprehensive testing across all platforms supported by DAZN, emphasizing thoroughness and quality assurance in our testing procedures.
- Tailored Solutions:
  Teams may require different tools and resources for their testing processes; for instance, some teams may rely on manual UI/UX testing devices, while others prioritize automation solutions.
- Integration Challenges:
  Incorporating in-house tools and services into our platform is essential to ensure seamless continuity for engineers, requiring careful integration efforts.
- Scalability and Maintenance:
  The platform needs to support a large number of engineers on a daily basis across multiple global sites, balancing scalability with efficient maintenance to avoid operational bottlenecks.

- Cost Considerations:
  Maintaining devices, infrastructure, and manpower across various locations can incur significant expenses, necessitating a cost-effective approach to resource allocation and management.

- Ownership and Accountability:
  Clear ownership and accountability structures are crucial to prevent issues such as maintenance gaps or device mismanagement.

To address these challenges, ongoing efforts are focused on streamlining processes, enhancing collaboration between teams, and leveraging automation wherever possible to optimize resource utilization and improve efficiency in our testing and development workflows.

## Solution — Lab as a Service

Our team, composed of seasoned experts in DevOps, Automation Infrastructure, and IT, recognized the need for a novel solution to address the challenges faced in managing Device Under Test (DUT) usage at DAZN. In the first quarter of 2023, we conducted a comprehensive assessment to identify and map out the major pain points associated with DUT utilization within the organization. This assessment culminated in the development of a pioneering approach coined “Lab as a Service,” aimed at alleviating the burden on developers and testers by handling all DUT-related tasks and physical infrastructure management.

During the second quarter of 2023, we introduced the Minimum Viable Product (MVP) setup, leveraging the Suitest framework — a vendor offering remote control and automation capabilities tailored for both HTML-based and native platforms. Our team meticulously orchestrated the deployment of all requisite physical infrastructure, assembling a dedicated lab housing 50 test devices within a spacious room at the DAZN Katowice office.

![DUT lab v1.0 (QA room — May 2023)](https://miro.medium.com/v2/resize:fit:640/format:webp/1*MDbJo7nq77qwWrRCpNvsrA.jpeg)_DUT lab v1.0 (QA room — May 2023)_

To bolster reliability and resilience, we implemented a comprehensive monitoring and alerting system, with Grafana serving as the frontend interface. Additionally, we integrated DUT recovery mechanisms to enhance the robustness of our testing environment.

![Grafana monitoring dashboard](https://miro.medium.com/v2/resize:fit:640/format:webp/1*8a9dpHjMLH1FZCQptCNplw.png)_Grafana monitoring dashboard_

The efficacy of these enhancements became apparent swiftly, as an increasing number of engineers gravitated towards utilizing the centralized lab infrastructure instead of configuring local devices. Leveraging the valuable feedback provided by users, we iteratively refined and enhanced the lab environment, ensuring that it continues to meet the evolving needs of our development and testing teams.

## Lab as a service 2.0

As the user base and the number of devices in the DAZN lab continued to surge, spurred by the establishment of a new DAZN site in India and the expansion of supported platforms throughout the latter half of 2023, we undertook further enhancements to the Katowice lab. These adjustments were aimed at accommodating the organization’s growth and fortifying its capacity to scale even further. Throughout the first quarter, we initiated a series of improvements targeting both the physical infrastructure and supporting features:

### Infrastructure enhancements

- **Relocation and Expansion:**

The DUT lab underwent relocation to a larger space, facilitating increased scalability and the seamless onboarding of additional platforms and devices. Leveraging existing racks more efficiently, we enabled the mounting of devices on both sides of each rack, resulting in a 50% boost in the number of TVs and a 40% increase in the number of set-top boxes (STBs), totaling 120 DUTs overall, with ample room for further expansion.

![DUT lab v2.0 (open space — January 2024)](https://miro.medium.com/v2/resize:fit:640/format:webp/1*IKUuw6aU2wQMzTBKCKS_bA.jpeg)_DUT lab v2.0 (open space — January 2024)_

- **Streaming Solutions:**

Collaborating with Suitest, we identified a cost-effective solution enabling the streaming of device HDMI output for STBs and gaming consoles. Additionally, we enhanced the TV streaming setup by refining the phone streaming app.

![Live DUT Streaming Phone camera setup](https://miro.medium.com/v2/resize:fit:720/format:webp/1*K460jjXRPqXUVQf8PVg0ng.jpeg)_Live DUT streaming mobile phone camera setup_

![HDMI grabber/splitter combo](https://miro.medium.com/v2/resize:fit:640/format:webp/1*s-cKLE8Oh1h5mliovKc91Q.jpeg)_HDMI grabber/splitter combo_

### Features enhancements

- **Collaboration with Suitest:**

Over the course of the year, we actively collaborated with Suitest to influence their roadmap, contributing to improvements in user experience and interface. A significant milestone was the introduction of a Unified UI integrating video streaming, remote control, and logs within the same browser window, enhancing productivity for lab users.

![Suitest Unified UI](https://miro.medium.com/v2/resize:fit:640/format:webp/1*Wg96k859tBYZwz13vVXG3Q.png)_Suitest Unified UI_

- **LR (Living Room) Launcher**:

In collaboration with the DAZN LivingRoom development team, we introduced LR Launcher, simplifying the workflow for manual testers and streamlining the onboarding process for partner platforms such as Sky and Movistar for the sake of ease of use when testing different development environments.

![LR Launcher](https://miro.medium.com/v2/resize:fit:640/format:webp/1*5AObQs1_Vdyd9cXhf5qF3w.png)_LR Launcher_

- **Enhanced Monitoring**:

With the growing user base and device count, robust monitoring became paramount. We revamped our monitoring service, streamlining processes and transitioning from an API-based solution to WSS notifications. Furthermore, we expanded monitoring coverage and implemented alerting for control units and streaming devices.

- **Automated Recovery**:

Leveraging smart plugs across all TV platforms, we implemented remote manual and automated recovery mechanisms for malfunctioning DUTs. Efforts are underway to develop similar mechanisms for cameras and control units, bolstering the resilience and efficiency of our lab environment.

- **Local Tunnel**:

A local tunnel solution providing exposure of a local development server allows developers and testers to seamlessly execute their code on a real physical lab device. This capability not only streamlines the testing process but also provides a practical and authentic environment for validating local code during development.

- **Lab Insights Report (BI)**:

An automated report detailing lab usage and testing trends. This report will provide valuable insights into usage patterns, enabling us to further optimize our services based on real data.

![BI Report](https://res.cloudinary.com/dtt2usukl/image/upload/v1716458627/Image_27-08-2023_at_11.58_sas9dh.jpg)_BI Report_

**VPN Change Region & VPN Console**:

A VPN solution that will empower lab users to seamlessly transfer DUTs between regions remotely. Using firewalls controlled by OPNSense, a lambda backend and a UI portal written in React (NextJS) users are able to testing of region-dependent content across all lab devices by switching regions as needed.

![VPN Console](https://res.cloudinary.com/dtt2usukl/image/upload/v1716458833/Image_23-05-2024_at_13.06_oxbech.jpg)_VPN Console_

## Service & Support

- **Continuous Improvement:**

Since the launch of the DUT lab in May 2023, we have continually bolstered our service and support infrastructure. This includes expanding our support engineering team, enhancing documentation and guides, and maintaining accessible support channels during working hours. Our commitment to swift issue resolution is underscored by our expanded remote resolution capabilities, facilitated by proactive monitoring and device error-based recovery processes.

- **Onboarding Optimization:**

Through the introduction of “Lab as a Service 2.0” and tailored training sessions, we have streamlined the onboarding process for new users, ensuring a seamless transition and maximizing productivity from the outset using official docs portal (using github pages and backstage).

These enhancements collectively represent our ongoing commitment to optimizing the DAZN lab environment, fostering innovation, and empowering our engineering teams to deliver exceptional quality across all platforms and devices.

## Acknowledgements

Special thanks to the amazing AP Nexus team for their dedication and hard work, it was a pleasure working with each one of you!

[Dennis Groisman](https://www.linkedin.com/in/dennis-groisman-9811b335/), [Daniel-Shahar Cohen](https://www.linkedin.com/in/danielshaharcohen/), [Rafal Muszalik](https://www.linkedin.com/in/rafa%C5%82muszalik/), [Blazej Milewski](https://www.linkedin.com/in/blazej-milewski/), [Alex Rivkin](https://www.linkedin.com/in/alex-rivkin-b037a6a1/), [Yoav Brown](https://www.linkedin.com/in/yoav-brown-970b55265/) and the one and only [Gal Moran](https://www.linkedin.com/in/gal-moran-a3454095/).

[![DAZN Lab as a Service](https://res.cloudinary.com/dtt2usukl/image/upload/v1716460903/play_video_lab_fnkajn.jpg)](https://res.cloudinary.com/dtt2usukl/video/upload/v1716460249/tvlab_v2_withmusic_twifar.mp4 'DAZN Lab as a Service - Nexus Team')_My team - Lab as a Service 2.0_

Article on [Medium (DAZN Engineering Blog)](https://medium.com/dazn-tech/lab-as-a-service-in-dazn-9bcc9d549970)


I'm incredibly proud to share the outstanding work our team has accomplished recently in upgrading our testing lab infrastructure.

Lab as a Service in DAZN


In this blog post i'll share a real world example of how to create an S3 bucket with a redirect URL using Cloudfront with Terraform.

This started at my work when we needed to create a simple redirect URL for a specific domain referring to the company's official documentation portal. Our goal was to create a simple URL that would redirect to the documentation portal for our specific team for our internal customers in the company, which would make it easier for them to access.

Our solution involves creating an S3 bucket configured to redirect requests, setting up a CloudFront distribution for global reach, responses, etc, and configuring a Route 53 DNS record to point to our CloudFront distribution.

Here's how these AWS services work together in our setup:

- AWS S3 Bucket: Hosts a redirect rule to our desired URL.

- AWS CloudFront: Distributes our content globally.

- AWS Route 53: Manages DNS to make our short URL accessible.

Let's start with our example:

**Setting Up the S3 Bucket for Redirection**

Our first step is to create an S3 bucket that will handle the URL redirection using Terraform:

```hcl
resource "aws_s3_bucket" "redirect_bucket" {
  bucket = "your-bucket-name"

  website {
    redirect_all_requests_to = "https://the-url-you-want-to-redirect-to.com"
  }
}
```

This snippet creates an S3 bucket and sets it up to redirect all incoming requests to our specified URL. The website block is used to configure the bucket as a static website, and the redirect_all_requests_to attribute specifies the URL to redirect to.

**Configuring the CloudFront Distribution**

Next, we set up a CloudFront distribution to serve our content globally. Here's the example configuration:

```hcl
resource "aws_cloudfront_distribution" "s3_distribution" {
  origin {
    domain_name = "${aws_s3_bucket.redirect_bucket.bucket}.s3-website.${var.region}.amazonaws.com"
    origin_id   = "S3-${aws_s3_bucket.redirect_bucket.id}"

    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "http-only"
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  # ... Additional configuration ...
}
```

This sets up a CloudFront distribution that points to our S3 bucket. Note how we configure the custom_origin_config to ensure secure content delivery.
Also, make sure that in the origin block, you specify the correct resource name for the S3 bucket (domain_name).

**Setting Up Route 53 for DNS Configuration**

For our short URL to be accessible, we need to configure a DNS record in Route 53:

```hcl
resource "aws_route53_record" "www" {
  zone_id = data.aws_route53_zone.selected.zone_id
  name    = "your-name"
  type    = "A"

  alias {
    name                   = aws_cloudfront_distribution.s3_distribution.domain_name
    zone_id                = aws_cloudfront_distribution.s3_distribution.hosted_zone_id
    evaluate_target_health = false
  }
}
```

This Terraform code creates an A record that points to our CloudFront distribution. Make sure to replace your-name with the desired domain name and to understand what's needed in terms or DNS record type.

Keep in mind, this is a simplified example of how to set up an S3 bucket with a redirect URL using CloudFront with Terraform. You may need to adjust/add configurations based on your specific requirements and use case.


Real world example of how to create an S3 bucket with a redirect URL using Cloudfront with Terraform.

S3 Bucket Redirect URL With Terraform


The common purposes of scheduling Lambda functions is to automate tasks such as nightly data backups, data fetching and processing, or any routine task that needs to run on a scheduled basis.

If you're using AWS as a cloud provider, once of the most common services to schedule your lambdas is EventBridge (formerly known as CloudWatch Events).

To show and example of how to create such thing, we'll use Terraform.

First, lets create the rule itself (eventually it'll be registered in EventBridge):

```hcl
resource "aws_cloudwatch_event_rule" "schedule" {
  name                = "${var.name_prefix}-schedule"
  description         = "Schedule for Updater Lambda Function (ap-vpn-console)"
  schedule_expression = var.schedule_expression
}
```

**name:** A unique identifier for the rule, using a variable for flexibility.

**description:** Clear description of what the rule does.

**schedule_expression:** When the Lambda will be triggered. Use the `schedule_expression` variable to easily change the timing without modifying the code directly.

Just in case you're not familiar with cron expressions, here is an example:

```hcl
variable "schedule_expression" {
  description = "The schedule expression for the lambda function"
  type        = string
  default     = "cron(0 2,10 * * ? *)"
}
```

This cron expression means the Lambda will run at 2:00 AM and 10:00 AM (UTC) every day.

![schedule-rule](/images/eventbridge-lambda-rule.jpeg)

Now let's hook it with our Lambda function:

```hcl
resource "aws_cloudwatch_event_target" "schedule_lambda" {
  rule      = aws_cloudwatch_event_rule.schedule.name
  target_id = "processing_lambda"
  arn       = aws_lambda_function.lambda.arn
}
```

**rule:** Links the target to the previously defined rule.

**target_id:** An identifier for the target.

**arn:** Amazon Resource Name for the Lambda function to execute.

Last thing we might want to do, is to allow EventBridge to invoke the Lambda function by defining its permissions:

```hcl
resource "aws_lambda_permission" "allow_events_bridge_to_run_lambda" {
  statement_id  = "AllowExecutionFromCloudWatch"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.lambda.function_name
  principal     = "events.amazonaws.com"
}
```

**statement_id:** A unique identifier for the statement.

**action:** Specifies the allowed action.

**function_name:** Links to the Lambda function.

**principal:** The AWS service that is allowed to perform the action.


Schedule your AWS Lambda tasks using Terraform and AWS EventBridge to automate routine tasks and improve efficiency.

Scheduling Lambdas Using AWS EventBridge and Terraform


Ensuring consistent application performance across multiple platforms is a cornerstone of quality software delivery. In this post, I'll share my approach for using GitHub Actions to automate testing across a variety of platforms, drawing from a real use case at my workplace.

Our solution utilizes GitHub Actions, a CI/CD service that enables automation of build, test, and deployment workflows directly within GitHub. The highlight of the approach is the dynamic matrix strategy, which handles both scheduled runs and on-demand tests across specified platforms.

**Scheduled and On-demand Testing**

In this example, the workflow is being triggered automatically at 22:00 every day for nightly tests. Additionally, we enabled manual triggers for specific platforms, allowing for targeted testing as needed. This flexibility is crucial for maintaining a fast-paced development cycle without compromising on quality.

```yaml
on:
  schedule:
    - cron: '0 22 * * *'
  workflow_dispatch:
    inputs:
      platform:
        description: 'Choose platform to run the test on'
        required: true
        type: choice
        options:
          - webos
          - tizen
          - playstation4
          - playstation5
          - xbox
```

This configuration leverages the `schedule` and `workflow_dispatch` events in GitHub Actions. The `cron` syntax schedules automated test runs, while `workflow_dispatch` allows for manual execution with platform-specific parameters.

**Dynamic Test Matrix Configuration**

The core of our workflow dynamically adjusts the test matrix based on the context of the trigger—whether it's a scheduled job covering all platforms or a manual job focusing on a specific platform.

```yaml
jobs:
  setup-test:
    runs-on: ubuntu-latest
    outputs:
      platform: ${{ steps.set-matrix.outputs.platform }}
    steps:
      - id: set-matrix
        run: |
          if [ "${{ github.event_name }}" = "schedule" ]; then
            echo 'platform=["webos", "tizen", "playstation4", "playstation5", "xbox"]' >> $GITHUB_ENV
          else
            echo 'platform=["${{ github.event.inputs.platform }}"]' >> $GITHUB_ENV
```

In this snippet, we employ conditional logic to populate the platform variable differently based on the trigger type. For scheduled runs, we test across all platforms, while manual triggers allow for focused testing on a selected platform. This strategy ensures that resources are allocated efficiently, focusing our testing efforts where they are most needed.
Of course, you could always do that with an external script or with a composite action, but this is a simple and effective way to achieve the same result.

**Executing the Tests with Dynamic Configuration**

Our approach to executing tests in this workflow benefits significantly from the dynamic nature of GitHub Actions, especially when it comes to handling multiple platforms which can be quite different from each other. This flexibility is further extended into our application's `package.json` file, where we define specific scripts for our testing needs. The `package.json` plays a crucial role in integrating the GitHub environment variables directly into our test execution process.

Without getting into the tests code, you can imagine that each platform has its own specific tests and configurations, and that's why we need to run the tests separately for each platform. This is where the dynamic matrix comes into play.

```yaml
- name: Run tests
  working-directory: ./src/lr-launcher-e2e
  env:
    npm_config_env: ${{ matrix.platform }}
  run: npm test
```

Here, the `npm test` command is executed within the context of each platform specified by the dynamic matrix. This ensures that our E2E tests are relevant and efficient, automatically adapting to the complexities of multi-platform support.

Here's a look at the relevant part of our `package.json` file:

```json
{
  "name": "lr-launcher",
  "version": "1.0.0",
  "scripts": {
    "pre-test": "node ./utils/setupTest.js --",
    "test": "mocha \"tests/*.test.js\" \"tests/$npm_config_env/*.test.js\" --no-timeouts --retries 1 --exit"
  },
  "dependencies": {
    "axios": "^1.6.2",
    "mocha": "^10.2.0"
  }
}
```

This script utilizes the `npm_config_env` environment variable, which we set dynamically based on the selected platform in our GitHub Actions workflow. By incorporating `$npm_config_env` into the mocha command, we ensure that the test runner picks up not only the general tests from tests/_.test.js but also platform-specific tests located in directories named after each platform (e.g., tests/webos/_.test.js).

This setup allows us to maintain a clear and organized test structure where platform-agnostic tests are separated from those that are platform-specific. It ensures that when our GitHub Actions workflow runs, it only executes the relevant tests for the specified platform, making our testing process both efficient and comprehensive.


Automate cross-platform testing with GitHub Actions, using dynamic matrices for efficient, targeted tests.

GitHub Actions for Dynamic Cross-Platform Testing


Makefiles are powerful tools in a developer's arsenal, capable of significantly improving efficiency and consistency in the build process. By understanding and utilizing Makefiles for Docker and Terraform, developers can streamline their workflows, reduce repetitive tasks, and focus more on the creative aspects of software development.

In this article, we'll explore the basics of Makefiles and how they can be used to automate the build process with examples of Docker and Terraform commands usage.

### What is a Makefile?

Incorporating a Makefile into your project can greatly streamline various development tasks, such as managing dependencies, running tests, building documentation,formatting code, performing static analysis, build/compile/transpile, cleaning up temporary files and more.

The flexibility of Makefiles allows you to automate routine processes in a structured and efficient manner. Here's a more detailed look at how you can leverage Makefiles in your projects:

### Basic Structure of a Makefile

A Makefile typically consists of targets, prerequisites, and recipes. Targets represent the goals you want to achieve, which could be actions like running your app or installing dependencies. Prerequisites are files or other targets needed before a target can be executed. Recipes are shell commands that make will execute.

For instance, you might have a `run` target to execute your Python application, a `setup` target for installing dependencies listed in `requirements.txt`, and a `clean` target for removing Python bytecode files or the `__pycache__` directory.

### Advanced Techniques

**Default Goals and Help Messages:** Setting a default goal, like `.DEFAULT_GOAL := help`, allows you to display a helpful message when make is run without a target.

**Virtual Environments:** Integrating virtual environments in your Makefile ensures that the right Python environment and dependencies are used. This can be crucial in projects with specific version requirements or multiple dependencies.

**Injecting Paths into `PYTHONPATH:`** This advanced trick allows Python to search for packages in specified file system paths without needing to install them, which can be particularly useful in projects with heavy dependencies or multiple branches.

### Some Practical Examples

**Running Tests with Coverage:** You can create a coverage target to run tests with coverage, specifying the commands for running the tests and generating a coverage report.

**Linting and Static Analysis:** A lint target can be set up to run various linting and static analysis tools like flake8, pylint, and mypy on your code.

**Task Dependencies and Parameters:** Makefiles allow you to specify dependencies between tasks. For example, a test target might depend on a lint target. You can also pass parameters to tasks, like specifying an IP and port for a serve task.

**Makefile for Docker Commands:**

By leveraging Makefiles in your Python projects, you not only save time but also create a more robust and maintainable development environment. These examples and advanced features illustrate just how versatile and powerful Makefiles can be in automating and streamlining your Python development workflow.

```makefile
export TOP := $(shell git rev-parse --show-toplevel)
export IMAGE_NAME := my-docker-image
export CONTAINER_NAME := my-container-image

build:
	docker build --build-arg JFROG_USERNAME=${JFROG_USERNAME} --build-arg JFROG_ARTIFACTORY_TOKEN=${JFROG_ARTIFACTORY_TOKEN} -t $(IMAGE_NAME) .

run:
	$(TOP)/scripts/run_docker.sh &

exec:
	docker exec -it $(CONTAINER_NAME) /bin/sh

delete:
	docker stop $(CONTAINER_NAME)
	docker rm $(CONTAINER_NAME)
```

In this example the `build` command creates a Docker image with build arguments (`make build`).

The `run` command runs a script to start the container in the background (`make run`). It runs a bash script that starts the container in the background and saves the container ID in a file.

The `exec` provides an interactive shell within the container for debugging proposes (`make exec`).

The `delete` stops and removes the container (`make delete`).

**Makefile for Terraform Commands:**

Terraform, an infrastructure as code software tool, also benefits from Makefiles, its commands can become quite complex. Here's an example on why it is useful to use Makefiles for Terraform:

```makefile
account := dev
namespace ?= $(shell echo $(USER) | awk '{print tolower($0)}')
service ?= $(shell basename `git rev-parse --show-toplevel`)

ifeq ($(account), dev)
    AWS_PROFILE := 123456
else ifeq ($(account), stage)
    AWS_PROFILE := 654321
else
    $(error Invalid account specified: $(account))
endif

init:
	terraform init -reconfigure \
		-backend-config=config/$(account)/backend.tfvars -backend-config=key=services/$(namespace)/$(service).tfstate

plan:
	terraform plan -out /tmp/$(service).tfplan \
		-var-file=config/$(account)/terraform.tfvars -var-file=config/$(account)/service.tfvars \
		-var "namespace=$(namespace)" -var "service_name=$(service)"

apply:
	terraform apply /tmp/$(service).tfplan

validate:
	@terraform validate

destroy:
	terraform destroy \
		-var-file=config/$(account)/terraform.tfvars -var-file=config/$(account)/service.tfvars \
		-var "namespace=$(namespace)" -var "service_name=$(service)"
```

This Makefile automates various Terraform commands like `init` which initializes a new or existing Terraform configuration (`make init`).
This command is particularly used to initialize the working directory containing Terraform configuration files, with -reconfigure to force re-initialization. It specifies backend configuration variables and the state file location.

The `plan` command creates an execution plan. This command creates a Terraform plan and outputs it to a temporary file. It specifies variable files and sets additional variables like namespace and service_name (`make plan`).

The `apply` command applies the changes required to reach the desired state of the configuration (`make apply`).

The `validate` command is used to validate the syntax and configuration of the Terraform files (`make validate`).

The `destroy` command is used to destroy all resources managed by the current Terraform configuration (`make destroy`).

In conclusion, Makefiles stand as a fundamental and powerful tool in software development, offering a versatile approach to automating and streamlining various aspects of the build process. Whether you're working with Docker containers or managing infrastructure with Terraform, Makefiles bring a level of efficiency and consistency that can significantly enhance your workflow. By effectively leveraging Makefiles, developers not only save valuable time but also ensure a more robust and maintainable development process, making them an essential component in any modern software project.


Makefiles are a cornerstone tool in the software development process, particularly useful for streamlining and automating the build process.

Why Makefile is One of My Essentials Tools in Every Software Project


At work, I manage a repository for a front-end application, which includes Markdown files that are compiled into a documentation portal.

Documentation, especially in Markdown format, is more than just words on a page. In my case, it's a user's guide, a developer's reference, and sometimes, even a product's first impression. Therefore, ensuring its quality, consistency, and accuracy is critical, and this is where Markdown linters come into play.

### Essential Tools for Markdown Linting I Used

Our approach involves three key tools, each serving a unique purpose:

**Markdownlint:** For stylistic consistency and syntax correctness.

**Markdown-link-check:** To validate all hyperlinks are functional.

**cspell** (Code Spell Checker): For ensuring correct spelling throughout your documentation.

In my case, we used to run these linters on each pull request/commit that was uploaded to the repository.

### Linters

**Markdownlint:**

```bash
npm install markdownlint-cli
```

Ensures Markdown files follow best practices and maintain a consistent style. Of course, as anything the following different rules, it can be customized to your needs.

Markdown lint follows its defined rules, you can find it [here](https://github.com/markdownlint/markdownlint/blob/main/docs/RULES.md).

To modify it, you can use a configuration file, `.markdownlint.json`, which can be placed in the root of your repository.

For example:

```json
{
  "MD001": false,
  "MD013": false,
  "MD028": false,
  "MD029": false,
  "MD036": false,
  "MD033": {
    "allowed_elements": ["iframe", "url"]
  }
}
```

This config file ignore certain rules defined by the tool, and allows specific elements that are not allowed by default.

**Markdown-link-check:**

```bash
npm install markdown-link-check
```

Checks for broken links in your Markdown files, ensuring all external references are valid. Note that this tools fails to check internal links from obvious reasons, therefore you can configure it to ignore them.

You can use `mlc_config.json` file to configure the tool, and place it in the root of your repository.

```json
{
  "ignorePatterns": [
    {
      "pattern": "^https://github.com/myorganization/"
    }
  ]
}
```

myorganization is the name of your organization in GitHub in this example.

**cspell:**

```bash
npm install cspell
```

Catches spelling mistakes, a critical aspect of maintaining professional and clear documentation. Although sometimes you might have words which aren't really a typo, but are not in the dictionary, therefore you can add them to the dictionary.

You can use `cspell.json` file to configure the tool, and place it in the root of your repository.

````json
{
  "words": ["tizen", "webos"],
  "ignoreRegExpList": [
    "\\!\\[[^\\]]*\\]\$[^)]*\$",
    "(?<=```[\\s\\S]*?)\\S+",
    "\\[.*?\\]\$https?://[^\$]*\\)"
  ]
}
````

Note that I also added regex expression to ignore certain patterns, such as links, code blocks, and images alt text.

### Setting Up GitHub Actions Workflow

Integrating these tools into a GitHub Actions workflow ensures that every pull request adheres to your documentation standards. Here’s how you can configure the workflow for each tool:

```yaml
name: Documentation Checks

on:
  pull_request:
    paths:
      - 'docs'
      - 'docs/**'

jobs:
  markdown-lint:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Setup-env
        uses: ./.github/actions/setup-env
        with:
          node-version: '18.16'

      - name: Run markdownlint
        run: npm run lint

  link-checker:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Setup-env
        uses: ./.github/actions/setup-env
        with:
          node-version: '18.16'

      - name: Run markdown-link-check
        run: npm run link-check

  spell-checking:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Setup-env
        uses: ./.github/actions/setup-env
        with:
          node-version: '18.16'

      - name: Run cspell
        run: npm run spell-check
```

This is a simple workflow that runs on every pull request, and runs each linter separately.
In this case I specified the paths to the docs folder, but you can change it to your needs.

In the beginning of the workflow, I used a very simple composite github action to install node and npm, and then run the linters so i won't have to duplicate the same steps in each job. It's a simple action, but it's very useful, you can improve it to your needs.

If your'e interested in the composite action, here it is:

```yaml
name: 'Setup Environment'
description: 'Checkout and setup Node.js'
inputs:
  node-version:
    description: 'NodeJS version'
    required: true
    default: '18.16'

runs:
  using: 'composite'
  steps:
    - name: Setup Node
      uses: actions/setup-node@v3
      with:
        node-version: ${{ inputs.node-version }}

    - name: install dependencies
      shell: bash
      run: npm install
```

![markdown-linters-workflow](/images/markdown-workflows.jpeg)

By integrating these Markdown linters into your GitHub Actions workflow, you elevate the standard of your project's documentation. This setup not only ensures consistency and correctness but also encourages a culture of quality and attention to detail within your development team. Remember, good documentation reflects the quality of your code and your commitment as a developer.


Do you have a repo that contains important documentation? Streamline your documentation with Markdown linters to enhance documentation quality.

Markdown Linters for Your Docs Repo


In continuation to my [previous post](./hosted-machines-monitoring), I took the OpenTelemetry Collector Contrib project and wrapped it in a minimal CLI using the [Appeal library](https://github.com/larryhastings/appeal/). This CLI is also compatible for all the common OS's (Windows, MacOS and Linux).

The goal was to make the OTEL binary a little bit more focused for our team use case, and easy for installation and usage.

## So What is Appeal?

Appeal is a command-line argument processing library for Python, like argparse, optparse, getopt, docopt, Typer, and click. But Appeal takes a refreshing new approach - the easiest approach I've seen so far. You're basically write Python functions, and Appeal translates them into command-line options and arguments.

Although its official documentation declares it wasn't tested on Windows yet, I found it to be working just fine (on Windows 10).

## My Use Case

I wanted to create a CLI that will be able to download the latest OTEL binary from the official GitHub repository, install it, run it, stop it and uninstall it without the user to worry about anything.
So in my use case at work I created the following commands: Download, Install, Run, Stop, Uninstall and Help is being generated for us automatically due to appeal's nature.

To start the CLI, I created `otel_cli` with a shebang so it'll be able to run as a script.
In addition to that i've added the imports used, initialization of the Appeal object and some global variables that will be used later.

```python
#!/usr/bin/env python3

import os
import shutil
import signal
import socket
import tarfile
import subprocess
import appeal
import ssl
import platform
import sys

from urllib.request import urlretrieve

app = appeal.Appeal()

ssl._create_default_https_context = ssl._create_unverified_context

CORALOGIX_DOMAIN = 'somedomain.coralogix.com'
CORALOGIX_APP_NAME = 'CI-Machine'
OTEL_PID = 'otelcol.pid'
_OTEL_CONTRIB_DIR = 'otel-contrib-dir'

#...commands...#

if __name__ == '__main__':
    app.main()

```

Now let's dig in into the commands themselves.

**download command**

```python
def get_machine_type():
    arch = platform.machine()
    if arch in ['AMD64', 'x86_64']:
        return "amd64"
    elif arch in ['arm64', 'aarch64']:
        return "arm64"
    else:
        print("Failed to detect architecture")
        raise ValueError(f"Unsupported architecture: {arch}")


def get_os_type():
    if platform.system() == 'Darwin':
        print("Detected MacOS")
        return "darwin"
    elif platform.system() == 'Linux':
        print("Detected Linux OS")
        return "linux"
    elif platform.system() == 'Windows':
        print("Detected Windows OS")
        return "windows"
    else:
        raise ValueError(f"Unsupported OS: {platform.system()}")

@app.command()
def download(version="0.89.0", ignore_cache=True):
    '''Download the OpenTelemetry Collector binary for the current machine.'''
    machine_type = get_machine_type()
    os_type = get_os_type()
    filename = f"otelcol-contrib_{version}_{os_type}_{machine_type}.tar.gz"
    url = f"https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v{version}/{filename}"
    print(f"Downloading: {filename}...")
    if not os.path.exists(_OTEL_CONTRIB_DIR):
        os.mkdir(_OTEL_CONTRIB_DIR)
    urlretrieve(url, f'./{_OTEL_CONTRIB_DIR}/{filename}')
    print("Download completed.")

```

The `download` command is designed to fetch the OpenTelemetry Collector binary that matches the user's system architecture and operating system. This command leverages Python's built-in `platform` and `urllib` libraries to determine the system specifics and retrieve the necessary files.

`get_machine_type`: This function uses `platform.machine()` to identify the system's architecture. It supports both amd64 and arm64 architectures. If the architecture is not recognized, it raises a ValueError.

`get_os_type`: This function uses `platform.system()` to determine the operating system (Darwin for macOS, Linux, and Windows). Like `get_machine_type`, it raises a ValueError if the OS is not supported.

The download function first retrieves the machine type and OS type using the above functions. It constructs the filename for the binary to download, using the version number, OS type, and machine type.

The binary's URL is generated based on the filename. It checks if the directory `_OTEL_CONTRIB_DIR` exists and creates it if it doesn't. The binary is downloaded using `urlretrieve` and stored in `_OTEL_CONTRIB_DIR`. And eventually the function prints a message to indicate the completion of the download.

**install command**

```python
@app.command()
def install():
    '''Install the OpenTelemetry Collector binary for the current machine.'''
    files = [f for f in os.listdir(f'./{_OTEL_CONTRIB_DIR}') if f.endswith('.tar.gz')]
    if not files:
        raise FileNotFoundError("No tar.gz file found for installation.")
    tar_file = f'./{_OTEL_CONTRIB_DIR}/{files[0]}'
    print(tar_file)

    print(f"Installing {tar_file}...")
    with tarfile.open(tar_file, 'r:gz') as tar:
        tar.extractall(path=f'./{_OTEL_CONTRIB_DIR}')
    print("Installation completed. You can now run the application by running: otel_cli run")
```

The `install` command is responsible for setting up the OpenTelemetry Collector binary on the user's system. This command assumes that the binary has been downloaded using the download command and is present in a specific directory.

The function begins by searching for `.tar.gz` file within the `_OTEL_CONTRIB_DIR` directory. This is where the downloaded binary should be located. If no such file is found, it raises a FileNotFoundError.

Once the file is located, the script prints the path of the tar file, indicating the binary to be installed. Using Python's `tarfile` module, the script opens the .tar.gz file in read mode and extracts its contents to \_OTEL_CONTRIB_DIR.

After successfully extracting the files, the script prints a message indicating that the installation is complete and that the user can now run the application.

The use of `tarfile.open` in the with statement ensures that the tar file is correctly opened and closed after the operation, following Python's recommended practice for file handling. The `extractall` method is used to extract all contents of the tar file into the specified directory. This approach abstracts the extraction process from the user, making the installation process straightforward and user-friendly.

**run command**

```python
@app.command()
def run(config_file="config.yaml"):
    '''Run the OpenTelemetry Collector binary for the current machine.'''
    binary_name = 'otelcol-contrib'
    if sys.platform == 'win32':
        binary_name += '.exe'

    otel_col_path = os.path.join(os.getcwd(), _OTEL_CONTRIB_DIR, binary_name)
    if not os.path.isfile(otel_col_path):
        raise FileNotFoundError(f"{otel_col_path} binary not found. Please install first.")

    os.environ['CORALOGIX_SUBSYS_NAME'] = socket.gethostname()

    if os.path.isfile('.env'):
        with open('.env') as f:
            for line in f:
                key, value = line.strip().split('=', 1)
                os.environ[key] = value

    os.environ['CORALOGIX_DOMAIN'] = CORALOGIX_DOMAIN
    os.environ['CORALOGIX_APP_NAME'] = CORALOGIX_APP_NAME

    print("Starting OpenTelemetry Collector...")
    process = subprocess.Popen([os.path.join(os.getcwd(), _OTEL_CONTRIB_DIR, 'otelcol-contrib'), '--config', config_file])
    with open(OTEL_PID, 'w') as otel_pid:
        otel_pid.write(str(process.pid))
    print(
        f"OpenTelemetry Collector is running in the background with APP_NAME: "
        f"{os.environ['CORALOGIX_APP_NAME']} and Subsystem: "
        f"{os.environ['CORALOGIX_SUBSYS_NAME']}")
```

The `run` command in the CLI tool is designed to start the OpenTelemetry Collector with a specified configuration. This command is crucial as it activates the collector, allowing it to begin its telemetry data collection and processing.

The function starts by setting the default binary name to `otelcol-contrib`.
It then checks if the platform is Windows `(sys.platform == 'win32')`. For Windows, it appends .exe to the binary name, accommodating the typical executable format in Windows.

The script constructs the full path to the binary using `os.path.join`, combining the current working directory, the \_OTEL_CONTRIB_DIR, and the binary name.
If the binary does not exist at the constructed path, a FileNotFoundError is raised, prompting the user to install the binary first.

The script sets the `CORALOGIX_SUBSYS_NAME` environment variable to the hostname of the current machine, using `socket.gethostname()` (this is how the machine your'e monitoring will displayed in your future Coralogix dashboards).
It then reads environment variables from a `.env` file, if present, and sets them in the current environment.
The `CORALOGIX_DOMAIN` and `CORALOGIX_APP_NAME` are set to a predefined values (global variables above).

The script prints a message indicating the start of the OpenTelemetry Collector.
It uses `subprocess.Popen` to start the OpenTelemetry Collector in the background, passing the path to the binary and the configuration file as arguments.
The process ID (PID) of the started collector is written to a file named `OTEL_PID`. This PID is used later for stopping the collector.

After successfully starting the collector, the script prints a message with details about the running instance, including the APP_NAME and Subsystem, derived from the environment variables.

**stop command**

```python
@app.command()
def stop():
    '''Stop the OpenTelemetry Collector.'''
    if not os.path.exists(OTEL_PID):
        print("OpenTelemetry Collector is not running or pid file is missing.")
        return

    with open(OTEL_PID, 'r') as otel_pid:
        pid = int(otel_pid.read().strip())

    try:
        if sys.platform == 'win32':
            subprocess.run(['taskkill', '/F', '/PID', str(pid)], check=True)
        else:
            os.kill(pid, signal.SIGTERM)

        os.remove(OTEL_PID)
        print("OpenTelemetry Collector has been stopped.")
    except ProcessLookupError:
        print("Process not found. It may have already been stopped.")
    except Exception as e:
        print(f"Error stopping the process: {e}")
```

The `stop` command in the CLI tool is designed to safely terminate the OpenTelemetry Collector that's running in the background. This command is crucial for stopping the collector's processes and ensuring that resources are properly released.

The function first checks if the `OTEL_PID` file exists. This file contains the process ID (PID) of the running OpenTelemetry Collector. If the file does not exist, it indicates that the collector is not running, and the function returns after printing an appropriate message.

To read the file with the PID in it, for Windows systems `(sys.platform == 'win32')`, it uses the `subprocess.run` method to execute the `taskkill` command, forcibly terminating the process with the specified PID.
For other operating systems, it uses the `os.kill` function with the SIGTERM signal to terminate the process. This is a more Unix/Linux traditional way of handling process termination.

The script then removes the `OTEL_PID` file, signifying that the collector has been stopped.

The script also includes exception handling for situations where the process may not be found (ProcessLookupError) or other unexpected errors occur. This ensures that any issues during the stopping process are gracefully caught and reported to the user.

Eventually, the script prints a message to inform the user that the OpenTelemetry Collector has been stopped successfully, providing clear feedback on the action's outcome.

**uninstall command**

```python
@app.command()
def uninstall():
    '''Uninstalls the OpenTelemetry Collector files and binaries.'''
    if os.path.exists(_OTEL_CONTRIB_DIR):
        print(f"Uninstalling OpenTelemetry Collector from {_OTEL_CONTRIB_DIR}...")
        shutil.rmtree(_OTEL_CONTRIB_DIR)
        print("Un-installation complete. OpenTelemetry Collector has been removed.")
    else:
        print("OpenTelemetry Collector is not installed or has already been uninstalled.")
```

The `uninstall` command is a crucial component of the CLI tool, enabling users to remove the OpenTelemetry Collector and its associated files from their system. This command ensures that users can cleanly and easily reverse the installation process.

The command starts by checking if the `_OTEL_CONTRIB_DIR` directory exists. This directory is where the OpenTelemetry Collector's binaries and related files are stored post-installation. If the directory exists, indicating that the collector is installed, the script prints a message to inform the user that the uninstallation process has begun.

It then uses the `shutil.rmtree` function to recursively delete the \_OTEL_CONTRIB_DIR directory, thereby removing all the files and binaries related to the OpenTelemetry Collector. This function is a powerful tool in Python for directory removal, capable of deleting a directory and all its contents.
After successful removal of the directory, the script prints a confirmation message indicating the completion of the uninstallation process.

If the \_OTEL_CONTRIB_DIR directory does not exist, the script assumes that the OpenTelemetry Collector was never installed or has already been uninstalled. In such cases, it prints a message stating that the OpenTelemetry Collector is not installed or has already been removed.
This check prevents the script from attempting to delete non-existent directories, which could lead to errors.

## Usage

For convenience, add the `otel_cli` executable to your PATH and update its symlink to point the python binary. You can do that by running the following command (MacOS):

```bash
ln -s $PWD/otel_cli /usr/local/bin/otel_cli
```

To download the OpenTelemetry collector, run the following command:

```bash
otel_cli download
```

Then you can install the OpenTelemetry collector as a service by running the following command:

```bash
otel_cli install
```

You can start the OpenTelemetry collector by running the following command:

```bash
otel_cli run
```

You can stop it by running the following command:

```bash
otel_cli stop
```

You can uninstall it by running the following command:

```bash
otel_cli uninstall
```

For further help simply run:

```bash
otel_cli help
```

And it'll output all the available commands and their descriptions (taken from the functions docstring).


In continuation to my previous post, I'll show you how to create a minimal and compatible CLI using the Appeal library.

Easy Minimal CLI Using Appeal Library


I encountered this task at work where I needed to monitor a local machines that are considered critical for us in terms of tests runners and CI.
The machines were running a few services that required some effective monitoring and alerting.

Recently, our team has transitioned from NewRelic to Coralogix for monitoring, and integrating it with OpenTelemetry Collector Contrib was done for all of our local machines. This post details the steps involved.

After a short research, I found out that OpenTelemetry Collector Contrib has a [hostmetrics receiver] that can be also used to monitor local machines and send the data directly to our new vendor - Coralogix.

OpenTelemetry provides a single set of APIs and libraries that standardize how you collect and transfer telemetry data. You can use these APIs and libraries to instrument your code and send telemetry data to a backend of your choice. The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry data. In addition, it removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source observability data formats (e.g. Prometheus, etc.) sending to multiple open-source or commercial back-ends.

**So let's start**

Download the otelcol-contrib binary for your platform from OpenTelemetry [GitHub](https://github.com/open-telemetry/opentelemetry-collector-releases/releases).

If you're using Coralogix as well, you'll need to obtain your Coralogix Private Key and set it along with other required environment variables:

```bash
CORALOGIX_DOMAIN
CORALOGIX_APP_NAME
CORALOGIX_SUBSYS_NAME
CORALOGIX_PRIVATE_KEY
```

Create a `config.yaml` file, including necessary metrics, scrapers, log level, and intervals. Refer to [OpenTelemetry's Host Metrics Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) for further information.

Add coralogix as an exporter in the config, as shown below as an example:

```yaml
receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      network:
      cpu:
      memory:
      disk:
exporters:
  coralogix:
    traces:
      endpoint: 'otel-traces.${CORALOGIX_DOMAIN}:443'
    metrics:
      endpoint: 'otel-metrics.${CORALOGIX_DOMAIN}:443'
    logs:
      endpoint: 'otel-logs.${CORALOGIX_DOMAIN}:443'
    domain: '$CORALOGIX_DOMAIN'
    private_key: '$CORALOGIX_PRIVATE_KEY'
    application_name: '$CORALOGIX_APP_NAME'
    subsystem_name: '$CORALOGIX_SUBSYS_NAME'
    application_name_attributes:
      - 'service.namespace'
    subsystem_name_attributes:
      - 'log.file.name'
      - 'service.name'
    timeout: 30s
  logging:
    verbosity: normal
service:
  pipelines:
    metrics:
      receivers:
        - hostmetrics
      exporters:
        - coralogix
```

Execute the binary with the config file flag:

```bash
./otelcol-contrib --config=config.yaml
```

And there you go. You should be able to see the metrics in your metrics vendor exploration by the subsystem name or application name.

![Coralogix-Widget](/images/coralogix-widget.png)


Learn how to seamlessly integrate OpenTelemetry Collector Contrib with Coralogix for efficient monitoring of local hosted machines.

Monitor Your Hosted Local Machines with OpenTelemetry Collector Contrib


I'm not really sure why, but for some reason I couldn't find many references on troubleshooting and setup NewRelic with FastAPI. I'm not sure if it's because FastAPI is still fairly new, or maybe it's because it's so simple to setup that no one really bothered to write about it, so I thought there's no harm on doing so.

When I started this task at work, I couldn't believe how intuitive and easy it was to integrate the NewRelic agent and getting amazing set of metrics and insights on my application without much effort.
Monitoring applications is a critical part of any software development cycle, and NewRelic provides an amazing Application Performance Monitoring (APM) tool for this purpose.

So here we're going to integrate NewRelic with FastAPI and Docker, allowing us to collect valuable data and monitor our applications more efficiently:

1. To start off, we need to install the NewRelic Python agent by running the following command:

```bash
pip install newrelic
```

2. Next, we need to import the NewRelic agent in our application file:

```python
import newrelic.agent
```

3. Set Up NewRelic Configuration;

There are two options here - Either download the `newrelic.ini` template file from NewRelic and fill in your valid New Relic license key.
You can find details about the configuration file [here](https://docs.newrelic.com/docs/apm/agents/python-agent/configuration/python-agent-configuration/#agent-configuration-file).

For Docker users, environment variables can be used as well, and this is actually my recommended way to do this. You just need to set `NEW_RELIC_APP_NAME` and `NEW_RELIC_LICENSE_KEY`.
Then pass in as an environment variable to your Docker container (either via Dockerfile or docker-compose.yml file).
On our case, we're fetching the license key from Vault using Terraform, which will be covered in a different blog post at some point.

4. Continue on and add an ENTRYPOINT for the NewRelic agent in your Dockerfile above the CMD command:

```dockerfile
ENTRYPOINT ["newrelic-admin", "run-program"]
```

If you're running the app without Docker, you can run your application with with additional command of newrelic agent, for example, here's how you can use it with gunicorn start command:

```bash
newrelic-admin run-program gunicorn app:app --config 'gunicorn.conf.py' --preload $*
```

5. To set up NewRelic, we initialize the agent and register our application:

I'd rather to wrap the two init agents commands with my own setup function, so I can pass in the application name as an argument:

```python
def setup_newrelic(app_name: str):
    newrelic.agent.initialize()
    newrelic.agent.register_application(name=app_name, timeout=10.0)
```

Then, we initialize the NewRelic agent in our gunicorn/Uuvicorn application:

```python
if __name__ == '__main__':
    setup_newrelic(app_name=f'{os.environ.get("NEW_RELIC_APP_NAME")}')
    uvicorn.run(app, host='0.0.0.0', port=int(os.environ.get('SERVER_PORT')))
```

By doing this, you will have enabled APM, which provides a wealth of useful metrics.

![APM-example](/images/apm.png)

6. Creating a Custom Event

Finally, to create a custom event in NewRelic, we can do the following:

```python
newrelic.agent.record_custom_event(event_name='ExampleEventName',
                value={'status_code': response.status_code},
                application=newrelic.agent.register_application())
```

The event_name must be a string, and value requires a dictionary with as many attributes as you wish to include.

Here's an example of a custom metric I created that helped me debug a large application where I wanted to get all exceptions types:

![custom-metric](/images/custom_metric_nr.png)

That's it! You now have NewRelic integrated with your FastAPI application running in a Docker environment. The possibilities for monitoring and analyzing your application are now significantly broader and more robust, helping you ensure that your application is always running at its best.


Enhance your FastAPI application's monitoring with NewRelic. Gain insights and improve performance with very minimal and simple setup steps.

DevOps Articles

All DevOps Articles