Integrating with GitHub Actions

GitHub Actions makes it easy to automate all your software development workflows. It allows you to build, test, and deploy your code right from GitHub. Workflows can be kicked off by GitHub events like push, issue creation, new release, or even a manual workflow dispatch.

A common use case for GitHub Actions is automatically building and testing your application whenever code changes are submitted. To improve visibility into your automated tests and manage all your test cases and test results—both manual and automated—more easily, you can send your automated test results directly from GitHub to TestRail. In this article, we will explore how you can take advantage of GitHub Actions and the TestRail CLI to easily accomplish this task. 

 

Please note that the TestRail CLI requires that a compatible JUnit report is generated by your test automation framework.

Using the TestRail CLI in GitHub workflow files

Automated processes can be configured on GitHub by using workflow configuration files. These are YAML files that allow you to specify your processes in a structured manner and make use of GitHub Actions for each step of the way to make it easier to configure your execution environment, as well as execute any action you require.

The easiest way to integrate TestRail with GitHub Actions is by configuring your workflow files to use the TestRail CLI to send automated test results to TestRail. Below is a sample of how a workflow file using the TestRail CLI to send test results to TestRail would look like.

name: send-results-to-testrail
on: [workflow_dispatch]
jobs:
  send-test-results-to-testrail:
    runs-on: ubuntu-latest
    steps:
- name: Checkout uses: actions/checkout@v3
- name: Execute automated tests
uses: ... - name: Python setup if: always() uses: actions/setup-python@v3 with: python-version: '3.x' - name: TestRail CLI upload results if: always() run: | pip install trcli trcli -y \ -h https://TESTRAIL_INSTANCE.testrail.io/ \ --project "PROJECT NAME" \ -u USER_EMAIL \ -p PASSWORD \ parse_junit \ --title "Automated Tests from GitHub workflow" \ --run-description ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} \ -f "junit-report.xml"

Breaking down the file, we can see notice a couple of interesting details:

  • The trigger event on: [workflow_dispatch] means this workflow will only run if manually executed. There are other events that can trigger a workflow, such as pushing code, creating a pull request, etc.
  • On jobs we have one item called execute-tests, which means we only have one job on this workflow.
  • The execute-tests job defines its agent with runs-on: ubuntu-latest and contains the steps below steps, which we will detail on the following sections.
    1. Checkout 
    2. Execute automated tests
    3. Python setup
    4. TestRail CLI upload results

You can learn more about workflows features on the official GitHub workflows documentation.

Step 1: Checkout

The Checkout step uses the actions/checkout@v3 GitHub Action which executes the code checkout on the agent for your repository.

- name: Checkout
  uses: actions/checkout@v3

Step 2: Execute automated tests

The Execute automated tests step is a placeholder. This is the point in time where the automated tests you have in your codebase will be executed and generate the JUnit report required to send the results to TestRail. This can be accomplished either by using a GitHub Action or by inserting your custom execution script.

name: Execute Automated tests
  uses: ...

Step 3: Install Python on the agent

The Python setup uses the actions/setup-python@v3 GitHub Action to configure Python on the agent. Because the TestRail CLI is a tool developed in Python, it is required that it is installed on the environment before you are able to install and run it.

Also, notice the if: always() condition, which forces this step to run, no matter if the automated test run was successful or not, because, by default, most test automation frameworks will exit with an error code if any test failed.

- name: Python setup
  if: always()
  uses: actions/setup-python@v3
  with:
    python-version: '3.x'

Step 4: Install the TestRail CLI and upload test results

The TestRail CLI upload results step runs a custom shell script to install and call the TestRail CLI in order to parse the JUnit report generated by your test automation framework and send the results to TestRail. 

The TestRail CLI is a Python packages hosted on the Python Package Index (PyPI) so, in order to install it, we simply execute the command pip install trcli.

After the TestRail CLI is installed, there are a few mandatory arguments such as your TestRail instance address and credentials, the project you want to report to, the title for your test run and the path to the JUnit report. The run description argument is not mandatory, but we recommend using it to add a link from TestRail to your GitHub workflow execution. This link can be obtained by composing the URL using GitHub embedded variables, such as on the snippet below.

For more information about arguments, please check the TestRail CLI documentation.

- name: TestRail CLI upload results
  if: always()
  run: |
    pip install trcli
    trcli -y \
      -h https://TESTRAIL_INSTANCE.testrail.io/ \
      --project "PROJECT NAME" \
      -u USER_EMAIL \
      -p PASSWORD \
      parse_junit \
      --title "Automated Tests from GitHub workflow" \
      --run-description ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} \
      -f "junit-report.xml"

Working example using Cypress

Below is an example of a GitHub workflow file which, in simple terms, runs Cypress tests and sends the test results straight to TestRail.

name: cypress-tests
on: [workflow_dispatch]
jobs:
  execute-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Cypress run
        id: cypress
        uses: cypress-io/github-action@v4
        with:
          start: npm start
      - name: Python setup
        if: always()
        uses: actions/setup-python@v3
        with:
          python-version: '3.x'
      - name: TestRail CLI upload results
        if: always()
        run: |
          pip install trcli
          junitparser merge --glob "reports/TEST-*" "reports/junit-report.xml"
          trcli -y \
            -h https://TESTRAIL_INSTANCE.testrail.io/ \
            --project "PROJECT NAME" \
            -u USER_EMAIL \
            -p PASSWORD \
            parse_junit \
            --title "Automated Tests from GitHub workflow" \
            --run-description ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} \
            -f "reports/junit-report.xml"

Breaking down the file, there are only two different worthy of note between the generic GitHub workflow we went through before and this one:

  1. The second step was replaced by a Cypress run that uses the cypress-io/github-actions@v4 GitHub Action, which install the Cypress project dependencies and execute the commands on the with parameter, triggering an automated test run.
  2. On the TestRail CLI upload results, there is an extra command to merge the reports generated by Cypress. This is necessary because the TestRail CLI requires only one report and Cypress generates multiple reports. You can learn more about this on the documentation about Integrating with Cypress

Creating your own repository

We're going to setup the GitHub workflow above using our Cypress sample project, which you can know more about by checking our documentation about Integrating with Cypress.

By simply following the steps below you'll have your GitHub workflow with test execution and results reporting to TestRail in no time.

  1. Create a new repository on GitHub
  2. Download the files on the  Cypress sample project 
  3. Add the file cypress-tests.yml under the path .github/workflows and copy the contents from the example above
  4. Replace TESTRAIL_INSTANCE, PROJECT NAME, USER_EMAIL and PASSWORD on the file content using your details (we recommend that you do not replace the password directly and use GitHub secrets instead)
  5. Upload the project to your GitHub repository

Executing the workflow

  1. Go to the Actions page on your repository and you should be able to see your workflow on the list

    actions_page.PNG
  2. Since this is a manual dispatch workflow, to run it, first click the workflow name on the left and then the Run workflow button

    cypress_workflow.PNG
  3. You can then go into your workflow details to see what went wrong. The execution logs are divided by steps. On the image below we can see that the Cypress run failed

    workflow_details.PNG
  4. By drilling into the Cypress run logs, we can see that the execution went well, but one test failed. Although the test failed, the Python setup and TestRail CLI upload results steps were still executed because we used the if: always() condition on our test step to make sure we send the results to TestRail even when tests fail

    workflow_logs.PNG
  5. Go to your TestRail instance and see your Test Run results. Notice that you have a link back to your GitHub workflow so you can easily further debug your test results! 

    testrail_test_run_details.PNG

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.

Was this article helpful?
2 out of 3 found this helpful