Pytest is a Python testing framework that makes it easy to write small, readable tests, and can scale to support complex functional testing for applications and libraries. In this tutorial, we will explore how to integrate the test results generated by a Pytest automated test run within TestRail using the TestRail CLI. This will allow you to centralize your automated test results and take advantage of all the analytical and reporting capabilities TestRail provides.
Throughout this tutorial, we will be using a sample project to guide you through the process of setting up a Pytest automated tests project that is compatible with the TestRail CLI and uploading the generated test results.
After reading through this tutorial, you will be able to:
- Execute tests from a Pytest project
- Install the TestRail CLI
- Configure your TestRail instance
- Run the CLI
- See your test cases and test results on TestRail
To be able to install and run a simple Pytest project, all you need is Python to get started. Since the TestRail CLI that will be used to import the test results is also a tool developed using Python, this will also allow us to install and run it.
Download the version for your operating system and follow the install wizard instructions.
To make sure the install was successful, try executing the commands
Installing the sample project
Let’s start by fetching the sample project code and installing the required dependencies.
- Clone or download the sample project
- Open your command prompt on the project root folder and execute the command below
$ pip install -r "requirements.txt"
Pytest – Python functional testing framework
Exploring the sample project
Use your favorite IDE to open the sample project and start digging into the test files. We’ve kept the automated tests code simple so that the focus of this tutorial remains on how to import the execution results. These tests consist of simply opening a page on the browser using selenium and performing a few actions and validations, such as the one on the example below.
import pytest def test_sum_two_numbers(): assert 1 + 1 == 2 def test_sum_two_decimals(): assert 0.8 + 0.3 == 1.2 @pytest.mark.parametrize("test_input,expected", [("3+5", 8), ("2+4", 6)]) def test_sum_multiple_numbers(test_input, expected): assert eval(test_input) == expected
Executing the sample project
On the same command prompt, execute the command below to run the Pytest tests in the project and save the results in JUnit XML format.
$ pytest --junitxml "junit-report.xml" "./tests"
If the command was correctly executed, there should be a file called junit-report.xml on the reports folder, as per our command line options, with test results in JUnit XML format. This is the file which will be parsed by the TestRail CLI in order to create the test run and upload your test results to TestRail on the next step.
<?xml version="1.0" encoding="utf-8"?> <testsuites> <testsuite name="pytest" errors="0" failures="1" skipped="0" tests="5" time="0.063" timestamp="2022-07-22T11:51:34.060104" hostname="LAPTOP-09USROG2"> <testcase classname="tests.test_subtraction" name="test_subtract_two_numbers" file="tests\test_subtraction.py" line="0" time="0.001"/> <testcase classname="tests.test_sum" name="test_sum_two_numbers" file="tests\test_sum.py" line="3" time="0.000"/> <testcase classname="tests.test_sum" name="test_sum_two_decimals" file="tests\test_sum.py" line="7" time="0.000"> <failure message="assert (0.8 + 0.3) == 1.2">def test_sum_two_decimals(): > assert 0.8 + 0.3 == 1.2 E assert (0.8 + 0.3) == 1.2 tests\test_sum.py:9: AssertionError </failure> </testcase> <testcase classname="tests.test_sum" name="test_sum_multiple_numbers[3+5-8]" file="tests\test_sum.py" line="11" time="0.001"/> <testcase classname="tests.test_sum" name="test_sum_multiple_numbers[2+4-6]" file="tests\test_sum.py" line="11" time="0.000"/> </testsuite> </testsuites>
Importing results to TestRail
After the tests have been executed and the junit-report.xml file is generated, you can easily import your test results (and test cases!) to TestRail by using the TestRail CLI. This will bring visibility to your automated test runs and will enable you to look at the big picture of how you’re actually testing your app all within TestRail.
Installing the TestRail CLI
Given you already have Python installed on your machine, installing the TestRail CLI is as simple as executing the following command on your command line.
$ pip install trcli
Secondly, you need to configure your TestRail instance according to the instructions below.
- Enable the TestRail API by going to Administration > Site Settings, click on the API tab, and checking the Enable API option.
- Create a Custom Field in order to map your automated test cases code to the actual TestRail cases. You can do so by going to Administration > Customizations and clicking Add Field. After you’ve reached the field creation screen, there are two requirements for this custom field:
- The System Name must be automation_id
- The Type must be String
Sending results to TestRail
After you’re done installing the TestRail CLI and finished configuring your TestRail instance, you can upload your test results by simply using a one-liner such as the one below.
$ trcli -y \ > -h https://INSERT-INSTANCE-NAME.testrail.io \ > --project "My Project" \ > --title "Pytest Automated Test Run" \ > --username INSERT-EMAIL \ > --password INSERT-PASSWORD \ > parse_junit \ > -f junit-report.xml
Note that the file name after the
-f option should match the path to your report file in case you change its default location. All others options should be according to your TestRail instance and project. You can check other command line options by checking the README.md file on the project repository, the TRCLI documentation article, or the embedded CLI help through the commands below.
$ trcli --help $ trcli parse_junit --help
Visualizing the results on TestRail
Now, if you go to the Test Cases page in your TestRail project, you’ll notice that the TestRail CLI automatically created the test cases that were on your test results report. Notice that it added a unique Automation ID by combining the
name attributes on each test on the JUnit report. This Automation ID is used to map the tests on your automation code base to the test cases on TestRail. This means that each time you run the TestRail CLI, it first attempts to match an existing test case on TestRail and only creates a new one in case there is no test case with that Automation ID.
If you change the name of your tests, the name of the file or its location, the Automation ID for those tests will change and they won’t be mapped to the existing test cases on TestRail.
On the Test Runs & Results page, we can see that a test run with the name Pytest Automated Test Run was created. By opening it we can dig further into the details of each automated test result and perform a high level analysis of why any test is failing since the error message provided by the test automation framework is also registered on the test result, as you can see on the image below.
Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!
You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.