How to test requirements with Xray, Zephyr and RY Testing?

Writing requirements can be seen as a simple writing task, but we both know this can be a little tricky. With tools like Atlassian Confluence and Jira, you can take your requirement management to another level with traceability matrices, and so on. One important part of requirement management is: Testing. You need to test your requirements to make sure they are achievable. Jira doesn’t have a built-in testing solution so when looking for one, make sure to assess the requirements. We’re going to talk requirements testing management with a few of our tips to make your QA Engineer’s life easier by integrating Requirement Yogi to plug-ins like Xray, Zephyr and RY Testing and Compliance.

Table of content:

• Manage simple tests with RY Testing and Compliance?
• How does testing requirements with Xray work?
• What about Zephyr apps for Jira?
• Alternatives for Test Management in Jira

---

How to manage simple tests for your requirements in Confluence with RY Testing and Compliance?

For those of you that need a simple testing system, perfect for smaller projects, RY Testing and Compliance is an easy solution.

The genius of this app is that one can write test procedures in free form in a Confluence page. For the rest, it behaves as every simple testing app: Mark tests as success, failed or any custom status, with some comments, picture or evidence.

Screenshot of RY Testing & Compliance

To proceed, either write your page manually and put the Test Result macro, or use the blueprint to create a new RY Test Session page. You can generate a full page of tests for each requirements with the following options:

• Test requirements on the current page,
• All the requirements on the space,
• Some requirements via their key or a certain query if you do not have the Requirement Yogi plug-in,
• Add them manually with a blanc tab.

Now, you can specify the test associated to the requirement, its result, add Jira links, all in viewing mode. Click on the RY Testing and Compliance byline on the top of the page to edit on view mode. You can also check a live chart with the proportion of successes, fails or undone tests. 

The strength of this requirement testing app is its integration with Confluence’s editor and its accessibility for non-technical users.

How does Requirement Yogi integrate with RY Testing & Compliance?

Although RY Testing & Compliance can be used standalone, it provides the most benefit when used with Requirement Yogi:

• Requirement Yogi displays the test results in a pop-up of the requirement (The most recent for each test relationship),
• The test procedure pages can be automatically generated from a set of requirements,
• You can view the test results in Requirement Yogi’s traceability matrix, and therefore export them to Excel.

The full documentation of RY Testing and Compliance is available here.

Internal Structure

RY Testing and Compliance uses a relationship name for the links to requirements. Indeed, one requirement can be tested several times in diverse environments, the latest execution will be displayed by default for each relationship name. Also, it recognises Requirement Yogi’s baselines: A specific version of the requirement can be tested.

Other than that, test sessions are one-dimensional. There is no complex data structure to track the evolution of a test across several versions of a requirement. In short, if the test plan evolves or if you rerun the test procedure, you will simply copy the original test page with a new list of steps, without the ability to track how the test procedure has changed.

This is why an enterprise testing suite will have much more comprehensive data model behind the scenes. Xray and Zephyr are good examples of this.

How does testing requirements with Xray work?

First, let’s introduce Xray. This is the app we hear the most about. It seems widely-used for large projects. Its strength is to rely on Jira’s data model to create tests: Everything is a Jira issue! You therefore benefit from Jira’s native features: JQL search, customizable issues, permissions, workflows, REST APIs, exports, etc. It really helps to organize the data. Xray adds panels and a lot of coloured status bars, test steps, specialized forms to run the tests. Additionally, since tests and test runs are Jira issues, you can link issues to other issues using Jira’s native functionality.

They organize the various phases of the process with:

• Tests
• Preconditions
• Test plans
• Test sets
• Test executions (test runs)

They also rely heavily on metadata, which helps managing the several dimensions of testing:

• Environments
• Versions
• Labels
• Components

A test plan will include various test sets and tests. From a test plan, it is possible to create a test execution to begin the execution process. Test executions have a dedicated screen to run them, as you can see below, you have the ability to detail the test sets, create defects, post evidence and more, in the process.

An excellent point is that Xray allows automating the tests with BDD Cucumber scenarios. Learn more on Xray’s automated tests on this page.

Finally, so that you can evaluate what Xray brings to the table, Xray Reports are quite powerful. They allow you to view and analyze your plan strategy:

• the success and failure rates of the test plans,
• how many tests were run,
• and other metrics about the daily evolution of the tests.

The full Xray Testing Management document is available on their website.

How does Requirement Yogi integrate with Xray ?

As usual in the Requirement Yogi for Jira app, link your issues to requirements using Alt+Shift+R or the menu entry (“More” → “Requirement Yogi links”).

Thanks to those Jira-to-Requirement-Yogi links, you can build a Traceability Matrix in Confluence with Xray traceability. What we call the Traceability is actually an extract of your requirements with the data that you choose to display. On the following screenshot, you can see someone building a traceability matrix, complete with traces from Requirements to Jira issues to Xray tests.

It leads to the desired view of Xray status inside of Confluence / Requirement Yogi:

This integration allows project managers to have an overview of their projects, with requirements – story in Jira – tests. Now that testing requirements with Xray is clear, you might want to know other plug-ins.

What about the Zephyr apps for Jira?

In the “Testing & QA” section of the Atlassian Marketplace, Zephyr is also one of the top names that come up. We are going to focus on Zephyr Scale, as you can add it directly to your Jira instance, just like the others presented above. 

How is it like to write test cases using Zephyr?

When we tried the app, Zephyr seemed much more straightforward than Xray. It is probably due to their dedicated UI to write test cases. Despite being simple, they have 3 types of entities and are therefore just as capable as Xray for large and complex projects.

You can create tests directly on a Jira Issue, whether an Epic, Story, Bug or Task, in the section “Traceability”. You can also access this screen using the “Tests” section in the top bar.

Firstly, Test cases are singular tests, they have an objective, precondition and other details you can see below. They can be linked to Confluence pages or specific issues, have as many steps of the script as you want. Moreover, you can call to another test, or choose to add a script for automation testing. They also have a great version system, fitted for test history. 

Secondly, a Test cycle is an event formed by a group of tests, there can be numerous test cycles with the same tests or versions of tests. They are made to be run to clearly see the progress bar and an overview of the success and failure rate of the tests. It is easy to assign them to a certain tester. 

Test plans are the same way as in Xray, they are use to manage the strategy of your tests, to organise all your tests.

Lastly, reports in Zephyr Scale are very precise and there are a lot of variables to choose. You can create a report of all the text execution results through wanted periods of time, projects, etc. It then shows a lot of data to better plan and manage the way you execute your tests.

Executing tests

When you execute a test, versions of the test are shown and you can choose a specific one, then the details set up in the test case come up. In addition, you can start a timer on the top-right corner that will time accurately the execution of the script, it will go on when the steps are marked as successful until the last step is completed, but will stop when one of the test is failed. It will allow you to compare the actual start and end date of the planned one and the performance of the tests. 

How does Requirement Yogi integrate with Zephyr?

As we said in the previous chapter, Jira issues can be linked to requirements if you have the Requirement Yogi for Jira connector (free). On the screenshot below, you can see both the test cases and the “Requirement Yogi” sections on the same issue:

Just like Xray, you can retrieve the Zephyr test results directly on the Traceability Matrix of our app. The menu lets you tick a few options to filter: All executions or just the latest, and the ability to separate cells or group results in the same cell. Requirement Yogi also displays the standard Zephyr properties: Component, priority, status and version.

See the documentation about Zephyr Scale.

Alternatives for test management in Jira

There are a lot of plug-ins specialized in project testing and we’ve only shown you the top apps. Feel free to contact us if other apps become popular and an integration is needed with Requirement Yogi.

QMetry Test management (1,9k installations)

This app implements itself directly in your project, and in the same way as Zephyr, there is a special tab to manage your test cases, test cycles and test plans.

Once a test is created, you can link it to a Jira issue or the other way around. You can sort through all your test cases with folders and add many details to the individual tests, including the usual steps, parameters, attachements etc, but also estimated time. Then, you are able to create a test cycle to execute your tests, the interface is pretty clear and it is easy to report bugs and defects. They also provide detailed reports. They can integrate Cucumber and Selenium for your tests automation. The strength of this app compared to Xray or Zephyr is a simpler and intuitive interface wether you are just starting to get used to testing or not. It is very accessible to new users.

As to the requirement links, it works the same way as the other apps, you can quickly link any requirements of your project to an issue with Alt+Shift+R.

All-In-One Tests for Jira (2,7k install)

Just like the other apps presented along the article, AIO Test is an app that implements to Jira. However, this is a free solution, for cloud and server, that includes a lot of good features to improve tests management and automation. As their name says, their app assembles all the important steps of test organisation in one. In other words, with this simple and visually attractive app, you can create test cases, add them to test sets and test plans and create test cycle to execute them manually or with automation tools. Moreover, you can add a lot of details to the test cases and even time your test executions, link the defects or bugs created, etc. Their reporting interface is interactive and allows you to track effortlessly the progress of your project through the tests. 

Even though they call their links “Requirement Links” they only link to Jira issues, however, you don’t need to create Jira Issues when you use the Requirement Yogi for Jira as you can link any requirement from your project to issues with the simple command Alt+Shift+R. You can also access the tests details to your Requirement Yogi Traceability matrix in Confluence. It will show you a quick view of the results and the tests associated with their requirements. 

Requirements are at the center of your test strategy

When people talk about test management, they really talk about requirement management. It is important to find a comfortable tool to write requirements because they will determine both your test plans and project planning. Although you can use the built-in “Requirement” field of Xray, you will extract information much more thoroughly if you start with requirements in Confluence, then build your test plans and plannings from there.

In any case, Requirement Yogi integrates with the two most popular apps of the Atlassian Marketplace for test management, and the tool you choose depends on the size and complexity of your projects.

Visit the Atlassian Marketplace to know more about Requirement Yogi