New Test Track Release - Built from the Ground Up

The time is finally here, the rewrite of Test Track is finally on its way. With the rewrite comes with performance improvements but many new features and improvements as well. Below is a list of a high level list of what’s change, but these are more detailed further down this post.

• Ability to delete all test plan steps even when used in a test run

Comment badge fix

2FA Support

Login via SSO (Social Sign on)

Improved detection for Flaky Test Detection

Improvements to automated test run submission

Adding prerequisites for a test plan that are copied to a test run

Improved project selector

Improvements to task status with rare bug

Adding attachments to test plan

We hope you find these improvements useful, and there’s plenty more to come in the future, part of this rewrite was to improve our developer experience making it easier for us to maintain and develop new features. The detailed information about the improvements listed above are detailed below.

Ability to delete all Test Plan Steps

The old version only allowed deleting test steps that were not part of a test run. If a test step was attempted to be deleted on a step that had been used in a test run, you got an error indicating that it couldn’t be removed as it formed the history of a test run step. However, this now does a soft delete so the history is maintained, but it can be deleted from the test plan, so it is not included in future test runs.

2FA Support

Users can now enable 2FA (Two Factor Authentication) for your account if using email/password login instead of SSO. This supports 2FA via authentication apps like Auth and Google Authenticator, or via email.

Login via SSO

You can login Social Sign On (SSO) using the below social providers, if you have have any others that should be included, drop me an email at [email protected].

• GitHub

• Google

• Microsoft

Improved Flaky Test Detection

The process for detecting flaky tests has been completely rewritten bringing a more reliable detection. Originally, it only looked at that test run, and checked if within that test run flipped between something failed or warning and then to passed. Now however, it looks at the last 5 previous test runs and the state of each step to determine a more reliable and more realistic result as to whether a test step is considered flaky. If there are less than 5 test runs available, flaky test detection still works with a minimum of 2, however a moving threshold is used to determine when a test is flaky.

Improvements for Submitting Automated Test Runs

The API and therefore the test-track-cli has been updated to add extra features. The original way of working is still present where you can submit a file to Test Track, however, it now detects if the same file is submitted multiple times so it will get rejected to avoid duplicating results unnecessarily. However, some test management frameworks create multiple files, so now you can use the —directory instead and all the JUnit XML files will be collected and submitted to Test Track under a single test run. Not only that, but there is also now optional —branch option to specify the branch the test run is running against (if not provided defaults to main) and also —tags option to provide a comma separated list of tags that are applied to the test run.

Test Plan Prerequisites

Sometimes there are certain tasks that need to be performed before starting a test run, whether that’s running a command to reset your test environment, or seed data into a database to ensure a consistent state. Now when creating a test plan you have the option to create a prerequisite and this prerequisite is copied to the test run. You can optionally require that all users have to acknowledge that they have read and understood the prerequisite before they can make any changes to the test run, ensuring more consistent test results and less likely to have flaky tests due to an inconsistent test environment.

Project Selector

Previously, when viewing a project you had a project switcher that only gave you access to your favourited projects, however, now the switcher has all projects with a filter box to be able to quickly switch between projects instead of bouncing back and fourth between the project list page and the project details page. This also removes the need for the starred projects on the project list page.

Comment Badge Fix

When clicking on the comment badge within a test run step, it now opens the task in the side bar and automatically goes to the comments tab instead of just opening the task

Rare Task Update Graphic Bug

On a rare occasion, you could update one task, but a different tasks status would change as well. This was only a visual Test Track Rewrite Change Notes 2 update, and a page reload would correct the display, but the state change is now working correctly.

When Is Release Available

The new release is scheduled for Saturday 12 July @ 10am UTC. I normally try and do updates much quicker as usually its non service affecting, however, in this instance, this is a rare time where I have to take Test Track offline as its a completely new install and involves database migrations. You can view the maintenance window and register for updates at https://status.test-track.com/

Conclusion

I hope you like the updates in this new version. Although every step has been taken to avoid bugs and my standard regression tests have been completed to ensure everything works as expected, this is a rewrite so there might be the occasional bug. If you come across any bugs, or if you have feedback or a feature request, please drop me an email at [email protected] or raise a support ticket at https://support.devsosoftware.com.