A few months ago, we were brought in to test a complex corporate application with the view to verify outstanding issues and provide stability by applying test methodology and process.
At the time we did a lot of research on what testing tool options were available. We wanted something that was accessible from the internet (so we could share our progress and results with our client), allowed for the development of complex test scenarios (we were looking to test different builds in different User Acceptance Testing environments – at the same time), tracked defects, and helped record test scripts for later payback. We wanted the test assets that we created to be in a form that could be handed over to our client once we were done and to provide them with a resource they could use into the future to train new test analysts. We trialled a few options, including a combination of TestRail (for test management) and JIRA (defect management) that seemed promising as they integrated well with each other. But we finally settled on Visual Studio Team Services (VSTS) because it covered all the functions we needed plus it provided a platform for other forms of testing, if we required them in the future – including unit through to system testing, and load/performance testing. It also offered a desktop client for recording test scripts (Microsoft Test Manager syncs with VSTS). It was also free (for teams up to 5 people), and a moderate cost for basic users thereafter, giving us the opportunity to scale test efforts at a reasonable cost. VSTS also works with Power BI – which is something I think we need to explore to provide an easier to digest summary of test progress and metrics for clients.
Our setup essentially consists of:
- One project for all test cycles, cases and scripts for our application of focus
- Agile methodology was applied
- A User Story was created for each issue we needed to classify and test (under Work tab)
- Epics and Features were defined to classify define a heirarchy of system functionality – with User Stories being the leaves
- Iterations were used for tracking the Release an issue was targeted for testing under
- Tags were used for each User Story to identify a defect on an issue raised in the vendor’s defect management system
- An exploratory test cycle and more formal test cycles were defined as Test Plans (under the Test tab). Test cases and test steps were defined for each test plan.
- Code, Build and Release tabs weren’t used
We are still learning how to best use VSTS, but we do know (even with our basic setup) that it has helped us keep consistent, detailed and accurate test records and in turn providing a tools we needed to enforce good testing process.