Scenario-based Testing

Best Practices: Scenario-Based Testing

Scenario Tests

When you have all the test pieces, as well as sufficient in-depth understanding of the AUT, you can start building much broader tests to verify the overall functionality. Verifying business requirements, sometimes referred to as “use cases”. Behavior-Driven Development – BDD – is a very good approach for this sort of testing.

When it comes to testing the API, the Scenario tests should cover all of the business logic. That way, the UI team is allowed to concentrate on the user aspects: presentation and usability.

Scenario-based tests should be directly tied to user stories, which were probably provided to you by the product owner or some business stakeholder.

The product owner has to report on the current status of the project to stakeholders, and so it will help you to make these tests as understandable for her as possible. If your speciality is test automation, you may not have the in-depth domain knowledge to be able to correctly set up each of the use cases.

Under ideal circumstances, these tests should be the user stories, so that your business colleagues can both easily help you develop the test cases and understand the results. (See the World Of API Testing section article Test First.)

Consider the following example:

Story: Upsell products.

In order to generate future sales,
As a tool-store owner,
I want to print out sale coupons for related items at checkout.

Scenario: Upsell a hammer for minimum threshold of nails sold.

Given a customer purchases nails,
And the amount of nails is over 2 boxes,
Then the customer receives a 20% coupon code for a hammer at checkout.

Each of the lines in the above scenario can be translated into an automated test using a tool like HipTest. However, even tools that do not use Gherkin natively can be adopted to follow specification by example.

When developing the scenario-based tests, you will be facing two problem areas: test data, and time-sensitive tests.

Test Data

As the user stories become more detailed enough, and cover more and more edge cases, it will become increasingly more difficult to find appropriate test data to verify all edge cases.

However, you should always give preference to real-world test data – a copy of production data. Only as a second choice should you start to synthesize new data.

The reason for this is that in scenario-based testing, the business owner is the most interested role in the team. Your business team are often familiar with the data that is available in production, or they may even suggest particular data (user, account, et c) that is already available in production when writing the user stories. (See the Best-Practices section article Working With Data.)

Time-Sensitive Data

Time-sensitive tests are a special subcategory of scenario based testing, and deals with verifying functionality depending on exact time or date.

In the simplest case your test environment may be connected to a third-party environment that does not operate 24 hours, but only during business hours.

Worst case, you are working from a time zone that is directly opposite to when that environment operates, as is common with off-shoring.

In this case your tests will have to run from a scheduled service during hours of operation for the third-party environment. You would preferrably use a CI – Continuous Integration system.

In a scheduled environment you would have to account for the much longer turnaround time for developing your tests: develop test one day, wait for results next day, adjust tests, repeat.

A scheduled environment requires significantly more logging than what you would have normally.

This is because you don't have the ability to see the test in real-time, and you can examine the test inputs and outputs only after the fact.

A more complicated scenario is when you have to enter data one day, and get results another day. An example is a lottery system: you place bets one day, the lottery drawing could happen at the end of the week, and you get your results (wins and loses) the following day.

To manage this, you will need to develop some additional logging mechanism to keep track of your test's activity one day, and have that log read by your test suite at a later time.

In the example of a lottery, you will need to keep track of perhaps all the ticket IDs purchased, and when the right time comes check them all for winnings to be able to test redemption correctly.

Large Procedures

Probably the most complicated time-sensitive tests are verifying large procedures that your AUT runs periodically, such as end-of-moth or end-of-year reconciliations in a bank. To sufficiently test these procedures you will want to run them several times.

However, you will probably not have multiple months to wait for the end of the month to roll around. Furthermore, the end of the year reconciliation has to be tested before the actual end of the year comes around.

For these types of tests you will need special test environments, where you will be able to enter the necessary data, and then roll the time forward at a faster pace than the real-world clock.

Setting up such an environment will need a lot of coordination with the operations team, since things like the machine clock is normally synchronized automatically to a time server.

Setting up tests entirely inside of a virtual environment, will lessen some of the problems, as virtualization solutions normally provide a way to control the virtual machine's clock, as well as easy snapshots and rollback mechanisms.

Having automated deployments will make a lot of this setup much easier. Again, schedule extra time in your effort estimates to account for any unforeseen problems. (See the World Of API Testing section article DevOps Trends.)

Follow the Data

Lastly, there is a special case of CRUD tests that should be addressed somewhere in your scenario-based tests.

This is a case where one (or more objects) are a pre-requisite for another object. In other words, the update operation is an entirely different CRUD test in itself.

For example, consider an airline booking engine. One CRUD test could involve creating a new flight in the system.

And a second test would use this flight for CRUD operations of individual seats.

This is a more thorough end-to-end test, compared to separate CRUD test of a flight, and a separate CRUD test of seats on a flight that is already in the database.

Domain Specific Language

One often overlooked area, is that correct industry-specific language is being used by your application.

For example, in the banking industry, money that is entering or leaving the corporation is referred to as a “deposit” or “withdrawal”, respectively. Money that is moving between different internal accounts, within the same corporation, is referred to as “credit” and “debit”. There is no such thing as “uncredit”!

Using correct language makes it less like to confuse anyone integrating with your API. Using incorrect language makes your application seem unprofessional, and as was already mentioned, loss of customer confidence is a very difficult thing to get back.