API Testing Strategy Best Practices

Best Practices: Deciding What to Test

In the article Structuring Your Tests we discussed some techniques on how to break up your tests into individual test-case-sized pieces. In your test career there will inevitably come a situation when you're pressed for time, and you will have to prioritize what to test. This article presents one risk-based testing approach that will allow you to decide what should be tested and when. As was mentioned previously, most modern test frameworks have a method of grouping tests into some named groups, so that you can run just one or more of these groups. The categories described in this article are also part of best practices, and should be included in your named groups. Keep in mind that most frameworks allow multiple named groups to be added to any one test.

Most modern test frameworks have a method of grouping tests into some named groups, so that you can run just one or more of these groups. The categories described in this article are also part of best practices, and should be included in your named groups. Most frameworks allow multiple named groups to be added to any one test suite.

What Not to Test

The first thing you should do is look at your application as a whole, and decide what not to test! Most modern applications are very complex, and usually interface with one or more 3rd party applications. Testing functionality of the 3rd party application should always be out of scope for your test efforts. For example, if you have a shopping cart application, it will have to contact a payment processor during checkout – a 3rd party payment processor. In this case, there is very little value in having separate tests for testing checkout for insufficient balance, checkout for invalid credit card, checkout for expired credit card, etc. This is all functionality provided (and supposedly already tested) by the payment processor. It should be sufficient to test that your application denies the purchase for any payment denial, just once, and correctly presents the error message from the payment processor.

Laws and Regulations

Once you have eliminated all the scenarios that do not apply to your application, the first thing to look at is any laws that apply to your industry. Even if your application functions perfectly, but you break any law in production, quality of the remaining functionality no longer matters. In a best case scenario, your company will be fined and the PR fallout will take huge amount of resources to repair; worst case, your company will be shut down. Laws normally apply to protecting consumer privacy: authorization and authentication (discussed in the State of API Security article), and storing of private information (encrypting credit card numbers, dates of birth, etc.). However, different industries might have additional regulations that apply only to them. For example in the gambling industry, clients have to be not only over the age of consent, they also have to be physically within some geographical borders in order to be allowed to play.

One important note, often misunderstood in testing circles, is that there is a big difference between how the AUT – Application Under Test – stores passwords, and how your test framework stores passwords (or any other personal information). The AUT needs to abide by all applicable laws: not storing passwords in plain-text, salting encrypted passwords, etc. Your test framework is, in all likelihood, using mocked or scrubbed passwords and personal information that is made up just for test purposes: it is pure fiction. As such, there is no point in encrypting your test passwords, and your test personal information, in the framework. As an example, consider dates of birth. In the application, it is against the law to store birthdays in plain-text format. However, it is trivial to create a fictitious user account that has a date of birth less than 18 years old, if that is what your test scenario calls for. Encrypting this birthday in your test harness is pointless, but encrypting this birthday in the AUT is a requirement.

Money Flow

The next area of your application to look at is money flow. It is of course important that you are not losing track of your customers' money (although that should already be covered under the legal category above), but it is equally important that your company is not losing money. Tests in this area should include any incentives that your company may offer to customers. In some cases, like in the hospitality industry, there are discounts for complicated packages that a customer may decide to purchase. Packages could include things like a flight, a hotel, and a car rental with multi-night stay all purchased during a certain time period. The combinations of these discounts are often limited only by the imagination of the marketing department, so effectively unlimited.

It is becoming increasingly common for companies to defer various payments in order to generate extra interest. You should check with your accounting department on specific practices. Following are some of the questions that you should be asking. During complicated calculations, at what point is money rounded off to the nearest cent? Which rounding rule is used? Is the money flowing to/from the correct internal accounts? For batch processing, is the money flowing to/from accounts at the right time?

Technological Complexity

With increasing technological complexity, especially in situations where systems are geographically separated, it is important to ask what happens to the data when things go wrong? Is the transmission retried? When? How often? How many points of failure are there?

For example: When our payment processor goes offline (due to actual problems, or just maintenance), do we turn away all our customers? Do we assume to approve them for certain amount, and process all payments as a batch at a later time? Do any of these “emergency procedures” have legal implications?

For systems that claim 5-nines availability or more, there will have to be failover procedures. Setting up testing of failover is neither trivial nor cheap. It requires technological knowledge, staffing resources, scheduling and hardware. Generally, the more automated this process is, the more manpower will be required during testing. Everything has to be monitored during testing, to be able to track down what went wrong, or at least to prove that everything went right.

Regression Testing

After you are satisfied that no laws are being broken, and that you are not losing any money, the next area to look at is regressions: existing functionality. Nothing annoys a client (whether an actual end-customer or another service consuming your API) more, than having a feature that he used either break or outright dropped. Do not forget that when we are talking about APIs, most of your consumers are in fact going to be other developers. Change or loss of features for them means additional coding and testing on their end, because they still need to provide the same functionality to their customers as before. Regressions should also be the first priority when you are considering which tests to automate. This is because the main point of automation should always be to save your coworkers time, and when a new feature is developed the most time consuming activity is to go back and figure out what else broke or changed.

Regression as a category is probably going to be extremely broad. It effectively represents all the functionality of your application for all time. You can sort all your API methods by the amount of hits they receive. This list of API methods can easily be found with one trip to your production support, and asking the database team to pull this information out of the logs. You will then have to apply some logic to categorize and count the transactions.

Talk With Your Developers

Another great source of information to help you prioritize areas to test, is to have a discussion with your developers. There are two main questions that you should ask them.

1. If you were interfacing with this API, which parts would you need to understand right away? Use the API Hierarchy of Needs and work your way from bottom up.
2. When you were working on development, which parts were
   1. the trickiest,
   2. least clear in the specification,
   3. least unit tested,
   4. most rushed?

The answers will probably help you clarify some of the priorities which may have been previously unclear.