Negative Testing

Best Practices: Negative Testing

Negative testing is probably the most hated category of tests by development: the equivalent of whining. In code, this is handled by checking for every kind of wrong input the user can possibly supply, and a lot of developers consider this to be somebody else's problem: either a user problem, or the problem of the UI client. Nothing could be further from the truth. Catching bad inputs before they hit the business logic, or worse the database, makes perfect sense.

It is not uncommon to find cases where one side of the boundary will still be processed by the API, and the result will be a generic server error instead of a more graceful error telling you what the problem is. A normal response from the development team to such a defect is: “the UI will never allow that.” While that may be correct, it is also important to consider the security implications of such a decision. Very often, malicious crackers will not bother using the UI, but will instead resort to custom tools that directly access the API. Also, such a decision locks the company into the current implementation of the UI because now the business logic is being spread between different tiers of the application.

At minimum you want to discover generic server errors, which in a REST API show as an HTTP500 response, and in a SOAP API show as an unhandled SOAP fault. In the worst case these will display errors from the database revealing the underlying architecture and exposing your API to various forms of attack. These are particularly bad, and are covered further in the Security Vulnerability Testing article. The unfortunate thing for a tester is that there is no straightforward way of finding these, other than the boundary tests discussed previously and more exploratory testing.

Field Types

The very first negative test should be verification of all field types. For example, if a particular field requires an integer, then send in a string.

For SOAP APIs, field type tests should be quite uninteresting (and possibly out of scope), since field type conversion is normally handled by whatever framework development used to generate services. However, field type tests are very important for REST APIs as the type conversion is often done by the business logic. Pay particular attention to non-primitive fields, such as date-time fields, which have to have a particular format.

The second thing to check would be any allowed limits. Of particular interest should be strings and enums. Make sure that the service does not blow up if you send in a string that is 10k bytes long. Always think like a malicious cracker and use your exploratory testing skills.

Required Fields

Setting up a test that verifies the usefulness of error messages is actually quite easy:

1. Make a “blank” call – pass an empty body in the request.
2. Get back an error that you are missing some element.
3. Add the missing element to your input and make the call again.
4. Repeat from 2 until you get back a positive response.

The idea is that the API is self-documenting: error messages lead you to build a complete successful call.

This approach also verifies that all inputs are actually sanity tested in the code. Having helpful error messages is especially important for public APIs. And do not underestimate the importance of spelling and grammar, which when wrong it makes your whole application look less professional and lowers customer confidence.

Providing helpful error messages not only shows respect towards your fellow coders, it also provides a better user experience for your actual users and customers. Strong customer confidence is beyond price. (See the World Of API Testing section article Why Test? for more.)

Database Integrity

Another interesting test is something called “database integrity testing". Data that goes into the application database can become corrupt, especially in a test environment, but in production as well. This can happen due to number of reasons: previous versions of the API (with either buggy or incomplete features), application crashes, or simply by people manually editing the database directly.

You can run a read / retrieve call on every single record in the database, expecting the API to not throw any errors. If it does, that is something worth investigating. It does not immediately indicate an error, since the data can genuinely be corrupt and in that case most APIs will behave unpredictably, but it can reveal interesting and possibly undesirable behaviour of the API.