A test harness is an API that simulates the behavior of a real API or web service, providing predefined responses (AKA canned data) to the calling application. Using test harnesses is particularly useful in testing scenarios where the actual web service or API is not available, unreliable for testing purposes, or when you want to test specific conditions that are hard to replicate with the live system.
Characteristics of Test Harnesses:
Mocking and Stubbing: The test harness acts as a stand-in for the actual web service or API, mimicking its behavior. The calling application can be tested using a test harness while the real API is still under development. On one project, our development team always updated our test harnesses as one of their first tasks if there were planned changes to our web services.
Canned Responses: The harness is configured with a set of predefined responses to specific requests, allowing for consistent and predictable testing. In one instance, we configured our test harness to return a different set of responses based on the credit card number sent by the calling application. We also provided a set of default response files for card numbers that did not have specific response files set up.
Isolation: By using a test harness, you isolate the system under test from external dependencies, which is beneficial when external systems are down, under development, or have detrimental bugs that hold up UI testing. When it comes to external dependencies, these issues occur all the time. Simply switch the URL in a config file to use the test harness and continue testing the UI.
Controlled Testing Environment: It allows testing in a controlled environment where you can simulate various scenarios like system failures, edge cases, or rare data conditions. A test harness is valuable for testing maximum data lengths for fields, how the application handles garbage data from external sources, or any kind of data you want or need. From my experience, it is easier to mock up test harness data than to try to get someone to create data in an external system. While you wait for them to create the data (since you will eventually need it for end-to-end testing), mocking up data saves time.
Implementation Considerations:
Designing Mock Responses: Carefully design the mock responses to cover a wide range of scenarios, including typical, edge, and error cases. You can set up your test harness to return whichever set of data you need.
Integration with Test Frameworks: The test harness should be easily integrated with your existing test frameworks and suites. For the most part, this shouldn’t be an issue, because a test harness is just another API.
Configuration: It should be configurable to adapt to different testing scenarios without needing to change the test code. You can design your test harness however it works best for you. Different data can be returned based on a user, security profile, credit card number, or any piece of information sent in the API request. Always have a default set of data to fall back on.
Performance: While performance may not be the primary concern, the test harness should not significantly degrade the test suite’s performance. For the most part, a test harness will operate faster than any real API since there is minimal logic involved, and the API can reside on the Dev or Test server.
Test Harness Data Design
Files or database tables can be used to store test harness data. For one application I worked on, our test harness data resided in files located on our Test server. I’m not sure if I have the right terminology here, but the data in our initial files were in some generic .NET serialized format. It was challenging to work with. Since I didn’t have much experience at the time, I asked the lead developer if he could change our test harness so that it could just use the XML files that we already had from our web service responses. He made that change, and our test harness files became XML files.
As I mentioned above, we had it set up to return a file based on the card number sent to the test harness. We had a set of default files that were used if we didn’t have a specific set for the card number sent in.
These files do not have to be XML files. I’d suggest using whatever format your API is using. If your API returns JSON data, set up your test harness to use JSON data. A database could be used to store the data in tables, although I don’t see much benefit in doing this. It’s simpler for most testers to work with text files, whether the data is in XML, JSON, or another format. There could be a benefit to storing the files in a database, depending on the setup of the Dev or Test environment and which servers are regularly backed up.
Conclusion
There are many benefits to using test harnesses for testing. I really can’t think of any downside except for the time needed to implement and update it. Over the course of testing a project, the benefits to development and QA outweigh the few days needed to get a harness implemented. If no one on the Test team can do this, a skilled developer can implement one in no time.