Hidden Costs of Open Source API Testing Tools

In this article, I will discuss some considerations and potential limitations of adopting an open source API testing solution for a long term API strategy . Open Source has become an essential and popular resource for many tools and platforms used in API deployments. From operating systems such as Linux, to databases such as MySQL, and browsers such as Firefox, open source has a proven track record for cost-effective applications and tools. API testing involves the ability to test APIP, XML, and REST based messaging patterns against a API endpoint in order to assess the robustness, reliability, and resilience of the API. Comprehensive testing of a API involves 4 primary areas: Functional, Performance, Interoperability, and Security.
Functional testing provides the ability to verify the proper behavior of APIs and build regression test suites to automate testing and baseline expected behavior of APIs to quickly assess and validate functionality through the lifecycle of API revisions.
Performance testing via a concurrent, simultaneous loading agent framework can determine throughput and capacity statistics of the API across the range of input and client load variances to validate API Level Agreement rates and well as identify bottlenecks and potential architectural weaknesses and performance dependencies.
Interoperability testing maximizes interoperability by measuring both the design characteristics of a API as well as the run-time adherence to standards and best-practices. Isolating potential interoperability issues early on in the lifecyle can significantly optimize efforts of integration when exposing the API to trading partners and clients which may be build on a varying array of technologies and platforms.
Security testing assesses the risk posture and robustness of a API with regard to vulnerability, data-leakage, data privacy, and data integrity. Each API is unique based on the schema and message patterns which defines the input and response message structure of how to communicate with the API. In the case of APIP APIs for example, using the WSDL schema as the source, security tests can be built to create boundary condition tests for the API which then identify the robustness of the API handling inputs outside the range of expectation.
These 4 areas of testing together provide the comprehensive analysis and understanding of the resilience, reliability, and robustness of the API.

Open Source API Testing
The open source tools available today for API testing focus primarily on the functional testing aspects of a API. Since functional testing is often first in the API development lifecycle, and adopted early-on in the development and implementation phase, free open source tools become widely adopted by development teams both because they are free, and also because the use cases are often limited to simple unit testing of API messaging.
However, as APIs mature and move down the API lifecycle phases to system testing, integration, pre-production analysis and validation, the other perspectives of API testing need to play a role in the comprehensive assessment of the quality, robustness, and capabilities of a published API. It is in these areas where the open source API testing tools can fall short.

API Functional Testing Limitations
Generally the functional testing capabilities of an open source testing tool are adequate for the simple type of API deployments which do not have complex WSDLs, Schemas, or message patterns. Once the deployments gain more complexity however, the challenges of functional testing move from single request-response testing to scenario testing where the functional behavior is measured not by one request-response, but rather several transactions each dependent on the other as a business functional unit. Having the ability to test these types of functional scenarios effectively requires the ability to maintain state between one test result and the next.
 

API Performance Testing Limitations
While there are several open source performance testing products on the market, these are primarily tools used in the static web testing paradigms. When dealing with APIP and XML -based transactions, the static data testing behaviors of these performance harnesses does not allow for the unique-wire signature requirements of the message patterns as they occur in actual API transactions. When running performance tests with a web-based testing platform, the results are that the end-points become inundated with static messages which are not characteristic of actual traffic patterns. In fact, in many cases, the API endpoint itself is supposed to reject these static messages as replay-attacks on the API.
Another consideration of performance testing is the level of security and identity provisions that messages may be required to carry in order to access the API. Static open source performance testing harnesses do not provide solutions for message security and message identity requirements.
 

API Interoperability Testing Limitations
The promise of API provides a open, reusable architecture which lowers cost through the reuse ROI factor. The challenge of API however is the ability to widely inter-operate with other technologies that also communicate via APIP, XML, and REST. Interoperability testing involves both design-time analysis of API characteristics, such as WSDL and schema, as well as run-time assessment of a API robustness in terms of consuming and handling message patterns that may fall outside the expected structures. Many open source toolkits leverage the available WS-I analysis framework to provide a means to assess the design-time characteristics of a WSDL and schema according to published profiles, and also provides some run-time analysis reporting of message patterns. However, the Open Source toolkits do not provide the ability to generate messages that fall outside these expected patterns, which is the key to measuring the actual posture of the run-time API. In fact, it is testing of messages that are not expected where the true measure of a API interoperability posture can be determined.

API Security Testing Limitations.
Security testing falls across many areas. From a threat perspective, security testing involves integrity and structure of messages with injection attacks at the parameter and data structure levels in order to assess the behavior and resilience of the API endpoint when faced with data values and message structures outside of the expected format. From a trust perspective, security testing involves PKI with encryption, signatures, and identity tokens. This requires testing frameworks that understand the various emerging standards from W3C and OASIS in order to support the wide range of security message formats and also requires the means to retrieve and utility X509 and Private Keys from a variety of sources including Windows Keystore, Java Keystore, SmartCards, PKCS12 files, etc. Many open source testing tools are designed for general testing and message creation, but lack the in-depth security and identity features to be considered viable for this type of testing.
Conclusion
Adopting a free open source tool for API testing seems the simplest, most cost effective choice for developers and testers early on. However, you should plan and consider the implications of a longer term strategy with an open source testing tool. The many other aspects of API testing which contribute to a comprehensive testing solution across the entire API lifecycle that go beyond simple testing paradigms.
Before adopting a tool, consider whether down the road you may need something more. You may find that paying for a solution ends up costing much less than not paying for one.