Evaluating the SOA testing tools
SOA testing is one of the most under- explored subjects in software testing and often it’s being confused with web service testing. Web services are one of the ways to achieve service oriented architecture but it’s not synonymous to SOA. Hence if you have a SOA application, don’t just restrict your testing to the web services, the scope should include all the components of the application: the service provider, the service directory, service consumer as well as the communication layer between these components.
SOA provides the conceptual model for development, deployment and management of distributed and dynamic system. It’s one step ahead of n-tier architecture model in terms of flexibility and interoperability, hence it has become so popular in last few years. But its full potential can be realized through discipline and standardization of right policy framework, design rules and documentation of different components so that they should act together to achieve the common business goal.
SOA testing tool is not just a functional testing tool neither just the web service testing tool, it can and should be leveraged for the overall SOA governance of the system through different level, from application front end to middleware to data base to message and communication layers. The best approach of getting the best out of the tool is to hook it up with the development (IDE) and unit testing tool used.
There are many popular open source tools available for SOA testing apart from the commercial ones. If we compare the open source and commercial tools:
1. Open source tools have low initial investment compared to commercial tools
2. An Open source tool from a passionate community (which doesn’t have a commercial version) is preferable than the free edition of a commercial entity. My personal experience, the free edition of the commercial company is only for namesake and advertisement; you will hardly find any feature that can be useful for your project.
3. Choosing an open source tool, we need to compromise on the features, service, support and any existing defects in the tool. Paid tool support and service is always better.
4. The running cost of using an open source tool may exceed the license fee of the commercial tool. The hidden costs are like : Resource cost on developing the customized features, risk and delay on known defects in the tool, delay in project because of support not available, Training cost, cost of migration to a commercial tool when required, time spent on searching for solutions when hit a roadblock
5. Personally I have found open source tools mostly lack the non-functional and security testing features compared to their commercial counterparts.
So if you want to start exploring the SOA testing world, start with an open source tool, but carefully choose the appropriate tool when your future strategy is in place. The best open source tool (in my opinion) would be SOAP UI and its commercial version SOAP UI Pro license fees are also quite affordable. Apart from that you can try the free edition of SOAP Sonar (personal Edition) or the free edition of Test Maker from Push to test. Among the commercial tools, I would rate the Para soft SOATest the best, followed by iTKO LISA and HP’s Service tester.
When we evaluate some tool or any language or platform to use, we compare its features with certain industry standards or benchmarks. The benchmark should be based on many factors including (but not limited to) your project requirement, user experience, best practices and future requirement; after sale support, its scalability and adaptability to new technology trends etc.
Let us discuss about the base standard for evaluating SOA testing tools for a typical SOA project (feel free to modify as per your project requirement). As SOA architecture is meant for platform independent, the first parameter should be platform and technology support for the SOA layers. We should also check the Specialized Platform Support. To be platform independent the SOA architecture should follow certain policies and standards acceptable across the applications as well as industry best practices hence Policy Enforcement is needed.
We should evaluate how easily test cases or suites can be developed and its integration with the standard development or test management tools. Hence the features: Test Authoring and Integration with other IDE and test management tools. We need to check the suitability for all types of testing: Manual Testing, Load/Performance Testing, Security Testing and Run time Error Detection during execution of the test suite. SOA application also includes the data base, business logic as well as the front end web application and RIA hence we need to verify its feature for all the application tiers: Advanced Web App Testing, Multilayer Verification and End-to-End Testing. Business logic and user requirement changes very often, so we need Continuous Regression Testing. Test harness and stubs are required to start the testing early and minimize the fixing cost, so we need to check the tool’s capability for Simple Stubbing. At last but not the least its reporting features for reporting and circulating the test results: Reporting & Admin
SOA application consists of many loosely coupled components, and many sensitive data and messages flow through the service interfaces. With increased complexity of business need and security standards we need the features like Complex Messages & WSDLs, Dynamic Messaging Processing, Cryptography & PKI, Identity, Performance, Security, and Service Score Card.
Among these broader features, there can be some functionality which is must for the project, something can be nice to have or something is not needed right away but can be a future need. So accordingly specify the must have, nice to have and future capabilities within the above broader category.