Automated Testing

Automated testing as a strategy

Automated testing is a great and powerful tool for ensuring consistent code coverage, performing fast regression tests, validating builds and locating potential code problems.  The introduction of automated testing into your processes should give you a noted increase in productivity and QA throughput. While it is true that the automated testing will improve your coverage and productivity, it is not the answer to every 'QA bottleneck' issue in history.  Beware of jumping to conclusions about what automated testing will bring to the table and instead focus on where it gains the most value.  It will never replace human testing in its entirety.  It simply will not replace manual testing in aspects such as product exploration and environmentally varied user story testing.

Automated tests will remove much of the burden from a human resource by testing a single action or logical group of actions repeatedly.  That is the great gift of automated testing, but it cannot entirely take the place of a human with platform knowledge exploring the product for cause and effect testing. It is important to focus your testing investments where they will have the most return.

Mike Cohn created a test automation pyramid indicating a good break down of test investments.  I have taken that and applied it to the platforms we are dealing with daily.  This pyramid illustrates a healthy distribution of test investment for complete platform coverage and explains how the investment in tests should focus at the unit level and then reduce up through the application layers.

Unit Tests

Unit tests should be your highest investment in code coverage.  These tests are created by the developers as they write the corresponding code. In a SOA world, every line of code in a service should have a corresponding unit test.  This includes all methods, extensions, data contract validations, CRUD operations and authentication routines.  These should have a test created as part of the development process.  Even when a modification is made to existing service, the modification must have corresponding tests to prove it is functioning as requested.  By its nature, a unit test will inherently give your platform total code coverage at a very base level.  It is important that all unit tests deal with every possible code path within the method being tested.  Cyclomatic complexity limitation is a standard specific metric that should be enforced with scripts, tools or code reviews, but will keep your unit tests, as well as your methods, clear, concise and focused.  For the sake of focus, I am avoiding the CC soapbox for today. This flavor of automated testing gives you the most absolute code coverage and is the base for automated testing.

Integration Tests

Integration testing is created by QA automation employees and is meant to validate functionality of entire user stories.  This is where other considerations such as performance should be measured. Integration tests should be SOA tests, meaning that the integration test plays the role of the presentation layer in executing a chain of tasks comprising user stories. These tests are constructed to validate and measure larger functional items that consist of many small service calls. A good example of an integration test is a long story consisting of customer creation, order creation, payment creation, editing of account credits and debits and comparing the end result against known values for outcome, performance and A/R.  The integration test provides regression testing at the user story level and needs to be done to all USL and SDK-API methods, as they are the outer most points in the SOA.  This type of automated testing offers the most user story coverage and should comprise the primary basis for validating the quality and functionality of your platform.

UI Tests

The UI test is the process of ensuring a UI meets specifications. UI tests are created by QA automation employees and are simply a robotic replacement for a human moving through the platform and executing the elements within a UI.  UI testing is a valid and valuable platform for product stability, but is ultimately a less indicative test of total platform quality than the areas before it.

Human Testing

Human testing should be approached as primarily an exploratory testing methodology.  In this environment, a person familiar with the platform spends the time varying individual environment variables and testing the functionality of the platform.  This is the layer at which testing no longer reflects a series of repeatable actions and begins to reflect an individual creatively trying to find flaws in a system. This is a valuable test concerning ensuring that a platform will be stable when not being used correctly or encountering anomalies in the standard environment. 
If approached with the proper expectations and focus, automated testing can be the answer to many aspects of QA that seem slow and redundant.  If we ensure the proper level of investment at each area of testing, the platform coverage can be complete and result in a much faster and more thorough acceptance process.