Automated testing as a strategy
Automated testing is a great and
powerful tool for ensuring consistent code coverage, performing fast regression
tests, validating builds and locating potential code problems. The
introduction of automated testing into your processes should give
you a noted increase in productivity and QA throughput. While it is true
that the automated testing will improve your coverage and productivity, it is
not the answer to every 'QA bottleneck' issue in history. Beware of
jumping to conclusions about what automated testing will bring to the table and instead focus on where it gains the most value. It will never replace human testing
in its entirety. It simply will not replace manual testing in aspects
such as product exploration and environmentally varied user story
testing.
Automated tests will remove much of the burden from a human
resource by testing a single action or logical group of actions repeatedly. That is the great gift of automated testing, but it
cannot entirely take the place of a human with platform knowledge exploring the
product for cause and effect testing. It is important to focus your
testing investments where they will have the most return.
Mike Cohn
created a test automation pyramid indicating a good break down of test
investments. I have taken that and applied it to the platforms we are
dealing with daily. This pyramid illustrates a healthy distribution of
test investment for complete platform coverage and explains how the investment
in tests should focus at the unit level and then reduce up through the
application layers.
Unit Tests
Unit tests should be your highest
investment in code coverage. These tests are created by the developers as
they write the corresponding code. In a SOA world, every line of code in a
service should have a corresponding unit test. This includes all methods,
extensions, data contract validations, CRUD operations and authentication
routines. These should have a test created as part of the development process.
Even when a modification is made to existing service, the modification must
have corresponding tests to prove it is functioning as requested. By its nature,
a unit test will inherently give your platform total code coverage
at a very base level. It is important that all unit tests deal with every
possible code path within the method being tested. Cyclomatic complexity limitation is a
standard specific metric that should be enforced with scripts, tools or code
reviews, but will keep your unit tests, as well as your methods, clear, concise
and focused. For the sake of focus, I am avoiding the CC soapbox for
today. This flavor of automated testing gives you the most absolute code
coverage and is the base for automated testing.
Integration Tests
Integration testing is created by QA
automation employees and is meant to validate functionality of entire user
stories. This is where other considerations such as performance should be
measured. Integration tests should be SOA tests, meaning that the integration
test plays the role of the presentation layer in executing a chain of tasks
comprising user stories. These tests are constructed to validate and measure
larger functional items that consist of many small service calls. A good
example of an integration test is a long story consisting of customer creation,
order creation, payment creation, editing of account credits and debits and
comparing the end result against known values for outcome, performance and
A/R. The integration test provides regression testing at the user story
level and needs to be done to all USL and SDK-API methods, as they are the
outer most points in the SOA. This type of automated testing offers the
most user story coverage and should comprise the primary basis for validating
the quality and functionality of your platform.
UI Tests
The UI test is the process of
ensuring a UI meets specifications. UI tests are created by QA automation
employees and are simply a robotic replacement for a human moving through the
platform and executing the elements within a UI. UI testing is a valid and
valuable platform for product stability, but is ultimately a less indicative
test of total platform quality than the areas before it.
Human Testing
Human testing should be approached
as primarily an exploratory testing methodology. In this environment, a
person familiar with the platform spends the time varying individual
environment variables and testing the functionality of the platform. This is
the layer at which testing no longer reflects a series of repeatable actions
and begins to reflect an individual creatively trying to find flaws in a
system. This is a valuable test concerning ensuring that a platform will be
stable when not being used correctly or encountering anomalies in the standard
environment.
If approached with the proper
expectations and focus, automated testing can be the answer to many aspects of
QA that seem slow and redundant. If we ensure the proper level of
investment at each area of testing, the platform coverage can be complete and
result in a much faster and more thorough acceptance process.