Almost everyone does unit testing during development, system testing during QA, and occasionally performance testing just before production. Integration testing typically gets dropped, and hostile testing isn’t even on anyone’s radar. That is a mistake. All are essential to high quality software.
Unit Testing
At its core a unit test is a functional test for a unit of code. Consider building a bridge. Like software, it is built by first creating pieces: the foundation, the support pillars, the suspension, etc. The pieces are then assembled to create the bridge. In software, the code units are the pieces. Once they are completed, you assemble them to create the system.
As you’re assembling the pieces, it would behoove you to check each piece before assembly of the bridge. The last thing that you want is to build the bridge and have it fall down because of a flaw in the support pillars. It would be better to test the pillars at the beginning of constructions to ensure they will hold enough weight. If they don’t, you still have time to correct the flaw, or adjust your design.
It also protects you against sudden changes in the pieces. If you need to switch out the concrete for the foundation, you can apply the same test to the new concrete to ensure it matches your needs.
Keep in mind that the weight you expect may be wrong because of a flaw in your design. In such cases, the tests of the support pillars will pass, but the bridge will still fall down. Unit tests by themselves are not complete. You can’t rely on them alone.
Good OO dictates that a unit of code encapsulates data and behavior. A good unit test will ensure that piece of code correctly manipulates the data independent of the context in which it will be used.
A good unit test for a foundation pillar in a bridge is to ensure it can support weight up to 10,000 lbs. You may be interested in using the pillars for a smaller bridge that only weighs 2,000 lbs. per pillar. The unit test at 10,000 lbs. will give you confidence your pillars will work for similar, but larger bridges.
Your code is an independent API. Pretend many developers will use it in every system in the world. Then design your unit tests. Don’t get in the mindset of testing the code only in the context you intend to use it for. The context will almost certainly change many times before the code reaches the end of its life. Ensuring each unit of code works in any context will make maintenance much easier.
Consider testing a custom implementation of java.util.List. You may just need the get and add methods for the system you are writing the code for. To be complete, you should implement all the methods properly. You should also test every method. You should also test adding nulls, or other edge cases even if they are impossible in the context in which the new List will be used. The code using the List may change, or it may be reused in another system.
There are numerous other benefits to unit testing. I’ll allow the unit test zealots to make those points. I will add one more thing. Most everyone considers a unit to be one class. That may not necessarily be the most efficient use of unit testing. Many classes we write are there to tie stuff together, which sometimes results in small classes and methods. Consider expanding the scope of “unit” until you have a group of classes with interesting behavior to test.
For example, in my current project we have a service layer and a DAO layer. Often the implementation of a service method is just to call the same method on the DAO layer. In such cases it might be more efficient to write one unit test that test both instead of two separate tests.
Integration Testing
Many developers don’t do integration testing with the excuse that it duplicates either the system test or the unit tests. They are wrong. The difference is in the focus. Unit tests focus on the functionality of a unit of code. System tests focus on the functionality of the system as a whole. Integration tests focus on the interaction between 2 or more units, components, or systems.
Often the tests exercise the same code, but the pass criteria is different. It isn’t the intention of the integration test to ensure that component A or component B is functioning properly. It should only check that component A and component B communicate with each other properly.
Consider the bridge that we are building. It would be a good idea to check that the support pillar can properly attach to the frame, and the connection can support the expected weight. The benefit is to find flaws in how the components fit together before the bridge is built and falls down. The fact that the integration test also tests the pillar and the support frame is incidental. The focus is to test how well the 2 pieces fit together.
Automated integration tests have many of the same benefits that automated unit tests have. It too can ensure bugs are caught early during development, and help provide a safety net when doing refactoring. The difference is in the kind of bugs caught.
System Test
System test ensures the system as a whole functions properly. System testing our bridge will ensure it won’t fall down when a car drives across it.
Everyone should understand what system testing is and its benefits. I won’t explain it further. I would, however, like to add a bit more. First, agile development has changed the scope of system testing, and the job of the system tester.
Back in the day, testers job started when the developers completed the system. Ideally the testers would only need to regression test the system only a hand full of times as developers fix all the bugs. In most cases it wasn’t worth the effort to automate those tests. Agile has changed that. Now the system needs to be tested a few times every few weeks. Agile development methodology has made automated system testing necessary.
This has changed the job of the testers from clicking buttons on the screen to writing scripts that click the buttons on the screen. Testers now need to be programmers. In exchange, they get a new title: QA Engineer.
I’m all for keeping metrics on system testing, but keeping track of the number of bugs found doesn't tell you anything about the quality of your system test process. It is like keeping track of the number of bad drugs the FDA keeps off the market, or the number of lead based toys are caught before making it to Wal-Mart. A much more interesting statistic is the number of dangerous toys being sold, or the number of bugs found in production. In other words, look at the number of bugs missed in system testing. Which is better, a system test that finds 1000 bugs and allows 100 into production, or one that finds 100 and allows 0 into production? The number of bugs found says more about the development process then the testing process.
Performance Testing
Like the other types of testing, performance testing has a unique focus. The pass/fail criteria should be related to the number of calls that can be made, and/or the lag time of each call.
The scope, however, can very. You can write a performance test at a unit level to ensure a particular algorithm is efficient. You can write one at the integration level to ensure a service or external call will be fast enough. More popular though, is to do performance testing at the system level to ensure the it has the performance and scalability required.
In the context of our bridge, performance testing would be placing weight on a pillar or the foundation until it fails, or more likely, it could be placing weight on the completed bridge to ensure it won’t fail when it is full of cars.
Remember the idiom: premature optimization is the root of all evil. That does not translate into: premature performance testing is the root of all evil. If you have tight performance, memory, or scalability goals, it may be worthwhile to do performance testing at a unit and integration levels during development, and include those tests with other automated unit and integration tests. Be smart about this. It may not be necessary to test everything, and additional tests can be added once a performance problem is found.
You should not optimize the code until it fails to meet your system wide goals. Wait until the system is put together and do system level testing. If it fails then, you can use the unit and/or integration level tests while your doing your optimizations. You can then tighten your goals on them to ensure future changes don’t hinder the performance.
I will spare you the details of how to do proper system level performance testing. There are numerous resources that you can find on that intraweb thingy.
Hostile Testing
The focus for this testing is on what happens when things go wrong. I don’t mean testing expected errors. That should be defined in the requirements and covered by the system tests. I mean unexpected errors.
If your system retrieves customer information from an external system, then retrieving a non-existent customer is a system test. Testing what happens when the external system doesn’t return at all is a hostile test.
Consider the bridge. A hostile test is ensuring the bridge can withstand high winds, earth quakes, car crashes, boat crashes, etc. The purpose is to ensure the bridge does not collapse under these circumstances. That would take an already bad situation and make it many times worse.
Likewise hostile testing on software systems is done to ensure one failure doesn’t cascade into a much bigger disaster. A failure in one system should not cause any system that communicates with it to fail also, or at least they should fail in a controlled manner.
Hostile testing is usually geared towards communications with external systems (including databases, web servers, etc). The external systems will often need to be mocked, which is easier to do at an integration level then a system level. However, it isn't that hard to mock external systems at a socket level if your just going to drop the connection, or return random data.
Michael Nygard has explained hostile testing in greater detail. I would highly recommend looking through his blogs and articles and picking up his book Release It!
Conclusion
Most development shops concentrate on system testing, and unit testing. Some even do performance testing. Testing is an important part of any development effort. It needs to be as complete as possible.
The fact that integration, performance, and hostile testing is time consuming and hard is no excuse. If you skip any of them, you simply do not know what will happen. Weigh that risk against the cost of testing. Is it still too expensive?
What do you think? Are all forms of testing worthwhile in every case? Is there any other forms of testing that i missed? Please leave your rating +1 Martin Fowler (brilliant) or +1 Paris Hilton (stupid).
No comments:
Post a Comment