Patterns for System Testing

By: D.E. DeLano, L. Rising
Published in: PLoPD3
Pages: 503-525. See also PHand, 97-119.

Testing patterns for developers and managers, as well as testers.

Category: Testing

Summary: Testing patterns for developers and managers, as well as testers.

Pattern: The Tester's More Important Than the Test Cases

Pages: 506-507

Assign tasks to testers based on their experience and talent. No matter how effective the test cases are, the testing results are dependent on the tester.

Category: Organization and Process, Testing

Pattern: Designers Are Our Friends

Pages: 507-508

How should testers work with designers? Build rapport with designers. Approach designers with the attitude that the system has problems that require cooperation to resolve. Designers and testers have a common goal. Use Get Involved Early and Document the Problem

Category: Customer Interaction, Organization and Process, Testing

Pattern: Get Involved Early

Pages: 508-509

You're a system tester working on a large software project. To maximize support from the design community, establish a working relationship with the designers early in the project, for example, learn the system and the features along with the designers or attend reviews of requirements and design documentation. Invite designers to reviews of test plans. Use Designers Are Our Friends. Don't wait until you need to interact with a designer; by that time it's too late. Trust must be built over time.

Category: Customer Interaction, Organization and Process, Testing

Pattern: Time to Test

Pages: 509-510

Start testing when an area is available, but not before. Reach agreement with designers that the area is ready for testing.

Category: Organization and Process, Testing

Pattern: Take Time

Pages: 510

You're using Get Involved Early and Designers Are Our Friends. When designers are behind schedule, give them the time they ask for. Use Time to Test on other areas. You'll save effort in the long run; testing a poorer-quality system takes more time.

Category: Organization and Process, Testing

Pattern: "Unchanged" Interfaces

Pages: 511-512

How should testing of third-party interfaces be scheduled? Don't fall into the trap of assuming that unchanged interfaces will function correctly in the new system. Test the third-party interface independently when it is chosen. When the project is ready to include the third-party software, immediately test to ensure the third-party functionality behaves as expected.

Category: Testing

Pattern: Ambiguous Documentation

Pages: 512-513

To pinpoint possible problem areas, study the documentation. Look for areas that seem ambiguous or poorly defined. If the designers can tell you everything you need to know about a feature, it probably works. It's what they can't tell you that needs attention.

Category: Testing

Pattern: Use Old Problem Reports

Pages: 513

Examine the old problem reports to find areas that should be targeted for testing. You can't test for all old problems, so concentrate on those from the last valid snapshot.

Category: Testing

Pattern: Problem Area

Pages: 514

What areas of the system should receive concentrated testing, regardless of the features being implemented? Keep a list of problem areas and the test cases to target them. These areas can be identified by: (1) talking to experienced system testers; (2) applying Ambiguous Documentation and Use Old Problem Reports, and (3) looking for areas that designers avoid.

Category: Testing

Pattern: Busy System

Pages: 515

What conditions should be considered for testing to find the most problems? Test in a busy environment, using simulators to provide levels of activity on the system. You don't need to stress the system; use a level of activity that the system could expect to experience regularly during busy times.

Category: Testing

Pattern: Don't Trust Simulations

Pages: 515-516

You're using simulations in your test plans. To configure the testing environment, supplement simulations with real world testing. Testing is not complete without real-world scenarios. A simulator can run a test case successfully a hundred times, but the test may fail when performed by a human because of unpredictable behaviors that are introduced. Use End User Viewpoint

Category: Testing

Pattern: End User Viewpoint

Pages: 516-517

To test new features without repeating any testing, test outside the normal scope of features. Don't use tests already run for feature testing. Use End User Viewpoint.

Category: Testing

Pattern: Unusual Timing

Pages: 517

What additional testing should be done that might not be covered by the test plans for an area? Test unusual timing. Run tests more quickly or slowly than normal. Abort tests in the middle of execution. Use End User Viewpoint. Some of these tests may be difficult to perform; those are exactly the ones that should be executed.

Category: Testing

Pattern: Multiple Processors

Pages: 518

When the system runs on multiple processors, test across all processors. A designer might have only considered one processor, assuming that it would work for all processors. Problems that occur on one processor will probably occur on other processors. Tests that pass one processor may fail on another.

Category: Testing

Pattern: Scratch 'n Sniff

Pages: 518-519

Once testing has started, what's a good strategy for determining the next test case to run? Test areas where problems have already been found. Problems tend to be found in clusters.

Category: Testing

Pattern: Strange Behavior

Pages: 519

You've tested a feature on a previous release and on the new release the feature is working but not exactly as expected. Take any unusual behavior as an indication of a possible problem and follow up. This should be done even if the problem is not related to the current test. Be wary when familiar tests produce results that might be acceptable but not exactly what was expected.

Category: Testing

Pattern: Killer Test

Pages: 519-520

Development is drawing to a close. The system is stable. To give a quick evaluation of the overall health of the system, use a favorite killer test to be run at any time. The test should provide good system coverage and be expected to fail, in some manner, most of the time.

Category: Testing

Pattern: Document the Problem

Pages: 520-521

You're using Get Involved Early and Designers Are Our Friends. To communicate problems found in testing, write a problem report. Don't argue with the designer. Don't accept a well-intentioned promise or document the problem informally.

Category: Testing

Pattern: Adopt-A-Problem

Pages: 522-523

You're using Designers Are Our Friends. A problem has been uncovered without a clear-cut solution. Adopt the problem. Stick with it until it is resolved. Use Document the Problem. Retest the problem periodically to gather data on it. Be aware of Pet Peeve

Category: Testing

Pattern: Pet Peeve

Pages: 523

You're using Designers Are Our Friends, Document the Problem, and Adopt-A-Problem. The validity of the problem has been debated to the point of holding up progress. Be sure the problem doesn't become a thorn in the designer's side. You might consider bringing in a third party (for example, the requirements writers) to resolve the impasse. At this point, stop following the status of the problem.

Category: Testing