1. Comprehensive (overall) Test Plan
1 Purpose
This document will provide a description of the testing activities to be
performed on HummingBird during development. It will list out a general
test approach, as well as individual test to be performed. This document
will also describe the circumstances of the tests as well as a bug tracking
system, and code control.
2 Overall Testing Approach
HummingBird will be modular and its development will be by module so its
testings will be modular. Each of the major parts of the software rely
on at least one other part, so modules will be tested in incremental steps.
At each increment the module will be tested with cooked data and real data
so that unusual and normal situations will both be tested.
For each implemented feature a list of tests will be run to try out
the new module. Once determined that the feature runs well in normal use
patterns it will be put through additional tests designed to break the
module.
At the point where more than one module is integrated the compatibility
of both of the modules will be tested. To determine how well the two function
together as well as to determine if one module can break another. This
will continue as more modules are added to the system.
At each stage of implementation and integration the system will be somewhat
functional. The parts will not fill all requirements but will allow a user
to interact with them on some level. When integrating this will allow the
system to be used as a user might at each stage. Testing as a user will
occur whenever the functionality allows.
Because each module of the system is important and depends almost equally
on the rest of the system, testing will be about equal on each module.
There maybe a sightly larger amount of testing on the user interface because
of the variety of possible input from a user.
Near the end of the project the system will be checked for portability.
The system as a whole will be compiled and tested on the target platforms.
This testing will also test the packaging and build scripts to see if they
work properly.
HummingBird will be automatically tested as well, a large database of
test data, that could be entered into configuration sources. But parts
of HummingBird are interactive and can not be tested automatically, these
parts must be tested manually. The user interface tests will be this kind
of test. They will entail the tester to click on buttons and enter in data,
to test the functionality.
3 Entrance and Exit Criteria
A section of code will be ready to test when minimal amount of functionality
is present. As more and more functionality is added more and more tests
will be performed. Once a module has tested through the pre-defined and
automated test list it will be passed on to integration testing. At this
point integration tests will occur until the modules pass all functional
tests. Once the whole system is complete a run though every test will be
done until all tests are passed.
4 Automated Regression Testing
Regression testing on HummingBird will occur when new modules in the system
are added. Each new module will be build on other modules so the interaction
and dependences must be checked at every addition. It will not be necessary
to test the whole system every time a source level change it made, only
when a major module is integrated into the system whole.
An automated test suite will be created that will run as an option during
the build process. The tests will be composed of Expect scripts, and C
programs to test the functional requirements of code modules. The source
will be compile into object code with the tests and be run, the expected
output will be compared to the actual output as it occurs. Any differences
in the expected output will be show and the test name will be printed.
The automated test suite will also contain full system tests. Where
multiple configurations of the entire HummingBird system are created. Each
configuration is installed in a running Hummer and cooked data is sent
to the system. The many pieces output from the Hummer will be compared
to the expected output.
The the tests will be designed to cover most every module or function.
The greatest effort will be to test the branches of every module.
5 Test Environment
The test environment will be the development machine and then target machines.
HummingBird will be developed on a Sun Ultra-1 Creator3D using SunOS
5.5 (or Solaris 2.5). The target machines will be:
-
Sun Ultra-1 Creator3D using SunOS 5.5 (or Solaris 2.5)
-
HP 9000 series workstation using HP-UX B.10.xx
-
i386 based workstation using FreeBSD 2.2-STABLE
-
i386 based workstation using Linux 2.0.xx
6 Code Control
Concurrent Versions System (CVS) 1.9.6 will be used to maintain source
code control. CVS is built upon Revision Control System (RCS) and keeps
record and changes of the code. The system can report the version of the
code per file and binary. Also groups of files can be ``tagged'' so that
they can be retrieved as a package. By creating a tag for a group of files
we can manage the test versions.
7 Defect Reporting and Tracking
GNATS, the GNU Problem Report Management System, version 3.2 will be used
to keep a log of defects. GNATS keeps a search-able record of submitted
bugs, as well as problem tracking. Each problem exists in one state: analyzed,
closed, feedback, open, or suspended. The problem can be moved between
these states, and records of what time, date who and why are kept. GNATS
makes it easy to keep detailed problem logs by leading the user through
the necessary information to log the full problem.
Jamie Marconi
12/13/1997