The Varieties of Testing Experience

There are many different types of tests that are commonly discussed in the literature on this subject. In fact, there are at least six main types of tests, and the boundaries between them can be fuzzy at times. Nevertheless, it is possible to make at least a few generalizations which can help you navigate the lingo found in most texts and web sites. This article is designed to give you basic definitions for each of the following types of tests:

  • Unit Tests
  • Integration Tests
  • Functional Tests
  • Stress, Load and Performance Tests
  • Acceptance Tests
  • Regression Tests

Unit Tests

As you will see, unit tests can be used to perform any of the six types of test. However, there is a pure form of unit testing that exists on a separate plain from the five other types of tests. When defined in this strict manner, unit tests represent the lowest level of tests. This “pure” or “strict” form of unit testing is very fine grained, and works on a method by method basis. One class is tested at a time, with mocks standing in for other classes. Only public methods are covered. Other rules, many of which will be discussed in depth, are covered in various sections throughout the rest of this text. For now, the key point to grasp is that unit testing, defined in the strictest sense, represents the lowest, and most detailed, level of the six common types of tests. It is possible to use unit tests to perform other types of tests, such as integration or functional tests. However, the “pure” form of unit testing represents a unique kind of test distinct from any other form of testing.

NOTE: When I say that unit tests can be very fine grained, or very low level, I do not mean to imply that unit tests are difficult to write. They are, in fact, quite easy to write. Nevertheless, they work at very detailed and precise level of the development process.

Integration Tests

While unit tests work on a method by method level, Integration tests explore the ways that large entities work together. The classic example would be testing if two classes can interoperate. But really any two abstractions, such as a class, package, assembly, service, or a subsystem, would do. The point is that two distinct entities are being tested to see if they play well together. A lot of people, including myself, use unit test technologies to to run many integration tests. However, there are some kinds of integration tests that can best be run without using unit tests. A good tester is in part defined or recognized by their ability to know when to use unit testing, and when to turn to some other technology.

NOTE: When running unit tests, people frequently use mock objects to simulate what happens when one class needs to interact with another class. Two classes are definitely involved in a typical mock test scenario. However, I think it is simplest to think of mock tests as being part of unit testing, and not a true example of an integration test.

Functional Tests

Functional tests demonstrate whether a subsystem or class meets a requirement. For instance, a functional test on a compiler might check to see whether the compiler can compile Hello World. A functional test on an FTP client might test whether the client could connect, download, or list a directory.

Functional tests are sometimes run on whole applications, while unit tests and integration tests are run on subsystems or methods. A lot of people, including myself, use unit testing technology to run at least some functional tests. Once you start testing a whole application, however, you are not really in the realm of unit testing any longer.

Both unit tests and integration tests are often run at a fairly fine grained level that is very close to the metal, and generally of interest primarily to programmers. Functional tests exist on a somewhat higher, and less technical, level. It is not just a programming team or QA that is interested in the results of these kinds of tests. Managers like functional tests. Managers are frequently working their way through a checklist of requirements, and they can use functional tests to see if there are items on their list they can check off.

Stress, Load and Performance Tests

Stress, load and performance tests verify that an application can handle a load or perform in predefined period of time, or under particular circumstances. Can a server handle 50 clients? How about 250? Or consider the compiler project mentioned in previous paragraphs. How long does it take to compile Hello World? How long does it take to compile an application with 500,000 lines of code? Can it be compiled in a reasonable period of time?

You generally can’t use unit tests to do this kind of thing, though there may be a few simple tests you could run inside a unit testing framework. A number of powerful commercial applications, including TestComplete (which is owned in part by my employer, Falafel), are designed to make this kind of testing easy. A whole class of tools, called profilers, are also designed to fill this role in the development cycle.

Acceptance Tests

Acceptance tests represent the highest, most abstract level of testing. They resides at the opposite end of the spectrum from the “strict” unit test as defined earlier in this section. Unit tests are typically very detailed and very technical, while acceptance tests work with entire applications, and usually concern the major features found in the interface of an application.

Acceptance tests are often run by the customer, and are designed to show whether or not the application meets an agreed upon specification. Acceptance tests are one way of helping a customer decided whether or not they are satisfied with a delivered product. This kind of test often includes at least some functional and some performance tests, but only those that directly impact the end user of an application. Issues like appearance, ease of use, etc, are part of this level of tests. I think of acceptance tests as being run by the client, or a by a very disengaged manager. Unit, functional tests and integration tests help the developer get the application ready to pass an acceptance test.

Regression Tests

Regression tests are not really another type of test so much as another way of looking at the testing process. These tests are designed to detect if new bugs are being introduced into existing code. If a programmer changes an existing code base, then comes back and runs tests to confirm that everything still works, then he or she is running a regression test. Unit, integration, functional, stress and acceptance tests can all be run as regression tests. All the matters is that you are looking back to confirm that nothing is broken after a change was made to the application. The change might be the addition of a new feature, the removal of an existing feature, or simply the refactoring of existing code without adding or deleting any features. Regardless of the type of change that was made, we run regression tests to confirm that the change did not break any existing code.

Unit tests of all kinds are frequently run as regression tests. As you will see, developers automate unit tests to run at least daily. By viewing the results of these automated tests, developers can confirm that nothing has been broken by any recent changes introduced into a code base. When used in this manner, unit tests can be viewed as a common, and very powerful, form of regression test.

Summary

All six types of tests covered in this section of the text are important. An experienced team of developers will use a wide variety of tests to ensure that their code is working properly. Different types of tests will appeal to different types of developers. It is not possible to assert that any one type of test is more important than another. Nevertheless, there is no question that many programmers find themselves particularly interested in unit testing since it can become so intimately bound up with the process of developing an application. Though unit testing is probably the most recently developed of all six types of tests, it appears to me that it is written about more often, and discussed more heatedly, than all the other types of tests combined. This does not mean that it is necessarily more important than the other types of tests, but only that it is more interesting, and of more direct importance to programmers from the perspective of their day to day development cycle.

No comments yet

Leave a Reply

You must be logged in to post a comment.