Overview of code coverage metrics

Unit tests are essential in software engineering nowadays. The quality of software begins with unit testing, but they are useless if there isn’t a metric or an indicator that shows how good the unit tests are. This is called code coverage. There are different kinds of code coverage metrics.

Statement Coverage

Let’s say we’ve got the following code

int myMethod(int x)
{
    do
    {
        /* some stuff */
    }while(x == 0);

    return -42;
}

A unit test (in this case CPPUnit light) for this could be

TEST(Quick, test)
{
    CHECK(myMethod(1) == -42);
}

Statement coverage means, as you might expect, that every statement is covered, which is the case with the test above. But there are several flaws in this. The potential endless loop isn’t covered or even detectable. Also, depending on the compiler, there might be machine code, which is still untested.

Decision or Branch Coverage

Again another example:

int myMethod(bool a, bool b)
{
    int result = 0;

    if(a && b){
            result = 1;
    }

    return result;
}

A unit tests for this could be:

TEST(Quick, test)
{
    CHECK(myMethod(true, false) == 0);

    CHECK(myMethod(true, true) == 1);
}

With branch coverage, every control flow needs to be tested and that means two tests. With code coverage only one test would be necessary.

Decision and Condition Coverage

If you look at the last unit test example, you can see that variable a wasn’t changed. Decision and condition coverage means, that every variable has to have every possible condition. In this case

TEST(Quick, test)
{
    CHECK(myMethod(true, false) == 0);

    CHECK(myMethod(false, true) == 1);
}

Modified Condition and Decision Coverage

A much more intensified approach is the modified condition and decision coverage (MC/DC). MC/DC is often used in a high risk environment. It is also used by NASA (see [1]). It’s supposed to show that every condition could have influence on the control flow. In this case it means three tests (n condition – 1).

TEST(Quick, test)
{
    CHECK(myMethod(true, true) == 1);

    CHECK(myMethod(false, true) == 0);

    CHECK(myMethod(true, false) == 0);
}

Which metric should I use?

There is no correct answer for this. It depends on the purpose of the system or device. For a lower risk, just choose statement coverage or branch coverage. If there is a higher risk, branch or decision and condition coverage is always a good choice. If you develop spacecrafts or an artificial lung, as a customer I would highly appreciate the modified condition and decision coverage.

[1]: http://shemesh.larc.nasa.gov/fm/papers/Hayhurst-2001-tm210876-MCDC.pdf

Advertisements
This entry was posted in C/C++, Development and tagged , , , , . Bookmark the permalink.

One Response to Overview of code coverage metrics

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s