Qafoo GmbH - passion for software quality
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:Author: 		Manuel Pichler
:Date:			Wed, 03 Apr 2013 07:52:04 +0200
:Revision: 		9
:Copyright:		All rights reserved


========================
Code Coverage with Behat
========================

:Keywords:
    php, testing, behat, behavior driven development, bdd, mink, sahi, symfony,
    ant

:Description:
    Code coverage does not make much sense for Behat tests, except for if you
    use them on legacy code as wide-coverage test cases, before starting to
    refactor. This blog post shows you how to generate code coverage
    information for your Behat test scenarios.

:Abstract:
    There is generally no point in having code coverage for Behat test cases
    because of their nature: The purpose of an acceptance test is to assert a
    certain behavior of an application, not to technically test a piece of
    code.  Therefore, there is no point in checking for uncovered code pieces
    in order to write a Behat test for it.

    That said, there is still a scenario where you want to peek at code
    coverage of Behat tests: When creating them as wide-coverage tests before
    starting to refactor legacy code. Behat in combination with Mink provides
    you with a great tool for such tests.

There is generally no point in having code coverage for Behat test cases
because of their nature: The purpose of an acceptance test is to assert a
certain behavior of an application, not to technically test a piece of code.
Therefore, there is no point in checking for uncovered code pieces in order to
write a Behat test for it.

.. note::
    Refactoring is an important skill for your team to make software
    maintainable and keep it like that. Base these capabilities on the sound
    basis of an `individual Qafoo training`__ for your team.

__ /services/training.html


That said, there is still a scenario where you want to peek at code coverage of
Behat tests: When creating them as wide-coverage tests before starting to
refactor legacy code. Behat in combination with Mink provides you with a great tool for
such tests.

Before you can start with refactoring legacy code you need tests to ensure that
you don't break working functionality. Web acceptance tests on basis of Behat
and Mink are a great tool to realize these. But how can you detect if the code you
are about to refactor is touched by at least one test? Code coverage can be of
assistance there.

-----------
Preparation
-----------

Since Behat does not ship with code coverage (for very good reason), you need
some hand work to get that done, but not much. In order to get started, you
need to install the `PHP_CodeCoverage library`__ and `phpcov`__, most probably
via PEAR using::

    $ pear config-set auto_discover 1
    $ pear install -a pear.phpunit.de/phpcov

__ https://github.com/sebastianbergmann/php-code-coverage
__ https://github.com/sebastianbergmann/phpcov

------------------------
Collecting Code Coverage
------------------------

Since the Behat tests stimulate your application through
external calls, it is not possible to generate code coverage right from the
test code. Instead, you need to trigger the code coverage collection from your
application code::

    <?php

    // ... forbid production access here ...

    $calculateCoverage = file_exists("/tmp/generate-behat-coverage");

    if ($calculateCoverage) {
        require 'PHP/CodeCoverage/Autoload.php';

        $filter = new PHP_CodeCoverage_Filter();
        $filter->addDirectoryToBlacklist(__DIR__ . "/../vendor");
        $filter->addDirectoryToWhitelist(__DIR__ . "/../src");

        $coverage = new PHP_CodeCoverage(null, $filter);
        $coverage->start('Behat Test');
    }

    // ... run your application here ...

    if ($calculateCoverage) {
        $coverage->stop();

        $writer = new PHP_CodeCoverage_Report_PHP;
        $writer->process($coverage, __DIR__ . "/../log/behat-coverage/" . microtime(true) . ".cov");
    }

At first the code detects if code coverage information should be gathered by
checking if the file ``/tmp/generate-behat-coverage`` exists. You can touch
and delete that one manually or from your test setup.

The next code block loads and initializes the code coverage collection,
creates a filter for 3rd party code and starts the code coverage collection.
After that, the comment indicates to run your application, which might e.g. be
a Symfony2 kernel handling call.

The final lines write the code coverage information into a file for further
processing. It will create a dedicated file for each request, where these files
then need to be merged later.

-------------
Running Tests
-------------

With the shown code in place, you can trigger a Behat test run with code
coverage using the following Ant code, for example::

    <target name="behat-coverage" depends="clean, initialize">
        <delete dir="${commons:logsdir}/behat-coverage" />
        <mkdir dir="${commons:logsdir}/behat-coverage" />

        <touch file="/tmp/generate-behat-coverage" />
        <antcall target="behat" />
        <delete file="/tmp/generate-behat-coverage" />

        <exec executable="phpcov" failonerror="false" dir="${basedir}">
            <arg value="--merge" />
            <arg value="--html" />
            <arg value="${commons:logsdir}/../coverage/behat" />
            <arg value="${commons:logsdir}/behat-coverage" />
        </exec>
    </target>

.. note::
    Get a Qafoo expert on-site to `support you with mastering challenges`__ in
    various areas of high quality code, including software architecture,
    design, automated testing and more.

__ /services/consulting.html


The Ant target first cleans up code coverage from previous runs. It then
touches the file that indicates to the application to run code coverage,
executes Behat and removes the trigger file again. Then the ``phpcov`` utility
is executed to merge the results of all requests into a single coverage report
and generate HTML from it.

----------
Conclusion
----------

Code coverage is completely out of scope for acceptance tests. However, if you
abuse Behat to create wide-coverage tests before refactoring, it might be of
help to you to see what is still missing before you start hacking.

..
   Local Variables:
   mode: rst
   fill-column: 79
   End: 
   vim: et syn=rst tw=79


Trackbacks
==========


Comments
========