During recent presentations at FOSDEM 2008 and DrupalCon 2008 Boston, founder Dries Buytaert presented the idea of bringing full testing of Drupal Core ( here and here ) to the Drupal community. When a system changes as rapidly as Drupal does, it becomes increasingly important to provide quantifiable reporting on the health of the system so that growth may continue. Without such reporting, Drupal 7 will realize a substantially shorter development cycle before a 7 month code freeze is enacted around July 15th, 2008. This will allow testers and developers a 7 month period to guarantee the quality of Drupal's next major release.
By providing an automated system of testing and reporting for Drupal, the Community will be able to extend the development cycle an additional 4 months placing the code freeze around November 15th, 2008. The 4 months spent testing and fixing bugs could instead be applied to providing new functionality and innovation. Using an automated testing and reporting system would allow us to release a better system without sacrificing quality.
At the time Dries was announcing the problem, several projects were underway to help realize the ideals of full test coverage for Drupal Core. These projects are leading the way to providing Drupal Core testing:
Additionally, the following g.d.o groups exist to forward this project:
In addition to these efforts, the weekend of April 19-20th will see the first of the Test Code Sprints aimed at furthering progress and solidifying a framework to continue the great work already in progress.
The workflow framework created by SimpleTest and SimpleTest Auto provide the basis for a solid testing infrastructure. SimpleTest is the vehicle that tests are being written for and processed in while SimpleTest Auto is the mechanism that creates test instances, initiate tests on them and reports findings from the SimpleTests. What is missing from this equation is Code coverage, a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing.
Jimmy Berry took great effort to compile Functional SimpleTest Status - List of all tests which comes the closest to what a Code Coverage report could contain. However, manual entry of this data is a time consuming process that is dependent upon the tests ... it has no concept of what is really getting tested at the code level.
The concept of DCOV (Drupal Code Coverage) is to allow developers, testers, reviewers and just about anyone with an interest to see what parts of Drupal Core are currently covered by test scripts, whether they pass or fail and other basic metrics including function execution times, invocation counts, etc. This will provide additional basic profiling on various aspects of the tests that are performed as part of the function test cycles. A prime example of usage of this data would be to evaluate what percentage of a documents code was run during the processing of a specific SimpleTest. In addition, the community at large should be able to run tests locally reporting findings back to the testing group.
The first public reference of Code Coverage is recorded in 1963 by the Association for Computing Machinery placing it amongst the first techniques invented for systematic software testing. Finding working examples relevant to Drupal are not difficult because Drupal is written in PHP. The PHP Quality Assurance Team has pioneered the techniques of Code Coverage as they apply to the PHP language. Their landing page serves as a clean and efficient method for communication of the groups objectives, possibly serving as a module for Drupal.
The true efforts of the PHP Quality Assurance Team's hard work are visible in the reporting component of their Code coverage testing known as GCOV. The GCOV site is an automated system that checks out from various RELEASES and SNAPHOTS of PHP from CVS, builds them and then runs the PHPT test suite. It reports data consumed from the LTP GCOV extension. The PHPT unit testing is the method of choice for PHP as it is better suited to the nature of PECL and Internals constructs for PHP than SimpleTest is for Drupal. The actual method of testing (whether it be PHPT, SimpleTest or some other system) is not DCOV's purpose. DCOV is a coverage tool that's designed to show how far the tests went, what PHP user space functions were executed (with pass/fail data acquired from the test layer above) and the other metrics previously alluded to earlier. If you follow the link above and browse, then the ideals of what DCOV seeks to become for Drupal won't be lost on the reader.
However, lets draw some examples as links and discuss the types of output the DCOV project is intended to offer Drupal:-
The basic proposal of the DCOV project is to provide useful feedback on how the Drupal Core code base is performing, how functions run with some primitive profiling metrics and how the tests themselves are matching to the code base either in CVS or from CVS with patches applied.
(Note, any final DCOV implementation will be designed to work with those patches Derick will commit. We don't want to find ourselves in a position of hacking Xdebug for the DCOV platform to work. Should patches not be committed, we'll work around them. An example of one such patch that already has been fed back is "limitation of code coverage scope". When running Simpletest AND Drupal it's currently found that to a greater degree, Xdebug is working on Simpletest execution far more than Drupal execution and wasting cycles, data, etc. We don't want to run Xdebug on Simpletest, just on Drupal. The first patch allows to target where Xdebug actually performs data acquisition and thus allows Xdebug to work in a more test oriented role rather than do everything).
The production of DCOV would lead to a central testing center (website) that publishes the various metrics derived from all the automated tests. However, it's intended to extend this site in the medium term to become a true QA site for Drupal where developers can upload test cases for evaluation and inclusion and disseminate essential quality metrics on the Drupal project as a whole.
The end result of the DCOV project will tie together and leverage the existing unit and auto-browser based testing suites into a completely automated test environment with full reporting made available to those that need the data to push forward with development.
Generally speaking, the DCOV project aims to provide details similar to Jimmy Berry's excellent post though in an automated fashion and providing far more details about each test case and realistic code coverage based on actual functions executed during the test run.
Realistically, the DCOV project needs to be delivered with a reasonably short time frame to aid and assist the Drupal 7 release cycle. The following developers are currently prepared to commit to DCOV and deliver a working framework by months end:-