DCOV - The Drupal Code Coverage Project
The Problem
During recent presentations at FOSDEM 2008 and DrupalCon 2008 Boston, founder Dries Buytaert presented the idea of bringing full testing of Drupal Core ( here and here ) to the Drupal community. When a system changes as rapidly as Drupal does, it becomes increasingly important to provide quantifiable reporting on the health of the system so that growth may continue. Without such reporting, Drupal 7 will realize a substantially shorter development cycle before a 7 month code freeze is enacted around July 15th, 2008. This will allow testers and developers a 7 month period to guarantee the quality of Drupal's next major release.
By providing an automated system of testing and reporting for Drupal, the Community will be able to extend the development cycle an additional 4 months placing the code freeze around November 15th, 2008. The 4 months spent testing and fixing bugs could instead be applied to providing new functionality and innovation. Using an automated testing and reporting system would allow us to release a better system without sacrificing quality.
What We Have
At the time Dries was announcing the problem, several projects were underway to help realize the ideals of full test coverage for Drupal Core. These projects are leading the way to providing Drupal Core testing:
Additionally, the following g.d.o groups exist to forward this project:
In addition to these efforts, the weekend of April 19-20th will see the first of the Test Code Sprints aimed at furthering progress and solidifying a framework to continue the great work already in progress.
What is Missing
The workflow framework created by SimpleTest and SimpleTest Auto provide the basis for a solid testing infrastructure. SimpleTest is the vehicle that tests are being written for and processed in while SimpleTest Auto is the mechanism that creates test instances, initiate tests on them and reports findings from the SimpleTests. What is missing from this equation is Code coverage, a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing.
Jimmy Berry took great effort to compile Functional SimpleTest Status - List of all tests which comes the closest to what a Code Coverage report could contain. However, manual entry of this data is a time consuming process that is dependent upon the tests ... it has no concept of what is really getting tested at the code level.
The DCOV Proposal
The concept of DCOV (Drupal Code Coverage) is to allow developers, testers, reviewers and just about anyone with an interest to see what parts of Drupal Core are currently covered by test scripts, whether they pass or fail and other basic metrics including function execution times, invocation counts, etc. This will provide additional basic profiling on various aspects of the tests that are performed as part of the function test cycles. A prime example of usage of this data would be to evaluate what percentage of a documents code was run during the processing of a specific SimpleTest. In addition, the community at large should be able to run tests locally reporting findings back to the testing group.
The first public reference of Code Coverage is recorded in 1963 by the Association for Computing Machinery placing it amongst the first techniques invented for systematic software testing. Finding working examples relevant to Drupal are not difficult because Drupal is written in PHP. The PHP Quality Assurance Team has pioneered the techniques of Code Coverage as they apply to the PHP language. Their landing page serves as a clean and efficient method for communication of the groups objectives, possibly serving as a module for Drupal.
The true efforts of the PHP Quality Assurance Team's hard work are visible in the reporting component of their Code coverage testing known as GCOV. The GCOV site is an automated system that checks out from various RELEASES and SNAPHOTS of PHP from CVS, builds them and then runs the PHPT test suite. It reports data consumed from the LTP GCOV extension. The PHPT unit testing is the method of choice for PHP as it is better suited to the nature of PECL and Internals constructs for PHP than SimpleTest is for Drupal. The actual method of testing (whether it be PHPT, SimpleTest or some other system) is not DCOV's purpose. DCOV is a coverage tool that's designed to show how far the tests went, what PHP user space functions were executed (with pass/fail data acquired from the test layer above) and the other metrics previously alluded to earlier. If you follow the link above and browse, then the ideals of what DCOV seeks to become for Drupal won't be lost on the reader.
However, lets draw some examples as links and discuss the types of output the DCOV project is intended to offer Drupal:-
- Sample 1. This link shows the output of a test run based on libraries the PHPT test suite "touched" on. DCOV would produce a similar output, however, it would be "file based" (ie, includes/database.inc, etc).
- Sample 2. This shows an example of source code that was "touched on" by the tests thus revealing functions that were missed in an obvious to read presentation. Again, DCOV would produce a similar output showing what functions a test run actually invoked. Color coding can be used to indicate functions invoked that passed all their parent tests, or failed or fact simply haven't been invoked by any test script at all
The basic proposal of the DCOV project is to provide useful feedback on how the Drupal Core code base is performing, how functions run with some primitive profiling metrics and how the tests themselves are matching to the code base either in CVS or from CVS with patches applied.
Structure
- The DCOV platform.
- Stage 1 is expected to implement code coverage analysis and deliverables are expected to include a Drupal DCOV system which, at it's heart, will encompass Xdebug and additional software as a wrapper to read the results of Xdebug's code coverage. Already some minor additional feature enhancements have been made to Xdebug to enable better placement within an automated test environment (as opposed to the single enviroment Xdebug expects to find itself). These enhancements to Xdebug are being fed back to Derick Rethans (author and maintainer of Xdebug) for consideration for inclusion for the benefit of the greater Xdebug community.
(Note, any final DCOV implementation will be designed to work with those patches Derick will commit. We don't want to find ourselves in a position of hacking Xdebug for the DCOV platform to work. Should patches not be committed, we'll work around them. An example of one such patch that already has been fed back is "limitation of code coverage scope". When running Simpletest AND Drupal it's currently found that to a greater degree, Xdebug is working on Simpletest execution far more than Drupal execution and wasting cycles, data, etc. We don't want to run Xdebug on Simpletest, just on Drupal. The first patch allows to target where Xdebug actually performs data acquisition and thus allows Xdebug to work in a more test oriented role rather than do everything).
- Stage 2 of DCOV is expected to enhance the platform to include functional tracing (including parameters) for executed tests that fail. This way, DCOV can supply a code path analysis to developers explicitly for failed tests to assist in bug/problem location. Additionally, DCOV will export to the testing layer runtime information as required. So, if a test assertion fails, the testing layer will be able to query DCOV for additional details of the runtime enviroment at assertion failure. However, this would require that the tests themselves become DCOV aware which is more work on those creating .test scripts. For this reason, this aspect of Stage 2 is deferred until such time as full maturity within the testing architecture arrives.
- Stage 1 is expected to implement code coverage analysis and deliverables are expected to include a Drupal DCOV system which, at it's heart, will encompass Xdebug and additional software as a wrapper to read the results of Xdebug's code coverage. Already some minor additional feature enhancements have been made to Xdebug to enable better placement within an automated test environment (as opposed to the single enviroment Xdebug expects to find itself). These enhancements to Xdebug are being fed back to Derick Rethans (author and maintainer of Xdebug) for consideration for inclusion for the benefit of the greater Xdebug community.
- The dcov.module. This Drupal module would be used to provide relevant reporting on the DCOV data, similar to the approach taken by the PHP QA Team and would be comprised of the following reports slanted towards Drupal Testing:
- An overview report - listing each CVS tag that has been monitored by the testing system including relevant statistics and information about the project as a whole.
- A tag detail report - highlighting test cases run, passes, failures, exceptions and other relevant information.
- A code coverage report - providing an overview of the coverage in an easy to understand format.
- A results report - This would be similar to the compile report and test failures reports used by the PHP team but would be slanted toward Drupal.
- A system report - To allow for multiple developers to test on multiple systems and report their findings back to the Drupal testing system.
- Various test scripts. TBA
Medium term goals
The production of DCOV would lead to a central testing center (website) that publishes the various metrics derived from all the automated tests. However, it's intended to extend this site in the medium term to become a true QA site for Drupal where developers can upload test cases for evaluation and inclusion and disseminate essential quality metrics on the Drupal project as a whole.
Summary/Conclusion
The end result of the DCOV project will tie together and leverage the existing unit and auto-browser based testing suites into a completely automated test environment with full reporting made available to those that need the data to push forward with development.
Generally speaking, the DCOV project aims to provide details similar to Jimmy Berry's excellent post though in an automated fashion and providing far more details about each test case and realistic code coverage based on actual functions executed during the test run.
Realistically, the DCOV project needs to be delivered with a reasonably short time frame to aid and assist the Drupal 7 release cycle. The following developers are currently prepared to commit to DCOV and deliver a working framework by months end:-
- Kevin Bridges (cyberswat). Responsible for :-
- DCOV for Drupal reality (delivery).
- The dcov.module, Drupal analysis, reporting, etc
- Andy Kirkham (AjK). Responsible for :-
- The DCOV platform.
- Xdebug integration/modification/patching.
- Wrapper scripts/module assistance.
|
|