HTML Tidy Regression Testing
============================

This directory contains test cases and testing scripts for conducting HTML Tidy
regression testing. A successful regression test must be performed prior to
merging code into master.

Testing can be implemented in any language; included test scripts are deemed to
be compatible across a broad range of systems.


Regression Testing Methodology
------------------------------

The directory `cases` includes test input, test configuration, and expected test
results for each test case. A test is considered "passed" if the expected output
matches the generated output.

cases/
  case-xxx-expect.html    The expected HTML generated by Tidy
  case-xxx-expect.txt     The expected English output generated by Tidy
  case-xxx.conf           The configuration used for this test. Optional.
  case-xxx.html           The input file to be Tidy'd.
  case-xxx-expect.nfo     Optional test notes that can be put into final report.
  case-xxx-expect.err     Tidy exit code for cases without expect-txt.
  config_default.conf     The default configuration used when no other present.

Multiple-configuration tests:
  case-xxx-expect-n.html  The expected HTML generated by Tidy
  case-xxx-expect-n.txt   The expected English output generated by Tidy
  case-xxxx-n.conf        Additional configurations will run multiple times.

The sample scripts include options for generating the `-expect` files.
Any `html` file may be instead `xml` or `xhtml` also.


Testing Specifications
----------------------

The testing applications shall find each *.html, *.xml, and *.xhtml file in
the `cases` directory (FILE).

The for each such file a list of matching *.conf files shall be found, deferring
to the default if not found (CONF).

For each FILE for each CONF Tidy OUTPUT and MESSAGES shall be captured.

OUTPUT and MESSAGES shall be compared to the matching `-expect.*` file to
generate a RESULT. Comparison method is not specified (e.g., can be DIFF, MD5,
etc.).

Missing `-expect.*` files shall be logged as an ERROR.

The RESULT and/or ERROR shall be logged. Results not meeting expectations shall
be saved into a directory results-5.x.x/, where 5.x.x indicates the version
number of Tidy.

results-5.x.x/
		all_results.txt       A summary of the entire test process.
		case-xxx-fail.html    The Tidy's OUTPUT.
		case-xxx-fail.txt     The Tidy's MESSAGES.
		case-xxx-diff.txt     The DIFF of the test case and failed result. Optional.
		case-xxx-report.txt   A description of the test failure. Optional.
		case-xxx-n-fail.html  Multiple-config pattern for Tidy's OUTPUT.
		case-xxx-n-fail.txt   Multiple-config pattern for Tidy's MESSAGES.
		(original files)      Option. Original files can be copied here for easy,
		                      quick reference.


Additional Specifications
-------------------------

A testing script shall be capable of running a single test given a command line
parameter to a specific *.html/xhtml/xml file. In cases where multiple configs
are given, all multiple tests will be executed.

A testing script shall be capable of using a tidy executable in a location
specified in a command line argument.

A testing script shall be capable of generating `-expects.*` files into a given
path via a command line argument.

Case names shall not consist of hyphens, which are used exclusively to delimit
`case-` from the beginning of file names and `-n` for multiple configuration
test cases.


HTML Tidy Regression Testing Exception Log
==========================================

Test conducted using `/usr/local/bin/tidy`, version 5.1.12
Results generated on DATE.

Summary:
========

     Number of case files: 458
Total test configurations: 461

        Case files tested: 209
    Configurations tested: 211
   Configurations aborted: 250
    Configurations passed: 4 of 211
           Configs failed: 207 of 211


Missing Expectations Files:
===========================

Case File           For Configuration     Requires Errors          Requires Markup
---------           -----------------     ---------------          ---------------
case-9743.html      default_config.conf   case-9743-expect.txt 
case-5656.html      case-5656.conf        case-5656-expect.txt     case-5656-expects.html 


Failed Tests:
=============

Case File           For Configuration      Markup   Errors
---------           -----------------      ------   ------
case-1234.html      default_config.conf    PASSED   FAILED
case-2345.xml       case-2345-1.conf       FAILED   FAILED

See files in directory results-5.1.12/ for failed results.
  



