I have developed a BlackBox Testing framework / facility where you could add Kettle Transformations and test the results against a Golden file.

2. The Input, Output, Temp, Golden directories must be named strictly as Input, Output, Temp, and Golden respectively. The build.xml creates the directories with these names. The Input directory must consist of all the input files required to run the transformations. The Transformations directory must consist of all the Transformations for each test case. The Output directory is where the transformation's output must be directed. The Golden directory is where the the golden files for the test cases must exist.

3. The Transformation file names must start with Test_Case_#.ktr [eg: Test_Case_1.ktr, Test_Case_34.ktr]

4. The Output file name must have the following format: Test_Case_#_Out_Actual.[txt|csv] [eg: Test_Case_1_Out_Actual.csv, Test_Case_34_Out_Actual.txt]

5. The sub-directories of the Golden directory must be numeric. [for example: 2.5.1, 3.0.1, 3.0.2]

6. The Golden directory is searched for the file to be compared against starting from the highest version then backwards. For example: If Test_Case_5_Out_Actual.txt exists in both 3.0.2 and 2.5.1, the search feature looks for the file 3.0.2 first and then goes to the directory with lower version numbers. The search stops as soon as the file is found in 3.0.2.

7. The exceptions encountered while executing a particular test case or while comparing Output file to Golden are logged and we would continue with next test case. To verify the results of the execution of the transformations, examine the logs generated.

8. doTests() method executes all the Transformations in the Transformation directory and compares all the files in Golden directory.

**Currently there is no support for testing Jobs.

9. The program looks to execute specific test cases or ALL test cases. To execute specific test cases, add the test case number to the testIDs property in the testing-config.xml file. For example: the configuration below executes test case number 1, 2, 3 and 7.

<property>
<name>testIDs</name>
<value>1,2,3,7</value>
</property>

To run ALL test cases remove any entry in the value node. For example:

<property>
<name>testIDs</name>
<value></value>
</property>

10. You can specify the logging level ranging from 1 to 6 in the testing-config.xml. For example:

        <property>
            <name>logLevel</name>
            <value>1</value>
        </property>

Where 1 provides minimal logging and 6 provides max logging information.

11. testing-config.xml holds the properties to be run for the Black Box Testing.

Sample testing-config.xml file:

<pentaho-testing>
    <test id="pdi3"
        strategy="org.pentaho.testing.strategy.PDI3Strategy">
        <property>
            <name>transformationsDirPath</name>
            <value>C:\Kettle Blackbox Tests\Transformations</value>
        </property>
        <property>
            <name>goldenDirPath</name>
            <value>C:\Kettle Blackbox Tests\Golden</value>
        </property>
        <property>
            <name>outputDirPath</name>
            <value>C:\Kettle Blackbox Tests\Output</value>
        </property>
        
        <property>
            <name>logLevel</name>
            <value>1</value>
        </property>
        <property>
            <name>testIDs</name>
            <value>1,2,3,7</value>
        </property>
    </test>
</pentaho-testing>

12. Sample Golden directory with the Golden files in 3.0.3 directory :

13. Sample Output directory (after executing the test cases 1,2,3,7):

14. Sample Log (after executing the test cases at level #1):

INFO 27-11 13:47:00,062 (PDI3Strategy.java:runTrans:28) -Running: C:\Kettle Blackbox Tests\Transformations\Test_Case_1.ktr
INFO 27-11 13:47:02,687 (VfsLog.java:info:122) -Using "C:\DOCUME~1\User\LOCALS~1\Temp\vfs_cache" as temporary files store.
INFO 27-11 13:47:04,671 (PDIStrategy.java:compareActualToOutput:339) -Comparing files C:\Kettle Blackbox Tests\Output\Test_Case_1_Out_Actual.csv , C:\Kettle Blackbox Tests\Golden\3.0.3\Test_Case_1_Out_Actual.csv
INFO 27-11 13:47:04,687 (PDIStrategy.java:compareActualToOutput:343) -Comparison succeeded.
INFO 27-11 13:47:04,687 (PDI3Strategy.java:runTrans:28) -Running: C:\Kettle Blackbox Tests\Transformations\Test_Case_2.ktr
INFO 27-11 13:47:06,125 (PDIStrategy.java:compareActualToOutput:339) -Comparing files C:\Kettle Blackbox Tests\Output\Test_Case_2_Out_Actual.txt , C:\Kettle Blackbox Tests\Golden\3.0.3\Test_Case_2_Out_Actual.txt
INFO 27-11 13:47:06,140 (PDIStrategy.java:compareActualToOutput:343) -Comparison succeeded.
INFO 27-11 13:47:06,140 (PDI3Strategy.java:runTrans:28) -Running: C:\Kettle Blackbox Tests\Transformations\Test_Case_3.ktr
INFO 27-11 13:47:07,250 (PDIStrategy.java:compareActualToOutput:339) -Comparing files C:\Kettle Blackbox Tests\Output\Test_Case_3_Out_Actual.txt , C:\Kettle Blackbox Tests\Golden\3.0.3\Test_Case_3_Out_Actual.txt
INFO 27-11 13:47:07,250 (PDIStrategy.java:compareActualToOutput:343) -Comparison succeeded.
INFO 27-11 13:47:07,250 (PDI3Strategy.java:runTrans:28) -Running: C:\Kettle Blackbox Tests\Transformations\Test_Case_7.ktr
INFO 27-11 13:47:08,484 (PDIStrategy.java:compareActualToOutput:339) -Comparing files C:\Kettle Blackbox Tests\Output\Test_Case_7_Out_Actual.txt , C:\Kettle Blackbox Tests\Golden\3.0.3\Test_Case_7_Out_Actual.txt
INFO 27-11 13:47:08,484 (PDIStrategy.java:compareActualToOutput:343) -Comparison succeeded.