In the Matrix

Update (4/18): This post does not seem to render well in blogger. You could find a well formatted version here.

If you belong to a testing organization its very difficult for you to do without test matrices. As a matter of fact you would be using test matrices semi-consciously in your day to day life. Managing money, time or girlfriends. You need matrices.

But on a more serious note, test matrices can be used for a variety of things. You could record a consistent set of tests using the test matrices. It can be used as proof of coverage of a certain criteria of tests, to serve as a checklist and refine/improve area in which a defect occurred.

Lets take an example to make things clearer.

Consider some new font sizes, size 75, 80, 85 and 90 which have been included as a standard in your Word application. You need to test whether they work well with your Word application across different platforms of Windows.

	Windows 2000	Windows XP Pro	Windows XP SP2
Font size = 75
Font size = 80
Font size = 85
Font size = 90

Figure 1

What have we achieved here? We have set up two axis. The Y-axis contains font size and X-axis the OS. So we can put down the first step as determining the axis variables and putting them in such a form where you have minimum one test per row and one test per column. The test performed is recorded as the intersection of row and column. Once the test is performed, you could lodge the result as a pass or a fail.

You could use other notations in your test matrix too. P=Pass, or N=Note, or F=Fail etc. A test could pass but you would like to add a note to it. Very possible.

	Windows 2000	Windows XP Pro	Windows XP SP2
Font size = 75	PASS
Font size = 80		PASS	PASS
Font size = 85			PASS
Font size = 90		PASS

Figure 2

Some points to note from the above:

Highlighting gives us a better idea of test coverage.
Complete testing of a huge matrix is not feasible. The matrix would give us a good idea of test coverage for key areas.

You could apply your mind in different ways to eliminate or identify problems.

	Windows 2000	Windows XP Pro	Windows XP SP2
Font size = 75
Font size = 80	FAIL
Font size = 85		PASS	PASS
Font size = 90

Figure 3

From Figure 3, we could conclude that there might be a bug with Windows 2000, there might be a bug with font size 80, both of the previous or there could be a bug when you try font size 80 on Windows 2000.

Lets test on the same axes as the failure and we would be able to make a much better guess to the problem. We perform two more tests.

	Windows 2000	Windows XP Pro	Windows XP SP2
Font size = 75
Font size = 80	FAIL	PASS
Font size = 85	FAIL	PASS	PASS
Font size = 90

Figure 4

Ah! Now we know the problem is platform specific.

What if the matrix read:

	Windows 2000	Windows XP Pro	Windows XP SP2
Font size = 75
Font size = 80	FAIL	PASS
Font size = 85	PASS	PASS	PASS
Font size = 90

Figure 5

Figure 5 will help us conclude that the problem is specific to Windows 2000 and font size 80.

One more matrix which could come about would be:

	Windows 2000	Windows XP Pro	Windows XP SP2
Font size = 75
Font size = 80	FAIL	FAIL
Font size = 85	PASS	PASS	PASS
Font size = 90

Figure 6

Figure 6 shows a font specific bug.

So on, test matrices can be used for any pair of criteria's. Sometimes, a bug might be found which might not be listed under any criteria in the test matrix. The matrix can easily be expanded to include this criteria. Another advantage I see of test matrices might be the fact that if a defect is found after release, one can always go back to the matrix and look for areas not tested or areas which were tested and still had a bug (this could be poor testing).