XSpec in practice: testing an XSLT stylesheet

XSpec describes what a stylesheet must produce, in a separate XML file that never touches the code under test. That separation is what makes the description maintainable: a test written inside the stylesheet disappears at the first rewrite, and nobody notices.

The first useful test fits in fifteen lines. The other half of the work is knowing exactly what "it passes" means.

The first test

<x:description stylesheet="invoice.xsl"
               xmlns:x="http://www.jenitennison.com/xslt/xspec">
  <x:scenario label="an invoice with no line">
    <x:context>
      <invoice/>
    </x:context>
    <x:expect label="produces a zero total">
      <total>0</total>
    </x:expect>
  </x:scenario>
</x:description>

x:context supplies the input document, x:expect describes the expected result. The scenario fails when the two trees differ.

"The two trees", not "the two files"

🔴 The comparison is made on the result tree, before serialisation. Nothing decided at the moment bytes are written is visible: not the encoding, not the XML declaration, not the indentation. A stylesheet whose output opens as unreadable characters passes the whole XSpec suite green, because the suite never reaches that step.

⚠️ Text nodes, on the other hand, are part of the tree. An x:expect re-indented to stay readable introduces whitespace the real result does not have, and the scenario fails on a difference you cannot see in the report. It is worth knowing this before spending an afternoon on it: the culprit is your own indentation, not the stylesheet.

To compare only part of the result, select evaluates an expression against what was produced:

<x:expect label="the total alone" select="/invoice/total" test="xs:integer(.) eq 0"/>

⚠️ The classic inversion is to assume this select applies to the input context. It applies to the output.

Scenarios nest

A nested scenario inherits its parent's x:context, which saves repeating the input for every assertion:

<x:scenario label="an invoice with two lines">
  <x:context href="invoice.xml"/>
  <x:scenario label="the total"><x:expect select="/invoice/total"/></x:scenario>
  <x:scenario label="the line count"><x:expect select="count(//line)"/></x:scenario>
</x:scenario>

⚠️ A child that redeclares x:context replaces the parent's, it does not extend it. And a relative href resolves against the .xspec file, never against the directory the command was launched from.

Three ways to pass without checking anything

🔴 A green test is information only if red was possible. Three cases produce exactly the expected signal without having measured anything:

The last two are reported in the HTML output — provided somebody opens it. ⚠️ A pending set for an afternoon and forgotten behaves exactly like a passing test for anyone reading only the overall colour.

What a green test does not tell you

• nothing about performance: the same output in ten seconds or ten minutes passes alike;
• nothing about the file produced, only about the tree;
• nothing about the inputs you did not write, which is the limit of every example-based test.

When a scenario fails and the report does not make the reason obvious, the next step is to watch the run itself — see setting a breakpoint in an XSLT stylesheet.

What this article does not cover

• testing functions and XSLT 3.0 packages, which have their own call syntax;
• running the suite in continuous integration, where the exit code matters more than the report;
• schemas: XSpec checks an expected result, never validity.

The validating and comparing chapter of the manual describes the tree comparison the whole approach rests on.