XML elements for ixml test catalogs

C. M. Sperberg-McQueen

7 November 2021, last rev. 22 February 2022

Introduction
Known issues
Element reference list

Introduction

This document lists the elements in an XML vocabulary for ixml test catalogs and test-result reports, giving a short description of each. Fuller description and discussion may be added later if felt to be needed; for now this is mostly an alphabetical reference list to explain what the elements are intended to mean and how they are to be used.

The vocabulary described was developed by C. M. Sperberg-McQueen as a proposal for the invisible-XML Community Group at the World Wide Web Consortium, but it has not yet been considered by the group, still less adopted, and it has no official status of any kind.

The vocabulary describes two different types of document:

Test catalogs describe a potentially complex set of test cases for an invisible XML processor; they can be organized internally into multiple sets of test cases, and test sets can nest.

Each test set may specify an ixml grammar in either ixml or XML form, together with a set of test cases and optionally nested test sets. An optional grammar-test element signals that the grammar itself should be regarded as a test case and identifies the expected result of processing the grammar.

Each test case in a test set supplies input to be parsed against the grammar for the test set, and specifies a result.

Grammars and test-case input can be given inline or in separate files. The expected results can also be given inline or externally.
Test reports describe the results of running a given processor against the tests in a test catalog.

A metadata section provides information about the test catalog used and the processor tested. Then, for each test set and test case, the result produced by the processor is reported and whether it is or is not the result expected in the test catalog.

Note: the current version of this vocabulary does not yet define elements for test reports. Real soon now.

A sample test catalog is given in the doc directory of the ixml-tests Git repository (where this document is also to be found) in both XML and HTML (for situations where dynamic XSLT processing is not possible).

Known issues

Among the shortcomings of the current state of the test catalog schema are these.

No reporting of test results. XML vocabularies are needed to allow a test harness to report the detailed results of running a test suite and comparing the output to the expected results, and to allow the developers of an implementation to report (perhaps in somewhat less detail) which tests they have run and not run, and of those they ran which tests they passed and which they failed.

An initial sketch of such a vocabulary is expected soon, but it's not here yet.

Awkward treatment of ambiguous sentences. An ambiguous sentence has more than one parse tree, and the choice of which tree to return is implementation-dependent (or possibly implementation-defined), so that a correct implementation may easily produce a parse tree which is correct but does not match the parse tree specified in the test case.

The current approach is to allow multiple parse trees, with the meaning being that if the processor produces any of them, it passes that test. If the processor returns a parse forest instead of a single tree, every tree in the returned forest should be among those specified in the result. This has the drawback that in some cases the set of parse trees specified will not be and cannot be complete: in some grammars, some sentences have an infinite number of parse trees.

For practical purposes, however, we can expect there to be a finite number of processors using any given test suite, each of which will return a finite number of trees. If a processor fails a test on an ambiguous grammar, then the tree returned by the processor should be checked manually and added to the specified result. Unless a processor returns a different result each time it parses the test input, the second and later runs against the test suite should be problem-free.

There are of course alternatives:

Allow an assertion about the result, expressed in XPath. This is the approach taken in the QT test suite and in XSpec. Drawback: unless the assertion is very thorough, the test case will not fully specify the required test result, allowing incorrect results to pass the test.
Allow a set of assertions about the result, expressed in XPath. To pass the test, a processor's result must satisfy all assertions. Drawback: as before, though it's easier to be thorough.
Allow an and/or tree of assertions about the result, expressed in XPath. To pass the test, a processor's result must satisfy all children of an and and at least one child of an or. Drawback: logical conjunction and disjunction are already built in to XPath so it's not really necessary to provide a new set of machinery for the purpose.

As the examples of the QT3 test suite and XSpec show, all of these involve building additional machinery both into the vocabulary for the test catalog and also into the test harness. An XPath assertion must be evaluated in a dynamic context and must say something about the result returned, so we now end up having to build a dynamic context and bind the result to some well known variable like $result, so that the assertion can say something useful about the result.

The current approach has the comparative advantage that it does not require an XPath evaluator: one could in principle handle the comparison of test results to the expected output by writing out temporary files and calling a diff program. The occasional false negative in the test results seems a small price to pay for that advantage.

Element reference list

app-info

(i.e. application information)

includes information needed by any application which is to process the test catalog; this is an extension mechanism to allow the inclusion of arbitrary XML elements in the catalog. May occur wherever metadata is allowed: as a child of test-catalog, test-set, test-case, etc.

The process by which a test case is generated may produce, as a side effect, a ‘raw parse tree’ in which all nonterminals are present. This can be helpful in diagnosing problems in an ixml parser, so it may be thought useful to record the raw parse tree, even though there is no standard place for it in the schema. The following example shows a raw parse tree embedded in a test case.

<test-case xml:id="S-1" name="S1">
   <created by="Gingersnap (CMSMcQ)" 
            on="2021-01-26"/>
   <test-string/>
   <result>
      <assert-xml>
         <S/>
      </assert-xml>
   </result>
   <app-info>
      <raw-parse-tree xmlns="">
         <S/>
      </raw-parse-tree>
   </app-info>
</test-case>

assert-not-a-grammar

signals that the grammar specified is not a conforming ixml grammar (and thus cannot be used to parse the test input). Correct behavior in this case is to report that the input cannot be parsed using the grammar and optionally describe the nature of the problem.

Remarks

Appears as child of result.

The spec does not require a particular way of signaling errors in the grammar, so each processor may require a different setup.

Within the result element for a grammar test, there is no useful distinction between this element and assert-not-a-sentence. Casuists can argue the case each way. (The purported grammar is not a grammar, so this element applies; it is also not a sentence in the language defined by the grammar for ixml grammars, so the assert-not-a-sentence also applies. If the problem with the grammar is not failure to follow the grammar for grammars but violation of some extra-grammatical constraint like having only one rule per nonterminal, the input is a sentence but is not a grammar. Etc., etc., etc.) In practice both elements will and should be treated as interchangeable in that particular case.

This element can contain any elements of the metadata class (so: description, app-info, doc), which leads to a lack of parallelism with assert-xml and assert-xml-ref. Remove?

assert-not-a-sentence

signals that the test input is not a sentence in the language described by the grammar indicated; correct behavior in this case is to report that the input cannot be parsed using the grammar and optionally describe the nature of the problem.

Remarks

Appears as child of result.

The spec does not require a particular way of signaling failure to parse the input, so each processor may require a different setup.

This element can contain any elements of the metadata class (so: description, app-info, doc), which leads to a lack of parallelism with assert-xml and assert-xml-ref. Remove?

Within the result element for a grammar test, there is no useful distinction between this element and assert-not-a-grammar; in that context, the two should in practice be treated as interchangeable.

assert-xml

signals that the test input is a sentence in the language described by the grammar indicated and contains an acceptable XML representation of one parse of the sentence.

Remarks

Appears as child of result.

For ambiguous sentences, more than one parse is correct, and this element may occur more than once.

assert-xml-ref

signals that the test input is a sentence in the language described by the grammar indicated and give a URI for the prescribed XML representation of one parse of the sentence.

Remarks

Appears as child of result.

For ambiguous sentences, more than one parse is correct, and this element may occur more than once.

catalog-date

gives the release date of the test catalog describing the tests whose results are included in a test report.

Remarks

Occurs as a child of metadata at the top of the test report.

The catalog-date will normally just repeat the information given in the release-date attribute on the test-catalog element. The purpose of including it in the test report is to help document which version of the test catalog was used, if the test catalog is revised.

catalog-uri

gives the URI of the test catalog describing the tests whose results are included in a test report.

Remarks

Occurs as a child of metadata at the top of the test report.

code

marks inline code fragments in prose.

Remarks

Typical typographic treatment is to display in a monospaced font.

created

a metadata element containing information about the creation of a test catalog, test set, or test case.

Attributes

who

specifies who created the item.

when

specifies the date the item was created.

Remarks

The element is allowed but optional on individual test cases and grammar tests. If no creation date is given for an individual test case, it is assumed to have been created at the same time as the test set as a whole; if it was added later, it can (and ideally should) have its date of creation and revision history noted. When a test suite is changed over time, it can be helpful if every test case has a complete revision history.

description

contains a sequence of paragraphs containing a natural-language description of a test catalog, test set, or test case.

Remarks

Ideally, the description element should be able to contain prose tagged in XHTML, or in the XML vocabulary of the author's choice. Currently this is not the case.

doc

refers to external documentation for the test catalog, test set, or test case, if there is any.

Attributes

href

contains a URI at which the external documentation can be found.

Remarks

If a test case exhibits a particularly tricky situation which has been extensively discussed or analysed by the community group (or anyone else), for example, it's helpful to be able to provide pointers instead of having to reproduce the discussion inside the test catalog, or omitting it entirely.

emph

marks emphasized text in prose.

Remarks

This element is intended to mark rhetorical emphasis or stress, not arbitrary special typographic treatment. (If we turn out to need markup for arbitrary typographic treatment, we'll want to introduce hi for typographic highlighting, or seg or span with a style or rendering element.)

Note: if we turn out to need markup for arbitrary typographic treatment, we would do better to specify that the prose should be marked up in XHTML.

grammar-result

provides the result, in a test report, of running a given grammar-test in the test catalog.

Attributes

result

Summarizes the result of running the test; the value is one of:

pass, indicating that the reported result agrees with the expected result
fail, indicating that the test was run and the reported result did not agree with the expected result
not-run, indicating that the grammar test was not run, for whatever reason (the reason can be given in a description element)
other, indicating that none of the other values apply (an explanation can be given in a description or app-info element)

Remarks

A child of test-set-results.

The results of the grammar test can be given in greater or lesser detail. The minimum grammar-result element gives the result of the test. If desired, however, the test result can also contain or point to the grammar tested, the expected result, and the reported result.

For convenience, the schema allows the grammar-result element to be the root element of a document.

grammar-test

indicates that the grammar associated with a given test set should be treated as a test case and checked for conformance; contains a result element indicating expected result of the test.

Remarks

A child of test-set.

If the grammar is given in an ixml-grammar element or pointed to with an ixml-grammar-ref element, the grammar should be parsed with the grammar for ixml and the result compared with the expected result. The extra-grammatical constraints imposed by the spec should also be checked. If the grammar given is in XML (using the vxml-grammar or vxml-grammar-ref elements), the processor should check it for structural correctness (the spec requires that the XML grammar “can be derived from an ixml grammar in ixml form by parsing as described in this specification”) and for conformance to the extra-grammatical or non-structural constraints in the ixml specification.

ixml-grammar

contains a grammar in ixml form.

The following example shows an ixml grammar for arithmetic expressions.

     <ixml-grammar>
{ Grammar for simple arithmetic expressions, 
  for testing of test-case generation and
  grammar-manipulation tools.

  Revisions:
  2020-12-30 : CMSMcQ : made grammar
}
expr: s*, term+addop.
term:  factor+mulop.
factor:  num; var;
         '(', expr, ')'; 
         '[', expr, ']'; 
         '{', expr, '}'.
num:  digit+, ('.', digit*)?.
var:  ['a'-'z'].
addop: s*, ["+-"], s*.
mulop: s*, ["*/×÷"], s*.
digit: ['0'-'9'].
s: #20; #9; #a.       
     </ixml-grammar>

Remarks

Appears as a child of test-set and gives the grammar for the test cases of the set.

This is one of several elements that can be used to specify a grammar; grammars may be given in ixml or XML form, inline or externally.

ixml-grammar-ref

(i.e. reference to an external grammar in ixml form)

points to an external resource containing an ixml grammar.

Attributes

href

contains a URI at which the external grammar can be found.

Remarks

Appears as a child of test-set and gives the grammar for the test cases of the set.

This is one of several elements that can be used to specify a grammar; grammars may be given in ixml or XML form, inline or externally.

metadata

contains a short sequence of metadata elements describing the contents of a test report.

Remarks

Children include name (of the report), report-date, processor, processor-version, catalog-uri, and catalog-date.

modified

contains an entry in the change log for a test case or test set describing a modification.

Attributes

change

describes the change.

who

specifies who made the change.

when

specifies the date the change was made.

Remarks

name

gives the name assigned to a test report.

Remarks

Occurs as a child of metadata at the top of the test report.

The analogous name of a test catalog is given as an attribute on test-catalog, not as an element.

p

a paragraph of natural-language prose.

Remarks

A very small number of phrase-level elements are provided for the markup of the prose; if the set proves inconveniently small, it can easily be grown.

processor

indicates the processor used in the testing described in a a test report.

Remarks

Occurs as a child of metadata at the top of the test report.

processor-version

gives the version number or identifier, if any, of the processor for which a test report was prepared.

Remarks

Occurs as a child of metadata at the top of the test report. Optional.

report-date

gives the date on which a test report was prepared.

Remarks

Occurs as a child of metadata at the top of the test report.

reported-not-a-grammar

in a test report, signals that the processor reported that the grammar supplied for a test is not a conforming ixml grammar (and thus cannot be used to parse the test input).

Remarks

Appears as child of result.

reported-not-a-sentence

in a test report, signals that the test input is not a sentence in the language described by the grammar supplied for the test.

Remarks

Appears as child of result.

reported-xml

in a test report, signals that the processor parsed the test input as a sentence in the language described by the grammar supplied; contains the XML element produced as output.

Remarks

Appears as child of result.

For ambiguous sentences, more than one parse is correct, and this element may occur more than once.

reported-xml-ref

in a test report, signals that the processor parsed the test input as a sentence in the language described by the grammar supplied; gives a URI for the XML output produced by the processor.

Remarks

Appears as child of result.

For ambiguous sentences, more than one parse is correct, and this element may occur more than once.

result

in a test catalog specifies the expected result of a test, using the assert-xml, assert-xml-ref or the assert-not-a-sentence elements depending on what is to be asserted and how; in a test report may optionally recapitulate the expected result and may give the output reported by the processor.

Remarks

Unlike a test catalog, which is required to specify the expected result exactly once, a test report may give the expected result in multiple forms (i.e. both inline and by reference to an external document). The same holds true for the reported result.

test-case

contains input and expected result for one test case.

Attributes

name

a name by which the test case can be referred to; should be unique within the test set

The following example shows a test case for the grammar S: {nil}. (not shown, because it's specified earlier in the catalog). The input is the empty string, as specified here by the test-string element. The expected output is an empty S element, as specified here by the assert-xml element within result.

<test-case xml:id="S-1" name="S1">
   <created by="Gingersnap (CMSMcQ)" 
            on="2021-01-26"/>
   <test-string/>
   <result>
      <assert-xml>
         <S/>
      </assert-xml>
   </result>
</test-case>

Remarks

A processor being tested should parse the specified input using the grammar supplied by a containing test-set element. The processor passes the test if the processor's output matches the expected result (specified in the result element).

test-catalog

contains a collection of test cases organized into one or more test sets and released at the same time.

Attributes

name

a name by which the collection of tests can be referred to

release-date

date of the test suite described by this catalog; more specifically, date of its release or official publication

Remarks

When a suite of related tests is published, it may happen that there are different test-catalog files (and test-catalog elements) for different groups of tests within the larger collection. It is a good idea for there to be one test catalog that includes all of the others, so as to provide a single point of entry for processing the entire collection of tests. The test-set-ref element is used to include other test-catalog files.

test-report

contains a collection of test-case results derived by running the tests from a given test catalog through a particular processor.

Remarks

The overall structure of the test report mirrors the structure of the test catalog: for each test-set in the catalog, there will normally be a test-set-results element in the report, for each grammar-test a grammar-result, and for each test-case and test-result.

test-result

provides the result, in a test report, of running a given test-case in the test catalog.

Attributes

name

The name of the test case.

result

Summarizes the result of running the test; the value is one of:

pass, indicating that the reported result agrees with the expected result
fail, indicating that the test was run and the reported result did not agree with the expected result
not-run, indicating that the grammar test was not run, for whatever reason (the reason can be given in a description element)
other, indicating that none of the other values apply (an explanation can be given in a description or app-info element)

Remarks

A child of test-set-results.

The results of the test case can be given in greater or lesser detail. The minimum test-result element gives the name of the test case and the result. If desired, however, the test result can also contain or point to the grammar used for the test, the input string, the expected result, and the reported result.

For convenience, the schema allows the test-result element to be the root element of a document. It may be desired, for example, that a test harness produce a full test report for a given test catalog with only summary information on each test, and that a detailed report be written out for each failing test case as a separate document.

test-set

a set of test cases, nested test sets, or both.

Attributes

name

a name by which the test set can be referred to; should be unique within the test catalog

Remarks

The grammar for a test set may be given in any of several ways:

in ixml form inline, using an ixml-grammar element, or
in ixml form in an external resource, using an ixml-grammar-ref element, or
in XML form inline, using the vxml-grammar element, or
in XML form in an external resource, using the vxml-grammar-ref element.

Here, vxml is short for ‘visible XML’, as distinct from ‘invisible XML’.

The grammar-test element can be used to specify that the grammar itself is to be treated as a test case and parsed or checked against the grammar for ixml grammars. In this case, the test set need not contain any test cases.

Test sets can be nested to reflect any desired structure for the test suite. Nested test sets can take the form of child test-set elements or of child test-set-ref elements, which give the URI of a subordinate test set. (In practice the test set reference is usually to an XML document rooted in test-catalog, rather than test-set, but the effect is the same: all test sets in the external resource are logically subordinate to the parent of the test-set-ref element.)

If a grammar is specified by a test-set element or by one of its ancestor test-set elements, then the test set can contain any number of test cases. A grammar specified on a descendant overrides any grammar specified on an ancestor; the first grammar found, when ascending from the test case, is the one used.

Note that grammars are not inherited across references to external test sets.

test-set-ref

(i.e. reference to an external test set)

points to an external resource with a set of test cases.

Attributes

href

contains a URI at which the external test set can be found.

Remarks

Despite the name of the element, the usual target of a test-set-ref element will be a document rooted in a test-catalog element.

test-set-results

a collection of test results for a test-set in a test catalog.

Attributes

name

the name of the source test set

test-string

contains a string to be used as input in a test case.

Remarks

If the string is long or otherwise inconvenient to include literally in the test catalog, the test-string-ref element may be used instead.

test-string-ref

points to a string to be used as input in a test case.

Attributes

href

contains a URI at which the external input can be found.

Remarks

If the string is short and simple, it may be more convenient to include it directly in the test catalog; when this is so, the test-string element may be used instead of the test-string-ref element.

vxml-grammar

(i.e. reference to grammar in visible-XML form)

contains a grammar in XML form.

Remarks

Note that the schema for test catalogs does not constrain the XML used in the content of this element; in cases where the purpose of the test is to check the behavior of a processor when given XML input which is not a valid XML representation of an ixml grammar, the XML content of the vxml-grammar element will not in fact conform to the schema for the XML form of ixml grammars.

This is one of several elements that can be used to specify a grammar; grammars may be given in ixml or XML form, inline or externally.

vxml-grammar-ref

(i.e. reference to grammar in visible-XML form)

points to a grammar in XML form.

Attributes

href

contains a URI at which the external grammar can be found.

Remarks

This is one of several elements that can be used to specify a grammar; grammars may be given in ixml or XML form, inline or externally.