CROHME 2014 Task 2:
Mathematical Expression Recognition
Task Description and System Evaluation
In this task, systems must segment, classify and parse symbols in individual handwritten expressions. All training and testing expressions will conform to a LaTeX grammar (the part IV grammar from CROHME 2013). XML and human-readable versions of these grammars are available to participants through the CROHMELib library. CROHMELib also provides a java parser for testing whether a LaTeX expression is legal for the part IV grammar. The grammar includes vertical layout structures such as fractions, square roots, subscripts, superscripts, and limits above and/or below summations and integrals. Grid-based and tabular structures such as matrices, choice notation, and cases in function definitions are not included in this task (see Task 3).
Systems will be evaluated using the same metrics as CROHME 2013.- Expression level metrics
- Expression recognition rate (used to rank systems)
- Expression rate allowing 1, 2 or 3 symbol or spatial relationship labeling errors in the Symbol Relation Tree (SRT), without changing the edge structure of the SRT.
- Symbol level metrics
- Recall and Precision for symbol segmentation
- Recall and Precision for symbol detection, requiring both correct segmentation and classification
- Recall and Precision for spatial relationships between symbol pairs in:
- Symbol Relation Trees, and
- SRT's with inherited spatial relationships (producing a Directed Acyclic Graph (DAG) over symbols)
- Stroke level metrics
Stroke level metrics are computed over a complete labeled directed graph over strokes, with each stroke labeled by the class of symbol to which it belongs, and each directed edge labeled by the spatial relationship from the symbol to which the parent stroke belongs, to the symbol to which the child stroke belongs (including 'merge'/'same symbol' and 'no relationship').
Each of the following will be computed for label graphs constructed from Symbol Relation Trees (which becomes a DAG over strokes), as well as SRT + inherited relationship DAGs.
- Label Hamming distances for strokes and relations (ΔB)
- Weighted Hamming distance (ΔE)
- Confusion matrices for symbol classification (each stroke is labeled by the class of symbol it belongs to) and spatial relationships (for directed and undirected stroke pairs).
Input file format
The input file format is the CROHME InkML format used in previous competitions. These files may be visualized using the CROHME InkML Viewer. A description is provided here.
Training data set
The training data set is the same used for CROHME 2013 (part IV), available from the TC11 download page.
New expressions will be created for the Test data set, by a set of writers that may be distinct from those in the training data. All new expressions will conform to the part IV grammar.
System inputs / outputs
Systems will be called with two arguments, the names of the input inkml file, and the output label graph file.
The output will be a simple Comma Separated Value format (CSV) which represent the Label Graph (LG) of the expression. Label Graphs represent structure at the stroke level. An introduction to the .lg format is available here, and the labels to use for spatial relationships are described here). Each stroke is represented by a node labeled with the class of its associated symbol, and all stroke pairs are have two labeled directed edges between them. Edge labels represent whether two strokes are unrelated, belong to the same symbol, or belong to two symbols in a spatial relationship (relations: Right, Above, Below, Inside (square root), Superscript, or Subscript). The LgEval library will be used to compare label graph files.
Remarks:
- Inkml files will not be accepted as system output. However, conversion scripts from CROHME .inkml to .lg format, and to create .inkml files from an .inkml file and .lg interpretation (e.g. for viewing results in the CROHME InkML Viewer) are provided in CROHMELib.
- .lg files should encode a Directed Acyclic Graph (DAG) over strokes, which represents a Spatial Relation Tree (SRT) over symbols without inherited edges. Inherited edges will be generated during evaluation (using tools provided in the LgEval library).
- The comma ',' class will be written 'COMMA' to avoid confusions in the CSV format.
- The prime symbol (e.g. x', $x^\prime$ or $x\prime$) must be represented using the Superscript (Sup) relationship.