CROHME 2014 Task 1:
Isolated Symbol Recognition
Task Description and System Evaluation
The systems will be tested under two conditions: 1) with valid symbols, and 2) with valid symbols along with mis-segmented symbols ("junk"). In both cases, the Top-N recognition rate from Top-1 through Top-10 will be computed. In the condition with junk samples, the False Positive and False Rejection Rates will be also computed.
The 101 valid symbol classes are provided here.
Input file format
The input file format is the CROHME InkML format, with only one symbol per file. Training file contents:
- Unique identifier (UI) constructed from the UI for the expression from which the symbol is taken,
- The true symbol class (101 classes + junk/reject class),
- Stroke list (using the same stroke identifiers provided in the full expression .inkml file),
- Segmentation giving the list of strokes in the symbol, along with the segment identifier (from the original expression .inkml file).
NOTE: Training files with ground truth contain all of the information above; test files will not contain the true symbol class or segmentation information. We recommend using an XML parsing library that will allow different tags to be easily selected from the XML parse tree for an .inkml file, and avoid requiring the symbol class and segmentation information in your program that runs your recognizers after training.
Training data set
The training set is generated from expressions in the CROHME 2013 (part IV) training dataset. This full expression set if available from the TC11 download page. A script to generate all isolated symbols is available for registered participants.
The "junk" class should be used to identify invalid symbols (i.e. incorrect segmentations passed to your classifier, when strokes from a symbol are incorrectly merged with strokes from another symbol, and/or strokes from a symbol are separated into multiple symbols). Most advanced word/text/math recognition systems depend on the construction of a segmentation trellis where each node is a segmentation hypothesis. Within such a trellis, most nodes (symbol hypotheses) are invalid. The proportion of valid symbols to junk in the competition test set will be roughly 1:1 (i.e. roughly as many incorrectly segmented symbols as valid symbols).
Training "junk" samples are also generated from the CROHME 2013 (part IV) training dataset using a script that partitions all stroke sequences in time order up to four strokes, removing valid segmentations. The same script will be used to generate symbols in the test data set.
System inputs
The systems will be called with one argument which is a file containing the list of all inkml files (one per line). Please do not load/unload resources between symbols (e.g. load all parameters set from training once before processing the list of symbols in the input file).
System outputs
The output format uses a simple Comma Separated Value format (CSV) printed on the standard output containing one line per symbol with the UI and the 10 best labels (or less), provided in decreasing order of confidence (i.e. the best class is listed first, followed by the second-best, etc.). Note that the comma ',' class will be written 'COMMA' to avoid confusion and the junk class will be indicated by the string 'junk'. Example of a valid output:
MfrDB3907_85801, a, b, c, d, e, f, g, h, i, j
MfrDB3907_85802, 1, |, l, COMMA, junk, x, X, \times
If the participants agree to share results of their systems after the competition, it would be interesting to also provide a score associated with each class. To do this, an optional second line per symbol should be provided, starting with 'scores' that contains the scores for each class provided in the first line, with scores in descending order. Example of valid output:
MfrDB3907_85801, a, b, c, d, e, f, g, h, i, j
scores, 1, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1
MfrDB3907_85802, 1, |, l, COMMA, junk, x, X, \times
scores, 10, 8.001, 2, 0.5, 0.1, 0,-0.5, -1, -100