This new CROHME edition will propose 3 tasks on the topic of handwritten math expressions with 3 different modalities: on-line, off-line and bimodal.
Same Evaluation for all tasks
We propose using the same evaluation metric for the 3 tasks to allow a fair comparison of the different participating systems in the different modalities at the expression level. Furthermore, as in the previous CROHME competitions, we will provide a detailed evaluation of the systems which provides the necessary information. All these metrics are provided in the CROHMELib and LgEval tools.
Thus the two levels of evaluation are:
expression level discarding the segmentation: that is possible whatever the modality, and possible for all recognition systems. The expression is represented as a tree of symbols containing their layout: Symbol Layout Tree (SLT). It considers only the symbolic form of a recognized formula, ignoring stroke/image segmentation. This form can be produced from MathML (presentation version), LaTeX string, or Stroke Label Graph (used in earlier CROHME).
metrics at stroke/bounding box level if provided by the participants: The ground truth of expression is given at the symbol level, which means that the segmentation and the label of each symbol are provided, plus the spatial relationship between the symbols. From the beginning of CROHME, we provide evaluation metrics (and tools) using this accurate ground truth: primitive and symbol recognition rate, relation recognition rate, … in this case, the expression level recognition rate means that each symbol and relation has been perfectly localized and recognized.
Task 1. Online HME Recognition
For the traditional task in CROHME, participants must convert a list of handwritten strokes captured as a list of poly-lines from a tablet or similar devices to a Symbol Layout Tree (SLT) or a Presentation MathML tree, or a LaTeX string. The SLT format links the raw strokes to the output expression and allows a detailed evaluation.
Note that the participants should use strictly the online information and not the offline information.
Participants will be ranked by the expression rate of their system.
Task 2. Offline HME Recognition
For offline recognition of handwritten expressions, for a given input image, participating systems must produce a Symbol Layout Tree (SLT), a LaTeX string, or a MathML representation tree as output. The SLT format links the bounding boxes of symbols to the output expression and allows a detailed evaluation.
Note that the participants should use strictly the off-line information and not the online information.
Participants will be ranked by the expression rate of their system.
Task 3. Bimodal On+Off HME Recognition
For the bimodal expression recognition, both the list of handwritten strokes and the corresponding images will be provided to the systems. The two signals will be coherent, in the sense that the image and the pen path can be aligned because they come from the same acquisition. Systems will be able to merge information from both worlds, using early, mid or post-merging of the information. For the evaluation, stroke-level ground-truth and bounding-box information are available as in task 1 and 2, thus a detailed evaluation is possible: from symbol segmentation and classification to full expression recognition. For the ranking of participants, the expression level will be used, using the symbol tree, as in task 1 and 2.
Note that the participants in this task can use both the online and offline information.
Participants will be ranked by the expression rate of their system.