Ranking Handwriting Legibility with Foundation Models: Our Research Published in Scientific Reports
Background and Motivation
Legibility describes how clearly letters and words can be read, and it is the visible result of the writing process. Poorly legible handwriting in children can point to difficulties in the underlying motor and cognitive processes and to developmental disorders, and it is linked to later success at school. Identifying the children who struggle early makes targeted support possible. Today, teachers, researchers, and occupational therapists assess legibility with rating scales, which takes time and depends on the person rating. Automated systems exist, but many of them rely on digitizing tablets or special pens that record the writing movement, which limits where they can be used. Methods that work on a plain image of the handwriting avoid this requirement, yet for alphabetic scripts they remain rare, partly because legibility is subjective and annotated data is scarce.
Our Approach: Pairwise Comparisons and a Foundation Model
The paper was written by master's student Meike Bauer and PhD candidate Marco Lents as equal first authors, together with PhD candidate Francesco Di Salvo, master's students Erik Schmidt and Lukas Pieger, Professor Christian Ledig, and our partners from STABILO and the Schreibmotorik Institut. In 2024, 202 children aged ten to twelve at two schools in Germany each copied the same ten short German sentences by hand. The scanned samples form the Handwriting Assessment for Legibility and Ordering (HaLO) dataset with 1,907 images. Rating a single sample on an absolute scale is demanding for people and leads to noisy ratings, so the annotators were instead shown two samples at a time and asked which one reads more fluently. Eight annotators made 19,070 such pairwise judgments. A Bayesian Bradley-Terry model, a statistical model for paired comparisons, turns these judgments into a reference ranking of all samples from least to most legible.
For the automatic assessment, each image passes through the frozen vision encoder of Pixtral 12B, a multimodal foundation model pretrained on natural images and documents, which accepts images of any size and aspect ratio, so a line of handwriting needs no resizing. A single linear layer with 1,025 parameters, a so-called linear probe, maps the resulting image representation to a legibility score. It is trained on the pairwise judgments, and the scores of all samples together form a ranking that can be compared with the human reference.
Results
On the test set, the linear probe identified the more legible sample of a pair in 83 percent of cases, with an area under the ROC curve of 0.92. Because human judgments are not perfectly consistent, even a model fitted directly to the annotations cannot be right every time: the Bradley-Terry model reaches 90 percent and an area under the curve of 0.97, which marks the practical ceiling for this dataset. Deeper networks on the same representations brought no significant improvement, and the same probe on the CLIP and DINOv2 encoders reached lower values of 0.87 and 0.89.
The ranking produced by the model agrees with the human reference ranking as closely as the rankings of two human raters agree with each other: Kendall's tau of 0.70 and Spearman's rho of 0.89 for the model, against an average of 0.71 and 0.89 between humans. When four annotators rated samples on a five-point scale, their agreement with each other ranged from 0.45 to 0.66 (Krippendorff's alpha), while their agreement with the model ranged from 0.61 to 0.70. The model therefore matches the consistency of human raters on the ranking and meets or exceeds it on individual ratings. An analysis of the encoder's representation space further shows that properties such as cursive writing and thin strokes are encoded in it linearly, and that samples written in cursive tend to receive lower legibility scores. The image representations, the annotations, and the code are publicly available under the MIT license.
You can explore the full paper here. Explore the dataset here and the code here.
About Scientific Reports
Scientific Reports is an open access journal of the Nature Portfolio that publishes research from the natural sciences, psychology, medicine, and engineering. According to the 2025 Journal Citation Reports, it has a two-year impact factor of 4.9 and is the second most-cited journal in the world, with more than 1,061,414 citations in 2025. As the paper is open access, it can be read by teachers, therapists, and researchers without a subscription.
From Three Master's Theses to One Paper
The paper is the result of three master's theses at our chair, all supervised by Professor Christian Ledig, that built on each other. Erik Schmidt and Lukas Pieger laid the foundation together. In close collaboration with STABILO, they designed a standardized writing test and a rating tool and collected the handwriting of the children with the STABILO DigiPen, a ballpoint pen with motion sensors that writes on ordinary paper. The result was a first version of the dataset with absolute legibility ratings.
In his thesis "Deep Learning based Evaluation of Handwriting Legibility using a Sensor Enhanced Ballpoint Pen", Erik Schmidt asked whether legibility can be determined from the sensor data of the pen. His models overfit the training data and relied mainly on whether a sample was written in cursive or in print letters. This showed the limits of the sensor-based route and pointed to writing style as a strong factor in perceived legibility, a link that the paper later confirms in the representations of the foundation model. Lukas Pieger took the image-based route in his thesis "Deep Learning based Legibility Evaluation using Images of Children's Handwriting", co-supervised by Jens Barth from STABILO. His image-based models performed on par with the sensor-based models for overall legibility and clearly better for more specific legibility features. He also found that ratings by experts and non-experts did not differ significantly, while the agreement between raters remained low.
Meike Bauer took this low agreement as the starting point of her thesis "Deep Learning Based Assessment of Handwriting Legibility via Pairwise Image Comparisons and Ranking". She re-annotated the handwriting samples with pairwise comparisons, designed the Bradley-Terry ranking procedure, and showed that pairwise judgments are much more consistent than absolute ratings. Her models already identified the more legible sample of a pair in up to 84 percent of cases. Together with Marco Lents, who further improved the model, carried out the final experiments, and co-wrote the manuscript, she extended her thesis into the published paper: the HaLO dataset as a public resource, the estimate of the performance ceiling, the comparison with human raters on rankings and on individual ratings, the comparison of encoders, and the analysis of the representation space. After graduating, Meike Bauer joined PwC.
Research and Industry Working Together
The project is a joint effort of our chair, STABILO International GmbH, and the Schreibmotorik Institut e.V., both based in Heroldsberg. Tim Hamann and Jens Barth from the Digital Product Development of STABILO provided the STABILO DigiPen, contributed to the study design, and collected the handwriting samples in the schools together with our students. Tal Hoffmann, head of the Schreibmotorik Institut, and Susanne Salata, occupational therapist and research associate at the institute, contributed their expertise on handwriting and guided the scientific background on legibility. Tal Hoffmann, Jens Barth, and Susanne Salata also volunteered as annotators for the pairwise comparisons in Meike Bauer's thesis. The Schreibmotorik Institut is a non-profit research institute outside the university system that works on handwriting, its connection to reading and writing skills, and the support of children who find writing difficult.
Each partner brought something the others could not provide alone. STABILO contributed the writing instrument and the experience of a company that develops products for writing, the Schreibmotorik Institut contributed knowledge of how handwriting develops and how it is assessed in practice, and our chair contributed the machine learning methods and the students who turned both into research. For the students, this meant working on a real question with real data from the first day of their theses and discussing their results regularly with partners from industry and practice. The work was carried out within the KIBEL project, funded by the Bavarian State Ministry of Economic Affairs, Regional Development and Energy as part of the Bavarian Collaborative Research Programme, with additional support from the Hightech Agenda Bayern.
In November 2025, TV Oberfranken reported on the project, and you can find the report here.
Looking Forward
The paper also names the next questions: whether the results carry over to other languages, scripts, and age groups, whether text recognition software fails on the same samples that people find hardest to read, and how a smaller image encoder could make screening possible directly on a tablet. Marco Lents continues this line of research in his PhD, which asks how artificial intelligence can give pupils feedback on the legibility of their handwriting.
We sincerely thank Meike Bauer, Erik Schmidt, and Lukas Pieger for their dedication and effort, and our partners at STABILO and the Schreibmotorik Institut for a close and trusting collaboration. Their work shows that a thesis at the xAILab Bamberg can become part of a publication in an international journal, and we warmly invite motivated students who would like to contribute to research of this kind to get in touch with us about a thesis.
