Early Detection of Writing Disorders using Computer Vision

This project aims to detect early learning writing disorders (such as dysgraphia) in a non-invasive way using video analysis. The system captures and reconstructs the complete 3D trajectory of a pen using synchronized cameras. The major innovation lies in the AI’s ability to analyze in-air movements (when the pen does not touch the paper), a crucial motor planning phase that is completely invisible to standard digital tablets. This end-to-end pipeline opens the way to 100% video-based automated clinical screening.

Context

This project is part of the AVIAREPT research thesis led by Lauren Sismeiro within the EuroMov Digital Health in Motion laboratory (University of Montpellier / IMT Mines Ales). Completed in partnership with Remy Plastre, this work represents our end-of-studies R&D mission as engineering students specializing in Artificial Intelligence.

• The Problem: Dysgraphia affects 5 to 10% of children. Current clinical tests (BHK test on paper) ignore temporal dynamics. Graphics tablets (e.g., Wacom) capture movement but suffer from a critical blind spot: loss of signal as soon as the pen is lifted more than 1 cm.
• Our Mission (R&D): Technically assist Lauren and build a non-invasive hardware and software system capable of tracking 100% of the pen’s movements in 3D space to capture these invisible cognitive markers.

Data

To capture ground truth and train models, a complex synchronized acquisition environment operating at 30 FPS was designed from scratch:

• Video Hardware: 2 Intel RealSense D435i cameras (RGB + Depth) and 1 Sony webcam.
• 2D Reference: 1 Wacom Intuos Pro tablet (2D position, coordinates, pressure, tilt).
• 3D Reference: A Qualisys Motion Capture (MoCap) system operating at 90 Hz with infrared markers.

Methodology

Challenge 1: The Choice of Ground Truth (MoCap vs. Tablet)

To evaluate the accuracy of our 3D reconstruction, we needed a reliable ground truth. Two options were available: the graphics tablet (very accurate but limited to 2D) or the optical MoCap system (3D tracking via infrared markers, but requiring mathematical interpolation of the pen tip position).

• Synchronization: To compare these two systems, we tested three temporal synchronization methods (by first contact detection, by velocity/acceleration correlation, and manually/visually). Visual and manual alignment proved to be the most accurate.
• Observation: The accuracy of the MoCap system is significantly lower than that of the tablet for movements as fine and micro-metric as writing. Furthermore, the tablet is not a perfect horizontal plane in 3D, and the conversion scales were biased. The MoCap was therefore discarded as a millimeter-level ground truth for contact writing, but retained to track broad movements when the pen is lifted.

Challenge 2: 2D Video Tracking and Postural Analysis

The first step in transforming raw pixels into physical coordinates was isolating the pen in the video streams.

• Pen Tracking (See intro video): Training a custom YOLOv8-pose model to simultaneously detect two key points (the tip and the top of the pen). This model was applied across two different viewing angles (front camera and side camera), demonstrating high robustness against occlusions (hands hiding the pen) and lighting variations.
• Postural Tracking: In parallel, MediaPipe was deployed to track upper body posture (shoulders, elbows, head) by associating 2D key points with depth maps from RealSense cameras, enabling analysis of the patient’s overall posture during writing.

Challenge 3: 3D Trajectory Reconstruction

The final hurdle was reconstructing the 3D coordinates of the pen’s tip and top from the 2D detections in the front and side views. We explored two distinct approaches:

• Deep Learning Approach (MLP): A Multilayer Perceptron (MLP) was trained using the coordinates from both 2D planes (front and profile) to directly predict the coordinates on the tablet plane. The goal was for the model to learn to correct the lens distortion effects of the cameras on its own.
• Geometric Approach (Epipolar Triangulation): A more traditional computer vision method consisting of projecting 2D coordinates into a unified 3D space. This required a rigorous calibration of intrinsic and extrinsic camera parameters using a grid (inspired by the work of Bruno Henriques, a PhD student at IMT Mines Ales specializing in this hardware). This robust approach was chosen for the final reconstruction.

Results

• YOLOv8-pose Tracking: The model achieves a mAP@50 exceeding 0.93, with a median localization error of only 3.28 pixels, ensuring smooth tracking despite rapid hand movements.
• Signal Validation: The video-reconstructed trajectories were compared with the Wacom tablet, validating the relevance of the purely visual approach for kinematic analysis.
• We submitted an abstract for a publication to PFIA 2026 / IEEE FG 2026.