# Stephen Watt

**The Mathematics of Mathematical Handwriting Recognition**

**Stephen M. Watt**

Computer Science Department

London, Ontario CANADA

www.csd.uwo.ca/~watt

*The*University*of*Western OntarioLondon, Ontario CANADA

www.csd.uwo.ca/~watt

Abstract:

Accurate computer recognition of handwritten mathematics offers to provide a

natural interface for mathematical computing, document creation and

collaboration. Mathematical handwriting, however, provides a number of

challenges beyond what is required for the recognition of handwritten natural

languages. For example, it is usual to use symbols from a range of different

alphabets and there are many similar-looking symbols. Many writers are

unfamiliar with the symbols they must use and therefore write them incorrectly.

Mathematical notation is two-dimensional and size and placement information is

important. Additionally, there is no fixed vocabulary of mathematical

``words'' that can be used to disambiguate symbol sequences. On the other hand

there are some simplifications. For example, symbols do tend to be

well-segmented. With these charactersitics, new methods of character

recognition are important for accurate handwritten mathematics input.

We present a geometric theory that we have found useful for recognizing

mathematical symbols. Characters are represented as parametric curves

approximated by certain truncated orthogonal series. This maps symbols to the

low-dimensional vector space of series coefficients. The Euclidean distance in

this space is closely related to the variational integral between two curves

and may be used to find similar symbols very efficiently. Training data sets

with hundreds of classes are seen to be almost linearly separable, allowing

classification by ensembles of linear SVMs. We have seen that the distance to

separating planes provides a reliable confidence measure for classifications.

In this setting, we find it particularly effective to classify symbols by their

distance to the convex hulls of nearest neighbors from known classes.

Additionally, by choosing the functional basis appropriately, the series

coefficients can be computed in real-time, as the symbol is being written and,

by using trancated series for integral invariant functions,

orientation-independent recognition is achieved.

The beauty of this theory is that a single, coherent view provides several

related geometric techniques that give a high recognition rate and do not

rely on peculiarities of the symbol set.

Accurate computer recognition of handwritten mathematics offers to provide a

natural interface for mathematical computing, document creation and

collaboration. Mathematical handwriting, however, provides a number of

challenges beyond what is required for the recognition of handwritten natural

languages. For example, it is usual to use symbols from a range of different

alphabets and there are many similar-looking symbols. Many writers are

unfamiliar with the symbols they must use and therefore write them incorrectly.

Mathematical notation is two-dimensional and size and placement information is

important. Additionally, there is no fixed vocabulary of mathematical

``words'' that can be used to disambiguate symbol sequences. On the other hand

there are some simplifications. For example, symbols do tend to be

well-segmented. With these charactersitics, new methods of character

recognition are important for accurate handwritten mathematics input.

We present a geometric theory that we have found useful for recognizing

mathematical symbols. Characters are represented as parametric curves

approximated by certain truncated orthogonal series. This maps symbols to the

low-dimensional vector space of series coefficients. The Euclidean distance in

this space is closely related to the variational integral between two curves

and may be used to find similar symbols very efficiently. Training data sets

with hundreds of classes are seen to be almost linearly separable, allowing

classification by ensembles of linear SVMs. We have seen that the distance to

separating planes provides a reliable confidence measure for classifications.

In this setting, we find it particularly effective to classify symbols by their

distance to the convex hulls of nearest neighbors from known classes.

Additionally, by choosing the functional basis appropriately, the series

coefficients can be computed in real-time, as the symbol is being written and,

by using trancated series for integral invariant functions,

orientation-independent recognition is achieved.

The beauty of this theory is that a single, coherent view provides several

related geometric techniques that give a high recognition rate and do not

rely on peculiarities of the symbol set.