Yet another blog about WPF, Surface, SL, MVVM, NUI....

Specifications

The HandWritingToText controls translate the text written with it's hand by the user into "computer text". The final purpose is to triggers some actions when a specific keyword is recognized and it'll so be able to recognize only one word at a time and not a whole text.

1. The recognized text will be published via an event and available trough a DependencyProperty,
2. The control will be fully customizable via properties,
3. It is delivered into an assembly which can be used in any WPF project.

We will follow all the tips and tricks given in this previous post : How to create your own control library.

Creating the hand writing to text control

Requirements

The first thing we need is the engine which will translate our hand written text to a real text. This tools is made available to us into the “IAWinFx dlls” of the Windows SDK. Starting from here, I had an hard time finding all of them because they are placed in a lot of differents folder. You have to install the FULL windows SDK with all the samples for all the languages or you won’t be able to find them all. The seeked files are :

1. IACore.dll,
2. IALoader.dll,
3. IAWinFX.dll,
4. Microsoft.Ink.Analysis.dll,
5. Microsoft.Ink.JournalReader.dll

After adding these DLLs as a reference to your project you have to a little configuration. As they are compiled for an old version of the runtime and that we are using the 4.0 we needs to add this snippet in our App.config file :

  <startup useLegacyV2RuntimeActivationPolicy="true">
<supportedRuntime version="v4.0" />
</startup>

Build up the visual

The template of our control is pretty simple because it will be constituted of one single element : an InkCanvas. This control is a Template part that we name PART_theInkCanvas. An InkCanvas is simply an area that receives and displays ink strokes.

Here is the content of our control theme file :

<ResourceDictionary 
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:handWriting="clr-namespace:AmazingsWPFControls.HandWritingToText">
<!-- ================================================================= -->
<!-- HandWritingToText                                                               -->
<!-- ================================================================= -->
<Style x:Key="{x:Type handWriting:HandWritingToText}"
TargetType="{x:Type handWriting:HandWritingToText}">
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="{x:Type handWriting:HandWritingToText}">
<InkCanvas x:Name="PART_theInkCanvas" Background="Transparent" />
</ControlTemplate>
</Setter.Value>
</Setter>
</Style>
</ResourceDictionary>

You may only wants to recognize the words and not display what the user writes : to do so you’ll only have to set the strokes color to Transparent. But this is not as simple because you have to set the property on the DrawingAttributes of the inkCanvas which is not bindable because this is not a DependencyObject. So the solution I found is to add a dependency property “StrokeColor” on the control, and to set via the code-behind the color on each change of this property (the color must also be set when the control is loaded). The code is very classic and I let you dig in the source if you want to see it !

Recognize hand writing

Then comes the funny part : analyze the drawed strokes and find the real words behind then ! It’s here that the IAWinFX DLLs take all their meaning by providing the InkAnalyser class. This is a very nice tool which analyses the strokes you feed him with and returns you what it recognize.

It’s pretty straightforward to instantiate an InkAnalyser because no parameters are needed. But we need to set some parameters for its algorithm to make it works better:

• The AnalysesMode which is how the ink analyzer behaves before, after and during the analysis. In our case we want it to clear the cache automatically and to start the reconciliation process right after it had finished his recognition.
• The analysis hint which are informations we give to the analyzer to facilitate its guess. Each information is called a Factoid. In our case the hint will be applied to the whole inkCanvas (an hint can be defined to a restricted zone if wanted), we’ll be looking for words only (no full sentence) and the analyzer will limit its analysis of ink within the hint's area to conform to the hint's Factoid property. A complete list of the different factoid available can be found here : http://msdn.microsoft.com/en-us/library/ms818539.aspx

//Create the ink analyzer
theInkAnalyzer = new InkAnalyzer();
theInkAnalyzer.AnalysisModes = AnalysisModes.StrokeCacheAutoCleanupEnabled | AnalysisModes.AutomaticReconciliationEnabled;
theInkAnalyzer.ResultsUpdated += new ResultsUpdatedEventHandler(theInkAnalyzer_ResultsUpdated);
var hint = theInkAnalyzer.CreateAnalysisHint();
hint.Location.MakeInfinite();
hint.WordMode = true;
hint.CoerceToFactoid = true;

Feed the ink analyzer with the strokes

To get the strokes entered by the use we only have to get the InkCanvas from the template in the OnApplyTemplate methods and subscribe to its StrokeCollected event. Then in it's handler we’ll be able to feed the ink analyser.

But wait… If we let the user play with the component and writes whatever he wants to, the ink analyzer won’t be able to recognize any pattern : it will only be a child drawing for him !

So the solution I found is to define a maximum time for the user to write the keyword. This time is fully customizable and exposed as a DependencyProperty named TimeToEnterText. If this duration is elapsed, we clear the ink analyzer of all the strokes it possessed and start feeding it with the new ones :

public override void OnApplyTemplate()
{
base.OnApplyTemplate();
theInkCanvas = 
      base.Template.FindName("PART_theInkCanvas", this) as InkCanvas;
if (theInkCanvas == null)
    throw new ArgumentException("Cannot find the PART_theInkCanvas InkCanvas.");
theInkCanvas.StrokeCollected += 
      new InkCanvasStrokeCollectedEventHandler(theInkCanvas_StrokeCollected);
}
void theInkCanvas_StrokeCollected(object sender,
       InkCanvasStrokeCollectedEventArgs e)
{
//Check if the time has passes before reset
if (DateTime.Now – lastResetOfInkCanvas 
             > TimeSpan.FromMilliseconds(TimeToEnterText))
{
try
{
theInkAnalyzer.RemoveStrokes(theInkCanvas.Strokes);
}
catch (Exception)
{
Debug.WriteLine(e);
throw;
}
//Remove old strokes
this.theInkCanvas.Strokes.Clear();
//Add the last one added
theInkCanvas.Strokes.Add(e.Stroke);
lastResetOfInkCanvas = DateTime.Now;
}
theInkAnalyzer.AddStroke(e.Stroke);
//Launch the analysis
theInkAnalyzer.BackgroundAnalyze();
}

How to get the results of the analysis ?

Everything takes place in the handler of the InkAnalyzer ResultsUpdated event. The argument of type ResultsUpdatedEventArgs is first used to find if the analysis is successful by using it’s e.Status.Successful property. If there is some results we then have to get the inkAnalyzer to get them.

The results are available in the form of a collection of a base class : ContextNode. Here is a list of the different classes inheriting from the ContextNode :

What they represent is each time quite clear and the most important are maybe these tree :

• InkWordNode : represents  a collection of strokes that make up a logical grouping that forms a recognizable word. The text can be found via the GetRecognizedString method.
• InkDrawingNode : Represents a ContextNode for a collection of strokes that make up a drawing. For example it can represent a Rectangle, a Circle or any geometric shape. The shape can be found via the GetShape method on an instance.
• LineNode : Represents a ContextNode for a line of words.

In the event handler we iterate trough the results and raise an event for each word finded. The event is a routed one that we have declared in the control named TextEnteredEvent :

void theInkAnalyzer_ResultsUpdated(object sender, ResultsUpdatedEventArgs e)
{
if (e.Status.Successful)
{
ContextNodeCollection nodes = ((InkAnalyzer)sender).FindLeafNodes();
foreach (ContextNode node in nodes)
{
if (node is InkWordNode)
{
InkWordNode t = node as InkWordNode;
string recognizedString = t.GetRecognizedString();
RaiseTextEnteredEvent(recognizedString);
}
//For demonstration purpose only
else if (node is InkDrawingNode)
{
InkDrawingNode d = node as InkDrawingNode;
Shape shape = d.GetShape();
//Shape may be null here...
}
}
}
}

Additional thoughts

One more thing I faced in testing this control : it worked very nicely on my laptop but it was not recognizing a word on my another home computer. It take me a little time to figure out what was the problem : one computer is in English and the other is in French.

But don’t worry you can solve it and very easily. When you add a stroke in the analyzer you can customize it a little and one of the customization is to set the language code of the draw ! So if you face the same problem than me you can use this snippet (which is also specifying that the strokes are forming a word and not anything else) :

theInkAnalyzer.AddStroke(e.Stroke);
theInkAnalyzer.SetStrokeType(e.Stroke, StrokeType.Writing);
theInkAnalyzer.SetStrokeLanguageId(e.Stroke, 0x09);//Use EN-US languageID

Clean up your room !

Using the ink analyzer can lead to memory leaks and freeze when exiting the application so you have to be careful to clean-up all the used resources. To do so I make the control implement the IDisposable interface and call in it a method named ClearResources.

I also subscribe to the Application.Exit event if it exist to call the same method in the handler :

/// <summary>
/// Clears all resources : abort the current analyses, call dispose , ...
/// </summary>
private void ClearAllRessources()
{
lock (_locker)
{
if (!_isDisposed)
{
theInkAnalyzer.Abort();
theInkAnalyzer.Dispose();
_isDisposed = true;
}
}
}

How to use it in your application ?

Here are the XXX steps to use it in your application :

1. Add the amazing control libs to your project as a reference,
2. Defines the XML Namespace amazingControls : http://blog.lexique-du-net.com/wpf/AmazingWPFControls
3. Add the control in your visual tree,
4. Register to the TextEnteredEvent or bind yourself to the LastRecognizedWord property of the control.

<AmazingControls:HandWritingToText
xmlns:AmazingControls="http://blog.lexique-du-net.com/wpf/AmazingWPFControls"
TextEntered="HandWritingToText_TextEntered" StrokeColor="Red"
Height="450" Background="White" />

Where to find the resulting control ?

You can find it on codeplex in the amazing WPF controls project, at this address :

                                http://amazingwpfcontrols.codeplex.com/

(The post announcing the creation of this codeplex project can be read here)

Some links you may find useful

• Find New Meaning In Your Ink With Tablet PC APIs In Windows Vista
• List of available Factoid for you analysis hints
• MSDN page on digital ink