Tutorial

Requirements

Before starting this tutorial, there are some requirements in order to continue.

Introduction

This tutorial will guide you through the implementation of a custom annotation type. This includes both the backend (Analysis configuration, RegexAnnotator, HttpApiAnnotator, and a filter) and the editor (Annotation type and UX). All will be accompanied by (collapsed) code examples.

For this tutorial the following example use-case has been chosen:

  • Parts of a document can be in different languages, either "en" or "nl"

  • Find all occurrences of feet and meter(s) unit of measure using a RegexAnnotator

  • Check if the unit of measure is appropriate for the used locale using a custom HttpApiAnnotator

  • Filter which annotations to return to the client

  • Show the annotations in the editor

  • Convert unit of measure text in document

This use-case can also be seen in the analysis configuration below.

In a real world scenario the HttpApiAnnotator would probably also include the logic which is done by the RegexAnnotator, however this is done separately in this tutorial for the purpose of explaining multiple concepts of Content Quality.

The code samples below can also be found at the following GitHub repository: https://github.com/FontoXML/content-quality-tutorial

Project setup

This tutorial can be split into three parts:

  • The backend

  • HttpApiAnnotator

  • FontoXML Editor

To set up the tutorial project, a number of directories need to be created:

  1. Create a project directory in which all parts will be placed.

  2. Create a "backend" directory in the project directory.

  3. Run "fdt content-quality init --version 1.4.5" in the backend directory.

  4. Create an "httpapiannotator" directory in the project directory.

  5. Create an "editor" directory in the project directory.

  6. Place the FontoXML Editor code (the unbuild version) in the editor directory.

For this tutorial, all systems will run on their own server which will run on the following ports:

Port

Service

8080

FontoXML Editor on dev-server.

6000

Content Quality App on docker.

6005

The HttpApiAnnotator which implements the custom annotation type in your programming language of choice.

Backend configuration

Content Quality needs to be configured with an analysis configuration to specify which annotators need to be run when Content Quality annotates the content received from the FontoXML Editor. This configuration is build up out of Compositors, Annotators, and Filters.

The FontoXML Editor splits documents into fragments, for example paragraphs, and are all checked individually by sending them to the Content Quality App. The analysis configuration describes what needs to be done with the content fragments received from the FontoXML Editor. Each fragment is passed through the flow as described in the analysis configuration.

Edit the analysis configuration file "<project>/backend/Configuration/analysis.xml" and use the following configuration.

<project>/backend/Configuration/analysis.xml

XML

<?xml version="1.0" encoding="utf-8"?>
<analysis
	xmlns="http://schemas.fontoxml.com/fcq/1.0/analysis-configuration.xsd"
	xmlns:tutorial="urn:fontoxml:fcq:annotations:tutorial:1.0.0">

	<!-- The example use-case has multiple annotators and filters which depend on each other, start a sequential flow to execute them in order. -->
	<sequential>
		<!-- The RegexAnnotator finds all occurrences of a number followed by the supported units of measure. -->
		<regexAnnotator annotationTypeId="tutorial:unit-of-measure" ignoreCase="true"  pattern="\b(?&lt;value&gt;[-+]?[0-9]*[\.\,]?[0-9]+(e[-+]?[0-9]+)?)\s+(?&lt;unit&gt;feet|meter|meters)\b" />

		<!-- A custom HttpApiAnnotator which uses the unit-of-measure annotations from the regex annotator. -->
		<httpApiAnnotator endpoint="http://localhost:6005/api/annotate" inputAnnotationTypeIds="tutorial:unit-of-measure" />

		<!-- Remove the temporary unit-of-measure annotations because only the unit-of-measure-convert annotations are relevant for the editor. -->
		<removeAnnotations annotationTypeIds="tutorial:unit-of-measure" />
	</sequential>

</analysis>

HTTP API annotator

The HTTP API annotator is a webserver which exposes a endpoint and takes the input text and input annotations from the Content Quality App. The HttpApiAnnotator processes the input and returns new annotations. This can be implemented in any language or based on any web server which can expose a HTTP POST endpoint. This tutorial provides an example in both .NET Core and Node.js/Express.

Alongside a text representation of the fragment, a HTTP API Annotator receives all annotations from previous annotators, including the internal language annotator. All annotations have a range includeing a startIndex and length based on the string representation. A HTTP API Annotator can use these annotations if needed. If a annotator do not depend on the annotation of an other annotator, you can also configure the analysis to execute them in Parallel compositor (which themselves can have nested compositors).

An annotator only needs to return new annotations. The Content Quality App automatically merges them with the input annotations. The Parallel compositor also merges the annotations from all parallel flows.

.NET Core

Run "<project>/httpapiannotator/" (Or create a new "" in Visual Studio) and replace the "Program.cs" with the content of the collapsed code below this example.

<project>/httpapiannotator/Program.cs

C#

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Net;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.DependencyInjection;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

namespace FontoXml.ContentQuality.HttpApiAnnotator.Sample
{
	/// <summary>
	/// Program
	/// </summary>
	public class Program
	{
		public static IWebHost BuildWebHost(string[] args) =>
			WebHost.CreateDefaultBuilder(args)
				.UseStartup<Startup>()
				.UseKestrel(options => { options.Listen(IPAddress.Any, 6005); })
				.Build();

		public static void Main(string[] args) => BuildWebHost(args).Run();
	}

	/// <summary>
	/// Startup
	/// </summary>
	public class Startup
	{
		public void Configure(IApplicationBuilder app, IHostingEnvironment env) => app.UseMvc();
		public void ConfigureServices(IServiceCollection services) => services.AddMvc();
	}

	/// <summary>
	/// Request
	/// </summary>
	[JsonObject]
	public sealed class RequestDto
	{
		[JsonProperty("annotations", Required = Required.Always)]
		public List<AnnotationDto> Annotations { get; set; }

		[JsonProperty("text", Required = Required.Always)]
		public string Text { get; set; }
	}

	/// <summary>
	/// Response
	/// </summary>
	[JsonObject]
	public sealed class ResponseDto
	{
		[JsonProperty("results", Required = Required.Always)]
		public List<AnnotationDto> Annotations { get; set; } = new List<AnnotationDto>();
	}

	/// <summary>
	/// Annotation
	/// </summary>
	public class AnnotationDto
	{
		[JsonProperty("metadata", Required = Required.Default)]
		public JObject Metadata { get; set; }

		[JsonProperty("range", Required = Required.Always)]
		public AnnotationRangeDto Range { get; set; }

		[JsonProperty("type", Required = Required.Always)]
		public AnnotationTypeDto Type { get; set; }

		[JsonObject]
		public class AnnotationTypeDto
		{
			[JsonProperty("name", Required = Required.Always)]
			public string Name { get; set; }

			[JsonProperty("namespace", Required = Required.Always)]
			public string Namespace { get; set; }
		}

		[JsonObject]
		public class AnnotationRangeDto
		{
			[JsonProperty("length", Required = Required.Always)]
			public int Length;

			[JsonProperty("startIndex", Required = Required.Always)]
			public int StartIndex;
		}
	}

	/// <summary>
	/// Controller
	/// </summary>
	[Route("api/annotate")]
	public class AnnotateController : Controller
	{
		[HttpPost]
		[Produces("application/json")]
		public ResponseDto Annotate([FromBody] RequestDto request)
		{
			// Select all language annotations.
			var languageAnnotations = request
				.Annotations
				.Where(a => a.Type.Name.Equals("language") && a.Type.Namespace.Equals("urn:fontoxml:fcq:annotations:language:1.0.0"))
				.ToList();

			// Select all unit-of-measure annotations.
			var unitsOfMeasure = request.Annotations
				.Where(a => a.Type.Name.Equals("unit-of-measure") && a.Type.Namespace.Equals("urn:fontoxml:fcq:annotations:tutorial:1.0.0"));

			var response = new ResponseDto();
			foreach (var unitOfMeasure in unitsOfMeasure)
			{
				// Select the language annotation for this unit-of-measure
				var language = languageAnnotations.Single(lang => IsAnnotationWithinLanguage(unitOfMeasure, lang));

				// Skip this unit-of-measure because we do not know the language of that part of the content.
				if (language == null)
					continue;

				if (unitOfMeasure.Metadata["captures"]?["value"] == null
				    || unitOfMeasure.Metadata["captures"]["value"].Count() != 1
				    || unitOfMeasure.Metadata["captures"]["value"][0]["value"] == null

				    || unitOfMeasure.Metadata["captures"]?["unit"] == null
				    || unitOfMeasure.Metadata["captures"]["unit"].Count() != 1
				    || unitOfMeasure.Metadata["captures"]["unit"][0]["value"] == null)
					continue;

				var languageTag = language.Metadata["tag"].Value<string>();
				var unit = unitOfMeasure.Metadata["captures"]["unit"][0]["value"].Value<string>();
				var value = double.Parse(unitOfMeasure.Metadata["captures"]["value"][0]["value"].Value<string>().Replace(',', '.'), new CultureInfo("en-US"));

				string replacement;
				const double metersToFeetFactor = 0.304800610;
				if (unit.Equals("feet") && languageTag.Equals("nl"))
				{
					var meters = value * metersToFeetFactor;
					replacement = $"{Math.Round(meters, 2).ToString(new CultureInfo(languageTag))} meter";
				}
				else if (unit.Equals("meter") && languageTag.Equals("en"))
				{
					var feet = value / metersToFeetFactor;
					replacement = $"{Math.Round(feet, 2).ToString(new CultureInfo(languageTag))} feet";
				}
				else
					continue;

				response.Annotations.Add(new AnnotationDto
				{
					Type = new AnnotationDto.AnnotationTypeDto
					{
						Name = "unit-of-measure-convert",
						Namespace = "urn:fontoxml:fcq:annotations:tutorial:1.0.0"
					},
					Range = unitOfMeasure.Range,
					Metadata = new JObject
					{
						{ "replacement", replacement }
					}
				});
			}

			return response;
		}

		/// <summary>
		/// Helper
		/// </summary>
		private static bool IsAnnotationWithinLanguage(AnnotationDto candidate, AnnotationDto language)
		{
			var languageStartIndex = language.Range.StartIndex;
			var languageEndIndex = languageStartIndex + language.Range.Length;

			var unitStartIndex = candidate.Range.StartIndex;
			var unitEndIndex = unitStartIndex + candidate.Range.Length;
			return languageStartIndex <= unitStartIndex && languageEndIndex >= unitEndIndex;
		}
	}
}

Node.js / Express

Run "npm init" and "npm install --save express body-parser full-icu" in the "<project>/httpapiannotator/" directory and place the example code files the same directory.

<project>/httpapiannotator/index.js

JavaScript

const bodyParser = require('body-parser');
const express = require('express');

const app = express();
app.use(bodyParser.json());

const getLanguageForAnnotation = (languageAnnotations, candidateAnnotation) => {
	return languageAnnotations.find(languageAnnotation => {
		const languageStartIndex = languageAnnotation.range.startIndex;
		const languageEndIndex = languageStartIndex + languageAnnotation.range.length;
		const unitStartIndex = candidateAnnotation.range.startIndex;
		const unitEndIndex = unitStartIndex + candidateAnnotation.range.length;
		return languageStartIndex <= unitStartIndex && languageEndIndex >= unitEndIndex;
	});
};

const METERS_TO_FEET_FACTOR = 0.304800610;

app.post('/api/annotate', (req, res) => {
	// Select all language annotations.
	const languageAnnotations = req.body.annotations.filter(annotation =>
		annotation.type.name === 'language' &&
		annotation.type.namespace === 'urn:fontoxml:fcq:annotations:language:1.0.0');

	// Select all unit-of-measure annotations.
	const unitsOfMeasureAnnotations = req.body.annotations.filter(annotation =>
		annotation.type.name === 'unit-of-measure' &&
		annotation.type.namespace === 'urn:fontoxml:fcq:annotations:tutorial:1.0.0');

	const unitsOfMeasureConvertAnnotations = [];
	for (const unitOfMeasureAnnotation of unitsOfMeasureAnnotations) {
		// Select the language annotation for this unit-of-measure
		const languageAnnotation = getLanguageForAnnotation(languageAnnotations, unitOfMeasureAnnotation);
		if (!languageAnnotation) {
			continue;
		}

		const metadata = unitOfMeasureAnnotation.metadata;
		if (!metadata.captures ||
			!metadata.captures.unit ||
			!metadata.captures.unit[0] ||
			metadata.captures.unit[0].value === undefined ||
			!metadata.captures.value ||
			!metadata.captures.value[0] ||
			metadata.captures.value[0].value === undefined) {
			continue;
		}

		const languageTag = languageAnnotation.metadata.tag;
		const unit = metadata.captures.unit[0].value.toLowerCase();
		const value = parseFloat(metadata.captures.value[0].value.replace(',', '.'));

		let replacement;
		if (unit === 'feet' && languageTag === 'nl') {
			replacement = `${(value * METERS_TO_FEET_FACTOR).toLocaleString('nl', { maximumFractionDigits: 2 })} meter`;
		}
		else if (unit.startsWith('meter') && languageTag === 'en') {
			replacement = `${(value / METERS_TO_FEET_FACTOR).toLocaleString('en', { maximumFractionDigits: 2 })} feet`;
		}
		else {
			continue;
		}

		unitsOfMeasureConvertAnnotations.push({
			type: {
				name: 'unit-of-measure-convert',
				namespace: 'urn:fontoxml:fcq:annotations:tutorial:1.0.0'
			},
			range: unitOfMeasureAnnotation.range,
			metadata: {
				replacement
			}
		});
	}

	res.json({
		results: unitsOfMeasureConvertAnnotations
	});
});

const port = process.env.PORT || 6005;
app.listen(port, () => {
	console.log(`Content Quality HttpApiAnnotator tutorial app listening on port ${port}!`);
});

FontoXML Editor configuration

This tutorial assumes you have a FontoXML Editor with the fontoxml-content-quality add-on installed, and use a XML schema which has paragraph (of similar) elements and uses the xml:lang attribute for languages.

Wire the configuration

  • Add the to be created fcq-tutorial package the "<project>/editor/config/fonto-manifest.json":

    JavaScript

    {
    	"dependencies": {
    		"fcq-tutorial": "packages/fcq-tutorial"
    	}
    }
  • Configure the dev-server proxy endpoint to support Content Quality in "<project>/editor/config.json":

    JavaScript

    {
    	"proxy": {
    		"content-quality": "http://localhost:6000"
    	}
    }
  • Optional: Place the Content Quality button in the Masthead, usually the in "ToolsMasthead.jsx" or any other appropriate "*Masthead.jsx" file:

    XML

    <FxOperationButton operationName="toggle-content-quality" />
  • Optional: When you use an FontoXML Editor which uses a schema which need custom Content Quality configuration, see the content-quality-configuration documentation to configure Content Quality to work with the schema.

Create the annotation type configuration package

Create a new package in the FontoXML Editor to configure the "unit-of-measure-convert" annotation type by creating a "fcq-tutorial" directory in "<project>/editor/packages/". Place the following code files in this package using the specified names and paths.

<project>/editor/packages/fcq-tutorial/fonto-manifest.json

JSON

{
	"dependencies": {
		"fontoxml-content-quality": "platform/fontoxml-content-quality"
	}
}

<project>/editor/packages/fcq-tutorial/src/install.js

JavaScript

import registerAnnotationTypes from './api/registerAnnotationTypes.jsx';

export default function install() {
    registerAnnotationTypes();
}

<project>/editor/packages/fcq-tutorial/src/api/registerAnnotationTypes.jsx

JSX

import React from 'react';

import { Flex, Inlay, Label } from 'fds/components';
import FxOperationButton from 'fontoxml-fx/src/FxOperationButton.jsx';

import t from 'fontoxml-localization/src/t.js';

import ContentQualityDefaultSidebarListCardContent from 'fontoxml-content-quality/src/ContentQualityDefaultSidebarListCardContent.jsx';
import ContentQualityTextWithAnnotationStyle from 'fontoxml-content-quality/src/ContentQualityTextWithAnnotationStyle.jsx';
import contentQualityManager from 'fontoxml-content-quality/src/contentQualityManager.js';

const getContextMenuOperations = annotation => {
    if (!annotation || !annotation.metadata || !annotation.metadata.replacement) {
        return [];
    }

    const replacement = annotation.metadata.replacement;
    return [
        {
            label: replacement,
            operationName: 'content-quality-replace-text',
            operationData: {
                annotationId: annotation.id,
                text: replacement
            }
        }
    ];
};

const renderSidebarDetails = ({ annotation, next }) => {
    const operationData = {
        annotationId: annotation.id,
        text: annotation.metadata.replacement
    };

    return (
        <Flex flexDirection="column" justifyContent="flex-start" spaceSize="m">
            <Inlay isScrollContainer>
                <Label isBold>{t('Unit of measure does not match locale.')}</Label>
                <ContentQualityTextWithAnnotationStyle
                    text={t(
                        'Replace %style_start%{ANNOTATION_TEXT}%style_end% with {REPLACEMENT}.',
                        {
                            ANNOTATION_TEXT: annotation.text,
                            REPLACEMENT: annotation.metadata.replacement
                        }
                    )}
                    annotationType={annotation.type}
                />
            </Inlay>

            <Flex flex="0 0 auto" justifyContent="flex-end">
                <FxOperationButton
                    label={t('Replace')}
                    type="primary"
                    onClick={next}
                    operationName="content-quality-replace-text"
                    operationData={operationData}
                />
            </Flex>
        </Flex>
    );
};
const renderSidebarListCardContent = ({ annotation }) => (
    <ContentQualityDefaultSidebarListCardContent
        annotation={annotation}
        title={t('Unit of measure does not match locale')}
    />
);

export default function registerAnnotationTypes() {
    // Annotation: {urn:fontoxml:fcq:annotations:tutorial:1.0.0}unit-of-measure-convert
    contentQualityManager.registerAnnotationType(
        'urn:fontoxml:fcq:annotations:tutorial:1.0.0',
        'unit-of-measure-convert',
        {
            getContextMenuGroup: annotation => ({
                heading: t('Unit of measure'),
                operations: getContextMenuOperations(annotation)
            }),
            renderSidebarDetails,
            renderSidebarListCardContent,
            icon: 'globe',
            squiggleVariation: 'wavy-underline'
        }
    );
}

Example content

Place the following snippet to your default initial document, or use the DITA Concept example document below when using a DITA based FontoXML Editor. This can be done by placing it in "<project>/editor/dev-cms/files/tutorial.xml" and setting the scope documentIds to "["tutorial.xml"]" in "<project>/editor/config.json".

XML

<p xml:lang="en">
	This is an English sentence: Sculpture number 1 is 3 meter high.
</p>
<p xml:lang="nl">
	Dit is een Nederlandse zin: Beeldwerk nummer 2 is 5 feet hoog.
</p>

DITA Concept example document

XML

<?fontoxml-author author-id="anonymous" name="Anonymous"?>
<concept xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" id="concept_bbabc91f-f7dd-4c3d-afca-209fef688985" xsi:noNamespaceSchemaLocation="urn:fontoxml:names:tc:dita:xsd:concept.xsd:1.3" xml:lang="en">
	<title id="title_58ee97c6-1c84-46d9-80b1-1cb3aa03cefa">FontoXML Content Quality</title>
	<conbody id="conbody_3f940376-0f85-4f71-dca0-6b9dd0c7ebf3">
		<section id="id-982347e9-93fa-4458-84b0-c3abb741e053" xml:lang="en">
			<title id="id-30318727-0682-47e9-c99b-4b229247e925">English</title>
			<p id="id-0e5e14cd-b81e-4c12-d914-da0dd78df0d6">Sculpture number 1 is 3 meter high.</p>
		</section>
		<section id="section_173d33ee-ba87-400f-c0cc-52789dfcccb5" xml:lang="nl">
			<title id="title_cc09954a-6bf6-424b-dc22-8affb234a655">Nederlands</title>
			<p id="id-2f8b9e78-984b-4894-e8e3-3495626bff35">Beeldwerk nummer 2 is 5 feet hoog.</p>
		</section>
	</conbody>
</concept>

Running and testing the annotation type

In order to test the annotators in the FontoXML Editor we must start both backends and the dev-server. Start the following in parallel:

  1. In "<project>/backend" run: "fdt content-quality run"

    This step requires you to be logged into our Docker repository. See the requirements for more information.

  2. In "<project>/httpapiannotator"

    1. For .NET Core run: "dotnet run" (Or run it from within Visual Studio)

    2. For Node.js run: "node --icu-data-dir=node_modules/full-icu index.js"

  3. In "<project>/editor" run: "fdt editor run"

Open the FontoXML Editor in your browser at "http://localhost:8080".

  1. Load the example content.

  2. Open the "Quality check" toolbar and turn on Quality check.

  3. Wait for the content to be checked.

  4. There will be two annotations visible in the content (and sidebar), one for "3 meter" and one for "5 feet".

  5. Click the first one ("3 meter"), the sidebar will now show the details for this annotation.

  6. Click replace. The content will be changed using the suggested text replacement.

  7. There will only be one annotation left in the document; The "3 meter" text is replaced by "9.84 feet", which is valid for the English locale so there will be no new annotation created for it.

  8. Right click on the second one ("5 feet") and choose "1,52 meter". The content will be changed using the suggested text replacement.

  9. There will only no annotations left in the document, and the sidebar will state that all is done.