path: root/doc/context/sources/general/fonts/fonts/fonts-features.tex

% language=uk

\startcomponent fonts-features

\environment fonts-environment

% windows: seguiemj.ttf              (windows 10)
% public : emojionecolor-svginot.ttf (https://github.com/eosrei/emojione-color-font)

\startMPextensions
    vardef MyRectangle(expr n, w, h, x, y, c) =
        image (
            fill unitsquare xyscaled (w,h) shifted (x,y) withcolor c ;
            draw textext("\tttf " & decimal n) xsized (1/2) shifted (w/2,h/2) shifted (x,y) withcolor white ;
        )
    enddef ;
    vardef MyDot(expr x, y) =
        image (
            draw (x,y) withpen pencircle scaled (2/3) withcolor white ;
            draw (x,y) withpen pencircle scaled (1/2) withcolor black ;
        )
    enddef ;
\stopMPextensions

\startchapter[title=Features][color=darkmagenta]

\startsection[title=Introduction]

If you look into fonts, it is hard not to bump into kerns (spacing between
characters) and ligatures (combining multiple shapes into one) and apart from
monospaced fonts most \TYPEONE\ fonts have them. In the \OPENTYPE\ universe we
call these properties features and in such a font there can be many such
features.

For those who grew up with \TEX\ or still remember the times of eight bit fonts,
it is no secret that \TEX\ macro packages did some magic to get most out of a
font: replacing missing glyphs, fixing metrics, using commands to access shapes
that had a weird code point, to mention a few. As there is absolutely no
guarantee that an \OPENTYPE\ font does better, there is a good reason to continue
messing around with fonts. After all, it's what \TEX\ users seem to like:
control.

So, when we started writing support for \OPENTYPE\ quite soon a mechanism has
been created that permits adding our own features to the repertoire that comes
with a font. Because \OPENTYPE\ features demand a configuration and control
mechanism, it made sense to generalize that and provide a single interface.

This means that when we talk about font features, we don't limit ourselves to
those provided by the font, but also those provided by \CONTEXT. As with font
features, they are enabled per font.

Some of the extra features are sort of generic, others are very font specific and
their properties are somewhat bound to a font. Such features are defined in a
font goodie files. Consider these goodies a font extension mechanism.

Some features need information that only the engine can provide. This is why we
have analyzers. Some are generic, others are bound to scripts. They come in
action before features are applied. Rather special is applying features in
combination with paragraph building. This is something very specific to \CONTEXT\
but it depends on properties of the font. It falls into the category \quote
{optimizing}.

It is clear that when we talk of features many aspects of a font play a role. In
this chapter we will discuss all the mentioned aspects. There is quite a bit of
\LUA\ code shown in this chapter, but don't worry, users will seldom need to
tweak fonts this way. On the other hand it's good to see what is possible.

\stopsection

\startsection[title=Regulars]

\startsubsection[title=Introduction]

The \OPENTYPE\ specification, which can be found on the \MICROSOFT\ website
is no easy reading. Some of the concepts are easy to understand, like relative
positioning (that we call kerning in \TEX) or ligature substitution (as we
have ligatures in \TEX\ too). It makes no sense to discuss the bitwise composition
of an \OPENTYPE\ or \TRUETYPE\ file here. First of all, all we get to see is
a \LUA\ table, and in \CONTEXT\ even that one gets sanitized and optimized
into a more useable table. However, as the data that comes with a font is
a good indication of what a font is capable of, we will discuss some of it in
an appendix. In this section we will discuss the basic principles and categories
of features.

\stopsubsection

\startsubsection[title=Feature sets]

Because in the next examples we will demonstrate features, we need to know how
we can tell \CONTEXT\ what features to use. Although you can add explicit
feature definitions to a font specification, I strongly advice you not to do this
but use the more abstract mechanism of feature sets. These are defined as follows:

\starttyping
\definefontfeature
  [MyFeatureSet]
  [alpha=yes,
   beta=no,
   gamma=123]
\stoptyping

Such a set is bound to a font with the \type {*} specifier, as in:

\starttyping
\definefont
  [MyFontInstance]
  [MyNiceFont*MyFeatureSet at 12pt]
\stoptyping

In most cases the already defined \type {default} feature set will suffice. It often
makes sense to use that one as base for new definitions:

\starttyping
\definefontfeature
  [MyFeatureSet]
  [default]
  [alpha=yes,
   beta=no,
   gamma=123]
\stoptyping

The second argument can be a list, as in:

\starttyping
\definefontfeature
  [MyFeatureSet]
  [MyFirstSet,MySecondSet]
  [alpha=yes,
   beta=no,
   gamma=123]
\stoptyping

Of course you need to know what features a font support, and one way to find
out is:

\starttyping
mtxrun --script font --list --info --pattern=pagella
\stoptyping

Don't be too surprised if different fonts show different features and even similar
features can be implemented differently. Sometimes you really need to know the font,
but fortunately many fonts come with examples.

There are many features and there values are kept with the font when it gets
defined. This means that when you change a featureset, it will not affect already
defined fonts. Because fonts are often defined on demand, you need to be aware of
the fact that a redefinition of a featureset can have consequences for already
defined fonts. For instance, a bodyfont switch only sets up the fonts and delays
defining them.

Although features are a sort of abstractions it can be interesting to see what features
and values are actually used:

\starttyping
\usemodule[fonts-features] \showusedfeatures
\stoptyping

You will notice that we have more features than \OPENTYPE\ fonts can offer. This
is because in \CONTEXT\ features is a more general concept.

\showusedfeatures

\stopsubsection

\startsubsection[title=Main categories]

There are two (but potentially more) main groups of features: those that deal
with substitution, and those that lead to positioning. It is not really needed
to know the gory details, but it helps to know at least a bit about them as
it can help to track down issues with fonts.

There are several substitutions possible:

\startitemize[packed]
\startitem a single substitution replaces one glyph by another \stopitem
\startitem a multiple substitution replaces one glyph by one or more \stopitem
\startitem a ligature substitution replaces multiple glyphs by one glyph \stopitem
\startitem an alternate substitution replaces one glyph by one out of a set \stopitem
\stopitemize

Like it or not, but these categories are not always used as intended: they just
are a way of replacing one or more glyphs by one or more other glyphs. This means
that when for instance \type {ij} gets replaced by one glyph (given that the font
supports it) a ligature substitution is used, even when in fact we have to do
with a diftong that can be represented by one character.

No matter what features you will use, keep in mind that they are nothing more
than a combination of substitutions and positioning directives. So, the de facto
standard ligature building feature \type {liga} indeed uses a ligature
substitution, but other features with names that resemble no ligatures might use
this substitution as well.

An example of a single substitution is an oldstyle (\type {onum}) although it can
as well be implemented as a choice out of alternate glyphs. Another example is
smallcaps (\type {smcp}). Nowdays these are more or less standard features for a
grown up font, while in the past they came as separate fonts. So, instead of loading
an extra font, one sticks to one and selects the feature that does the
substitution.

A second category concerns relative positioning. Again we have several variants:

\startitemize[packed]
\startitem a single positioning moves a glyph over one of two axis and can change the width and|/|or height \stopitem
\startitem a pair positioning also moved glyphs but concerns two adjacent glyphs \stopitem
\startitem a cursive positioning operates on a range of glyphs and is used to visually connect them \stopitem
\stopitemize

In addition there are three ways to anchor marks onto glyphs:

\startitemize[packed]
\startitem a mark can be anchored on a base glyph \stopitem
\startitem a mark can be anchored on a specific (visual) component of a ligature \stopitem
\startitem a mark can be anchored on another mark \stopitem
\stopitemize

In base mode the single, alternate and ligature substitutions can rather easily
be mapped onto the traditional \TEX\ font handling mechanism and this is what
happens in base mode. A single substitution is just another instance of a glyph
so there we just replace the original index into the glyph table by another one.
In the case of an alternate we change the default index into one of several
possible replacements in the alternate set. Ligatures can be mapped onto \TEX s
ligature mechanism. The single positioning maps nicely on \TEX s kerning
mechanism and pairwise positioning is not applicable in base mode. In node mode
we don't do any remapping at loading time but delegate that to \LUA\ when
processing the node lists.

Marks are special in the sense that they normally only occur in scripts that also
use substitution and positioning which in turn means that some more housekeeping
is involved. After all, we need to keep track to what a mark applies. Of course a
font can provide regular latin accents as marks but that is ill practice because
cut and paste might not work out as expected. A proper font will support composed
characters and provide glyphs that have the accents built in. Marks are not dealt
with in base mode.

Talking of complex scripts, the above set of operations is far from enough. Take
for instance Arabic, where a sequence of 5~characters with 3~marks can easily
become two glyphs glued together with two marks only. In the process we can have
single substitutions, ligatures being built, marks being anchored and glyphs
being cursively positioned. But, in order to do this well, some contextual
analysis has to be done as well. Again we have several variants of this:

\startitemize[packed]
\startitem with contextual substitution a replacement takes place depending on a matching sequence of glyphs,
optionally preceded or followed by matches \stopitem
\startitem with contextual positioning shifting and anchoring happens based on a matching sequence of glyphs,
optionally preceded or followed by matches \stopitem
\startitem multiple contextual substitutions or positionings can be chained together \stopitem
\startitem this can also happen in the reverse order (for right|-|to|-|left scripts) \stopitem
\stopitemize

In practice there is no fundamental difference between these and we can collapse
them all in a sequence of lookups resulting in a sequence of whatever other
manipulation is wanted.

Given this, what is a feature? It's mostly a sequence of actions expressed in the
above. And although there is a whole repertoire of semi|-|standardized features
like \type {liga} and \type {onum}, there is no real hard coded support for them
in \CONTEXT. Instead we have a generic feature processor that deals with all of
them. A feature, say \type {abcd}, has a definition that boils down to a sequence
of lookups. A lookup is just a name that is associated to one of the mentioned
actions. So, \type {abcd} can do a decomposition (multiple substitution), then a
replacement (single substitution) based on neighbouring glyphs, then do some
ligature building (ligature substitution) and finally position the resulting
glyphs relative to each other (like cursive positioning and anchoring marks).

Imagine that we start out with 5 characters in the input. Instead of real glyphs
we represent them by rectangles. The third one is a mark.

\startlinecorrection
\startMPcode
  draw MyRectangle(1,2,6, 0,0,.5red    ) ;
  draw MyRectangle(2,2,4, 3,0,.5green  ) ; draw MyDot(4,4.25) ;
  draw MyRectangle(3,2,1, 6,5,.5blue   ) ; draw MyDot(7,4.75) ;
  draw MyRectangle(4,2,5, 9,0,.5yellow ) ;
  draw MyRectangle(5,2,5,12,0,.5magenta) ;
  currentpicture := currentpicture ysized(4cm) ;
\stopMPcode
\stoplinecorrection

In the next variant we see that four and five have been replaced by
number six. This is a ligature replacement.

\startlinecorrection
\startMPcode
  draw MyRectangle(1,2,6,0,0,.5red  ) ;
  draw MyRectangle(2,2,4,3,0,.5green) ; draw MyDot(4,4.25) ;
  draw MyRectangle(3,2,1,6,5,.5blue ) ; draw MyDot(7,4.75) ;
  draw MyRectangle(6,3,5,9,0,.5cyan ) ;
  currentpicture := currentpicture ysized(4cm) ;
\stopMPcode
\stoplinecorrection

The mark is an independent entity. Sometimes it has a width, sometimes it hasn't.
In both cases we can position it. Here we move the shape left and down. There are
two ways to do this: simple pairwise kerning but better is to use anchors. Here
we have one anchor per shape but there can be many.

\startlinecorrection
\startMPcode
  draw MyRectangle(1,2,6,0,0  ,.5red  ) ;
  draw MyRectangle(2,2,4,3,0  ,.5green) ;
  draw MyRectangle(3,2,1,3,4.5,.5blue ) ; draw MyDot(4,4.25) ;
  draw MyRectangle(6,3,5,6,0  ,.5cyan ) ;
  currentpicture := currentpicture ysized(4cm) ;
\stopMPcode
\stoplinecorrection

Next we apply some kerning. Of course the anchored marks need to move as well.

\startlinecorrection
\startMPcode
  draw MyRectangle(1,2,6,0,  0  ,.5red  ) ;
  draw MyRectangle(2,2,4,2.5,0  ,.5green) ;
  draw MyRectangle(3,2,1,2.5,4.5,.5blue ) ; draw MyDot(3.5,4.25) ;
  draw MyRectangle(6,3,5,5,  0  ,.5cyan ) ;
  currentpicture := currentpicture ysized(4cm) ;
\stopMPcode
\stoplinecorrection

Alternatively we can connect the shapes in a cursive way. The name cursive is
somewhat misleading as it just boils down to shifting. The cursive indicates that
the shifts accumulate within a word.

\startlinecorrection
\startMPcode
  draw MyRectangle(1,2,6,0,0  ,.5red  ) ;
  draw MyRectangle(2,2,4,2,0.5,.5green) ;
  draw MyRectangle(3,2,1,2,5  ,.5blue ) ; draw MyDot(3,4.75) ;
  draw MyRectangle(6,3,5,4,1  ,.5cyan ) ;
  currentpicture := currentpicture ysized(4cm) ;
\stopMPcode
\stoplinecorrection

\stopsubsection

\startsubsection[title={Single substitution}]

Single substitutions are probably the most used ones. For instance, when you
ask for small caps, a lot of glyphs get replaced. When using oldstyle numerals
only digits get replaced but even then each glyph has to be checked. This can be
demonstrated with the Latin Modern fonts.

\startlinecorrection
\scale
  [height=1cm]
  {\strut
   {\definedfont[lmroman10-bold*default]\$123.45}%
   \quad
   {\definedfont[lmroman10-bold*oldstyle]\$123.45}}
\stoplinecorrection

As you can see here, Latin Modern has an oldstyle dollar sign. If you don't like
that one, you're in troubles as it comes with the rest of the oldstyles. The only
way out is to apply the oldstyle numerals to digits only which involves more tagging
than you might be willing to add. So, whenever you choose a substitution, be aware
that you have not that much control over what gets substituted: it's the font that
drives it. Here are some examples:

\starttyping
\definefontfeature[capsandold][smallcaps,oldstyle]

\showotfcomposition{dejavu-serif*capsandold          at 24pt}{}{It's 2013!}
\showotfcomposition{cambria*capsandold               at 24pt}{}{It's 2013!}
\showotfcomposition{lmroman10regular*capsandold      at 24pt}{}{It's 2013!}
\showotfcomposition{texgyrepagellaregular*capsandold at 24pt}{}{It's 2013!}
\stoptyping

\definefontfeature[capsandold][smallcaps,oldstyle]

\blank \showotfcomposition{dejavu-serif*capsandold          at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{It's 2013!}} \blank
\blank \showotfcomposition{cambria*capsandold               at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{It's 2013!}} \blank
\blank \showotfcomposition{lmroman10regular*capsandold      at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{It's 2013!}} \blank
\blank \showotfcomposition{texgyrepagellaregular*capsandold at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{It's 2013!}} \blank

\stopsubsection

\startsubsection[title={Multiple substitution}]

In a multiple substitution a sequence of characters (glyphs) gets replaced by
another sequence. In fact, you might wonder why one||to||one, multiple||to||one
and multiple||to||multiple are not all generalized into this variant. Efficiency
is probably the main reason. \footnote {Isn't it strange that complex mechanisms
are designed to save a few bytes while at the same time we produce ridiculous
large pictures with cameras.} For instance the many||to||one is often used for
ligatures (\type {liga}) and as a consequence \type {liga} is often misused also
for non||ligatures.

One usage of multiple replacements is to avoid and or undo other replacements. In
the next example we see a language dependent \type {fi} ligature. Take the dutch
\type {ij} and \type {ie} diftongs. Here we need to prevent the \type {i}
becoming combined with the \type {f} as it would look weird. Among the solutions
for this are: context dependent ligatures (which involves a lot of rules), or
multiple to multiple replacements (looking at the \type {fij} sequence).

\startbuffer[definitions]
\definefontfeature[default-fijn-en][default][language=eng,script=latn]
\definefontfeature[default-fijn-nl][default][language=nld,script=latn]
\stopbuffer

\getbuffer[definitions] \typebuffer[definitions]

\starttyping
\definedfont[lmroman10-regular*default-fijn-en]\en effe fijn fietsen
\definedfont[lmroman10-regular*default-fijn-nl]\nl effe fijn fietsen
\stoptyping

This gives:

\startlinecorrection[blank]
\scale [width=\textwidth] \bgroup
    \framed [offset=overlay,frame=off,foregroundcolor=maincolor,align=normal,strut=no] \bgroup
        \definedfont[lmroman10-regular*default-fijn-en]\en effe fijn fietsen\vskip-1ex
        \definedfont[lmroman10-regular*default-fijn-nl]\nl effe fijn fietsen\par
    \egroup
\egroup
\stoplinecorrection

Of course from this result one cannot see what (combination of) substitution(s)
was used, but it's a nice exercise to work out a solution.

Multiple substitutions are mostly used for scripts more complex than latin or
special fonts like Zapfino where advanced contextual analysis happens.

\stopsubsection

\startsubsection[title={Alternate substitution}]

Alternates are simple one||to||one substitutions. Popular examples are small
capitials and oldstyle numerals.

A nice application of alternates is the punk font. This font is a Knuth original.
As part of experimenting with the \METAPOST\ library in the early days of
\LUATEX\ and \MKIV, runtime randomization was implemented. However, that variant
used virtual fonts and was somewhat resource hungry. So, in a later stage Khaled
Hosny made an \OPENTYPE\ version using \METAPOST\ output. Randomization is
implemented through the \type {rand} feature.

In \MKIV\ the \type {rand} feature is not really special and behaves just like
any other (stylistic) alternate. The only difference is that for this feature a
value of \type {yes} equals \type {random}. This also means that any feature that
uses alternates use them randomly.

\startbuffer
\definefontfeature[punknova-first] [mode=node,kern=yes,rand=first]
\definefontfeature[punknova-2]     [mode=node,kern=yes,rand=2]
\definefontfeature[punknova-yes]   [mode=node,kern=yes,rand=yes]
\definefontfeature[punknova-random][mode=node,kern=yes,rand=random]
\stopbuffer

\typebuffer \getbuffer

We use this is:

\startbuffer[sample]
The original punk font is designed by Don Knuth: xxxxxxxxxxxx
\stopbuffer

\startbuffer
\definedfont[punknova-regular                 at 15pt] \getbuffer[sample]
\definedfont[punknova-regular*punknova-first  at 15pt] \getbuffer[sample]
\definedfont[punknova-regular*punknova-2      at 15pt] \getbuffer[sample]
\definedfont[punknova-regular*punknova-yes    at 15pt] \getbuffer[sample]
\definedfont[punknova-regular*punknova-random at 15pt] \getbuffer[sample]
\stopbuffer

\typebuffer[sample]

\typebuffer

In order to illustrate the variants we show a sequence of \type {x}'s. There are
upto ten different variants per characters.

\startlines[color=maincolor] \getbuffer \stoplines

There is one pitfall with random alternates: if each run leads to a different
outcome, we can end up in oscillation: different shapes give different paragraphs
and we can get more pages or cross references etc.\ that can end up differently
so this is why \CONTEXT\ always uses the same random seed (which gets reset when
you purge the auxiliary files.

\stopsubsection

\startsubsection[title={Ligature substitution}]

A ligature is traditionally a combination of several characters into one. Popular
ligatures are \quote {fi}, \quote {fl}, \quote {ffi} and , \quote {ffl}.
Occasionally we see \quote {\ae}, \quote {\oe} and some more. Often ligatures are
language dependant. For instance in languages like Dutch and German there can be
compound words where one part ends with an \type {f} and the next part starts with
an \type {f} and that looks bad or at least not intuitive. To some extent one
can wonder if this tradition of ligatures is a good one. It definitely made
sense ages ago, but I wouldn't be surprised if they are often added to fonts
because the encoding vectors have them. After all, nothing prevents to go ahead
and come up with way more ligatures.

There can be many ligature features in a font. Although we support arbitrary
features, that is: those not registered as being official one way or the other,
the following are known by description:

\startluacode
context.starttabulate { "|lTCT{maincolor}|l|" }
for k, v in table.sortedhash(fonts.handlers.otf.tables.features) do
    if string.find(v,"ligature") then
        context.NC()
        context(k)
        context.NC()
        context(v)
        context.NR()
    end
end
context.stoptabulate()
\stopluacode

The \type {default} feature set has type {liga} as wel as the \TEX\ specific \type {tlig}
that replaces successive hyphen signs into en- and emdashes. The \type {arabic} feature
set also has \type {rlig} enabled.

Now, there is one thing you should realize when we discuss specific features and
the underlaying mechanisms: there is no real relationship between the features's
name and the mechanisms used: any feature can use any underlying mechanism or
combination. This is why deep down we see that what is internally called ligature
gets used for any purpose where multiple||to||one replacements happen, and why the
\type {liga} feature can use single substitutions or alternates to swap in
another rendering so that the dot of the \type {i} stays free of the preceding
\type {f}. And for some fonts relative positioning can be used to achieve a
ligature effect.

The next examples demonstrate how the \type {liga} feature deals with \type {ffi}.
Possible solutions are: replace all three at once, replace the first two first and
in a next step, combine a ligature and following character, replace one or more
components by variants that have no interference with the dot of the~\quote{i}.

\starttyping
\showotfcomposition{dejavu-serif*default          at 48pt}{}{ffi}
\showotfcomposition{cambria*default               at 48pt}{}{ffi}
\showotfcomposition{lmroman10regular*default      at 48pt}{}{ffi}
\showotfcomposition{texgyrepagellaregular*default at 48pt}{}{ffi}
\stoptyping

\blank \showotfcomposition{dejavu-serif*default          at 48pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{ffi}} \blank
\blank \showotfcomposition{cambria*default               at 48pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{ffi}} \blank
\blank \showotfcomposition{lmroman10regular*default      at 48pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{ffi}} \blank
\blank \showotfcomposition{texgyrepagellaregular*default at 48pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{ffi}} \blank

\stopsubsection

\startsubsection[title={Single positioning}]

Single positioning is also known as kerning, moving characters closer together so
that we get a more uniformly spaced sequence of glyphs. It is a mistake to think
that kerning is always needed! There are fonts that have hardly any kerns or no
kerns at all and still look good.

\start
    \showfontkerns                                                                       \blank
    \definedfont[dejavu-serif*default          at  8pt]Dejavu Serif:        \input tufte (E.R. Tufte)\blank
    \definedfont[cambria*default               at  9pt]Cambria:             \input tufte (E.R. Tufte)\blank
    \definedfont[lmroman10regular*default      at 10pt]Latin Roman Regular: \input tufte (E.R. Tufte)\blank
    \definedfont[lucidabrightot*default        at  8pt]Lucida Bright:       \input tufte (E.R. Tufte)\blank
    \definedfont[texgyrepagellaregular*default at  9pt]Pagella Regular:     \input tufte (E.R. Tufte)\blank
\stop

The next couple of examples show the action for a few words:

\blank \showotfcomposition{dejavu-serif*default          at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{We thrive}} \blank
\blank \showotfcomposition{cambria*default               at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{We thrive}} \blank
\blank \showotfcomposition{lmroman10regular*default      at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{We thrive}} \blank
\blank \showotfcomposition{lucidabrightot*default        at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{We thrive}} \blank
\blank \showotfcomposition{texgyrepagellaregular*default at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{We thrive}} \blank

\stopsubsection

\startsubsection[title={Pairwise positioning}]

This variant of positioning involved the first, second or both glyphs of a glyph
pair. The specification can influence the horizontal and vertical positions we
well as the widths of the positioned glyphs.

\startnotabene
    We need an example here.
\stopnotabene

\stopsubsection

\startsubsection[title={Mark positioning}]

Marks are (often) small symbols that represent accents (in latin) or vowels (in
arabic) that get attached to base glyphs. In the input stream they come after the
character that they apply to. Many fonts come with precomposed latin characters
which means that an \type {à} in the input is mapped directly onto its
corresponding shape. When the input contains an \type {a} followed by a \type{̀ }
input normalization will normally turn this into an \type {à}. But, when this
doesn't happen, the font machinery has to make sure that the mark gets positioned
right onto the base character. In traditional \TYPEONE\ fonts that more or less
happened automatically by overlaying the shapes. In \OPENTYPE\ (single)
positioning is used to place the mark right.

\startnarrowtyping
\showotfcomposition{dejavu-serif*default          at 24pt}{}{à a\utfchar{"0300} à}
\showotfcomposition{cambria*default               at 24pt}{}{à a\utfchar{"0300} à}
\showotfcomposition{lmroman10regular*default      at 24pt}{}{à a\utfchar{"0300} à}
\showotfcomposition{lucidabrightot*default        at 24pt}{}{à a\utfchar{"0300} à}
\showotfcomposition{texgyrepagellaregular*default at 24pt}{}{à a\utfchar{"0300} à}
\stopnarrowtyping

Of course a font can contain logic that replaces a sequence of base and mark into
precomposed characters with the right \UNICODE\ entry.

\blank \showotfcomposition{dejavu-serif*default          at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{à a\utfchar{"0300} à}} \blank
\blank \showotfcomposition{cambria*default               at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{à a\utfchar{"0300} à}} \blank
\blank \showotfcomposition{lmroman10regular*default      at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{à a\utfchar{"0300} à}} \blank
\blank \showotfcomposition{lucidabrightot*default        at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{à a\utfchar{"0300} à}} \blank
\blank \showotfcomposition{texgyrepagellaregular*default at 24pt}{}{\disabletrackers[otf.analyzing]\color[maincolor]{à a\utfchar{"0300} à}} \blank

You can imagine that when marks are bound to characters that have become
ligatures the anchoring is more complex as the font machinery has to keep track
of onto which component the mark goes. For this purpose marks as well as base
characters and base ligatures have anchors and feature lookups can explicitly
refer to them.

\stopsubsection

\startsubsection[title={Contextual analysis}]

What actually happens when turning a list of characters into a list of glyphs can
range from real simple to pretty complex. For instance the \type {smcp} feature
only has to run over the list and relate characters to a smallcaps shape. A
slightly more complex feature might also demand some positioning. One step further
is the use of contextual analysis, i.e. looking at previous, current and following
characters (or glyphs). Because features can range from simple to complex the actual
processing is not per feature! A font comes with a sequence of so called lookups that
relate to a feature, script and language. Also, each feature can use one||to||one,
multiple||to||one and many||to||many replacements as well as relative positioning.

So, what actually happens is not that a feature is processed, but that all features
are dealt with at the same time, in the order that the font prescribes. Enabling a
specific feature means that a step is executed, while a disabled feature skips all
steps that are tagged as belonging to that feature. And, as each feature can use
contextual analysis, you can imagine that the effective sequence of actions can be
a complex mix.

A nice example of a contextual substitution is the centered period character in
catalan in \type {ebgaramond}:

\startbuffer
\definefontfeature
  [example]
  [default]
  [locl=yes,script=latn,language=cat]

\definedfont[file:ebgaramond12-regular.otf*default at 40pt]l·l\quad
\definedfont[file:ebgaramond12-regular.otf*example at 40pt]l·l
\stopbuffer

\typebuffer

We show the boundingbox of the glyphs. The centered period between two l's is
is replaced by a raised variant with no width.

\blank \start \showglyphs \maincolor \midaligned{\getbuffer} \stop \blank

It will be clear that in order to use such features you need to know what the font
provides. For some fonts you need to explicitly enable the latin script (while others
use default). Such a feature can be part of localized support but that is no rule.
In that respect \OPENTYPE\ features are a rather unpredictable mess. For instance,
nothing prevents such a feature to be a ligature, and in case you find that strange,
especially ligature features are often abused for any purpose.

\stopsubsection

\startsubsection[reference=ligatures:hyphenation,title=Ligatures and hyphenation]

In this section we will say a few words on how hyphenation interferes with
(especially) ligature building. For this you need to know that:

\starttyping
effe
\stoptyping

But when hyphenation is permitted between the two \type{s}'s we actually have
internally:

\starttyping
ef{-}{}{}fe
\stoptyping

The first snippet comes at the end of a line, the second at the beginning of a
the next line and the last snippet is used when no hyphenation is needed. Such
triplets need to be taken into account when we do replacements and positioning
and also when we do contextual lookups.

An \OPENTYPE\ font is just a container that collects the following:

\startitemize[packed]
\startitem
    graphic representations of characters and symbols
\stopitem
\startitem
    information about what characters the shapes represent
\stopitem
\startitem
    rules about converting (sequences of) characters into one or more
    representations
\stopitem
\startitem
    rules about positioning representations relative to each other
\stopitem
\stopitemize

Although the way this information is stored is standardized, the rules are not.
You can imagine that there would be some standard way to turn an \type {f} and
\type {i} into an \quote{fi} but we already saw that this is not the case. Here
are some possibilities:

\startitemize[packed]
\startitem
    The two characters get their own standard glyph, maybe with some kerning.
\stopitem
\startitem
    The two characters are combined into one shape.
\stopitem
\startitem
    The \type{f} gets a narrow representation and is kept close to the standard
    \type{i}.
\stopitem
\startitem
    A standard \type {f} is kerned with a dotless \type{i} (not to be confused
    with the \UNICODE\ character).
\stopitem
\startitem
    A special \type {f} is combined with a special \type {i}.
\stopitem
\stopitemize

% maybe mark lig components when separate chars so that we can do spacing

If the two characters are represented by their own shape, some contextual
analysis takes place. Again there are several approaches to this:

\startitemize[packed]
\startitem
    When an \type{f} is seen in the input, the next character is checked and one
    or both gets replaced.
\stopitem
\startitem
    When an \type{i} is seen in the input, the previous character is checked and
    the \type {i} gets replaced.
\stopitem
\startitem
    When an \type{f} several following characters are checked, for instance to
    see if we need to take \type {ij} into account.
\stopitem
\stopitemize

Traditionally the \type {f} followed by an \type{f}, \type{l} and \type{i} get a
treatment, but some fonts also combine the \type {f} with \type {k}, \type {j},
\type {b}, \type {t} and more.

The \MKIV\ font handler is rather generic in the sense that it support what the
font requires. However, a complication is that the scripts (languages) that use
these diverse methods also expect hyphenation within such a ligature. Script like
Arabic that are more demanding don't hyphenate so there interference with
hyphenation is not a problem.

Some ligatures are sensitive for languages. In languages that have compound words
it might be undesirable to have a ligature at a word boundary, or in the Dutch
word \type {fijn} we like to have a nice glyph (or combinations) for \type {ij}
but no \type {fi} ligature. In a similar way hyphenation patterns can have
rules and it will be no surprise that the hyphenation mechanism can compete with
the ligature building for the best solution. This gets complicated by the fact
that there is no real way to recognize in the font handler if we really are
dealing with ligature building. Not only is the \type {liga} feature (and deep
down the ligature gsub handling) not bound to ligatures (but simply a
many|-|to|-|one mapper), some of the mentioned pseudo ligature builders use simple
substitution and kerning and there is no way to recognize that as a ligature.

Although it is possible to come up with a solution that is acceptable for many cases,
there is no way to predict what kind of tricks font designers will use. A hyphenation
point can be seen as follows:

\starttabulate[||||]
\NC \type{effe}      \NC \type{ef-fe}       \NC \type{e{f-}{f}{ff}e}       \NC \NR
\NC \type{efficient} \NC \type{ef-fi-cient} \NC \type{e{ffi-}{}{ffi}cient} \NC \NR
\stoptabulate

In the second case the larger ligatures has replaced the previous one. We could
have kept the first one because there are ways to manage two|-|step bounding
ligatures but it's not worth the trouble (read: way more complex code and
increased runtime for the whole mechanism). Here the \type {{ff}} and \type
{{ffi}} can be individual shapes or just one shape.

The three components of a hyphenation point: the pre, post and replacement text
need to be looked at independently so that we get the proper kerning with the
preceding and following characters. Also, in more complex (chained) lookups we
need to compare each element with its surrounding. A fully expanded solution tree
is too time consuming so we take some shortcuts and limits the checks to the
level that it has no big impact on performance. The occasionally needed
backtracking and inspection of components is currently quite reasonable. We need
to trade quality with convenience: the result should look okay but processing
speed should also be as high as possible. There is no need to let other scripts
or regular fonts suffer too much from excessive script demands of fonts that
could have be done better.

The complication is that we not only need to check and replace but also need to
check the kerning with preceding and following characters. We also need to take
the hyphen into account (here one, but there can also be one after the break.

It is for this reason that in \MKIV\ we have a (we think) acceptable mix of
heuristics around hyphenation points that deal with single and multiple
substitution as well as kerning. It will never be 100\% pertect but we consider
it better to drop an occasional hyphenation in favor of proper font handling. In
practice \TEX\ is clever enough to break a paragraph in lines within these
restrictions.

In \CONTEXT\ we have the traditional \TEX\ hyphenator but also provide an
extensible \LUA\ reimplementation. That one might become the default in future
versions. In traditional \TEX\ there are several low level hyphenation
representations: simple hyphen only points, injected by the hyphenator,
explicitly injected by the user or originating from a hyphen character. Then
there is the generic (pre, post, replace) discretionary that can be explicitly
injected by the user (or a macro). In \MKIV\ all hyphenation points get
normalized to this generic discretionary. There is no need for old|-|time
optimizations and a consistent (expanded) representation is easier to deal with
in other extensions. However, because the font handler is supposed to also work
outside \CONTEXT\ we need to deal with traditional cases too. But \unknown\ the
results might differ a bit.

\stopsubsection

    \startsubsection[title=Color]

    % TODO: use emojionecolor-svginot-archived.ttf

    A recent new (and evolving) addition to \OPENTYPE\ is colored glyphs. One variant
    (by \MICROSOFT) uses overlays and this method is quite efficient.

    \startbuffer
    \definefontfeature[colored][colr=yes]
    \definefontsynonym[Emoji][file:seguiemj.ttf*default,colored]

    \definesymbol[bug][\getglyphdirect{Emoji}{\char"1F41B}]
    \definesymbol[ant][\getglyphdirect{Emoji}{\char"1F41C}]
    \definesymbol[bee][\getglyphdirect{Emoji}{\char"1F41D}]
    \stopbuffer

    \typebuffer \getbuffer

    Here we see a \symbol[bug], \symbol[ant] and \symbol[bee], and they come in
    color! Once \UNICODE\ started adding such symbols (and more get added) the
    distinction between characters and symbols get even fuzzier. Of course one
    can argue that we communicate in pictograms but even then, given that
    mankind lasts a while, the \UNICODE\ repertoire will explode.

    \startplacefigure[title={A few emojis from \type {seguiemj.ttf}}]
        \startcombination [3*1]
            {\scale[width=.3\textwidth]{\symbol[bug]}} {\type{U+1F41B}: bug}
            {\scale[width=.3\textwidth]{\symbol[ant]}} {\type{U+1F41C}: ant}
            {\scale[width=.3\textwidth]{\symbol[bee]}} {\type{U+1F41D}: bee}
        \stopcombination
    \stopplacefigure

    Here we use \type {seguiemj.ttf}, a font that comes with \MSWINDOWS. Colors are
    achieved by combining glyphs rendered in different colors. A variant that uses
    \SVG\ instead of overlays is \type {emojionecolor-svginot.ttf}:

    \startbuffer
    \definefontfeature[svg][svg=yes]
    \definefontsynonym[Emoji][file:emojionecolor-svginot.ttf*default,svg]
    \stopbuffer

    \typebuffer \getbuffer

    This time we get \symbol[bug], \symbol[ant] and \symbol[bee] and they look
    quite different. Both fonts also have ligatures and you can wonder what sense
    that makes. It makes it impossible to swap fonts and as there is no standard
    one never knows what to expect.

    \startplacefigure[title={A few emojis from \type {emojionecolor-svginot.ttf}}]
        \startcombination [3*1]
            {\scale[width=.3\textwidth]{\symbol[bug]}} {\type{U+1F41B}: bug}
            {\scale[width=.3\textwidth]{\symbol[ant]}} {\type{U+1F41C}: ant}
            {\scale[width=.3\textwidth]{\symbol[bee]}} {\type{U+1F41D}: bee}
        \stopcombination
    \stopplacefigure

    \definefont[emoji][file:emojionecolor-svginot.ttf*default,svg]

    \def\FourFaces{\char128104\zwj\char128105\zwj\char128102\zwj\char128102\relax}

    \def\Man  {\char"1F468\relax}
    \def\Woman{\char"1F469\relax}
    \def\Boy  {\char"1F466\relax}
    \def\Girl {\char"1F467\relax}

    How do we know what faces add up to the ligature {\emoji\Man \zwj \Woman \zwj
    \Girl \zwj \Boy} and how are we supposed to know that there should {\darkgray
    \type {zwj}} in between? When we input four faces separated by zero width
    joiners, we get a four face symbol instead. The reason for having the joiners in
    between is probably to avoid unexpected ligatures. The sequence \type {man},
    \type {woman}, \type {boy}, \type {boy} gives \type {family}:
    %
    {\emoji\Man}   + {\darkgray \type {zwj}}
    {\emoji\Woman} + {\darkgray \type {zwj}}
    {\emoji\Boy}   + {\darkgray \type {zwj}}
    {\emoji\Boy}   = {\emoji\Man \zwj \Woman \zwj \Boy \zwj \Boy},
    %
    but two girls also work:
    %
    {\emoji\Man}   + {\darkgray \type {zwj}}
    {\emoji\Woman} + {\darkgray \type {zwj}}
    {\emoji\Girl}  + {\darkgray \type {zwj}}
    {\emoji\Girl}  = {\emoji\Man \zwj \Woman \zwj \Girl \zwj \Girl},
    %
    so does a mixture of kids:
    %
    {\emoji\Man}   + {\darkgray \type {zwj}}
    {\emoji\Woman} + {\darkgray \type {zwj}}
    {\emoji\Girl}  + {\darkgray \type {zwj}}
    {\emoji\Boy}   = {\emoji\Man \zwj \Woman \zwj \Girl \zwj \Boy},
    %
    although (at least currently):
    %
    {\emoji\Man}   + {\darkgray \type {zwj}}
    {\emoji\Woman} + {\darkgray \type {zwj}}
    {\emoji\Boy}   + {\darkgray \type {zwj}}
    {\emoji\Girl}  = {\emoji\Man \zwj \Woman \zwj \Boy \zwj \Girl},
    %
    gives twin boys. Of course the real family emoj is {\emoji\char"1F46A}.

    In our times for sure many combinations are possible, so:
    %
    {\emoji\Man}  + {\darkgray \type {zwj}}
    {\emoji\Man}  + {\darkgray \type {zwj}}
    {\emoji\Girl} + {\darkgray \type {zwj}}
    {\emoji\Girl} = {\emoji\Man \zwj \Man \zwj \Girl \zwj \Girl},
    %
    indeed gives a family, but I wonder at what point cultural bias will creep into
    font design. One can even wonder how clothing and haircut will demand frequent
    font updates: {\emoji\char"1F46B}, {\emoji\char"1F46C}, {\emoji\char"1F46D}.

    In the math alphabets we have a couple of annoying holes because some characters
    were already present in \UNICODE. The bad thing here is that we now always have
    to deal with these exceptions. But not so with emojis because here eventually all
    variants will show up. Where a character \type {A} in red or blue uses the same
    code point, a white telephone {\emoji\char"1F57E} and black telephone
    {\emoji\char"1F57F} have their own. And because obsolete scripts are already
    supported in \UNICODE\ and more get added, we can expect old artifacts also
    showing up at some time. Soon the joystick {\emoji\char"1F579} will be an unknown
    item to most of us, while the \MICROSOFT\ hololens migth get its slot.

    \startplacefigure[title={Will all animals come in stages of development?}]
        \startcombination [3*1]
            {\scale[width=.3\textwidth]{\emoji\char"1F423}} {\type{U+1F423}: hatching chick}
            {\scale[width=.3\textwidth]{\emoji\char"1F424}} {\type{U+1F424}: baby chick}
            {\scale[width=.3\textwidth]{\emoji\char"1F425}} {\type{U+1F425}: front-facing baby chick}
        \stopcombination
    \stopplacefigure

    For sure these mechanisms will evolve and to what extent we support them depends
    on what users want. At least we have the basics implemented.

    \stopsubsection

    \stopsection

\startsection[title=Extras]

\startnotabene
    Todo.
\stopnotabene

\stopsection

\startsection[reference=goodies,title=Goodies]

Goodies range from simple to complex. They share that they are defined in files
and loaded at runtime. There is a good change that when you read this, that there
are already more goodies than mentioned here. Here we will just mention a couple
of goodies. More details can be found in the files that ship with \CONTEXT\ and
have suffix \type {lfg}.

A goodie file is a regular \LUA\ file and is supposed to return a table. This
table collects data that is used for implementing the goodie or relates to a
regular feature. It can also provide information that is used for patching a
font. An example of a simple goodie file is the ones that accompanies the first
release of the \OPENTYPE\ Lucida fonts.

\starttyping
return {
  name = "lucida-opentype-math",
  version = "1.00",
  comment = "Goodies that complement lucida opentype.",
  author = "Hans Hagen",
  copyright = "ConTeXt development team",
  mathematics = {
    alternates = {
      italic = {
        feature = 'ss01',
        value = 1,
        comment = "Mathematical Alternative Italic"
      },
    }
  }
}
\stoptyping

This goodie file is only providing information about the meaning of a stylistic
alternate. These have abstract tags like \type {ss01} and in this case this
category collects alternative italic (calligraphic) shapes. Because math does
not follow the same rules as text, this feature is enabled explicitly.

In the goodie file of Xits math the alternates table has more entries:

\startnarrowtyping
alternates = {
  cal       = { ... comment = "Mathematical Calligraphic Alphabet" },
  greekssup = { ... comment = "Mathematical Greek Sans Serif Alphabet" },
  greekssit = { ... comment = "Mathematical Italic Sans Serif Digits" },
  monobfnum = { ... comment = "Mathematical Bold Monospace Digits" },
  mathbbbf  = { ... comment = "Mathematical Bold Double-Struck Alphabet" },
  mathbbit  = { ... comment = "Mathematical Italic Double-Struck Alphabet" },
  mathbbbi  = { ... comment = "Mathematical Bold Italic Double-Struck Alphabet" },
  upint     = { ... comment = "Upright Integrals" },
  vertnot   = { ... comment = "Negated Symbols With Vertical Stroke" },
}
\stopnarrowtyping

An alternate is triggered at the \TEX\ end with:

\starttyping
$ABC$ $\cal ABC$ $\mathalternate{cal}\cal ABC$
\stoptyping

This is an example of a dynamic feature that gets applied when enabled at a
specific location in the input. The \type {cal} is only recognized when it
is defined in a goodies file, where the value is defined (in all of the above cases
the value is~\type {1}).

The Xits math fonts has a goodie files that starts with:

\starttyping
return {
  name = "xits-math",
  version = "1.00",
  comment = "Goodies that complement xits (by Khaled Hosny).",
  author = "Hans Hagen",
  copyright = "ConTeXt development team",
  mathematics = {
    italics = {
      ["xits-math"] = italics,
    },
    alternates = {
\stoptyping

Here the \type {italics} variable is a table defined before the \type {return}
that looks as follows:

\starttyping
local italics = {
  defaultfactor = 0.025,
  disableengine = true,
  corrections   = {
   -- [0x1D44E] = 0.99,    -- a (fraction of quad)
   -- [0x1D44F] = 100,     -- b (font points)
      [0x1D453] = -0.0375, -- f
  }
}
\stoptyping

This rather specific table tells \CONTEXT\ that (when enabled) it has to apply
italic correction. It disables support built into the \TEX\ engine (which in the
case of \LUATEX\ is close to absent anyway). It will apply a default italic
correction of \type {0.025} but for some shapes a different value is used. Again
we have some commands at the \TEX\ end:

\starttyping
\setupmathematics[italics=1] % apply italics with we have an italic font
\setupmathematics[italics=2] % apply italics anyway
\setupmathematics[italics=3] % apply italics only when italic or bolditalic shapes
\setupmathematics[italics=4] % combination of 1 and 3
\stoptyping

An alternative is this:

\starttyping
\definefontfeature[mathextra][mathextra][collapseitalics=yes]
\stoptyping

This extends the \type {mathextra} feature to move the italic correction into
the character's width. Often this works out ok.

Because (definitely at the start of the \LUATEX\ project) we had no
proper \OPENTYPE\ math fonts, but at the same time wanted to move on
to \OPENTYPE\ and \UNICODE\ math and no longer struggle with all
those math families and definitions. The way out of this problem
is to define a virtual math font. The code for doing this is built
into the \MKIV\ core but is controlled by a goodie definition. Take
for instance Antykwa Math:

\startnarrowtyping
return {
  name = "antykwa-math",
  version = "1.00",
  comment = "Goodies that complement antykwa math.",
  author = "Hans, Mojca, Aditya",
  copyright = "ConTeXt development team",
  mathematics = {
    mapfiles = {
      "antt-rm.map",
      "antt-mi.map",
      "antt-sy.map",
      "antt-ex.map",
      "mkiv-base.map",
    },
    virtuals = {
      ["antykwa-math"] = {
        { name = "file:AntykwaTorunska-Regular", features = "virtualmath", main = true },
        { name = "mi-anttri.tfm", vector = "tex-mi", skewchar=0x7F },
        { name = "mi-anttri.tfm", vector = "tex-it", skewchar=0x7F },
        { name = "sy-anttrz.tfm", vector = "tex-sy", skewchar=0x30, parameters = true } ,
        { name = "ex-anttr.tfm", vector = "tex-ex", extension = true } ,
        { name = "msam10.tfm", vector = "tex-ma" },
        { name = "msbm10.tfm", vector = "tex-mb" },
      },
\stopnarrowtyping

Normally users will not define such tables but the keys give an indication of
what is involved. The same is true for the previously shown tables: they are just
provided in the \CONTEXT\ distribution.

Text fonts also can have goodies. We start with a rather dumb one and there
will be not that many of those. This one is needed to turn a \TYPEONE\ font
with a rather special encoding into a \UNICODE\ font. The next mapping is
possible because the dingbats are part of \UNICODE.

\starttyping
return {
  name = "dingbats",
  version = "1.00",
  comment = "Goodies that complement dingbats (funny names).",
  author = "Hans Hagen",
  copyright = "ConTeXt development team",
  remapping = {
    tounicode = true,
    unicodes = {
      a1   = 0x2701,
      a10  = 0x2721,
      a100 = 0x275E,
      a101 = 0x2761,
      a102 = 0x2762,
\stoptyping

Applying this encoding happens in two steps. Because goodies like this are just
features, we need to define a proper font feature set:

\starttyping
\definefontfeature
  [dingbats]
  [mode=base,
   goodies=dingbats,
   unicoding=yes]
\stoptyping

We have a base mode font, so no special processing takes place. The \type {goodies}
key is used to communicate the goodies file. The \type {unicoding} key is used
to apply the encoding. Of course this only works because the remapper code is present
in the core and is hooked in to the font initialization code. The \type {dingbats}
feature set is predefined, just as the font definition:

\starttyping
\definefontsynonym [ZapfDingbats] [file:uzdr] [features=dingbats]
\stoptyping

Here is a goodie file that I made a while ago:

\starttyping
return {
  name = "oxoniensis",
  version = "1.00",
  comment = "Oxoniensis test file for Thomas Schmitz.",
  author = "Hans Hagen",
  copyright = "ConTeXt development team",
  features = {
    lunatesigma = {
      type = "substitution",
      data = {
        sigma  = 0x03F2,
        sigma1 = 0x03F2,
        Sigma  = 0x03F9,
        phi    = phi1,
      },
    }
  },
}
\stoptyping

There is not that much to say about this, apart from that it's a sort of fake
feature that gets enabled as regular one:

\starttyping
\definefontfeature[test]
  [mode=node,
   kern=yes,
   lunatesigma=yes,
   goodies=oxoniensis]

\definefont[somefont][file:oxoniensis*test]
\stoptyping

A complete different kind of goodie is the following. At one of the \CONTEXT\ meetings
Mojca Miklavec discussed the possibility to have an additional mechanism for
defining combinations of fonts. Often fonts come in a set of four (regular, italic,
bold and bold italic). In \MKII\ the complexity of typescripts depends on the amount of
encodings that need to be supported but in \MKIV\ things are easier. For a set of four fonts
a typescript looks as follows:

\starttyping
\starttypescript [sans] [somesansfont] [name]
  \setups[font:fallback:sans]
  \definefontsynonym [Sans]           [file:somesans]  [features=default]
  \definefontsynonym [SansBold]       [file:somesansb] [features=default]
  \definefontsynonym [SansItalic]     [file:somesansi] [features=default]
  \definefontsynonym [SansBoldItalic] [file:somesansz] [features=default]
\stoptypescript
\stoptyping

We still have the abstract notion of a \type {Sans} font so that we can refer to
the regular shape without knowing the real name but the number of lines needed
is small. Such a definition can then be referred to using:

\starttyping
\starttypescript[somefontset]
  \definetypeface [somefontset] [rm] [serif] [someserif] [default]
  \definetypeface [somefontset] [ss] [sans]  [somesans]  [default]
  \definetypeface [somefontset] [tt] [mono]  [somemono]  [default]
  \definetypeface [somefontset] [mm] [math]  [somemath]  [default]
\stoptypescript
\stoptyping

So far things look simple. Given that many fonts follow a similar naming scheme
Wolfgang made a module that avoids such definitions altogether. However, being
involved in the development of the Antykwa fonts, Mojca ran into the situation
that not just four fonts were part of the set but many more. There are several
weight (think of light and heavy variants) as well as condensed variants and of
course the whole set is not per se a multiple of four.

In the meantime, in addition to the \type {file:} and \type {name:} accessors,
\CONTEXT\ had an additional one tagged \type {spec:} where a string made out of
weight, style, width etc.\ is turned into a (best guessed) font name. Therefore
the most natural way to deal with the many|-|fonts|-|in|-|a|-|set dilemma was to
provide an additional interface between this specification and the font set and
the most robust method was to define all in a goodie file.

In this case the goodies are loaded independent of the font, that is: not
as a feature. For instance:

\starttyping
\loadfontgoodies[antykwapoltawskiego]
\stoptyping

This file maps obscure fontnames onto the \type {spec:} interface so that
we can access them in a robust way.

\starttyping
\definefont
  [MyFontA]
  [file:Iwona-Regular*smallcaps]
\definefont
  [MyFontB]
  [file:AntykwaTorunska-Regular*smallcaps]
\definefont
  [MyFontC]
  [file:antpoltltcond-regular*smallcaps]
\definefont
  [MyFontD]
  [spec:antykwapoltawskiego-bold-italic-condensed-normal*smallcaps]
\definefont
  [MyFontE]
  [spec:antykwapoltawskiego-bold-italic-normal]
\stoptyping

The goodies file looks as follows:

\starttyping
return {
  name = "antykwa-poltawskiego",
  version = "1.00",
  comment = "Goodies that complement Antykwa Poltawskiego",
  author = "Hans & Mojca",
  copyright = "ConTeXt development team",
  files = {
    name = "antykwapoltawskiego", -- shared
    list = {
      ["AntPoltLtCond-Regular.otf"] = {
        weight = "light",
        style  = "regular",
        width  = "condensed",
      },
      ...
      ["AntPoltExpd-BoldItalic.otf"] = {
        weight = "bold",
        style  = "italic",
        width  = "expanded",
      },
    },
  },
  typefaces = {
    ["antykwapoltawskiego-light"] = {
      shortcut     = "rm",
      shape        = "serif",
      fontname     = "antykwapoltawskiego",
      normalweight = "light",
      boldweight   = "medium",
      width        = "normal",
      size         = "default",
      features     = "default",
    },
    ...
  },
}
\stoptyping

In addition to the files|-|to|-|specification mapping, there is
also a typeface specification table. This permits the definition
of a typeface in the following way:

\starttyping
\definetypeface
  [name=mojcasfavourite,
   preset=antykwapoltawskiego,
   normalweight=light,
   boldweight=bold,
   width=expanded]

\setupbodyfont
  [mojcasfavourite]
\stoptyping

When resolving the definition, the best possible match will be taken from the
typeface table in the goodie file. Of course this is not something that we expect
the average user to deliver and deal with.

As the Antykwa font is somewhat atypical and not used in everyday typesetting,
you might wonder if all this overhead makes sense. However, there are type
foundries that do ship their fonts in many weights and for those using a \LUA\
goodie file instead of many typescripts in \TEX\ coding makes sense. Take for
instance TheMix:

\starttyping
\loadfontgoodies
  [themix]

\definetypeface
  [name=themix,
   preset=themix-light]

\definetypeface
  [name=themix,
   preset=themixmono-light]

\setupbodyfont
  [themix]
\stoptyping

In this case the goodie file can serve as a template for more such fonts.
In order to be efficient this goodie file uses a couple of local
tables (we could have used metatables instead).

\starttyping
local themix = {
  name     = "themix",
  shortcut = "ss",
  shape    = "sans",
  fontname = "themix",
  width    = "normal",
  size     = "default",
  features = "default",
}

local themixmono = {
  name     = "themixmono",
  shortcut = "tt",
  shape    = "mono",
  fontname = "themixmono",
  width    = "normal",
  size     = "default",
  features = "default",
}
\stoptyping

The main goodie table defines a lot of weights:

\startnarrowtyping
return {
  name = "themix",
  version = "1.00",
  comment = "Goodies that complement TheMix (by and for sale at www.lucasfonts.com).",
  author = "Hans Hagen",
  copyright = "ConTeXt development team",
  files = {
    list = {
      ["TheMixOsF-ExtraLight"] = {
        name   = "themix",
        weight = "extralight",
        style  = "regular",
        width  = "normal"
      },
      ["TheMixOsF-ExtraLightItalic"] = {
        ...
      },
      ...
      ["TheMixOsF-Black"] = {
        ...
      },
      ["TheMixOsF-BlackItalic"] = {
        ...
      },
      ...
      --
      ["TheMixMono-W2ExtraLight"] = {
        name   = "themixmono",
        weight = "extralight",
        style  = "regular",
        width  = "normal"
      },
      ...
      ["TheMixMono-W9BlackItalic"] = {
        ...
      },
    },
  },
  typefaces = {
    ["themix-extralight"] = table.merged(themix, {
      normalweight = "extralight",
      boldweight   = "semilight"
    }),
    ["themix-light"] = table.merged(themix, {
      normalweight = "light",
      boldweight   = "normal"
    }),
    ...
    ["themixmono-bold"] = table.merged(themixmono, {
      normalweight = "bold",
      boldweight   = "black"
    }),
  },
}
\stopnarrowtyping

It's now time for some generic goodies. In the \CONTEXT\ distribution there
is a goodie file that (at the time of this writing) looks as follows:

\starttyping
local default = {
  analyze  = "yes",
  mode     = "node",
  language = "dflt",
  script   = "dflt",
}

local smallcaps = {
  smcp = "yes",
}

local function statistics(tfmdata)
  commands.showfontparameters(tfmdata)
end

local function squeeze(tfmdata)
  for k, v in next, tfmdata.characters do
    v.height = 0.75 * (v.height or 0)
    v.depth  = 0.75 * (v.depth  or 0)
  end
end

return {
  name = "demo",
  version = "1.01",
  comment = "An example of goodies.",
  author = "Hans Hagen",
  featuresets = {
    default = {
      default,
    },
    smallcaps = {
      default, smallcaps,
    },
  },
  colorschemes = {
    default = {
      [1] = {
        "one", "three", "five", "seven", "nine",
      },
      [2] = {
        "two", "four", "six", "eight", "ten",
      },
    },
    all = {
      [1] = {
        "*",
      },
    },
    some = {
      [1] = {
        "0x0030:0x0035",
      },
    },
  },
  postprocessors = {
    statistics = statistics,
    squeeze    = squeeze,
  },
}
\stoptyping

This demo file implements several goodies: featuresets, colors and
postprocessors. Keep in mind that a goodie file can provide whatever information
it wants but of course only known subtables will be dealt with.

The coloring of glyphs can happen by name, which assumes that glyph names are
used, or by number. Here we use generic glyph names, but for a specific font one
might need to provide a special goodie file. For instance, the color section of
the goodie file for the husayni font has entries like:

\startnarrowtyping
[3] = {
  "Ttaa.waqf", "SsLY.waqf", "QLY.waqf", "Miim.waqf", "LA.waqf", "Jiim.waqf",
  "Threedotsabove.waqf", "Siin.waqf", "Ssaad.waqf", "Qaaf.waqf", "SsL.waqf",
  "QF.waqf", "SKTH.waqf", "WQFH.waqf", "Kaaf.waqf", "Ayn.ruku", "Miim.nuun_high",
  "Siin.Ssaad", "Nuunsmall", "emptydot_low", "emptydot_high", "Sifr.fill",
  "Miim.nuun_low", "Nuun.tanwiin",
},
\stopnarrowtyping

Of course such a definition can only be made when the internals of the font are
known and in this case it concerns a pretty complex font.

\startbuffer
\definefontfeature
  [demo-colored]
  [goodies=demo,
   colorscheme=default,
   featureset=default]

\definefontfeature
  [demo-colored-all]
  [goodies=demo,
   colorscheme=all,
   featureset=default]

\definefontfeature
  [demo-colored-some]
  [goodies=demo,
   colorscheme=some,
   featureset=default]

\definefont[DemoFontA][MonoBold*demo-colored      at 10pt]
\definefont[DemoFontB][MonoBold*demo-colored-all  at 10pt]
\definefont[DemoFontC][MonoBold*demo-colored-some at 10pt]
\stopbuffer

\typebuffer \getbuffer

% \definecolor[colorscheme:1:1][s=.75]
% \definecolor[colorscheme:1:2][r=.75]
% \definecolor[colorscheme:1:3][g=.75]
% \definecolor[colorscheme:1:4][b=.75]
% \definecolor[colorscheme:1:5][c=.75]
% \definecolor[colorscheme:1:6][m=.75]
% \definecolor[colorscheme:1:7][y=.75]

% \definecolor[colorscheme:2:7][s=.75]
% \definecolor[colorscheme:2:6][r=.75]
% \definecolor[colorscheme:2:5][g=.75]
% \definecolor[colorscheme:2:4][b=.75]
% \definecolor[colorscheme:2:3][c=.75]
% \definecolor[colorscheme:2:2][m=.75]
% \definecolor[colorscheme:2:1][y=.75]

\startbuffer
\starttabulate[||||]
\NC
    \DemoFontA \resetfontcolorscheme   test 1234567890 \NC
    \DemoFontA \setfontcolorscheme  [1]test 1234567890 \NC
    \DemoFontA \setfontcolorscheme  [2]test 1234567890 \NC
\NR
\NC
    \DemoFontB \resetfontcolorscheme   test 1234567890 \NC
    \DemoFontB \setfontcolorscheme  [1]test 1234567890 \NC
    \DemoFontB \setfontcolorscheme  [2]test 1234567890 \NC
\NR
\NC
    \DemoFontC \resetfontcolorscheme   test 1234567890 \NC
    \DemoFontC \setfontcolorscheme  [1]test 1234567890 \NC
    \DemoFontC \setfontcolorscheme  [2]test 1234567890 \NC
\NR
\stoptabulate
\stopbuffer

\typebuffer \getbuffer

Here is an example that I made at the TUG 2013 conference in Japan,
after a presentation by Chof. The font (adapted by by Dohyun Kim) can
be downloaded from: \hyphenatedurl {http://ftp.ktug.org/KTUG/hcr-lvt/1.910_nomac/}.

\startbuffer[korean-demo]
\definefontfeature
  [korean-composed]
  [goodies=hanbatanglvt,
   colorscheme=default,
   mode=node,
   ljmo=yes,
   tjmo=yes,
   vjmo=yes,
   script=hang,
   language=kor]

\definefont
  [KoreanJMO]
  [hanbatanglvt*korean-composed]

\definecolor[colorscheme:100:1][r=.75]
\definecolor[colorscheme:100:2][g=.75]
\definecolor[colorscheme:100:3][b=.75]

\definecolor[colorscheme:101:1][g=.75,b=.75]
\definecolor[colorscheme:101:2][r=.75,b=.75]
\definecolor[colorscheme:101:3][r=.75,g=.75]
\stopbuffer

\typebuffer[korean-demo] \getbuffer[korean-demo]

\startbuffer
    % Hunminjeongeum: http://en.wikipedia.org/wiki/Hunminjeongeum
    나랏말ᄊᆞ미中듕國귁에달아문ᄍᆞᆼ와로서르ᄉᆞᄆᆞᆺ디아니ᄒᆞᆯᄊᆡ%
    사ᄅᆞᆷ마다ᄒᆡᅇᅧ수ᄫᅵ니겨나...% ᆯ로ᄡᅮ메便뼌安ᅙᅡᆫ킈ᄒᆞ고져ᄒᆞᇙᄯᆞᄅᆞ미니라
\stopbuffer

\startlinecorrection
\startcombination[1*3]
   {\framed{\startscript[hangul]\KoreanJMO                        \getbuffer\stopscript}} {no colorscheme}
   {\framed{\startscript[hangul]\KoreanJMO\setfontcolorscheme[100]\getbuffer\stopscript}} {colorscheme 100}
   {\framed{\startscript[hangul]\KoreanJMO\setfontcolorscheme[101]\getbuffer\stopscript}} {colorscheme 101}
\stopcombination
\stoplinecorrection

The goodie definition looks as follows (watch how we use ranges):

\starttyping
return {
    name = "hanbatanglvt",
    version = "1.00",
    comment = "Goodies that complement the hanbatanglvt fonts.",
    author = "Hans Hagen",
    colorschemes = {
        default = {
            { "0x01100:0x0115F" }, -- jamo_initial (r/c)
            { "0x01160:0x011A7" }, -- jamo_medial  (g/m)
            { "0x011A8:0x011FF" }, -- jamo_final   (b/y)
        }
    }
}
\stoptyping

This is much shorter (and efficent) that defining a whole vector, as in:

\starttyping
local f_uni_base = string.formatters["uni%04X"]
local f_uni_plus = string.formatters["uni%04X.y%s"]

local function range(first,last)
    local t = { }
    for i=first,last do
        t[#t+1] = f_uni_base(i)
        for j=0,19 do
            t[#t+1] = f_uni_plus(i,j)
        end
    end
    return t
end

return {
    name = "hanbatanglvt",
    version = "1.00",
    comment = "Goodies that complement the hanbatanglvt fonts.",
    author = "Hans Hagen",
    colorschemes = {
        default = {
            range(0x01100,0x0115F), -- jamo_initial (r/c)
            range(0x01160,0x011A7), -- jamo_medial  (g/m)
            range(0x011A8,0x011FF), -- jamo_final   (b/y)
        }
    }
}
\stoptyping

By using names we don't depend on \UNICODE\ which sometimes is needed when glyphs
have ended up in the private space. However, by default, after glyphs have been
mapped to colors, an extra pass will make sure that characters pushed into
private space will get the same mapping as their regular \UNICODE\ has gotten
(given that the number is known). Of course explicitly assigned colors will be
preserved.

Another generic demo feature is postprocessing. In principle one can
add additional postprocessors but for that the source code needs to
be consulted which in turn assumes some knowledge of fonts and \CONTEXT\
internals.

\startbuffer
\definefontfeature
  [justademoa]
  [default]
  [goodies=demo,
   postprocessor=squeeze]

\definefontfeature
  [justademob]
  [default]
  [goodies=demo,
   postprocessor=statistics]

\definefontfeature
  [justademoc]
  [default]
  [goodies=demo,
   postprocessor={statistics,squeeze}]
\stopbuffer

\typebuffer \getbuffer

The statistics just print some font parameters to the log so that one
is not showing up here. The squeeze looks as follows:

\startbuffer
\definefont[DemoFontD][Serif*default    at 30pt]
\definefont[DemoFontE][Serif*justademoa at 30pt]
\stopbuffer

\typebuffer \getbuffer

\startlinecorrection
\hbox\bgroup
    \ruledhbox{\color[maincolor]{DemoFontD height \& depth}}\quad
    \ruledhbox{\color[maincolor]{DemoFontE height \& depth}}
\egroup
\stoplinecorrection

The squeezer just makes the height and depth of glyphs a  bit smaller and it is
not that hard to imagine other manipulations. The demo goodie file is good
place to start playing with such things.

Because there is less standardization with respect to features than one might
suspect, goodie files provide a mean to define featuresets. We can use such a set
in another definition:

\starttyping
\definefontfeature
  [demo-smallcaps]
  [goodies=demo,
   featureset=smallcaps]
\stoptyping

Of course this only makes sense for more complex combinations. The already mentioned
husayni font comes with many features and most of these work together.

The basic goodie table looks as follows:

\startnarrowtyping
return {
  name         = "husayni",
  version      = "1.00",
  comment      = "Goodies that complement the Husayni font by Idris Samawi Hamid.",
  author       = "Idris Samawi Hamid and Hans Hagen",
  featuresets  = { },
  solutions    = { },
  stylistics   = { },
  colorschemes = { },
}
\stopnarrowtyping

We already saw the color schemes and now we will fill in the other tables. First
we define a couple of sets:

\startnarrowtyping
local basics = {
  analyze  = "yes",
  mode     = "node",
  language = "dflt",
  script   = "arab",
}

local analysis = {
  ccmp = "yes",
  init = "yes", medi = "yes", fina = "yes",
}

local regular = {
  rlig = "yes", calt = "yes", salt = "yes", anum = "yes",
  ss01 = "yes", ss03 = "yes", ss07 = "yes", ss10 = "yes", ss12 = "yes",
  ss15 = "yes", ss16 = "yes", ss19 = "yes", ss24 = "yes", ss25 = "yes",
  ss26 = "yes", ss27 = "yes", ss31 = "yes", ss34 = "yes", ss35 = "yes",
  ss36 = "yes", ss37 = "yes", ss38 = "yes", ss41 = "yes", ss42 = "yes",
  ss43 = "yes", js16 = "yes",
}

local positioning = {
  kern = "yes", curs = "yes", mark = "yes", mkmk = "yes",
}

local minimal_stretching = {
  js11 = "yes", js03 = "yes",
}

local medium_stretching = {
  js12="yes", js05="yes",
}
local maximal_stretching= {
  js13 = "yes", js05 = "yes", js09 = "yes",
}

local wide_all = {
  js11 = "yes", js12 = "yes", js13 = "yes", js05 = "yes", js09 = "yes",
}

local shrink = {
  flts = "yes", js17 = "yes", ss05 = "yes", ss11 = "yes", ss06 = "yes",
  ss09 = "yes",
}

local default = {
  basics, analysis, regular, positioning, -- xxxx = "yes", yyyy = 2,
}
\stopnarrowtyping

Next we define some featuresets and we use the default as starting point:

\startnarrowtyping
  featuresets = {
    default = {
      default,
    },
    minimal_stretching = {
      default, js11 = "yes", js03 = "yes",
    },
    medium_stretching = {
      default, js12="yes", js05="yes",
    },
    maximal_stretching= {
      default, js13 = "yes", js05 = "yes", js09 = "yes",
    },
    wide_all = {
      default, js11 = "yes", js12 = "yes", js13 = "yes", js05 = "yes",
      js09 = "yes",
    },
    shrink = {
      default, flts = "yes", js17 = "yes", ss05 = "yes", ss11 = "yes",
      ss06 = "yes", ss09 = "yes",
    },
  }
\stopnarrowtyping

When defining the font at the \TEX\ end we can now refer to for instance \type
{wide_all} which saves us some typing. However, it does not stop here. In a later
paragraph we will see how fonts can work in tandem with the parbuilder. For that
purpose the goodie table has a \type {solutions} subtable:

\startnarrowtyping
solutions = {
  experimental = {
    less = {
      "shrink"
    },
    more = {
      "minimal_stretching", "medium_stretching", "maximal_stretching", "wide_all"
    },
  },
}
\stopnarrowtyping

Here we define an experimental solution for optimizing the lines in a paragraph:
we can narrow words or we can widen them according to a specific featureset. In
order to reach the optimal solution the text will be retypeset under a different
feature regime.

{\em TODO: show how to apply.}

%D \starttyping
%D \setupfontsolutions[method={random,preroll},criterium=1,randomseed=101]
%D
%D \definefontsolution % actually only the last run needs to be done this way
%D   [FancyHusayni]
%D   [goodies=husayni,
%D    solution=experimental]
%D
%D \definedfont[husayni*husayni-default at 24pt]
%D \setupinterlinespace[line=36pt]
%D \righttoleft
%D \enabletrackers[parbuilders.solutions.splitters.colors]
%D \setfontsolution[FancyHusayni]
%D alb alb alb \par
%D \resetfontsolution
%D \disabletrackers[parbuilders.solutions.splitters.colors]
%D \stoptyping

Because there are a some 55 stylistic and 21 justification variants the
goodie file also provides a \type {stylistics} table and for tracing purposes
the {colorschemes} table is populated.

Yet another demonstration of manipulation is the following. Not all fonts come
with all combined glyphs. Although we have an auto|-|compose feature in \CONTEXT\
it sometimes helps to be specific with respect to some combinations. This is
where the \type {compositions} goodie kicks in:

\starttyping
local compose = {
  [0x1E02] = {
    anchored = "top",
  },
  [0x1E04] = {
    anchored = "bottom",
  },
  [0x0042] = { -- B
    anchors = {
      top = {
        x = 300,
        y = 700,
      },
      bottom = {
        x = 300,
        y = -30,
      },
    },
  },
  [0x0307] = {
    anchors = {
      top = {
        x = -250,
        y = 550,
      },
    },
  },
  [0x0323] = {
    anchors = {
      bottom = {
        x = -250,
        y = -80,
      },
    },
  },
}

return {
  name = "lm-compose-test",
  version = "1.00",
  comment = "Goodies that demonstrate composition.",
  author = "Hans and Mojca",
  copyright = "ConTeXt development team",
  compositions = {
    ["lmroman12-regular"] = compose,
  }
}
\stoptyping

Of course this assumes some knowledge of the font metrics (in base points) and
\UNICODE\ slots, but it might be worth the trouble. After all, one only needs to
figure it out once. But keep in mind that it will always be a kludge.

A slightly different way to define such compositions is the following:

\starttyping
local defaultunits = 193 - 30

local compose = {
                 DY = defaultunits,
 -- [0x010C] = { DY = defaultunits }, -- Ccaron
 -- [0x02C7] = { DY = defaultunits }, -- textcaron
}

-- fractions relative to delta(X_height - x_height)

local defaultfraction = 0.85

local compose = {
  DY = defaultfraction, -- uppercase compensation
}

return {
  name = "lucida-one",
  version = "1.00",
  comment = "Goodies that complement lucida.",
  author = "Hans and Mojca",
  copyright = "ConTeXt development team",
  compositions = {
    ["lbr"]  = compose,
    ["lbi"]  = compose,
    ["lbd"]  = compose,
    ["lbdi"] = compose,
  }
}
\stoptyping

Of course no one really needs this because \OPENTYPE\ Lucida fonts
have replaced the \TYPEONE\ versions.

The next goodie table is dedicated to the de facto standard \TEX\ font Latin
Modern. There is a bit of history behind this file. When we started writing
\CONTEXT\ there were not that many fonts available and so we ended up with a font
system that was rather well suited for the predecessor of Latin Modern, called
Computer Modern. And because these fonts came in design sizes the font system
was made such that it could cope efficiently with many files in a font set. Although
there is no additional overhead compared to small font sets, apart from more files,
there is some burden in defining them. And, as they are the default fonts, these
definitions slow down the initialization of \CONTEXT\ (which is due to the fact that
the large typescript definitions were loaded and parsed). So, at some point
the decision was made to kick out these definitions and move the burden of figuring
out the right size to \LUA. When Latin Modern is chosen as font the effect is the
same when design sizes are enabled. But, instead of many definitions (one for each
combination of size and style) we now have an option. A non|-|designsize typeface
is defined as follows:

\startnarrowtyping
\starttypescript [modern,modern-base]
  \definetypeface [\typescriptone] [rm] [serif] [modern] [default]
  \definetypeface [\typescriptone] [ss] [sans]  [modern] [default]
  \definetypeface [\typescriptone] [tt] [mono]  [modern] [default]
  \definetypeface [\typescriptone] [mm] [math]  [modern] [default]
  \quittypescriptscanning
\stoptypescript
\stopnarrowtyping

The designsize variant looks like this:

\startnarrowtyping
\starttypescript [modern-designsize]
  \definetypeface [\typescriptone]
    [rm] [serif] [latin-modern-designsize] [default] [designsize=auto]
  \definetypeface [\typescriptone]
    [ss] [sans]  [latin-modern-designsize] [default] [designsize=auto]
  \definetypeface [\typescriptone]
    [tt] [mono]  [latin-modern-designsize] [default] [designsize=auto]
  \definetypeface [\typescriptone]
    [mm] [math]  [latin-modern-designsize] [default] [designsize=auto]
  \quittypescriptscanning
\stoptypescript
\stopnarrowtyping

Of course there are accompanying typescripts that map the sans, serif, mono
and math styles onto files. The \type {designsize} magic uses the following
table. We show only part of the file, as it is in the \CONTEXT\ distribution.

\starttyping
return {
  name = "latin modern",
  version = "1.00",
  comment = "Goodies that complement latin modern.",
  author = "Hans Hagen",
  copyright = "ConTeXt development team",
  mathematics = {
    tweaks = {
      aftercopying = {
        mathematics.tweaks.fixbadprime, -- prime is too low
      },
    },
  },
  designsizes = {
    ["LMMathRoman-Regular"] = {
      ["4pt"]  = "LMMath5-Regular@lmroman5-math",
      ...
      ["12pt"] = "LMMath12-Regular@lmroman12-math",
      default  = "LMMath10-Regular@lmroman10-math"
    },
    ["LMMathRoman-Bold"] = { -- not yet ready
      ...
    },
    ["LMRoman-Regular"] = {
      ["4pt"]  = "file:lmroman5-regular",
      ...
      ["12pt"] = "file:lmroman12-regular",
      default  = "file:lmroman10-regular",
    },
    ["LMRoman-Bold"] = {
      ...
    },
    ["LMRoman-Demi"] = {
      default  = "file:lmromandemi10-regular",
    },
    ["LMRoman-Italic"] = {
      ...
    },
    ...
    ["LMRoman-Unslanted"] = {
        default  = "file:lmromanunsl10-regular",
    },
    ["LMSans-Regular"] = {
        ...
    },
    ["LMTypewriter-Regular"] = {
        ...
    },
    ...
    ["LMTypewriterVarWd-DarkOblique"] = {
      default  = "file:lmmonoproplt10-boldoblique",
    },
    ...
    ["LMTypewriter-CapsOblique"] = {
      default  = "file:lmmonocaps10-oblique",
    },
  }
}
\stoptyping

The \type {auto} option will choose a best fit compatible to the
\MKII\ implementation. When \type {default} is used instead, the
default filename will be taken. Of course one might wonder if
there will ever be similar goodie files because design sizes
are not that popular nowadays.

Not all fonts are perfect and of course the \LUATEX\ engine can have flaws as
well. For this reason we can implement patches. Here is another example of a
goodie file that has some more code than just a table:

\starttyping
local patches = fonts.handlers.otf.enhancers.patches

local function patch(data,filename,threshold)
  local m = data.metadata.math
  if m then
    local d = m.DisplayOperatorMinHeight or 0
    if d < threshold then
      patches.report("DisplayOperatorMinHeight(%s -> %s)",d,threshold)
      m.DisplayOperatorMinHeight = threshold
    end
  end
end

patches.register("after","analyze math","asana",
    function(data,filename) patch(data,filename,1350) end)

local function less(value,target,original)
  return 0.25 * value
end

local function more(value,target,original)
  local o = original.mathparameters.DisplayOperatorMinHeight
  if o < 2800 then
      return 2800 * target.parameters.factor
  else
      return value -- already scaled
  end
end

return {
  name = "asana-math",
  version = "1.00",
  comment = "Goodies that complement asana.",
  author = "Hans Hagen",
  copyright = "ConTeXt development team",
  mathematics = {
    parameters = {
      DisplayOperatorMinHeight         = more,
      StackBottomDisplayStyleShiftDown = less,
      StackBottomShiftDown             = less,
      StackDisplayStyleGapMin          = less,
      StackGapMin                      = less,
      StackTopDisplayStyleShiftUp      = less,
      StackTopShiftUp                  = less,
      StretchStackBottomShiftDown      = less,
      StretchStackGapAboveMin          = less,
      StretchStackGapBelowMin          = less,
      StretchStackTopShiftUp           = less,
    }
  }
}
\stoptyping

In fact, in addition to already mentioned \type {mapfiles} and
\type {virtuals} subtables, we can pass variables and
overload parameters.

\starttyping
return {
  name = "lm-math",
  ...
  mathematics = {
    mapfiles = {
      ...
    },
    virtuals = {
      ...
    variables = {
      joinrelfactor = 3, -- default anyway
    },
    parameters = { -- test values
     -- FactorA = 123.456,
     -- FactorB = false,
     -- FactorC = function(value,target,original)
     --   return 7.89 * target.factor
     -- end,
     -- FactorD = "Hi There!",
    },
  }
}
\stoptyping

This kind of goodie functionality is typical for the development of \LUATEX\ and
experimental math fonts and no user should ever be bothered with it. However, it
demonstrates that we're not stuck with only features built in the fonts.

% mathdimensions

It can be that a user is not satisfied by some aspects of a math font design.
There is not much that we can do about the shapes, but we can manipulate for
instance dimensions.

For this there are two mechanism available: automatically applied dimensional
fixes and a \type {mathdimensions} feature. Both work with the same goody
specification.

\starttyping
mathematics = {
  ...
  dimensions = {
  },
  ...
}
\stoptyping

The entries in a dimensions table are tables themselves. There can be many
of them so one can organize dimensional tweaks in groups. The \type {default}
group is always applied, while others are applied on demand. Say that want
to tweak all \type {±} and \type {∓}. \footnote {In fact, this mechanism is a
a response to a mail on the \CONTEXT\ mailing list.}

\starttyping
mathematics = {
  dimensions = {
    default = {
      [0x00B1] = { -- ±
        height = 500,
        depth  = 0,
      },
      [0x2213] = { -- ∓
        height = 500,
        depth  = 0,
      },
    },
  },
}
\stoptyping

This will give these two characters a different height and depth. However, this
will not have much effect in rendering (much larger dimensions might have).

\starttyping
mathematics = {
  dimensions = {
    default = {
      [0x00B1] = { -- ±
        yoffset =  100,
      },
      [0x2213] = { -- ∓
        yoffset = -100,
      },
    },
  },
}
\stoptyping

This will raise and lower the glyphs in their bounding boxes and give them
an appearance more close to their ancestors. But defined this way, they are
always applied and that might not be what we  want. So, we can do this:

\starttyping
mathematics = {
  dimensions = {
    signs = {
      [0x00B1] = { -- ±
        yoffset =  100,
      },
      [0x2213] = { -- ∓
        yoffset = -100,
      },
    },
  },
}
\stoptyping

This time the application is feature driven. As with all features, setting them
up has to happen {\em before} fonts are loaded. This will do the trick:

\starttyping
\definefontfeature [lm-math] [mathdimensions=signs]
\stoptyping

The \type {lm-math} feature is not defined by default but can be used for such
purposes. It {\em is} defined with the fontname:

\starttyping
\definefontsynonym
  [LMMathRoman-Regular]
  [file:latinmodern-math-regular.otf]
  [features={math\mathsizesuffix,lm-math},
   goodies=lm]
\stoptyping

A rather specialized goodie is the one that is used to specify math cut|-|ins. A
good quality math font has these kerns already defined but even then you might
want to add or replace some by your own. Here is an example of such a patch.
Normally there are multiple goodies defined in one file but we only show kerns
here:

\starttyping
local kern_200 = { bottomright = { { kern = -200 } } }
local kern_100 = { bottomright = { { kern = -100 } } }

return {
    name = "pagella-math",
    version = "1.00",
    comment = "Goodies that complement pagella.",
    author = "Hans Hagen",
    copyright = "ConTeXt development team",
    mathematics = {
        kerns = {
            [0x1D449] = kern_200, -- math italic V
            [0x1D44A] = kern_100, -- math italic W
        },
    },
}
\stoptyping

As with other goodies the file is loaded with:

\starttyping
\definefontsynonym
  [MathRoman]                     % names used in definitions
  [file:texgyrepagella-math.otf]  % the file to be loaded
  [features=math\mathsizesuffix,  % size dependent features
   goodies=pagella-math]          % the goodie file to be applied
\stoptyping

This is typically a goodie that is always applied and not driven by a feature.
After all, the values given are passed to the engine (after being scaled).

Most goodies are bound to fonts of collections of fonts. This is different for
treatments. These ship with the distribution but you can also provide your own.
As this is still somewhat experimental we just mention a few aspects. The entries
are filenames that point to tables.

\starttyping
return {
  name = "treatments",
  version = "1.00",
  comment = "Goodies that deals with some general issues.",
  author = "Hans Hagen",
  copyright = "ConTeXt development team",
  treatments = {
    ["adobeheitistd-regular.otf"] = {
      embedded = false, -- not yet used
      comment  = "this font is part of acrobat",
    },
    ["crap.ttf"] = {
      ignored = true,
      comment = "a text file with suffix ttf",
    },
    ["latinmodern-math.otf"] = {
      comment = "experimental",
    },
    ["rubish-regular.ttf"] = {
      comment = "check output for missing à and á",
    }
  },
}
\stoptyping

The comment entry in such a table becomes part of the message at the end
of a run:

\startnarrowtyping
mkiv lua stats  > loaded fonts: 2 files: latinmodern-math.otf (experimental), lmroman12-regular.otf
\stopnarrowtyping

The ignored flag signals the font name database builder to ignore the file. This
means that the font can still be known as file, but that its (name based)
properties are not collected. As you asked explicitly for a file, the file can
still be loaded. You can use this trick to avoid issues with the database builder
in case of a problematic file, but a real run will still try to load the file. After
all, you get what you ask for. If loading and usage is successful you get at least
the message reported at the end of the run.

\stopsection

\startsection[title=Analyzers]

An \OPENTYPE\ font is kind of special in the sense that it provides some
information on how to turn sequences of characters into sequences of glyphs. In
fact, if all fonts had a reasonable repertoire of glyphs most of the information
that concerns combining, remapping and shuffling the input and|/|or mapping onto
glyphs could as well happen in the renderer. This means that fonts have many of
their internal features tables in common, or more precisely could share many gsub
related issues, if only there had been some predefined sets of substitutional
features.

So, for most of the time, a feature processor just does what the font demands and
the font provides the information. There are however a few cases where font only
provide part of the logic. Take for instance the \type {init}, \type {medi},
\type {fina} and \type {isol} features that relate to positions in the word: the
start, the end, in the middle or isolated. For these features to work the engine
has to provide information about the state of a character (glyph) and this is where
analysis kicks in. Just watch this:

\startbuffer
\definefontfeature
  [default-with-analyze]
  [default]
  [script=latn,mode=node,
   init=yes,medi=yes,fina=yes,isol=yes]

\showotfcomposition
  {dejavu-serif*default-with-analyze at 24pt}
  {}
  {I don't wanna know tha\utfchar{"300}t!}
\stopbuffer

\typebuffer

In the tracer the different categories are colored. This kind of information is
especially important for typesetting Arabic. Normally \CONTEXT\ can figure out
itself when this is needed so you don't have to worry too much about this kind of
additional actions.

\blank \getbuffer \blank

\stopsection

\startsection[title=Processors]

    \startnotabene
        Todo.
    \stopnotabene

\stopsection

\startsection[title=Optimizing]

    \startnotabene
        Todo.
    \stopnotabene

\stopsection

\startsection[title=Tracing]

There are a lot of tracing options in \MKIV, but most will never be seen by users. Most
are enabled using the tracker mechanism. Some have a bit more visibility and have a dedicated
command to trigger them.

When something is going terribly wrong, you will always get a message but sometimes even an
end|-|user has to request for more information. An example are missing characters. There are
several ways to get them reported:

\starttyping
\enabletrackers[fonts.missing=replace]
\enabletrackers[fonts.missing=remove]
\enabletrackers[fonts.missing]
\stoptyping

For historic reasons we also have:

\starttyping
\checkcharactersinfont
\removemissingcharacters
\replacemissingcharacters
\stoptyping

which happens automatically when you enable the tracker. There is some extra
overhead involved so you might want to turn on this feature on only if you really
expect characters not to be present.

Say that we use Latin Modern fonts and ask for some of the rare fractions:

\startbuffer
\definedfont[lmroman10-regular*default-with-missing at 10pt]
a b c ½ ⅓ ¼ ⅕ ⅙ ⅛ Ɣ ɣ ʤ ʭ ʮ α β γ
\stopbuffer

\typebuffer

\enabletrackers[fonts.missing=replace]
We get this: \start \getbuffer \stop
\removeunwantedspaces . \space
In the log file you will find something like this:
\par \disabletrackers[fonts.missing]

\starttyping
fonts   > characters > start missing characters: lmroman10-regular.otf

missing > U+00194  Ɣ  LATIN CAPITAL LETTER GAMMA
missing > U+00263  ɣ  LATIN SMALL LETTER GAMMA
missing > U+002A4  ʤ  LATIN SMALL LETTER DEZH DIGRAPH
missing > U+002AD  ʭ  LATIN LETTER BIDENTAL PERCUSSIVE
missing > U+002AE  ʮ  LATIN SMALL LETTER TURNED H WITH FISHHOOK
missing > U+003B1  α  GREEK SMALL LETTER ALPHA
missing > U+003B2  β  GREEK SMALL LETTER BETA
missing > U+003B3  γ  GREEK SMALL LETTER GAMMA
missing > U+02153  ⅓  VULGAR FRACTION ONE THIRD
missing > U+02155  ⅕  VULGAR FRACTION ONE FIFTH
missing > U+02159  ⅙  VULGAR FRACTION ONE SIXTH
missing > U+0215B  ⅛  VULGAR FRACTION ONE EIGHTH

fonts   > characters > stop missing characters
\stoptyping

If you're lucky your editor will use a font that shows the missing characters (dejavu
monospace is a good candidate).

The replacement characters can help you to locate the spots where something is missing
so that an alternative can be considered. The replacements resemble the category
of the missing character.

\showmissingcharacterslegend

You can call up this legend after loading an extra module:

\starttyping
\usemodule[s][fonts-missing]

\showmissingcharacterslegend

\showmissingcharacters
\stoptyping

The last command shows a detailed list of missing characters

\showmissingcharacters

Here the characters are shown, because we use a monospaced font that happens to
have them. Of course this example uses characters that are rarely used and are
unlikely to show up in future versions of the Latin Modern fonts.

\startnotabene
    Here a few more relevant trackers will be mentioned.
\stopnotabene

\stopsection

% \startsection[title=Discretionaries]
%
% \startbuffer
% \definedfont[cambria*default]
% 12\discretionary
%     {3} {4} {5}%
% 67\par
% 12{\oldstyle\discretionary
%     {3} {4} {5}}%
% 67\par
% 12\discretionary
%     {3{\oldstyle3}} {{\oldstyle4}4} {5{\oldstyle5}5}%
% 67\par
% \stopbuffer
%
% The font handler has to do some magick to get features working with and across
% discretionaries. To some extend you can use font switches inside discretionaries
% but for sure border cases are not dealt with. This works:
%
% \startlinecorrection[blank]
% \startcombination[nx=4,ny=1,location=top]
%     {\framed[align=normal]{\enabledirectives [otf.alwaysdisc]\setupwhitespace[line]\getbuffer}} {1}
%     {\framed[align=normal]{\enabledirectives [otf.alwaysdisc]\hsize1mm\getbuffer}} {2}
%     {\framed[align=normal]{\disabledirectives[otf.alwaysdisc]\setupwhitespace[line]\getbuffer}} {3}
%     {\framed[align=normal]{\disabledirectives[otf.alwaysdisc]\hsize1mm\getbuffer}} {4}
% \stopcombination
% \stoplinecorrection
%
% The first two examples have \type {otf.alwaysdisk} enabled, the last two have it
% disabled.
%
% \typebuffer
%
% \stopsection

\startsection[title=Some remarks]

If you talk about features and fonts it is not difficult to end up speaking
\OPENTYPE . However, in \CONTEXT\ we use the term in a more general way, if only
because we provide more features. In traditional \TEX\ we have a few features:
ligatures, smallcaps and kerns, and to some extent we can see oldstyle numerals
also as feature. It is however important to notice that in \OPENTYPE\ ligatures
are just a synonym for combining multiple characters into on. From the user
interface point of view these operations are grouped into \type {liga}, \type
{dlig}, \type {clig} and \type {rlig} and for \TEX ies we have \type {tlig}. The
distinction is not as clear as one might think: any feature can use the ligature
builder. And as a consequence we see that happen too, for instance some fonts use
\type {ccmp} for constructing mandatory ligatures.

Some of these interpretations (or maybe even tricks) are side effects of for
instance user interfaces. If one can for instance not turn on or off the \type
{ccmp} feature, but can do that for \type {liga}, then one way to keep some
ligatures in for instance letter spaced text, is to put them into \type {ccmp},
assuming that this one will always be enabled. Eventually that then becomes a
sort of standard. Personally I don't like such pseudo standards but we have to
live with them.

Another example of such a standard is the used of non breakable spaces to
influence treatment of some Devanagari characters. Where \UNICODE\ has special
characters to influence mechanisms that combine and replace characters, the lack
of some triggers others to be used and eventually that becomes a standard.
Similar ambiguities arise with math: we have no way to indicate math (while we do
have ways to indicate a change in writing order).

Talking of math, take \OPENTYPE\ math: at some point there is a draft, that then
gets implemented in one word processor using one font, but omissions or
imperfections that surface (maybe because more fonts and engines are developed)
stay around because the initial implementation is published and frozen, simply
because there are many users that stick to expectations. Where \TEX ies accept a
few years of development, this is not true for commercial applications. \footnote
{Of course \HTML\ is the biggest example of this: we're stuck forever with open
tags without close tags, mixed uppercase and lowercase tags, attributes without
value or values without quotes.}

So, although there is without doubt progress, some annoyances stay. The \TEX\
community has always been able to adapt, and this is one reason why a \LUA\
implementation is nice: it gives room for experiments, extensions, variants, etc.
Of course it also makes a bit more independent, although one may wonder if that
matters any longer in a rapidly changing world. The original idea behind \TEX,
that it should be useable for ages, will survive, but users might see more
changes in a lifetime than foreseen when \TEX\ showed up.

\stopsection

\startsection[title=Different spaces]

The width of the space and its stretch and shrink are taken from the font. The so
called emspace is the reference for much spacing related parameters. It is the
width of character \type {0x2014}. The regular space width is taken from \type
{0x0020}, the space character. When there is no space character, in the case of a
monospaced font we take the emwidth, otherwise we take half the emwidth. As a
last resort we can take the average width of characters. And of even that fails
we take half of the font units. When there is no emwidth that one is set to the
font units.

In the \CONTEXT\ font loader we use a stretch that is 1/2 of the width of a space
and the shrink is 1/3 the width of a space, so we use values that are quite
similar to what \TEX\ always used.

You can overload these values when a font is loaded and the overload is
implemented as a feature. The next example demonstrates how this is done:

\startbuffer
\definefontfeature[morespace][spacing=0.50 plus 0.50 minus 0.250]
\definefontfeature[lessspace][spacing=0.25 plus 0.25 minus 0.125]

\definedfont[Serif*default]          \samplefile{klein}\blank
\definedfont[Serif*default,morespace]\samplefile{klein}\blank
\definedfont[Serif*default,lessspace]\samplefile{klein}\blank
\definedfont[Serif*default]          \samplefile{klein}\blank
\stopbuffer

\typebuffer \blank \getbuffer \blank

\stopsection

\startsection[title=Dynamic features]

We can enable and disable features any time in the input by using the
\type {\feature} command. he following example was posted on the list:

\startbuffer
\definefont
  [WeirdShapes]
  [file:libertiner*default]

\definefontfeature
  [hist]
  [hlig=yes]

\definefontfeature
  [rare]
  [dlig=yes]

\setupquotation
  [style={\feature[+][hist,rare]}]

\startlines
\WeirdShapes
strict {\feature[+][hist]strict}
wurtzite {\feature[+][rare]wurtzite}
\quotation{strict wurtzite}
\stoplines
\stopbuffer

\typebuffer

Or typeset:

\getbuffer

The \type {\feature} command takes as first argument a directive of what
do do:

\starttabulate[|T||]
\NC + more        \NC add set to previous set and combine with font set \NC \NR
\NC - less        \NC subtract set to previous set and combine with font set \NC \NR
\NC = new         \NC replace font set \NC \NR
\NC ! < reset     \NC forget sets and revert to font set \NC \NR
\NC > old default \NC make sure the current set is used on top of the font set \NC \NR
\stoptabulate

\stopsection

\startsection[title=Spacekerns]

Some fonts kern glyphs with spaces. Although \TEX\ doesn't really have spaces we do
support this. However, it's implemented as part of kerning so when you define such
kerns you need to hook it into for instance the \type {kern} feature:

\starttyping
\startluacode
    local kern = -50
    local pair = { [32] = kern }

    fonts.handlers.otf.addfeature {
        name    = "kern", -- spacekerns assume kern
        type    = "kern",
        data    = {
            A = pair, V = pair, W = pair,
            [32] = {
                A = kern,
                V = kern,
                W = kern,
            },
        }
    }
\stopluacode
\stoptyping

Of course this depends on font properties so one can wonder how useful this is.

\stopsection

\stopchapter

\stopcomponent