path: root/doc/context/sources/general/fonts/fonts/fonts-formats.tex

% language=uk

\startcomponent fonts-formats

\environment fonts-environment

\startchapter[title=Font formats][color=darkred]

\startsection[title=Introduction]

In this chapter the font formats as we know them will be introduced. The
descriptions will be rather general but more details can be found in the
appendix. Although in \MKIV\ we do support all these types eventually the focus
will be on \OPENTYPE\ fonts but it does not hurt to see where we are coming from.

\stopsection

\startsection[title=Glyphs]

A typeset text is mostly a sequence of characters turned into glyphs. We talk of
characters when you input the text, but the visualization involves glyphs. When
you copy a part of the screen in an open \PDF\ document or \HTML\ page back to
your editor you end up with characters again. In case you wonder why we make this
distinction between these two states we give an example.

\startplacefigure [location=here,reference=fig:character-glyph,title=From characters to glyphs.]
    \startcombination
        {\color[maincolor]{\definedfont[Serif*default       at 30pt]affiliation}} {upright}
        {\color[maincolor]{\definedfont[SerifItalic*default at 30pt]affiliation}} {italic}
    \stopcombination
\stopplacefigure

We see here that the shape of the \type {a} is different for an upright serif and
an italic. We also see that in \type {ffi} there is no dot on the \type {i}. The
first case is just a stylistic one but the second one, called a ligature, is
actually one shape. The 11 characters are converted into 9 glyphs. Hopefully the
final document format carries some extra information about this transformation so
that a cut and paste will work out well. In \PDF\ files this is normally the
case. In this document we will not be too picky about the distinction as in most
cases the glyph is rather related to the character as one knows it.

So, a font contains glyphs and it also carries some information about
replacements. In addition to that there needs to be at least some information
about the dimensions of them. Actually, a typesetting engine does not have to
know anything about the actual shape at all.

\startplacefigure [location=here,reference=fig:glyph-dimension-normal,title=The boundingbox of some normal glyphs.]
    \startcombination[9*1]
        {\ruledhbox{\color[maincolor]{\definedfont[Serif*default at 30pt]a}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[Serif*default at 30pt]b}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[Serif*default at 30pt]g}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[Serif*default at 30pt]l}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[Serif*default at 30pt]q}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[Serif*default at 30pt].}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[Serif*default at 30pt];}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[Serif*default at 30pt]?}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[Serif*default at 30pt]ffi}}} {}
    \stopcombination
\stopplacefigure

\startplacefigure [location=here,reference=fig:glyph-dimension-italic,title=The boundingbox of some italic glyphs.]
    \startcombination[9*1]
        {\ruledhbox{\color[maincolor]{\definedfont[SerifItalic*default at 30pt]a}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[SerifItalic*default at 30pt]b}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[SerifItalic*default at 30pt]g}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[SerifItalic*default at 30pt]l}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[SerifItalic*default at 30pt]q}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[SerifItalic*default at 30pt].}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[SerifItalic*default at 30pt];}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[SerifItalic*default at 30pt]?}}}   {}
        {\ruledhbox{\color[maincolor]{\definedfont[SerifItalic*default at 30pt]ffi}}} {}
    \stopcombination
\stopplacefigure

The rectangles around the shapes \in {figure} [fig:glyph-dimension-normal] and \in
{figure} [fig:glyph-dimension-italic] are called boundingbox. The dashed line
reflects the baseline where they eventually are aligned onto next to each other.
The amount above the baseline is called height, and below is called depth. The
piece of the shape above the baseline is the ascender and the bit below the
descender. The width of the bounding box is not by definition the width of the
glyph. In \TYPEONE\ and \OPENTYPE\ fonts each shape has a so called advance width
and that is the one that will be used.

\usemodule[fonts-kerns]

\startplacefigure [location=here,reference=fig:glyph-kerns,title={Kerning in Latin Roman, Cambria, Pagella and Dejavu.}]
    \scale[width=\textwidth]{\startcombination[1*4]
        {\color[maincolor]{\definedfont[name:lmroman10-regular*default     sa   1]\ShowKernedHBox{Very often glyphs get very small spaces inserted horizontally.}}} {}
        {\color[maincolor]{\definedfont[name:cambria*default               sa   1]\ShowKernedHBox{Very often glyphs get very small spaces inserted horizontally.}}} {}
        {\color[maincolor]{\definedfont[name:texgyrepagellaregular*default sa   1]\ShowKernedHBox{Very often glyphs get very small spaces inserted horizontally.}}} {}
        {\color[maincolor]{\definedfont[name:dejavuserif*default           sa 0.9]\ShowKernedHBox{Very often glyphs get very small spaces inserted horizontally.}}} {}
    \stopcombination}
\stopplacefigure

Another traditional property of a font is kerning. In \in {figure}
[fig:glyph-kerns] you see this in action. These examples
demonstrate that not all fonts need (or provide) the same kerns
(in points).

So, as a start, we have now met a couple of properties of a font.
They can be summarized as follows:

\starttabulate[|l|p|]
\NC mapping to glyphs   \EQ characters are represented by a shapes that have recognizable
                            properties so that readers know what they mean \NC \NR
\NC ligature building   \EQ a sequence of characters gets mapped onto one glyph \NC \NR
\NC dimensions          \EQ each glyph has a width, height and depth \NC \NR
\NC inter-glyph kerning \EQ optionally a bit of positive or negative space has to be inserted between glyphs \NC \NR
%NC italic correction   \EQ a correction is applied between an oblique shape and what follows \NC \NR
\stoptabulate

Regular font kerning is hardly noticeable and improves the overall look of the
page. Typesetting applications sometimes are capable of inserting additional
spaces between shapes. This more excessive kerning is not that much related to
the font and is used for special purposes, like making a snippet of text stand
out. In \CONTEXT\ this kind of kerning is available but it is a font independent
feature. Keep in mind that when applying that kind of rather visible kerning
you'd better not have ligatures and fancy replacements enabled as \CONTEXT\
already tries to deal with that as good as possible.

\stopsection

\startsection[title=The basic process]

In \TEX\ a font is an abstraction: the engine only needs to know about the
mapping from characters to glyphs, what the width, height and depth is, what
sequences need to be translated into ligatures and when kerning has to be
applied. If for the moment we forget about math, these are all the properties
that matter and this is what the \TEX\ font metric files that we see in the next
section provide.

Because one of the principles behind \LUATEX\ is that the core engine (the
binary) stays small and that new functionality is provided in \LUA\ code, the
font subsystem largely looks like it always has been. As users will normally use
a macro package most of the loading will be hidden from the user. It is however
good to give a quick overview of how for instance \PDFTEX\ deals with fonts using
traditional metric files.

\startFLOWchart[pdftex]
    \startFLOWcell
        \name {source}
        \location {1,1}
        \shape {action}
        \text {input}
        \connection [rl] {parser}
    \stopFLOWcell
    \startFLOWcell
        \name {parser}
        \location {2,1}
        \shape {action}
        \text {characters}
        \connection [rl] {builder}
    \stopFLOWcell
    \startFLOWcell
        \name {builder}
        \location {3,1}
        \shape {action}
        \text {glyphs}
        \connection [rl] {backend}
    \stopFLOWcell
    \startFLOWcell
        \name {backend}
        \location {4,1}
        \shape {action}
        \text {subset}
    \stopFLOWcell
\stopFLOWchart

\startplacefigure [location=here,reference=fig:tfm-pdftex,title={Several translation steps in a traditonal \TEX\ flow.}]
    \FLOWchart[pdftex]
\stopplacefigure

The input (bytes) gets translated into characters by the input parser. Normally
this is a one|-|to|-|one translation but there are examples of some translation
taking place. You can for instance make characters active and give them a
meaning. So, the eight bit represention of an editors code page \type {ë} can
become something else internally, for instance a regular \type {e} with an \type
{¨} overlayed. It can also become another character, which in the code page
would be shown as \type {á} but the user will not know this as by then this byte
is already tokenized. Another example is multibyte translation, for instance
\UTF\ sequences can get remapped to something that is known internally as being a
character of some kind. The \LUATEX\ engine expects \UTF\ so a macro package has
to make sure that translation to this encoding happens beforehand, for instance
using a callback that intercepts the input from file. \footnote {In \CONTEXT\ we
talk of input regimes and these can be mixed, although in practice most users
will stick to \UTF\ and never use regimes.}

So, the input character (sequence) becomes tokens representing a character. From
these tokens \TEX\ will start building a (linked) node list where each character
becomes a node. In this node there is a reference to the current font. If you
know \TEX\ you will understand that a list can have more than characters: there
can be skips, kerns, rules, references to images, boxes, etc.

At some point \TEX\ will handle this list over to a routine that will turn them
into something that resembles a paragraph or otherwise snippet of text. In that
stage hyphenation kicks in, ligatures get built and kerning is added. Character
references become glyph indices. This list can finally be broken into lines.

It is no secret that \TEX\ can box and unbox material and that after unboxing
some new formatting has to happen. The traditional engine has some optimizations
that demand a partial reconstruction of the original list but in \LUATEX\ we
removed this kind of optimization so there the process is somewhat simpler. We
will see more of that later.

When \TEX\ ships out a page, the backend will load the real font data and merge
that into the final output. It will now take the glyph index and build the right
data structures and references to the real font. As a font gets subset only the
used glyphs end up in the final output.

There is one tricky aspect involved here: re|-|encoding. In so called map files
one can map a specific metric filename onto a real font name. One can also
specify an encoding vector that tells what a given index really refers to. This
makes it possible to use fonts that have more than 256 glyphs and refer to any of
them. This is also the trick that makes it possible to use \TRUETYPE\ fonts in
\PDFTEX: the backend code filters the right glyphs from the font, remapping
\TEX's glyph indices onto real entries in the font happens via the encoding
vector. In \in {figure} [fig:tfm-bytes] we show a possible route for input byte
68.

\startFLOWchart[bytes]
    \startFLOWcell
        \name {source}
        \location {1,1}
        \shape {action}
        \text {bytes (68)}
        \connection [rl] {parser}
    \stopFLOWcell
    \startFLOWcell
        \name {parser}
        \location {2,1}
        \shape {action}
        \text {bytes (31)}
        \connection [rl] {builder}
    \stopFLOWcell
    \startFLOWcell
        \name {builder}
        \location {3,1}
        \shape {action}
        \text {index (31)}
        \connection [rl] {backend}
    \stopFLOWcell
    \startFLOWcell
        \name {backend}
        \location {4,1}
        \shape {action}
        \text {index (88)}
    \stopFLOWcell
\stopFLOWchart

\startplacefigure [location=here,reference=fig:tfm-bytes,title={From bytes to indices.}]
    \FLOWchart[bytes]
\stopplacefigure

As \LUATEX\ carries much of the bagage of older engines, you can still do it this
way but in \CONTEXT\ \MKIV\ we have made our live much simpler: we use unicode as
much as possible. This means that we effectively have removed two steps (see \in
{figure} [fig:tfm-luatex]).

\startFLOWchart[luatex]
    \startFLOWcell
        \name {source}
        \location {1,1}
        \shape {action}
        \text {input}
        \connection [rl] {builder}
    \stopFLOWcell
    \startFLOWcell
        \name {builder}
        \location {2,1}
        \shape {action}
        \text {glyphs}
    \stopFLOWcell
\stopFLOWchart

\startplacefigure [location=here,reference=fig:tfm-luatex,title={Simplified mapping in \LUATEX.}]
    \FLOWchart[luatex]
\stopplacefigure

There is of course still some work to do for the backend, like subsetting, but
the nasty dependency on the input encoding, font encoding (that itself relates to
hyphenation) and backend re|-|encoding is gone. But keep in mind that the
internal data structure of the font is still quite traditional.

Before we move on to font formats I like to point out that there is no space in
\TEX. Spaces in the input are converted into glue, either or not with some
stretch and|/|or shrink. This also means that accessing character 32 in
traditional \TEX\ will not end up as space in the output.

\stopsection

\startsection[title=\TEX\ metrics]

\appendixdata{\in[fontdata:tfm]}
\appendixdata{\in[fontdata:vf]}

Traditional font metrics are packaged in a binary format. Due to the limitations
of that time a font has at most 256 characters. In books dedicated to \TEX\ you
will often find tables that show what glyphs are in a font, so we will not repeat
that here as after all we got rid of that limitation in \LUATEX.

Because 256 is not that much, especially when you mix many scripts and need lots
of symbols from the same font, there are quite some encodings used in traditional
\TEX, like \type {texnansi}, \type {ec} and \type {qx}. When you use \LUATEX\
exclusively you can do with way less font files. This is easier for users,
especially because most of those files were never used anyway. It's interesting
to notice that some of the encodings contain symbols that are never used or used
only once in a document, like the copyright or registered symbols. They are often
accessed by symbolic names and therefore easily could have been omitted and
collected in a dedicated symbol font thereby freeing slots for more useful
characters anyway. The lack of coverage is probably one of the reasons why new
encodings kept popping up. In the next table you see how many files are involved
in Latin Modern which comes in a couple of design sizes. \footnote {The original
Computer Modern fonts have \METAFONT\ source files and (runtime) generated bitmap
files in whatever resolutions are needed for previewing and printing. The
\TYPEONE\ follow|-|up came in several sets, organized by language support. The
Latin Modern fonts have a few more weights and variants than Computer Modern.}

\starttabulate[|l|c|r|r|r|]
\HL
\NC \bf font format \NC \bf type \NC \bf \# files \NC \bf size in bytes \NC \bf \CONTEXT \NC \NR
\HL
\NC type 1   \NC tfm \NC 380 \NC  3.841.708 \NC \NC \NR
\NC          \NC afm \NC  25 \NC  2.697.583 \NC \NC \NR
\NC          \NC pfb \NC  92 \NC  9.193.082 \NC \NC \NR
\NC          \NC enc \NC  15 \NC     37.605 \NC \NC \NR
\NC          \NC map \NC   9 \NC     42.040 \NC \NC \NR
\HL[darkgray]
\NC          \NC     \NC 521 \NC 15.812.018 \NC mkii \NC \NR
\HL
\NC opentype \NC otf \NC  73 \NC  8.224.100 \NC mkiv \NC \NR
\HL
\stoptabulate

A \TFM\ file can contain so called italic corrections. This is an additional kern
that can be added after a character in order to get better spacing between an
italic shape and an upright one. As this is manual work, it's a not that advanced
mechanism, but in addition to width, height, depth, kerns and ligatures it is
nevertheless a useful piece of information. But, it's a rather \TEX\ specific
quantity.

Since \TEX\ showed up many fonts have been added. In addition support for
commercial fonts was provided. In fact, for that to happen, one only needs
accompanying metric files for \TEX\ itself and map files and encoding vectors
for the backend. Because a metric file also has some general information, like
spacing (including stretch and shrink), the ex|-|height and em|-|width, this
means that sometimes guesses must be made when the original font does not come
with such parameters.

At some point virtual fonts were introduced. In a virtual font a \TFM\ file has
an accompanying \VF\ file. In that file each glyph has a specification that tells
where to find the real glyph. It is even possible to construct glyphs from other
glyphs. In traditional \TEX\ this only concerns the backend, which in \PDFTEX\ is
built in. In \LUATEX\ this mechanism is integrated into the frontend which means
that users can construct such virtual fonts themselves. We will see more of that
later, but for now it's enough to know that when we talk about the representation
of font (the \TFM\ table) in \LUATEX, this includes virtual functionality.

An important limitation of \TFM\ files cq.\ traditional \TEX\ is that the number
of depths and heights is limited to 16 each. Although this results in somewhat
inaccurate dimensions in practice this gets unnoticed, if only because many
designs have some consistency in this. On the other hand, it is a limitation when
we start thinking of accents or even multiple accents which lead to many more
distinctive heights and depths.

Concerning ligatures we can remark that there are quite some substitutions
possible although in practice only the multiple to one replacement has been used.

Some fonts that are used in \TEX\ started out as bitmaps but rather soon
\TYPEONE\ outline fonts became the fashion. These are supported using the map
files that we will discuss later. First we look into \TYPEONE\ fonts.

\stopsection

\startsection[title=\TYPEONE]

\appendixdata{\in[fontdata:afm]}
\appendixdata{\in[fontdata:enc]}
\appendixdata{\in[fontdata:map]}

For a long time \TYPEONE\ fonts have dominated the scene. These are \POSTSCRIPT\
fonts that can have more that 256 glyphs in the file that defines the shapes, but
only 256 of them can be used at one time. Of course there can be multiple subsets
active in one document.

In traditional \TEX\ a \TYPEONE\ font is used by making a \TFM\ file from a so
called Adobe metric file (\AFM) that come with such a font. There are several
tool chains for doing this and \CONTEXT\ \MKII\ ships with one that can be of
help when you need to support commercial fonts. Projects like the Latin Modern
Fonts and \TEX\ Gyre have normalized a whole lot of fonts that came in several
more or less complete encodings into a consistent package of \TYPEONE\ fonts.
This already simplified live a lot but still users had to choose a suitable input
and font encoding for their language and|/|or script. As \TEX\ only cares about
metrics and not about the rendering, it doesn't consider \TYPEONE\ fonts as
something special. Also, as \TEX\ and \POSTSCRIPT\ were developed about the same
time support for \TYPEONE\ fonts is rather present in \TEX\ distributions.

You can still follow this route but for \CONTEXT\ \MKIV\ this is no longer the
recommended way because there we have changed the whole subsystem to use
\UNICODE. As a result we no longer use \TFM\ files derived from \AFM\ files, but
directly interpret the \AFM\ data. This not only removes the 256 limitation, but
also brings more resolution in height and depth as we no longer have at most 16
alternatives. There can also be more kerns. Of course we need some heuristics to
determine for instance the spacing but that is not different from former times.

Because most \TEX\ users don't use commercial fonts, they will not notice that
\CONTEXT\ \MKIV\ treats \TYPEONE\ fonts this way. One reason is that the free
fonts also come as wide fonts in \OPENTYPE\ format and whenever possible
\CONTEXT\ prefers \OPENTYPE\ over \TYPEONE\ over \TFM.

In the beginning \LUATEX\ only could load a \TFM\ file, which is why loading
\AFM\ files is implemented in \LUA. Later, when the \OPENTYPE\ loaded was added,
loading \PFB\ and \AFM\ files also became possible but it's slower and we see no
reason to rewrite the current code in \CONTEXT. We also do a couple of extra
things when loading such a file. As more \TYPEONE\ fonts move on to \OPENTYPE\ we
don't expect that much usage anyway.

\stopsection

\startsection[title=\OPENTYPE]

\appendixdata{\in[fontdata:otf]}

When an engine can deal with \UNICODE\ directly it also means that internally it
uses pretty large numbers for storing characters and glyph indices. The first
\TEX\ descendent that went wide was \OMEGA, later replaced by \ALEPH. However, this
engine never took off and still used its own extended \TFM\ format: \OFM. In fact,
as \LUATEX\ uses some of the \ALEPH\ code, it can also use these extended metric
files but I don't think that there are any useful fonts around so we can forget
about this.

We use the term \OPENTYPE\ for a couple of font formats that share the same
principles: \OPENTYPE\ (\OTF), \TRUETYPE\ (\TTF) and \TRUETYPE\ containers
(\TTC). The \LUATEX\ font reader presents them in a similar format. In the case
of a \TRUETYPE\ container, one does not load the whole font but selects an
instance from it. Internally an \OPENTYPE\ font can have the glyphs organized in
subfonts.

The first \TEX\ descendent to really go wide from front to back is \XETEX. This
engine can use \OPENTYPE\ fonts directly and for a whole category of users this
opened up a new world. Hoever, it is still mostly a traditional engine. The
transition from characters to glyphs is accomplished by external libraries, while
in \LUATEX\ we code in \LUA. This has the disadvantage that it is slower
(although that depends on the job) but the advantage is that we have much more
control and can extend the font handler as we like.

An \OPENTYPE\ font is much more complex than a \TYPEONE\ one. Unless it is a
quick and dirty converted existing font, it will have more glyphs to start with.
Quite likely it will have kerns and ligatures too and of course there are
dimensions. However, there is no concept of a depth and height. These need to be
deduced from the bounding box instead. There is an advance width. This means that
we can start right away using such fonts if we map those properties onto the
\TFM\ table that \LUATEX\ expects.

But there is more, take ligatures. In a traditional font the sequence \type {ffi}
always becomes a ligature, given that the font has such a glyph. In \LUATEX\
there is a way to disable this mechanism, which is sometimes handy when dealing
with mono|-|spaced fonts in verbatim. It's pretty hard to disable that. For
instance one option is to insert kerns manually. In an \OPENTYPE\ font ligatures
are collected in a so called feature. There can be many such features and even
kerning is a feature. Other examples are old style numerals, fractions,
superiors, inferiors, historic ligatures and stylistic alternates.

\starttabulate[|lT|l|l|l|l|]
\NC \type{onum} \NC \ruledhbox{\maincolor\DemoOnumLM\char45 1}
                \NC \ruledhbox{\maincolor\DemoOnumLM1234567890}
                \NC \ruledhbox{\maincolor\DemoOnumLM\char"A2}
                \NC \ruledhbox{\maincolor\DemoOnumLM\char"24} \NC \NR
%NC \type{lnum} \NC \ruledhbox{\maincolor\DemoLnumLM\char45 1}
%               \NC \ruledhbox{\maincolor\DemoLnumLM1234567890}
%               \NC \ruledhbox{\maincolor\DemoLnumLM\char"A2}
%               \NC \ruledhbox{\maincolor\DemoLnumLM\char"24} \NC \NR
\NC \type{tnum} \NC \ruledhbox{\maincolor\DemoTnumLM\char45 1}
                \NC \ruledhbox{\maincolor\DemoTnumLM1234567890}
                \NC \ruledhbox{\maincolor\DemoTnumLM\char"A2}
                \NC \ruledhbox{\maincolor\DemoTnumLM\char"24} \NC \NR
\NC \type{pnum} \NC \ruledhbox{\maincolor\DemoPnumLM\char45 1}
                \NC \ruledhbox{\maincolor\DemoPnumLM1234567890}
                \NC \ruledhbox{\maincolor\DemoPnumLM\char"A2}
                \NC \ruledhbox{\maincolor\DemoPnumLM\char"24} \NC \NR
\stoptabulate

To this all you need to add that features operate in two dimensions: languages
and scripts. This means that when ligatures are enabled for Dutch the \type {ij}
sequence becomes a single glyph but for German it gets mapped onto two glyphs.
And, to make it even more complex, a substitution can depend on circumstances,
which means that for Dutch \type {fijn} becomes \type {f ij n} but \type {fiets}
becomes \type {fi ets}. It will be no surprise that not all \OPENTYPE\ fonts come
with a complete and rich repertoire of rules. To make things worse, there can be
rules that turn \type {1/2} into one glyph, or transfer the numbers into superior
and inferior alternatives, but leaves us with an unacceptable rendered \type
{1/a}, given that the \type {frac} features is enabled. It looks like features
like this are to be applied to a manually selected range of characters.

The fact that an \OPENTYPE\ font can contain many features and rules to apply
them makes it possible to typeset scripts like Arabic. And this is where it gets
vague. A generic \OPENTYPE\ sub|-|engine can do clever things using these rules,
but if you read the specification for some scripts additional intelligence has to
be provided by the typesetting engine.

While users no longer have to care about encodings, map files and back|-|end
issues, they do have to carry knowledge about the possibilities and limitations
of features. Even worse, he or she needs to be aware that fonts can have bugs.
Also, as font vendors have no tradition of providing updates this is something
that we might need to take care of ourselves by tweaking the engine.

One of the problems with the transition from \TYPEONE\ to \OPENTYPE\ is that font
designers can take an existing design and start from that basic repertoire of
shapes. If such a design had oldstyle figures only, there is a good chance that
this will be the case in the \OPENTYPE\ variant too. However, such a default
interferes with the fact that the \type {onum} feature is one that we explicitly
have to enable. This means that writing a generic style where a font is later
plugged in becomes somewhat messy if it assumes that features need to be turned
on.

\TEX\ users expect more control, which means that in practice just an \OPENTYPE\
engine is not enough, but for the average font the \TEX\ model using the
traditional approach still is quite acceptable. After all, not all users use
complex scripts or need advanced features. And, in practice most readers don't
notice the difference anyway.

\stopsection

\startsection[title=\LUA]

\appendixdata{\in[fontdata:lua]}

As mentioned support for virtual fonts is built into \LUATEX\ and loading the so
called \VF\ files happens when needed. However, that concerns traditional fonts
that we already covered. In \CONTEXT\ we do use the virtual font mechanism for
creating missing glyphs out of existing ones or add fallbacks when this is not
possible. But this is not related to some kind of font format.

In 2010 and 2011 the first public \OPENTYPE\ math fonts showed up that replace
their \TYPEONE\ originals. In \CONTEXT\ we already went forward and created
virtual \UNICODE\ fonts out of traditional fonts. Of course eventually the
defaults will change to the \OPENTYPE\ alternatives. The specification for such a
virtual font is given in \LUA\ tables and therefore you can consider \LUA\ to be
a font format as well. In \CONTEXT\ such fonts can be defined in so called
goodies files. As we use these files for much more tuning, we come back to that
in a later chapter. In a virtual font you can mix real \TYPEONE\ fonts and real
\OPENTYPE\ fonts using whatever metrics suit best.

An extreme example is the virtual \UNICODE\ Punk font. This font is defined in
the \METAPOST\ language (derived from Don Knuths \METAFONT\ sources) where each
glyph is one graphic. Normally we get \POSTSCRIPT, but in \LUATEX\ we can also
get output in a comparable \LUA\ table. That output is converted to \PDF\
literals that become part of the virtual font definitions and these eventually
end up in the \PDF\ page stream. So, at the \TEX\ end we have regular (virtual)
characters and all \TEX\ needs is their dimensions, but in the \PDF\ each glyph
is shown using drawing operations. Of course the now available \OPENTYPE\ variant
is more efficient, but it demonstrates the possibilities.

\stopsection

\startsection[title=Files]

We summarize these formats in the following table where we explain what the file
suffixes stand for:

\starttabulate[|Tl|p|]
\HL
\NC tfm \NC This is the traditional \TEX\ font metric file format and it reflects
            the internal quantities that \TEX\ uses. The internal data structures
            (in \LUATEX) are an extension of the \TFM\ format. \NC \NR
\NC vf  \NC This file contains information about how to construct and where to
            find virtual glyphs and is meant for the backend. With \LUATEX\ this
            format gets more known. \NC \NR
\NC pk  \NC This is the bitmap format used for the first generation of \TEX\
            fonts but the typesetter never deals with them. Bitmap files are more
            or less obselete. \NC \NR
\HL
\NC ofm \NC This is the \OMEGA\ variant of the \type {tfm} files that caters for
            larger fonts. \NC \NR
\NC ovf \NC This is the \OMEGA\ variant of the \type {vf}. \NC \NR
\HL
\NC pfb \NC In this file we find the glyph data (outlines) and some basic
            information about the font, like name|-|to|-|index mappings. A
            differently byte|-|encoded variant of this format is \type {pfa}.\NC
            \NR
\NC afm \NC This file accompanies the \type {pfb} file and provides additional
            metrics, kerns and information about ligatures. A binary variant of
            this is the \PFA\ format. For \MSWINDOWS\ there is a variant that has the
            \type {pfm} suffix. \NC \NR
\NC map \NC The backend will consult this file for mapping metric file names onto
            real font names. \NC \NR
\NC enc \NC The backend will include (and use) this encoding vector to map
            internal indices to font indices using glyph names, if needed. \NC
            \NR
\HL
\NC otf \NC This binary format describes not only the font in terms of metrics,
            features and properties but also contains the shapes. \NC \NR
\NC ttf \NC This is the \MICROSOFT\ variant of \OPENTYPE. \NC \NR
\NC ttc \NC This is the \MICROSOFT\ container format that combines multiple fonts
            in one. \NC \NR
\HL
\NC fea \NC A (\FONTFORGE) feature definition file. Such a file can be loaded and
            applied to a font. This is no longer supported in \CONTEXT\ as we have
            other means to achieve the same goals. \NC \NR
\NC cid \NC A glyph index (name) to \UNICODE\ mapping file that is referenced
            from an \OPENTYPE\ font and is shared between fonts. \NC \NR
\HL
\NC lfg \NC These are \CONTEXT\ specific \LUA\ font goodie files providing
            additional information. \NC \NR
\HL
\stoptabulate

If you look at how files are organized in a \TEX\ distribution, you will notice
that these files all get their own place. Therefore adding a \TYPEONE\ font to
the distribution is not that trivial if you want to avoid clashes. Also, files
are simply not found when they are not in the right spot. Just to mention a few
paths:

\starttyping
<root>/fonts/tfm/vendor/typeface
<root>/fonts/vf/vendor/typeface
<root>/fonts/type1/vendor/typeface
<root>/fonts/truetype/vendor/typeface
<root>/fonts/opentype/vendor/typeface
<root>/fonts/fea
<root>/fonts/cid
<root>/fonts/dvips/enc
<root>/fonts/dvips/map
\stoptyping

There can be multiple roots and the right locations are specified in a
configuration file. Currently all engines can use the \DVIPS\ encoding and map
files, so luckily we don't need to duplicate this. For some reason \TRUETYPE\ and
\OPENTYPE\ fonts have different locations and you need to be aware of the fact
that some fonts come in both formats (just to confuse users) so you might end up
with conflicts.

In \CONTEXT\ we try to make live somewhat easier by also supporting a simple path
structure:

\starttyping
<root>/fonts/data/vendor/typeface
\stoptyping

This way files are kept together and installing commercial fonts is less complex
and error prone. Also, in practice we only have one set of files now: one of the
other \OPENTYPE\ formats.

If you want to see the difference between a traditional (\PDFTEX\ or \XETEX\ plus
\CONTEXT\ \MKII) setup or a modern one (\LUATEX\ with \CONTEXT\ \MKIV) you can
install the \CONTEXT\ suite (formerly known as minimals). If you explicitly
choose for a \LUATEX\ only setup, you will notice that far less files get
installed.

\stopsection

\startsection[title=Text]

This is not an in|-|depth explanation of how to define and load fonts in
\CONTEXT. First of all this is covered in other manuals, but more important is
that we assume that the reader is already familiar with the way \CONTEXT\ deals
with fonts. Therefore we limit ourselves to some remarks and expand on this a bit
in later chapters.

The font subsystem has evolved over years and when you look at the low level code
you will probably find it complex. This is true, although in some aspects it is
not as complex as in \MKII\ where we also had to deal with encodings due to the
eight bit limitations. In fact, setting up fonts is easier due the fact that we
have less files to deal with.

The main properties of a (modern) font subsystem for typesetting text are the
following:

\startitemize[n]
    \startitem
        We need to be able to switch the look and feel efficiently and
        consistently, for instance going from regular to bold or italic. So,
        when we load a font family we not only load one file, but often
        at least four: regular, bold, italic (oblique) and bolditalic
        (boldoblique).
    \stopitem
    \startitem
        When we change the size we also need to make sure that these related
        sets are changed accordingly. You really want the bold shapes to scale
        along with the regular ones.
    \stopitem
    \startitem
        Shapes are organized in serif, sans serif, mono spaced and math and for
        proper working of a typesetter that has math all over you need always
        need the math. Again, when you change size, all these shapes need to
        scale in sync.
    \stopitem
    \startitem
        In one document several families can be combined so the subsystem should
        make it possible to switch from one to the other without too much
        overhead.
    \stopitem
    \startitem
       Because section heads and other structural elements have their own sizes
       there has to be a consistent way to deal with that. It should also be
       possible to specify exceptions for them.
   \stopitem
\stopitemize

In the next chapters we will cover some details, for instance font features. You
can actually control these when setting up a body font, simply by redefining
the \type {default} feature set, but not all features are dealt with this way.
So let's continue the demands put on a font subsystem.

\startitemize[continue]
    \startitem
        Sometimes inter|-|character kerning is needed. In \CONTEXT\ this is not a
        property of a font because glyphs can be mixed with basically anything.
        This kind of features is applied independent of a font.
    \stopitem
    \startitem
        The same is true for casing (like uppercasing and such) which is not
        related to a font but applied to a selected (or marked) piece of the
        input stream.
    \stopitem
    \startitem
        Using so called \quotation {small caps} or \quotation {old style}
        numerals or \unknown\ can be dealt with by setting the default features
        but often these are applied selectively. As these are applied using the
        information in a font they do belong to the font subsystem but in
        practice they can be seen as independent (assuming that the font supports
        them at all).
    \stopitem
    \startitem
        Protrusion (into margins) and expansion (to improve whitespace) are
        applied to the font at load time because the engine needs to know about
        them. But they two can selectively be turned on and off. They are more
        related to line break handling than font defining.
    \stopitem
    \startitem
        Slanting (to fake oblique) and expanding (to fake bold) are regular
        features but are applied to the font because the engine needs to know
        about them. They permanently influence the shape.
    \stopitem
\stopitemize

We will discuss these in this manual too. What we will not discuss in depth is
spacing, even when it depends on the (main body) font size. These use properties
of fonts (like the ex|-|height or em|-|width and maybe the width of the space,
but normally they are controlled by the spacing subsystem. We will however
mention some rather specific possibilities:

\startitemize[continue]
    \startitem
        The \CONTEXT\ font subsystem provides ways to combine multiple fonts
        into one.
    \stopitem
    \startitem
        You can construct artificial fonts, using existing fonts or \METAPOST\
        graphics.
    \stopitem
    \startitem
        Fonts can be fixed (dimensions) and completed (for instance accented
        characters) when loading/
    \stopitem
    \startitem
        There are extensive tracing options, not only for applied features but
        also for loading, checking etc. There is a set of styles that can be
        used to study fonts.
    \stopitem
\stopitemize

Sometimes users ask for very special trickery and it no surprise then that some
of that is now widely know (or even discussed in detail). When we get notice of
that we can mention it in this manual.

So how does this all relate to font formats? We mentioned that when loading we
basically load some four files per family (and more if we use specific fonts for
titling). These files just provide the data: metric information, shapes and ways
to remap characters (or sequences) into glyphs, either of not positioned relative
to each other. In traditional \TEX\ only dimensions, kerns and ligatures
mattered, but in nowadays we also deal with specific \OPENTYPE\ features. But
still, as you can deduce from the above, this is only part of the story. You need
a complete and properly integrated system. It is no big deal to set up some
environment that uses font files to achieve some typesetting goal, but to provide
users with some consistent and extensible system is a bit more work.

There are basically three font formats: good old bitmaps, \TYPEONE\ and
\OPENTYPE. All need to be supported and expectations are that we also support
their features. But is should be noticed that whatever font you use, the quality
of the outcome depends on what information the font can provide. We can improve
processing but are often stuck with the font. There are many thousands of
fonts out there and we need to be able to use them all.

\stopsection

\startsection[title=Math]

In the previous section we already mentioned math fonts. The fonts are just one
aspect of typesetting math and math fonts are special in the sense that they have
to provide the relevant information. For instance a parenthesis comes in several
sizes and at some point turns in a symbol made out of pieces (like a top curve,
middle lines and bottom curve) that overlap. The user never sees such details. In
fact, there are ot that many math fonts and these are already set up so there is
not much to mess up here. Nevertheless we mention:

\startitemize [n]
    \startitem
        Math fonts are loaded in three sizes: text, script and scriptscript. The
        optimal relative sizes ar defined in the font.
    \stopitem
    \startitem
        There are direction aware math fonts and we support this in \CONTEXT.
    \stopitem
    \startitem
        Bold math is in fact a bolder version of a regular math font (that can
        have bold symbols too). Again this is supported.
    \stopitem
\stopitemize

The way math is dealt with in \CONTEXT\ is different from the way it is done
traditionally. Already when we started with \MKIV\ we moved to \UNICODE\ and
the setup at the font level is kept simple by delegating some of the work to
the \LUA\ end. We will see some of the mentioned aspects in more detail later.

Because of it's complexity and because in a math text there can be many times
activation of math fonts (and related settings) quite some effort has been put in
making it efficient. But you need to keep in mind that when we discuss math
related topics later on, this is hardly of concern. Math fonts are loaded only
once so manipulating them a bit has no penalty. And using them later on is hardly
related to the font subsystem.

Concerning formats we can notice that traditional \TEX\ comes with math fonts
that have properties that the engine can use. Because there were not many math
fonts, this was no problem. The \OPENTYPE\ math fonts however are also used in
other applications and therefore are a bit more generic. \footnote {Their
internals are now defined in the \OPENTYPE\ specification.} For this we not only
had to adapt the math engine in \LUATEX\ (although we kept that to the minimum)
but we also had to think different about loading them. In later chapters we will
see that in the transition to \UNICODE\ math fonts we implemented a mechanism for
combining \TYPEONE\ fonts into virtual \UNICODE\ fonts. We did that because it
made no sense to keep an old and new loader alongside.

There will not be thousands of math fonts flying around. A few dozen is already a
lot and the developers of macro packages can set them up for the users. So, in
practice there is not much that a user needs to know about math font formats.

\stopsection

\startsection[title=Caching]

Because fonts can be large and because we use \LUA\ tables to describe them
a bit of effort has been put into managing them efficiently. Once converted
to the representation that we need they get cached. You can peek into the cache
which is someplace on your system (depending on the setup):

\starttabulate[|l|p|]
\NC \type{fonts/data}    \NC font name databases \NC \NR
\NC \type{fonts/mp}      \NC fonts created using \METAPOST \NC \NR
\NC \type{fonts/one}     \NC type one fonts, converted from \type {afm} and \type
                             {pfb} files \NC \NR
\NC \type{fonts/otl}     \NC open type fonts, converted from \type {ttf}, \type {otf},
                             \type {ttc} and \type {ttx} files loaded using the
                             \CONTEXT\ \LUA\ loader \NC \NR
\NC \type{fonts/pdf}     \NC font shapes for color fonts \NC \NR
\NC \type{fonts/shapes}  \NC outlines of fonts (for instance for use in \METAFUN) \NC \NR
\NC \type{fonts/streams} \NC font programs for variable font instances \NC \NR
\stoptabulate

There can be three types of files there. The \type{tma} files are just \LUA\
tables and they can be large. These files can be compiled to bytecode where \type
{tmc} is for stock \LUATEX\ and \type {tmb} for \LUAJITTEX. The \type {tma} files
are optimized for space and memory (aka: packed) but you can expand them with
\type {mtxrun --script font}.

Fonts in the cache are automatically updated when you install new versions of a
font or when the \CONTEXT\ font loader has been updated.

\stopsection

\startsection[title=Paths]

The search for fonts happens on paths defined in \type {texmf.cnf}. The information
in there is used to generate a file database for fast access with priorities based
on file type. The \TDS\ is starting point. The environment variable driven paths
\type {OSFONTDIR} (set automatically) and \type {EXTRAFONTDIR} are taken into account.

In addition you can set \type {RUNTIMEFONTS} which is, when set, consulted at
runtime. You can also add a path in your style:

\starttyping
\usefontpath[c:/data/projects/myproject/fonts]
\stoptyping

although in general we recommend to put fonts in

\starttyping
<texroot>/tex/texmf-fonts/fonts/data]
\stoptyping

which is more efficient.

\stopsection

\stopchapter

\stopcomponent