2021-01-17 21:42:00

3 files changed, 154 insertions, 96 deletions

diff --git a/doc/context/documents/general/manuals/luametatex.pdf b/doc/context/documents/general/manuals/luametatex.pdf
index 635e56000..f5ece94d6 100644
--- a/doc/context/documents/general/manuals/luametatex.pdf
+++ b/doc/context/documents/general/manuals/luametatex.pdf
diff --git a/doc/context/sources/general/manuals/followingup/followingup-fonts.tex b/doc/context/sources/general/manuals/followingup/followingup-fonts.tex
index 2f7320b12..6bc10e929 100644
--- a/doc/context/sources/general/manuals/followingup/followingup-fonts.tex
+++ b/doc/context/sources/general/manuals/followingup/followingup-fonts.tex
@@ -23,7 +23,24 @@ Fonts come in sizes, and for instance Latin Modern has quite a few variants wher
 the shapes are adapted to the size. This means that when you need a 9pt regular
 shape alongside a 12pt one, two fonts have to be loaded. This is quite visible in
 math where we have three related sizes: text, script and scriptscript, grouped in
-so called families.
+so called families. When we scale the digit~2 to the same height you will notice
+that the text, script and scriptscript sizes look different (the last three are
+unscaled):
+
+\startlinecorrection
+\dontleavehmode\scale[frame=on,height=5ex]{$\textstyle         2$}\quad
+\dontleavehmode\scale[frame=on,height=5ex]{$\scriptstyle       2$}\quad
+\dontleavehmode\scale[frame=on,height=5ex]{$\scriptscriptstyle 2$}\quad\quad
+\dontleavehmode\scale[frame=on,height=2ex]{$\textstyle         2$}\quad
+\dontleavehmode\scale[frame=on,height=2ex]{$\scriptstyle       2$}\quad
+\dontleavehmode\scale[frame=on,height=2ex]{$\scriptscriptstyle 2$}\quad\quad
+\dontleavehmode\scale[frame=on,width=4em]{\colored[r=.6,a=1,t=.5]{$\textstyle         2$}}\hskip-4em
+\dontleavehmode\scale[frame=on,width=4em]{\colored[g=.6,a=1,t=.5]{$\scriptstyle       2$}}\hskip-4em
+\dontleavehmode\scale[frame=on,width=4em]{\colored[b=.6,a=1,t=.5]{$\scriptscriptstyle 2$}}\quad\quad
+\dontleavehmode{$\textstyle         2$}\quad
+\dontleavehmode{$\scriptstyle       2$}\quad
+\dontleavehmode{$\scriptscriptstyle 2$}\quad
+\stoplinecorrection
 
 Plenty has been written (in various documents that come with \CONTEXT) about how
 this all works together and how it impacts the design of the system, so here I
@@ -34,62 +51,82 @@ just give a short summary of what a font system has to deal with.
     In a bodyfont setup different sizes (9pt, 10pt, 12pt) can have their own
     specific set of fonts. This can result in quite some definitions that relate
     to the style, like regular, bold, italic, bolditalic, slanted and
-    boldslanted, etc. When possible loading the fonts is delayed.
+    boldslanted, etc. When possible loading the fonts is delayed. But in
+    \CONTEXT\ often the number of actually loaded fonts is not that large.
 \stopitem
 \startitem
     Some font designs have different shapes per bodyfont size. A minor
-    complication is that when one is missing some heuristic best match choice
-    might be needed.
+    complication is that when one is missing some heuristic best|-|match choice
+    might be needed. Okay, in practice only Latin Modern falls into this
+    category. Maybe \OPENTYPE\ variable fonts can be seen this way, but, although
+    we supported that right from the start, I haven't noticed much interest in
+    the \TEX\ community.
 \stopitem
 \startitem
     Within a bodyfont size we distinguish size variants. We can go smaller (x and
     xx), for instance when we use sub- and superscripts in text, or we can go
-    larger, for instance in titling (a, b, c, d, \unknown). Fortunately most of
+    larger, for instance in titles (a, b, c, d, \unknown). Fortunately most of
     the loading of these can be delayed too.
 \stopitem
 \startitem
     When instances are not available, scaling can be used, as happens for
     instance with 11pt in Computer Modern. Actually, this is why in \CONTEXT\ we
     default to 12pt, because the scaled versions didn't look as nice as the
-    others.
+    others (keep in mind that we started in the age of bitmaps).
 \stopitem
 \startitem
     Special features, like smallcaps or oldstyle numerals, can demand their own
     definitions. More loading and automatic definitions can be triggered by sizes
-    needed in for instance scripts and titling.
+    needed in for instance scripts and titles.
 \stopitem
 \startitem
     A document can have a mixed setup, that is: use different font designs in one
     document, so some kind of namespace subsystem is needed.
 \stopitem
 \startitem
-    In traditional eight bit engine the dependency of hyphenation on the encoding
-    of a font can result in the necessity to load a font multiple times in
-    different encodings, something that depends on the language mix used.
+    In an eight bit font world, we not only have text fonts but also collections
+    of symbols, and even in math there are additional symbol collections. In
+    \OPENTYPE\ symbols end up in text fonts, but there we have tons of emoji's
+    and color fonts. All has to be dealt with in an integrated way. And we're
+    not even talking of virtual fonts, (runtime) \METAPOST\ generated fonts, and
+    so on.
+\stopitem
+\startitem
+    In traditional eight bit engines the dependency of hyphenation on the
+    encoding of a font can result in the necessity to load a font multiple times
+    in different encodings, something that depends on the language mix used.
+    Interesting is that coming up with an European encoding (covering most Latin
+    languages) was not that hard, especially when one keeps in mind that many
+    eight bit encodings waste slots on seldom used symbols, but by that time
+    \OPENTYPE\ and and \UNICODE\ input started to dominate.
 \stopitem
 \startitem
     In the more modern \OPENTYPE\ fonts combinations of features can demand
-    additional instances (one can thing of language|/|script combinations,
+    additional instances (one can think of language|/|script combinations,
     substitutions in base mode, special effects like boldening, color fonts,
     etc.).
 \stopitem
 \startitem
-    Math is complicated by the fact that alphabets come from different fonts
-    (sometimes a font has several alphabets which means that some mapping can be
-    needed). Operating on the size, shape, encoding and style axes puts some
-    demands on the font system. Add to this (often) partial (due to lack of
-    fonts) bold support and it gets even more complicated.
+    Math is complicated by the fact that in traditional \TEX\ alphabets come from
+    different fonts which is why we have many so called families; sometimes a
+    font has several alphabets which means that some mapping can be needed.
+    Operating on the size, shape, encoding and style axes puts some demands on
+    the font system. Add to this (often) partial (due to lack of fonts) bold
+    support and it gets even more complicated. In \OPENTYPE\ all the alphabets
+    come from one font.
 \stopitem
 \startitem
     There is additional math auto|-|definition and loading code for the sizes
-    used in text scripts and titling.
+    used in text scripts and titles.
 \stopitem
 \stopitemize
 
 All this has resulted in a pretty complex subsystem. Although going \OPENTYPE\
 (and emulated \OPENTYPE\ with \TYPEONE\ fonts as we do in \MKIV) removes some
 complications, like encodings, it does also add complexity because of the many
-possible font features, either or not dependent on script and language.
+possible font features, either or not dependent on script and language. Text as
+well as math got simpler at the \TEX\ end (but that is compensated by quite a bit
+\LUA\ code that deals with new features).
 
 Also, in order to let a font subsystem not impact performance to much, let alone
 extensive memory usage, the \CONTEXT\ font subsystem is rather optimized. The
@@ -102,26 +139,27 @@ always been rather fast.
 \startsection[title={Reality}]
 
 In \MKIV\ and therefore also in \LMTX\ more font magic happens. The initial node
-lists that makes up a box or paragraph can get manipulated in several ways and
-often fonts are involved. The mentioned font features can be defines static (as
-part of the definition) or dynamic (resolved on the spot at the cost of some
-overhead). Characters can be remapped, fonts can be replaced. The math subsystem
-in \MKIV\ was different right from the start: we use a limited number of families
-(regular, bold, l2r and r2l), and stay abstract till the moment we need to deal
-with the specific alphabets. But still, in \MKIV, we have the families with three
-fonts.
+lists that make up a box or paragraph can get manipulated in several ways and
+often fonts are involved. The font features (smallcaps, oldstyle, alternates,
+etc.) can be defined static (as part of the definition) or dynamic (resolved on
+the spot at the cost of some overhead). Characters can be remapped, fonts can be
+replaced. The math subsystem in \MKIV\ was different right from the start: we use
+a limited number of families (regular, bold, l2r and r2l), and stay abstract till
+the moment we need to deal with the specific alphabets. But still, in \MKIV, we
+have the families with three fonts.
 
 In the \LUAMETATEX\ manual we show some math magic and we do so for different
-fonts. As a side effect, we set up half a dozen bodyfont collections. Even with
-delayed and shared font loading, we end up with 158 instances but quite a bit of
-them are math fonts, at least six per bodyfont size: regular and bold (boldened)
-text, script and scriptscript. Of course most are just copies with different
-scaling that reuse already loaded resources.
+fonts. As a side effect, we set up half a dozen bodyfont collections: Lucida,
+Pagella, Latin Modern, Dejavu, the math standard Cambria, etc. Even with delayed
+and shared font loading, we end up with 158 instances but quite a bit of them are
+math fonts, at least six per bodyfont size: regular and bold (boldened) text,
+script and scriptscript. Of course most are just copies with different scaling
+that reuse already loaded resources. In the final\PDF\ we have 21 subset fonts.
 
 If we look at the math fonts that we use today, there is however quite some
 overlap. It starts with a text font. From that script and scriptscript variants
 are derived, but often these variants use many text size related shapes too. Some
-shapes get alternatives (from the \type {ssty} feature), and the whole clone get
+shapes get alternatives (from the \type {ssty} feature), and the whole clone gets
 scaled. But, much of the logic of for instance extensibles is the same.
 
 A similar situation happens with for instance large \CJK\ fonts: there are hardly
@@ -130,19 +168,25 @@ dimensions, and these fonts can be real huge!
 
 Actually, when we talk about features, in many cases in \CONTEXT\ you don't
 define them as part of the font. For instance small caps can best be triggered by
-using a dynamic feature: applied to a specific stretch of text. When the font
-handler does its work there are actually four cases: no features get applied,
-something that for instance happens with most monospaced fonts, base mode is
-used, which means that the \TEX\ machinery takes care of constructing ligatures
-and injecting kerns, and node mode, where \LUA\ handles the features. The fourth
-case is a special case of node mode where a different features set is applied.
-\footnote {We also have so called plug mode where an external rendered can do the
-work but that one is only around for some experiments during Idris' font
-development.} At the cost of some extra overhead (for each node mode run) dynamic
-features are quite powerful and save quite some memory and definitions. The
-overhead comes from much more testing regarding the font we deal with because
-suddenly the same font can demand different treatments, depending on what dynamic
-features is active.
+using a dynamic feature: applied to a specific stretch of text. In fact, often
+features like superiors of fractions only work well on characters that fit the
+bill and produce weird side effects otherwise (a matter of design completeness).
+When the font handler does its work there are actually four cases: no features
+get applied, something that for instance happens with most mono spaced fonts,
+base mode is used, which means that the \TEX\ machinery takes care of
+constructing ligatures and injecting kerns, and node mode, where \LUA\ handles
+the features. The fourth case is a special case of node mode where a different
+feature set is applied. \footnote {We also have so called plug mode where an
+external renderer can do the work but that one is only around for some
+experiments during Idris Hamid's font development.} At the cost of some extra
+overhead (for each node mode run) dynamic features are quite powerful and save
+quite some memory and definitions. \footnote {The generic font handler that is
+derived from the \CONTEXT\ one doesn't implement this so it runs a little
+faster.} The overhead comes from much more testing regarding the font we deal
+with because suddenly the same font can demand different treatments, depending on
+what dynamic features are active. \footnote {Originally this model was introduced
+for a dynamic paragraph optimization subsystem for Arabic but no one really uses
+it because there there are no suitable fonts.}
 
 Although the font handling is responsible for much of the time spent in \LUA, it is
 still reasonable given what has to be done. Actually, because we have an extensible
@@ -183,8 +227,8 @@ e{\glyphxscale 2500 ffi}cient
 
 So, in order to deal with this kind of scaling, we now operate not only on the
 font (id) and dynamic feature axes, but also on the scales of which we have three
-variants: glyph scale, glyph xscale and glyph y scale). There is actually also a
-state dimension but we leave that for now. Think of flagging glyphs as initial or
+variants: glyph scale, glyph xscale and glyph yscale). There is actually also a
+state dimension but we leave that for now; think of flagging glyphs as initial or
 final. This brings the number of axis to six. It is important to stress that in
 these examples the same font instance is used!
 
@@ -197,22 +241,22 @@ go together but for now I see no real gain in that.
 
 \startsection[title={Math}]
 
-Given what is written in the previous sections, a logical question is if we can
-apply scaling to math and the answer is \quotation {yes, we can}. But we actually
-go a bit further and that is partly due to some other properties of the engine.
+Given what is written in the previous sections, a logical question would be
+\quotation {Can we apply scaling to math?} and the answer is \quotation {Yes, we
+can!}. But we actually go a bit further and that is partly due to some other
+properties of the engine.
 
 From \PDFTEX\ the \LUATEX\ engines inherited character protrusion and glyph
 expansions, aka hz. However, where in \PDFTEX\ copies of the font are made that
 carry the expanded dimensions, in \LUATEX\ at some point this was replaced by an
 expansion field in the glyph and kern nodes. So, instead of changing the font id
 of expanded glyphs, the same id is used but with the applied expansion factor set
-in the glyph. A side effect was that in places where dimensions are needed, calls
-to functions that calculate the expanded widths on request (as these can change
-during linebreak calculations) in combination with accessing font dimensions
-directly, we now always use accessor functions. This level of abstraction is even
-more present in \LUAMETATEX. This means that we have an uniform interface to
-fonts and as a side effect scaling is to be dealt with in only a few places in
-the code.
+in the glyph. A side effect was that in places where dimensions are needed, we
+call functions that calculate the expanded widths on request (as these can
+change during linebreak calculations) in combination with accessing font
+dimensions directly. This level of abstraction is even more present in
+\LUAMETATEX. This means that we have an uniform interface to fonts and as a side
+effect scaling is to be dealt with in only a few places in the code.
 
 Now, in math we have a few more complications. First of all, we have three sizes
 to consider and we also have lots of parameters that depend on the size. But, as
@@ -236,9 +280,9 @@ the math engine needs a specific character at a given size (text, script,
 scriptscript) we just follow that chain.
 
 In an \OPENTYPE\ math font the script and scriptscript sizes are specified as
-percentages of the text size. So, when the dimensions of a glyph are needed, we
-just scale on the fly. Again this adds some overhead but I'm pretty sure that no
-user will notice this.
+percentages of the text size. When the dimensions of a glyph are needed, we just
+scale on the fly. Again this adds some overhead but I'm pretty sure that no user
+will notice this.
 
 So, to summarize: if we need a character at scriptscript size, we access the text
 size glyph, check for a pointer to a script size, go there, and again check for a
@@ -246,8 +290,11 @@ smaller size. We just use what fits the bill. And, when we need dimensions we
 just scale. In order to scale we need the relative size, so we need to set that
 up when we load the font. Because in \CONTEXT\ we also can assemble a virtual
 \OPENTYPE\ font from \TYPEONE\ fonts, it was actually that (old) compatibility
-feature that took most time to adapt, not so much because it is complicates but
-because in \LMTX\ we have to bypass some advanced loading mechanisms.
+feature, the one that implements \TYPEONE\ bases \OPENTYPE\ math, that took most
+time to adapt, not so much because it is complicates but because in \LMTX\ we
+have to bypass some advanced loading mechanisms. Because we can scale in two
+dimensions the many (font related) math parameters also need to be dealt with
+accordingly.
 
 The end result is that for math we now only need to define two fonts per bodyfont
 setup: regular and bold at the natural scale (normally 10pt) and we share these
@@ -283,12 +330,12 @@ manual we get these stats:
 \stoptyping
 
 So, we win on all fronts when we use this glyph scale mechanism. The magic
-primitive that deals with this is \type {\glyphscale} which accepts a number,
-where \type {1200} and \type {1.2} both mean scaling to 20\percent\ more than
-normal. But one can best not use this primitive directly.
+primitive that deals with this is \type {\glyphscale} primitive that accepts a
+number, where \type {1200} and \type {1.2} both mean scaling to 20\percent\ more
+than normal. But one can best not use this primitive directly.
 
 A specific font can be defined using the \type {\definefont} command. In \LMTX\ a
-regular scaler can be followed by two scale factors. The nest example
+regular scaler can be followed by two scale factors. The next example
 demonstrates this:
 
 \startbuffer
@@ -311,11 +358,10 @@ only shows the two scales but also introduces the offset.
 
 \getbuffer
 
-Here is another one:
+In compact mode this is one font. Here is another example:
 
 \startbuffer
-\definetweakedfont[squeezed] [xscale=0.9]
-\definetweakedfont[squeezed] [xscale=0.9]
+\definetweakedfont[squeezed][xscale=0.9]
 
 \startlines
 $a = b^2 + \sqrt{c}$
@@ -330,13 +376,13 @@ $a = b^2 + \sqrt{c}$
 Watch this:
 
 \startbuffer
-\startcombination[4*1]
+\startcombination[3*1]
     {\bTABLE
         \bTR \bTD foo \eTD \bTD[style=\squeezed] $x = 1$ \eTD \eTR
         \bTR \bTD oof \eTD \bTD[style=\squeezed] $x = 2$ \eTD \eTR
      \eTABLE}
     {local}
-    {\bTABLE[style=squeezed]
+    {\bTABLE[style=\squeezed]
         \bTR \bTD $x = 1$ \eTD \bTD  $x = 3$ \eTD \eTR
         \bTR \bTD $x = 2$ \eTD \bTD  $x = 4$ \eTD \eTR
      \eTABLE}
@@ -404,20 +450,28 @@ gives:
 \stoplinecorrection
 
 Glyphs can have offsets and these are used for implementing \OPENTYPE\ features.
-However, they are also available at the \TEX\ end. Take this
+However, they are also available at the \TEX\ end. Take this example where we use
+the new \type {\glyph} primitive (a variant of \type {\char} that takes
+keywords):
 
 \startbuffer
 \ruledhbox{
-    \ruledhbox{\glyph yoffset 1ex options 0 123}
+    \ruledhbox{\glyph yoffset 1ex options 0 123} % left curly brace
     \ruledhbox{\glyph xoffset .5em yoffset 1ex options "C0 123}
-    \ruledhbox{oeps{\glyphyoffset 1ex \glyphxscale 800 \glyphyscale\glyphxscale oeps}oeps}
+    \ruledhbox{oeps{\glyphyoffset 1ex \glyphxscale 800
+                    \glyphyscale\glyphxscale oeps}oeps}
 }
 \stopbuffer
 
-This example demonstrates that the \type {\glyph} primitive takes keywords: \type
-{xoffset}, \type {yoffset}, \type {xscale}, \type {yscale}, \type {options},
-\type {font} and \type {id} where the last two take a font identifier or font id
-(an positive number).
+\typebuffer \getbuffer
+
+This example demonstrates that the \type {\glyph} primitive takes quite some
+keywords: \type {xoffset}, \type {yoffset}, \type {xscale}, \type {yscale}, \type
+{left}, \type {right}, \type {raise}, \type {options}, \type {font} and \type {id}
+where the last two take a font identifier or font id (an positive number). For
+this article it's enough to know that the option indicates that glyph dimension
+should include the offset. In a moment we will see an alternative that doesn't
+need that.
 
 \startbuffer
 \samplefile{jojomayer}
@@ -441,7 +495,7 @@ To quote Jojo Mayer:
 \stopnarrower
 
 Keep in mind that this can interfere badly with font feature processing which also
-used offsets. It might work out okay, or not.
+used offsets. It might often work out okay vertically, but less well horizontally.
 
 The scales, as mentioned, works with pseudo scales but that is sometimes a bit
 cumbersome. This is why a special \type {\numericscale} primitive has been
@@ -477,8 +531,8 @@ the slanted version could use some more left shifting of the E.
 
 \typebuffer[definition,example]
 
-This gives is the \TEX\ logos (but of course we normally use the real ones
-instead).
+This gives the \TEX\ logos but of course we normally use the more official
+definitions instead.
 
 \startnarrower
     \darkred \getbuffer[definition,example]
@@ -490,7 +544,7 @@ suboptimal. Here is a better alternative:
 
 \startbuffer[definition]
 \def\UnKernedTeX
-  {T\glyph left .2ex raise -.4ex `E\glyph left .4ex `X\relax}
+  {T\glyph left .2ex raise -.4ex `E\glyph left .2ex `X\relax}
 \stopbuffer
 
 \typebuffer[definition]
@@ -501,7 +555,7 @@ The result is the same:
     \darkgreen \getbuffer[definition,example]
 \stopnarrower
 
-But anyway: don't over|-|do it. We could deal with such cases for decades without
+But anyway: don't over|-|do it. We have dealt with such cases for decades without
 these fancy new features. The next example show margins in action:
 
 \startlinecorrection
@@ -565,23 +619,25 @@ dimensions of a glyph.
 
 Discussing the implementation in the engine makes no sense here, also because
 details might change. However, it is good to know that quite some properties
-travel with the glyph nodes, for instance the scales and offsets. The dimensions
-(width, height and depth) are not stored in the glyph node but calculated from
-the font, scales and optionally the offsets and expansion factor. One problem is
-that the more clever (and nice) solutions we cook up, the more it might impact
-performance. So, I will delay some experiments till I have a more powerful
-machine.
+travel with the glyph nodes, for instance the scales, margins, offsets, language,
+script and state properties, control over kerning, ligaturing, expansion and
+protrusion, etc. The dimensions (width, height and depth) are not stored in the
+glyph node but calculated from the font, scales and optionally the offsets and
+expansion factor. One problem is that the more clever (and nice) solutions we
+cook up, the more it might impact performance. So, I will delay some experiments
+till I have a more powerful machine.
 
 One reason for {\em not} storing the dimensions in a glyph node is that we often
 copy those nodes or change character fields in the font handler and we definitely
 don't want the wrong dimensions there. At that moment, offsets and margin fields
-don't reflect features yet, so copying them is no big deal. However, dimensions
-are rather character bound so every time a character is set, we also would have
-to set the dimensions. Even worse, when we can set them, the question arises if
-they actually were already set explicitly. So, this is a can of worms we're not
-going to open: the basic width, height and depth of the glyph as specified in the
-font is used and combines with actual dimensions (likely already scaled according
-the glyph scales) in offset and margin fields.
+don't reflect features yet, so copying them is no big deal because at that moment
+these are still zero. However, dimensions are rather character bound so every
+time a character is set, we also would have to set the dimensions. Even worse,
+when we can set them, the question arises if they actually were already set
+explicitly. So, this is a can of worms we're not going to open: the basic width,
+height and depth of the glyph as specified in the font is used and combined with
+actual dimensions (likely already scaled according the glyph scales) in offset
+and margin fields.
 
 Now, I have to admit that especially playing with using margins to glyphs instead
 of font kerns is more of an experiment to see what the consequences are than a
diff --git a/doc/context/sources/general/manuals/luatex/luatex-modifications.tex b/doc/context/sources/general/manuals/luatex/luatex-modifications.tex
index 747945f55..bf4233165 100644
--- a/doc/context/sources/general/manuals/luatex/luatex-modifications.tex
+++ b/doc/context/sources/general/manuals/luatex/luatex-modifications.tex
@@ -615,6 +615,8 @@ primitives (for as far their functionality is still around) you now can do this:
 \protected\def\pdfmapline             {\pdfextension mapline }
 \protected\def\pdftrailer             {\pdfextension trailer }
 \protected\def\pdfglyphtounicode      {\pdfextension glyphtounicode }
+\protected\def\pdfrunninglinkoff      {\pdfextension linkstate 1 }
+\protected\def\pdfrunninglinkon       {\pdfextension linkstate 0 }
 \stoptyping
 
 The introspective primitives can be defined as: