2021-07-06 18:47:00

7 files changed, 180 insertions, 49 deletions

diff --git a/doc/context/documents/general/manuals/luametatex.pdf b/doc/context/documents/general/manuals/luametatex.pdf
index a741c40f6..aa07db6da 100644
--- a/doc/context/documents/general/manuals/luametatex.pdf
+++ b/doc/context/documents/general/manuals/luametatex.pdf
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex b/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex
index 4b4156d5d..c3cdc16a9 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex
@@ -111,6 +111,25 @@ The \LUAMETATEX\ version number starts at~2 in order to prevent a clash with
 \LUATEX, and the version commands are the same. This is a way to indicate that
 these projects are related.
 
+The \type {status} library also provides some information including what we get
+with the three mentioned primitives:
+
+\starttabulate[|l|l|]
+\DB field                   \BC value                               \NC \NR
+\TB
+\NC \type {filename}        \NC \cldcontext{status.filename}        \NC \NR
+\NC \type {banner}          \NC \cldcontext{status.banner}          \NC \NR
+\NC \type {luatex_engine}   \NC \cldcontext{status.luatex_engine}   \NC \NR
+\NC \type {luatex_version}  \NC \cldcontext{status.luatex_version}  \NC \NR
+\NC \type {luatex_revision} \NC \cldcontext{status.luatex_revision} \NC \NR
+\NC \type {luatex_verbose}  \NC \cldcontext{status.luatex_verbose}  \NC \NR
+\NC \type {copyright}       \NC \cldcontext{status.copyright}       \NC \NR
+\NC \type {development_id}  \NC \cldcontext{status.development_id}  \NC \NR
+\NC \type {format_id}       \NC \cldcontext{status.format_id}       \NC \NR
+\NC \type {used_compiler}   \NC \cldcontext{status.used_compiler}   \NC \NR
+\LL
+\stoptabulate
+
 \stopsubsection
 
 \stopsection
@@ -413,10 +432,18 @@ mode, but in \LUAMETATEX\ there is no error message and the box the height and
 depth are equally divided. Of course in text mode there is no math axis related
 offset applied.
 
-It is possible to change or add to the attributes assigned to a box:
+It is possible to change or add to the attributes assigned to a box with \lpr
+{boxattribute}:
 
 \starttyping
-\boxattr 0 123 456
+\boxattribute 0 123 456
+\stoptyping
+
+You can set attributes of the current paragraph specification node with \lpr
+{parattribute}:
+
+\starttyping
+\parattribute 123 456
 \stoptyping
 
 \stopsubsection
@@ -1396,7 +1423,65 @@ the acceptable range is from 0 to 65535.
 
 Both rule drawing commands take an optional \type {xoffset} and \type {yoffset}
 parameter. The displacement is virtual and not taken into account when the
-dimensions are calculated.
+dimensions are calculated. A rule is specified in the usual way:
+
+\obeydepth
+
+\startbuffer
+\blue \vrule
+    height 2ex depth 1ex width 10cm
+\relax
+\stopbuffer
+
+\startlinecorrection
+\getbuffer
+\stoplinecorrection
+
+There is however a catch. The keyword scanners in \LUAMETATEX\ are implemented
+slightly different. When \TEX\ scans a keyword it will (case insensitive) scan
+for a whole keyword. So, it scans for \type {height} and when it doesn't find it
+it will scan for \type {depth} etc. When it does find a keyword in this case it
+expects a dimension next. When that criterium is not met it will issue an error
+message.
+
+In order to avoid look ahead failures like that it is recommended to end the
+specification with \type {\relax}. A glue specification is an other example where
+a \type {\relax} makes sense when look ahead issues are expected and actually
+there in traditional scanning the order of keywords can also matter. In any case,
+when no valid keyword is seen the characters scanned so far are pushed back in
+the input.
+
+The main reason for using an adapted scanner is that we always permit repetition
+(consistency) and accept an arbitrary order. Because we have more keywords to
+process the scanner quits at a partial failure. This prevents some push back and
+also gives an earlier warning. Interesting is that some \CONTEXT\ users ran into
+error messages due to a missing \type {\relax} and found out that their style has
+a potential flaw with respect to look ahead. One can be lucky for years.
+
+Back to rules, there are some extra keywords, two deal with an offset, and four
+provide margins. The margins are a bit special because \type {left} and \type
+{top} are the same as are \type {right} and \type {bottom}. They influence the
+edges and these depend on it being a horizontal or vertical rule.
+
+\obeydepth
+
+\startbuffer
+\blue \vrule
+    height 2.0ex depth 1.0ex width 10cm
+\relax
+\white \vrule
+    height 1.0ex depth 0.5ex width  9cm
+    xoffset -9.5cm yoffset .25ex
+\relax
+\blue \vrule
+    height .5ex depth 0.25ex width  8cm
+    xoffset -18cm yoffset .375ex top 1pt
+\relax
+\stopbuffer
+
+\startlinecorrection
+\getbuffer
+\stoplinecorrection
 
 Two new primitives were introduced: \lpr {nohrule} and \lpr {novrule}. These can
 be used to reserve space. This is often more efficient than creating an empty box
@@ -1413,6 +1498,11 @@ The \prm {vsplit} primitive has to be followed by a specification of the require
 height. As alternative for the \type {to} keyword you can use \type {upto} to get
 a split of the given size but result has the natural dimensions then.
 
+\starttyping
+\vsplit 123 to   10cm % final box has the required height
+\vsplit 123 upto 10cm % final box has its natural height
+\stoptyping
+
 \stopsubsection
 
 \startsubsection[title={Images and reused box objects},reference=sec:imagesandforms]
@@ -2226,6 +2316,8 @@ the noalign primitive. Of course we can also apply the \prm {global} and \lpr
 \NC \type {enforced}   \NC all is permitted (but only in zero mode or ini mode) \NC \NR
 \NC \type {aliased}    \NC the macro gets the same flags as the original \NC \NR
 \HL
+\NC \type {untraced}   \NC the macro gets a different treatment in tracing \NC \NR
+\HL
 \stoptabulate
 
 The not yet discussed \lpr {instance} is just a flag with no special meaning
@@ -2267,6 +2359,10 @@ modules or macros of unpredictable origin will probably suffer from warnings and
 errors when de mode is set to non zero. In \CONTEXT\ we're okay unless of course
 users redefine instances but there a warning or error is kind of welcome.
 
+There is an extra prefix \lpr {untraced} that will suppress the meaning when
+tracing so that the macro looks more like a primitive. It is still somewhat
+experimental so what gets displayed might change.
+
 \stopsection
 
 \startsection[title={Constants with \lpr{integerdef} and \lpr {dimendef}}]
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-fonts.tex b/doc/context/sources/general/manuals/luametatex/luametatex-fonts.tex
index 9a09f705c..573511f32 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-fonts.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-fonts.tex
@@ -56,6 +56,9 @@ the font level:
                                            character entries for math|-|specific keys \NC \NR
 \NC \type {oldmath}     \NC boolean    \NC this key flags a font as representing an old school \TEX\
                                            math font and disables the \OPENTYPE\ code path \NC \NR
+\NC \type {mathcontrol} \NC bitset     \NC this controls various code paths in the math engine,
+                                           like enforcing the traditional code path \NC \NR
+\NC \type {textcontrol} \NC bitset     \NC this controls various code paths in the text engine \NC \NR
 \LL
 \stoptabulate
 
@@ -229,6 +232,42 @@ The default value is~0, and can be left out. That signifies a \quote {normal}
 ligature where the ligature replaces both original glyphs. In this table the~\type {|}
 indicates the final insertion point.
 
+The \type {mathcontrol} bitset is mostly there for experimental purposes. Because
+there is inconsistency in the \OPENTYPE\ math fonts with respect to for instance
+glyph dimensions, it is possible to force the traditional code path. We just mention
+the possible flags:
+
+\startluacode
+    context.starttabulate { "|||" }
+    context.DB() context("value") context.BC() context("effect") context.NC() context.NR()
+    context.TB()
+    for k, v in table.sortedhash(tex.getmathcontrolvalues()) do
+        context.NC() context("0x%04X",k) context.NC() context(v) context.NC() context.NR()
+    end
+    context.LL()
+    context.stoptabulate()
+\stopluacode
+
+The \type {textcontrol} field is used to control some aspects of text processing.
+More options might be added in the future.
+
+\startluacode
+    context.starttabulate { "|||" }
+    context.DB() context("value") context.BC() context("effect") context.NC() context.NR()
+    context.TB()
+    for k, v in table.sortedhash(tex.gettextcontrolvalues()) do
+        context.NC() context("0x%04X",k) context.NC() context(v) context.NC() context.NR()
+    end
+    context.LL()
+    context.stoptabulate()
+\stopluacode
+
+In \CONTEXT\ these are interfaced via pseudo features. The math control flags of
+a font can be overloaded by \lpr {mathcontrolmode} on the spot and the set
+controls of a font can be queried by \lpr {fontmathcontrol}. The text control
+flags in a font always win over the ones set by other parameters, like \lpr
+{hyphenationmode}. They can be queried with \lpr {fonttextcontrol}.
+
 \stopsection
 
 \startsection[reference=virtualfonts,title={Virtual fonts}]
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-introduction.tex b/doc/context/sources/general/manuals/luametatex/luametatex-introduction.tex
index f78b9f3e7..edf1bdd78 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-introduction.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-introduction.tex
@@ -29,7 +29,7 @@ journals, but all are in the \CONTEXT\ distribution. I did the same with the
 development of \LUAMETATEX.
 
 The \LUAMETATEX\ engine is, as said, a follow up on \LUATEX. Just as we have
-\CONTEXT\ \MKII\ for \PDFTEX\ and \XETEX, we have \MKIV\ for \LUATEX. For
+\CONTEXT\ \MKII\ for \PDFTEX\ and \XETEX, we have \MKIV\ for \LUATEX\ so for
 \LUAMETATEX\ we have yet another version of \CONTEXT: \LMTX. By freezing \MKII,
 and at some point freezing \MKIV, we can move on as we like, but we try to remain
 downward compatible where possible, something that the user interface makes
@@ -56,6 +56,18 @@ been compensated by (hopefully) harmless optimization of code in the engine and
 some more core functionality. In the process system dependencies have been
 minimalized.
 
+One side effect of opening up is that what normally is hidden gets exposed and
+this is also true for all kind of codes that are used internally to distinguish
+states and properties of commands, tokens, nodes and more. Especially during
+development these can change but the good news is that they can be queried so on
+can write in code independent ways (in \LUATEX\ node id's are an example). That
+also means more interface related commands, so again lean and mean is not
+applicable here, especially because the detailed control over the text, math,
+font and language subsystems also results in additional commands to query their
+state. And, as the \METAPOST\ got extended, that subsystem is on the one hand
+leaner and meaner because backend code was dropped but on the other hand got a
+larger code base due to opening up the scanner and adding a feedback mechanism.
+
 This manual started as an adaptation of the \LUATEX\ manual and therefore looks
 similar. Some chapters are removed, others were added and the rest has been (and
 will be further) adapted. It also discusses the (main) differences. Some of the
@@ -64,7 +76,9 @@ new primitives or functions that show up in \LUAMETATEX\ might show up in
 this manual as reference for \LUATEX ! As long as we're experimenting we can
 change things at will but as we keep \CONTEXT\ \LMTX\ synchronized users normally
 won't notice this. Often you can find examples of usage in \CONTEXT\ related
-documents and the source code so that serves a reference too.
+documents and the source code so that serves a reference too. More detailed
+explanations can be found in documents in the \CONTEXT\ distribution, if only
+because there we can present features in the perspective of useability.
 
 For \CONTEXT\ users the \LUAMETATEX\ engine will become the default. As
 mentioned, the \CONTEXT\ variant for this engine is tagged \LMTX. The pair can be
@@ -91,16 +105,6 @@ in, so if you want to complain about \LUAMETATEX, don't bother me. Of course, if
 you really need professional support with these engines (or \TEX\ in general),
 you can always consider contacting the developers.
 
-% And yes, I'm really fed up with receiving mails or seeing comments where there's
-% this 'always need to be present' negative remark (nagging) about the program,
-% documentation, development, support, etc. present, probably to put the writer on
-% a higher stand, or maybe to compensate some other personal shortcoming ... who
-% knows. This 'I need to make my stupid point' behaviour seems to come with the
-% internet and it also seems to increase, but that doesn't mean that I want to deal
-% with those unpleasant people for the sake of the larger "tex good". Therefore, I'm
-% quite happy in the nearly always positive and motivating ConTeXt bubble. It's also
-% why I (start) avoid(ing) certain mailing lists and don't really follow forums.
-
 \blank[big]
 
 Hans Hagen
@@ -117,35 +121,6 @@ Hans Hagen
 \NC \LUATEX\ Team \EQ Hans Hagen, Hartmut Henkel, Taco Hoekwater, Luigi Scarso \NC \NR
 \stoptabulate
 
-\vfilll
-
-{\bf remark:} \LUAMETATEX\ development is mostly done by Hans Hagen and in
-adapting the macros to the new features Wolfgang Schuster, who knows the code
-inside||out is a instrumental. In the initial phase Alan Braslau, who love
-playing with the three languages did extensive testing and compiled for several
-platforms. Later Mojca Miklavec make sure all compiles well on the buildbot
-infrastructure. After the first release more users got involved in testing. Many
-thanks for their patience! The development also triggered upgrading of the wiki
-support infrastructure where Taco Hoekwater and Paul Mazaitis have teamed up. So,
-progress all around.
-
-{\bf remark:} When there are non|-|intrusive features that also make sense in
-\LUATEX, these will be applied in the experimental branch first, so that there is
-no interference with the stable release. However, given that in the meantime the
-code bases differs a lot, it is unlikely that much will trickle back. This is no
-real problem as there's not much demand for that anyway.
-
-{\bf remark:} Most \CONTEXT\ users seem always willing to keep up with the latest
-versions which means that \LMTX\ is tested well. We can therefore safely claim
-that end of 2019 the code has become quite stable, although after that in some
-areas there were substantial additions. There are no complaints about performance
-(on my 2013 laptop this manual compiles at 24.5 pps with \LMTX\ versus 20.7 pps
-for the \LUATEX\ manual with \MKIV). After updating some of the \CONTEXT\ code to
-use recently added features by the end of 2020 I could do more than 25.5 pps but
-don't expect spectacular bumps in performance (I need a new machine for that to
-happen). Probably no one notices it, but memory consumption stepwise got reduced
-too. And \unknown\ the binary is still below 3~MegaBytes on all platforms.
-
 \stopchapter
 
 \stopcomponent
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex b/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex
index ed4b269b6..8b707c6dd 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex
@@ -77,7 +77,7 @@ warnings about unrecognized options are suppressed.
 
 Command line processing happens very early on. So early, in fact, that none of
 \TEX's initializations have taken place yet. The \LUA\ libraries that don't deal
-with \TEX\ are initialized early.
+with \TEX\ are initialized rather soon so you have these available.
 
 \LUAMETATEX\ allows some of the command line options to be overridden by reading
 values from the \type {texconfig} table at the end of script execution (see the
@@ -207,11 +207,16 @@ some care you can deal with \UNICODE\ just fine.
 
 There are some more libraries present. These are discussed on a later chapter.
 For instance we embed \type {luasocket} but contrary to \LUATEX\ don't embed the
-related \LUA\ code. An adapted version of \type {luafilesystem} is also included.
+related \LUA\ code. The \type {luafilesystem} module has been replaced by a more
+efficient one that also deals with the \MSWINDOWS\ file and environment
+properties better (\UNICODE\ support in \MSWINDOWS\ dates from before \UTF8
+became dominant so we need to deal with wide \UNICODE16).
+
 There are more extensive math libraries and there are libraries that deal with
-encryption and compression. At some point some of these might become so called
-optional libraries (read: the handful that we provide interfaces for but that get
-loaded on demand).
+encryption and compression. There are also some optional libraries that we do
+interface but that are loaded on demand. The interfaces are as minimal as can be
+because we so much in \LUA, which also means that one can tune behaviour to
+usage better.
 
 \stopsection
 
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex b/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex
index 52b73978c..f720b8f6f 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex
@@ -483,6 +483,21 @@ that over time system dependencies have been handles with \TEX\ change files, th
 \WEBC\ infrastructure, \KPSE\ features, compilation variables and flags, etc. In
 \LUAMETATEX\ we try to minimize all that.
 
+When it became unavoidable that we output more detail, it also became clear that
+it made no sense to stay log and trace compatible. Some is controlled by
+parameters in order to stay close the original, but \CONTEXT\ is configured such
+that we benefit from the new possibilities. Examples are that in addition to
+\type {\meaning} we have \type {\meaningfull} that also exposes macro properties,
+and \type {\meaningless} that only exposes the body. The \type {\untraced} prefix
+will suppress some in the log, and we set \type {\tracinglevels} to 3 in order to
+get details about the input and grouping level. When there's less shown than
+expected keep in mind that \LUAMETATEX\ has a somewhat optimized saving and
+restoring of meanings so less can happen which is reflected in tracing. When node
+lists are serialized (as with \type {\showbox}) some nodes, like discretionaries
+report more detail. The compact serializer, used for instance to signal overfull
+boxes, also shows a bit more detail with respect to non|-|content nodes. I math
+more is shown if only because we have more control and additional mechanisms.
+
 \stopsubsection
 
 \startsubsection[title=Parsing]
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex.tex b/doc/context/sources/general/manuals/luametatex/luametatex.tex
index 48279e1a8..3f0e7efd1 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex.tex
@@ -128,6 +128,7 @@
     \component luametatex-codes
     \component luametatex-registers
     \component luametatex-statistics
+    \component luametatex-remarks
 \stopbackmatter
 
 \stopdocument