2020-12-04 20:21:00

8 files changed, 164 insertions, 103 deletions

diff --git a/doc/context/documents/general/manuals/luametatex.pdf b/doc/context/documents/general/manuals/luametatex.pdf
index 8c82fe6ee..946f6db26 100644
--- a/doc/context/documents/general/manuals/luametatex.pdf
+++ b/doc/context/documents/general/manuals/luametatex.pdf
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex b/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex
index 5b8f7558a..67556a99a 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex
@@ -1,4 +1,4 @@
-    % language=uk
+% language=uk
 
 \environment luametatex-style
 
@@ -10,7 +10,7 @@ As \LUAMETATEX\ is a leaner and meaner \LUATEX, this chapter will discuss
 what is gone. We start with the primitives that were dropped.
 
 \starttabulate[|l|pl|]
-\NC fonts       \NC \type {\letterspacefont}
+\BC fonts       \NC \type {\letterspacefont}
                     \type {\copyfont}
                     \type {\expandglyphsinfont}
                     \type {\ignoreligaturesinfont}
@@ -18,7 +18,7 @@ what is gone. We start with the primitives that were dropped.
                     \type {\leftghost}
                     \type {\rightghost}
                 \NC \NR
-\NC backend     \NC \type {\dviextension}
+\BC backend     \NC \type {\dviextension}
                     \type {\dvivariable }
                     \type {\dvifeedback}
                     \type {\pdfextension}
@@ -28,14 +28,14 @@ what is gone. We start with the primitives that were dropped.
                     \type {\draftmode}
                     \type {\outputmode}
                 \NC \NR
-\NC dimensions  \NC \type {\pageleftoffset}
+\BC dimensions  \NC \type {\pageleftoffset}
                     \type {\pagerightoffset}
                     \type {\pagetopoffset}
                     \type {\pagebottomoffset}
                     \type {\pageheight}
                     \type {\pagewidth}
                 \NC \NR
-\NC resources   \NC \type {\saveboxresource}
+\BC resources   \NC \type {\saveboxresource}
                     \type {\useboxresource}
                     \type {\lastsavedboxresourceindex}
                     \type {\saveimageresource}
@@ -43,11 +43,11 @@ what is gone. We start with the primitives that were dropped.
                     \type {\lastsavedimageresourceindex}
                     \type {\lastsavedimageresourcepages}
                 \NC \NR
-\NC positioning \NC \type {\savepos}
+\BC positioning \NC \type {\savepos}
                     \type {\lastxpos}
                     \type {\lastypos}
                 \NC \NR
-\NC directions  \NC \type {\textdir}
+\BC directions  \NC \type {\textdir}
                     \type {\linedir}
                     \type {\mathdir}
                     \type {\pardir}
@@ -56,14 +56,14 @@ what is gone. We start with the primitives that were dropped.
                     \type {\pagedirection}
                     \type {\bodydirection}
                 \NC \NR
-\NC randomizer  \NC \type {\randomseed}
+\BC randomizer  \NC \type {\randomseed}
                     \type {\setrandomseed}
                     \type {\normaldeviate}
                     \type {\uniformdeviate}
                 \NC \NR
-\NC utilities   \NC \type {\synctex}
+\BC utilities   \NC \type {\synctex}
                 \NC \NR
-\NC extensions  \NC \type {\latelua}
+\BC extensions  \NC \type {\latelua}
                     \type {\lateluafunction}
                     \type {\openout}
                     \type {\write}
@@ -74,7 +74,7 @@ what is gone. We start with the primitives that were dropped.
                     \type {\closein}
                     \type {\ifeof}
                 \NC \NR
-\NC control     \NC \type {\suppressfontnotfounderror}
+\BC control     \NC \type {\suppressfontnotfounderror}
                     \type {\suppresslongerror}
                     \type {\suppressprimitiveerror}
                     \type {\suppressmathparerror}
@@ -82,10 +82,10 @@ what is gone. We start with the primitives that were dropped.
                     \type {\suppressoutererror}
                     \type {\mathoption}
                 \NC \NR
-\NC whatever    \NC \type {\primitive}
+\BC whatever    \NC \type {\primitive}
                     \type {\ifprimitive}
                 \NC \NR
-\NC ignored     \NC \type {\long}
+\BC ignored     \NC \type {\long}
                     \type {\outer}
                     \type {\mag}
                 \NC \NR
@@ -100,38 +100,41 @@ and they don't work well in \LUATEX\ anyway. We only have two directions left.
 The primitive related extensions were not that useful and reliable so they have
 been removed. There are some new variants that will be discussed later. The \type
 {\outer} and \type {\long} prefixes are gone as they don't make much sense
-nowadays and them becoming dummies opened the way to something new, again to be
-discussed elsewhere. I don't think that (\CONTEXT) users will notice it. The
-\type {\suppress..} features are now default and can't be changed so related
-primitives are gone.
+nowadays and them becoming dummies opened the way to something new: control
+sequence properties that permit protection against as well as controlled
+overloading of definitions. I don't think that (\CONTEXT) users will notice these
+prefixed being gone. The definition and parsing related \type {\suppress..}
+features are now default and can't be changed so related primitives are gone.
 
 The \type {\shipout} primitive does no ship out but just erases the content of
-the box, if that hasn't happened already in another way.
-
-The extension primitives relate to the backend (when not immediate) and can be
-implemented as part of a backend design using generic whatsits. There is only one
-type of whatsit now. In fact we're now closer to original \TEX\ with respect to
-the extensions.
+the box, if that hasn't happened already in another way. A macropackage should
+implement its own backend and related shipout. Talking of backend, the extension
+primitives that relate to backends can be implemented as part of a backend design
+using generic whatsits. There is only one type of whatsit now. In fact we're now
+closer to original \TEX\ with respect to the extensions.
 
 The \type {img} library has been removed as it's rather bound to the backend. The
 \type {slunicode} library is also gone. There are some helpers in the string
 library that can be used instead and one can write additional \LUA\ code if
-needed. There is no longer a \type {pdf} backend library.
+needed. There is no longer a \type {pdf} backend library but we have an up to
+date \PDF\ parsing library on board.
 
 In the \type {node}, \type {tex} and \type {status} library we no longer have
 helpers and variables that relate to the backend. The \LUAMETATEX\ engine is in
-principle \DVI\ and \PDF\ unaware. There are only generic whatsit nodes that can
-be used for some management related tasks. For instance you can use them to
-implement user nodes. More extensive status information is provided in the
-overhauled status library.
+principle \DVI\ and \PDF\ unaware. There are, as mentioned, only generic whatsit
+nodes that can be used for some management related tasks. For instance you can
+use them to implement user nodes. More extensive status information is provided
+in the overhauled status library.
 
 The margin kern nodes are gone and we now use regular kern nodes for them. As a
 consequence there are two extra subtypes indicating the injected left or right
 kern. The glyph field served no real purpose so there was no reason for a special
 kind of node.
 
-The \KPSE\ library is no longer built|-|in. Because there is no backend, quite
-some file related callbacks could go away. The following file related callbacks
+The \KPSE\ library is no longer built|-|in, but one can use an external \KPSE\
+library, assuming that it is present on the system, because the engine has a so
+called optional library interface to it. Because there is no backend, quite some
+file related callbacks could go away. The following file related callbacks
 remained (till now):
 
 \starttyping
@@ -152,6 +155,13 @@ gets reallocated.
 trace_memory
 \stoptyping
 
+When you use the overload protect mechanisms, a callback can be plugged in to handle
+exceptions:
+
+\starttyping
+handle_overload
+\stoptyping
+
 The (job) management hooks are kept:
 
 \starttyping
@@ -187,7 +197,7 @@ ago in \TEX\ engines by command line options. Talking of options, only a few are
 left. All input goes via \LUA, even the console.
 
 We took our time for reaching a stable state in \LUATEX. Among the reasons is the
-fact that most was experimented with in \CONTEXT. It took many man|-|years to
+fact that most was experimented with in \CONTEXT. It took many years of work to
 decide what to keep and how to do things. Of course there are places when things
 can be improved and it might happen in \LUAMETATEX. Contrary to what is sometimes
 suggested, the \LUATEX|-|\CONTEXT\ \MKIV\ combination (assuming matched versions)
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex b/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex
index f4dbfb15d..057fddde4 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex
@@ -19,7 +19,7 @@ and there will be no \type {luametatex} variants. This is because we consider
 \LUAMETATEX\ to be \LUATEX 2\high{+}.
 
 Contrary to the \LUATEX\ engine \LUAMETATEX\ enables all its primitives. You can
-clone (a selection of) primitives with a different prefix, like:
+clone (a selection of) primitives with a different prefix, like this:
 
 \starttyping
 \directlua { tex.enableprimitives('normal',tex.extraprimitives()) }
@@ -27,8 +27,11 @@ clone (a selection of) primitives with a different prefix, like:
 
 The \type {extraprimitives} function returns the whole list or a subset,
 specified by one or more keywords \type {core}, \type {tex}, \type {etex} or
-\type {luatex}. \footnote {At some point this function might be changed to return
-the whole list always}.
+\type {luatex}. When you clone all primitives you can also do this:
+
+\starttyping
+\directlua { tex.enableprimitives('normal',true) }
+\stoptyping
 
 But be aware that the curly braces may not have the proper \prm {catcode}
 assigned to them at this early time (giving a \quote {Missing number} error), so
@@ -42,23 +45,8 @@ it may be needed to put these assignments before the above line:
 More fine|-|grained primitives control is possible and you can look up the
 details in \in {section} [luaprimitives]. There are only three kinds of
 primitives: \type {tex}, \type {etex} and \type {luatex} but a future version
-might drop this and no longer make that distinction as it no longer serves
-a purpose.
-
-\stopsubsection
-
-\startsubsection[title={Experiments}]
-
-There are a few extensions to the engine regarding the macro machinery. Some are
-already well tested but others are (still) experimental. Although they are likely
-to stay, their exact behaviour might evolve. Because \LUAMETATEX\ is also used
-for experiments, this is not a problem. We can always decide to also add some of
-what is discussed here to \LUATEX, but it will happen with a delay.
-
-There are all kinds of small improvements that might find their way into stock
-\LUATEX: a few more helpers, some cleanup of code, etc. We'll see. In any case,
-if you play with these before they are declared stable, unexpected side effects
-are what you have to accept.
+might drop this and no longer make that distinction as it no longer serves a
+purpose apart from the fact that it reveals some history.
 
 \stopsubsection
 
@@ -231,6 +219,34 @@ Fonts are loaded via \LUA\ and a minimal amount of information is kept at the
 about what a character (or glyph) number represents (a \UNICODE\ or index) as it
 only is interested in dimensions.
 
+In \TEX\ the number of registers is 256 and \ETEX\ bumped that to 32K. One reason
+for a fixed number is that these registers are fast ways to store data and
+therefore are part of the main lookup table (used for data and pointers to data
+as well as save and restore housekeeping). In \LUATEX\ the number was bumped to
+64K but one can argue that less would also do. In order to keep the default
+memory footprint reasonable, in \LUAMETATEX\ the number of languages, fonts and
+marks is limited. The size of some tables can be limited by configuration
+settings, so they can start out small and grow till configured maximum which is
+smaller than the absolute maximum. The following table shows all kind of defaults
+as reported by \typ {status.getconstants()}.
+
+\startluacode
+    context.starttabulate { "|T|r|" }
+    for k, v in table.sortedhash(status.getconstants()) do
+        context.NC() context(k) context.NC() context(v) context.NC() context.NR()
+    end
+    context.stoptabulate()
+\stopluacode
+
+Because we have additional ways to store integers, dimensions and glue, we might
+actually decide to decrease the maximum of the registers: if 64K is not enough,
+and you work around it, then likely 32K might do as well. Also, we have \LUA\ to
+store massive amounts of data. One can argue that saving some 1.5MB memory (when
+we go halfway) is not worth the effort in a time when you have to close a browser
+in order to free the gigabytes it consumes, but there is no reason not to be lean
+and mean: a more conservative approach to start with creates headroom for going
+wild later.
+
 \stopsubsection
 
 \stopsection
@@ -276,6 +292,12 @@ language related information, the expansion factor, etc. Now that we have access
 to these nodes from \LUA\ it makes sense to be able to carry more information
 with a node and this is where attributes kick in.
 
+It is important to keep in mind that there are situations where nodes get created
+in the current context. For instance, when \TEX\ builds a paragraph or page or
+constructs math formulas, it does add nodes and giving these the current
+attributes makes no sense and can even give weird side effects. In these cases,
+the attributes are inherited from neighbouring nodes.
+
 \stopsubsection
 
 \startsubsection[title={Attribute registers}]
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex b/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex
index 803820de1..ed4b269b6 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex
@@ -70,6 +70,20 @@ There are less options than with \LUATEX, because one has to deal with them in
 \LUA\ anyway. There are no options to enter a safer mode or control executing
 programs. This can easily be achieved with a startup \LUA\ script.
 
+Next the initialization script is loaded and executed. From within the script,
+the entire command line is available in the \LUA\ table \type {arg}, beginning
+with \type {arg[0]}, containing the name of the executable. As consequence
+warnings about unrecognized options are suppressed.
+
+Command line processing happens very early on. So early, in fact, that none of
+\TEX's initializations have taken place yet. The \LUA\ libraries that don't deal
+with \TEX\ are initialized early.
+
+\LUAMETATEX\ allows some of the command line options to be overridden by reading
+values from the \type {texconfig} table at the end of script execution (see the
+description of the \type {texconfig} table later on in this document for more
+details on which ones exactly).
+
 The value to use for \prm {jobname} is decided as follows:
 
 \startitemize
@@ -92,20 +106,6 @@ The value to use for \prm {jobname} is decided as follows:
 \stopitem
 \stopitemize
 
-Next the initialization script is loaded and executed. From within the script,
-the entire command line is available in the \LUA\ table \type {arg}, beginning
-with \type {arg[0]}, containing the name of the executable. As consequence
-warnings about unrecognized options are suppressed.
-
-Command line processing happens very early on. So early, in fact, that none of
-\TEX's initializations have taken place yet. The \LUA\ libraries that don't deal
-with \TEX\ are initialized early.
-
-\LUAMETATEX\ allows some of the command line options to be overridden by reading
-values from the \type {texconfig} table at the end of script execution (see the
-description of the \type {texconfig} table later on in this document for more
-details on which ones exactly).
-
 So let's summarize this. The handling of what is called jobname is a bit complex.
 There can be explicit names set on the command line but when not set they can be
 taken from the \type {texconfig} table.
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex b/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex
index ff401a1ba..cc9fd3b8d 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex
@@ -39,9 +39,18 @@ not|-|so|-|expected changes. These are sometimes a side|-|effect of a new
 (conflicting) feature, or, more often than not, a change necessary to clean up
 the internal interfaces. These will also be mentioned.
 
+Again we stress that {\em this is not a \TEX\ manual, nor a tutorial}. If you are
+unfamiliar with \TEX\ first play a little with a macro package, take a look at
+the \TEX\ book, make yourself familiar with the concepts and macro language. That
+will likely take days and not hours. Also, many of the new concepts introduced in
+\LUATEX\ and \LUAMETATEX\ are explained in documents that come with the \CONTEXT\
+distribution, articles and presentations. It doesn't pay of to repeat that here,
+especially not in a time when users often search instead of read from cover to
+cover.
+
 \stopsubsection
 
-\startsubsection[title=Changes from \TEX\ 3.1415926]
+\startsubsection[title=Changes from \TEX\ 3.1415926...]
 
 \topicindex {\TEX}
 
@@ -58,12 +67,14 @@ most still comes from original Knuthian \TEX. But we divert a bit.
     {languages}, \type {fonts}, \type {libraries}, etc. There are some artifacts
     of the conversion to \CCODE, but these got (and get) removed stepwise. The
     documentation, which actually comes from the mix of engines (via so called
-    change files), is kept as much as possible. Of course we want to stay as
-    close as possible to the original so that the documentation of the
+    change files), is a mix of what authors of the engines wove into the source,
+    and most is of course from Don Knuths original. In \LUAMETATEX\ we try to
+    stay as close as possible to the original so that the documentation of the
     fundamentals behind \TEX\ by Don Knuth still applies. However, because we use
     \CCODE, some documentation is a bit off. Also, most global variables are now
     collected in structures, but the original names and level of abstraction were
-    mostly kept. On the other hand, opening up had its impact on the code.
+    mostly kept. On the other hand, opening up had its impact on the code, so
+    that makes some documentation a bit off too. Adapting that all will take time.
 \stopitem
 
 \startitem
@@ -88,7 +99,8 @@ most still comes from original Knuthian \TEX. But we divert a bit.
     stack, input levels, the hash table and table of equivalents, etc. now all
     start out small and are enlarged when needed, where maxima are controlled in
     the usual way. In principle the initial memory footprint is smaller while at
-    the same time we can go real large.
+    the same time we can go real large. Because we have wide memory words some
+    data (arrays) used for housekeeping could be reorganized a bit.
 \stopitem
 
 \startitem
@@ -138,14 +150,6 @@ most still comes from original Knuthian \TEX. But we divert a bit.
     supporting \OPENTYPE\ math.
 \stopitem
 
-\startitem
-    When detailed logging is enabled more detail is output with respect to what
-    nodes are involved. This is a side effect of the core nodes having more
-    detailed subtype information. The benefit of more detail wins from any wish
-    to be byte compatible in the logging. One can always write additional logging
-    in \LUA.
-\stopitem
-
 \stopitemize
 
 \stopsubsection
@@ -445,6 +449,11 @@ more details anyway.
 
 \startsubsection[title=Logging]
 
+When detailed logging is enabled more detail is output with respect to what nodes
+are involved. This is a side effect of the core nodes having more detailed
+subtype information. The benefit of more detail wins from any wish to be byte
+compatible in the logging. One can always write additional logging in \LUA.
+
 The information that goes into the log file can be different from \LUATEX, and
 might even differ a bit more in the future. The main reason is that inside the
 engine we have more granularity, which for instance means that we output subtype
@@ -465,6 +474,34 @@ that over time system dependencies have been handles with \TEX\ change files, th
 
 \stopsubsection
 
+\startsubsection[title=Parsing]
+
+Token parsers have been upgraded for the sake of \LUA, \type {\csname} handling
+has been extended, macro definitions can be more flexible so there code was
+adapted, more conditionals also brought some changes. But we build upon the
+(reorganized) \TEX\ foundation so the basics can definitely be recognized.
+
+Because of interfacing in \LUA\ the internal token and node organization has
+been normalized (read: we cannot cheat because all is kind of visible). On
+the one hand this can come with a performance penalty but that is more than
+compensated by extensions, optimized parsers and such. Still the fact that we
+are \UTF\ based (32 bit) makes the machinery slower than the 8~bit original.
+The reworked \LUAMETATEX\ engine is substantially faster than the \LUATEX\
+predecessor.
+
+The handling of conditionals has been adapted so that we can have flatter
+branches (\type {\orelse} cum suis). This again has some consequences for
+parsing. Because parsing alignments is rather interwoven in general parsing and
+expansion the handling of related primitives has been slightly adapted (also for
+the sake of \LUA\ interfacing) and dealing with \type {\noalign} situations is a
+bit more convenient.
+
+This are just a few of the adaptations and most of this happened stepwise with
+testing in the \CONTEXT\ code base. It will be clear that \LUAMETATEX\ is a quite
+different extension to the original. You're warned.
+
+\stopsubsection
+
 \stopsection
 
 \stopchapter
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-preamble.tex b/doc/context/sources/general/manuals/luametatex/luametatex-preamble.tex
index 1897d0f3e..5a2dae818 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-preamble.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-preamble.tex
@@ -81,6 +81,17 @@ see:
 [par] <=> [hlist] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
 \stoptyping
 
+Actually, the above representation is one view, because in \LUAMETATEX\ we can
+choose for this:
+
+\starttyping
+[par] <=> [glue] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
+\stoptyping
+
+where glue (currently fixed) is used instead of an empty hlist (think of a \type
+{\hbox}). Options like this are available because want a certain view on these
+lists from the \LUA\ end and the result being predicable is part of that.
+
 It's also good to know beforehand that \TEX\ is basically centered around
 creating paragraphs and pages. The par builder takes a list and breaks it into
 lines. At some point horizontal blobs are wrapped into vertical ones. Lines are
@@ -99,7 +110,9 @@ kick in much \LUA\ code, you will notices that performance drops. Don't blame an
 bother the authors with performance issues. In \CONTEXT\ over 50\% of the time
 can be spent in \LUA, but so far we didn't get many complaints about efficiency.
 Adding more callbacks makes no sense, also because at some point the performance
-hit gets too large. There are plenty ways to achieve one goals.
+hit gets too large. There are plenty ways to achieve one goals. For that reason:
+take remarks about \LUATEX, features, potential, performance etc.\ with a natural
+grain of salt.
 
 Where plain \TEX\ is basically a basic framework for writing a specific style,
 macro packages like \CONTEXT\ and \LATEX\ provide the user a whole lot of
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-tex.tex b/doc/context/sources/general/manuals/luametatex/luametatex-tex.tex
index 48266aa2e..0ef178526 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-tex.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-tex.tex
@@ -102,30 +102,6 @@ mind when preloading files into a bytecode register in \INITEX.
 
 \stopsubsection
 
-\startsubsection[title={Chunk name registers}]
-
-\libindex{name}
-\libindex{setluaname}
-\libindex{getluaname}
-
-There is an array of 65536 (0--65535) potential chunk names for use with the
-\prm {directlua} and \lpr {latelua} primitives.
-
-\startfunctioncall
-lua.name[<number> n] = <string> s
-<string> s = lua.name[<number> n]
-\stopfunctioncall
-
-If you want to unset a \LUA\ name, you can assign \type {nil} to it. The function
-accessors are:
-
-\startfunctioncall
-lua.setluaname(<string> s,<number> n)
-<string> s = lua.getluaname(<number> n)
-\stopfunctioncall
-
-\stopsubsection
-
 \startsubsection[title={Introspection}]
 
 \libindex{getstacktop}
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex.tex b/doc/context/sources/general/manuals/luametatex/luametatex.tex
index a7ece66d8..c75e8ec86 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex.tex
@@ -38,6 +38,9 @@
 % 20200610 : 258 pages
 % mingw all in: 10.6
 
+% 20201204 : 266 pages
+% mingw all in: 10.3
+
 % 20200720      : 258 pages / all in
 %
 % mingw         : 10.6 (sometimes less)