2020-11-30 10:23:00

5 files changed, 327 insertions, 45 deletions

diff --git a/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex b/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex
index 7ddfde2c5..ef527732b 100644
--- a/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex
+++ b/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex
@@ -521,9 +521,27 @@ quit.
 I have to admit that I don't really need it but it made some macros that I was
 redefining behave better, so there is some self|-|interest here. Anyway, I
 considered some other features, like picking up a detokenized argument but I
-don't expect that to be of much use. In the meantime we ran out of reasonable characters,
-but some day \type {#?} and \type {#!} might show up, or maybe I find a use for \type {#<}
-and \type {#>}.
+don't expect that to be of much use. In the meantime we ran out of reasonable
+characters, but some day \type {#?} and \type {#!} might show up, or maybe I find
+a use for \type {#<} and \type {#>}. A summary of all this is given here:
+
+\starttabulate[|T|i2l|]
+\FL
+\NC +   \NC keep the braces \NC \NR
+\NC -   \NC discard and don't count the argument \NC \NR
+\NC /   \NC remove leading an trailing spaces and pars \NC \NR
+\NC =   \NC braces are mandate \NC \NR
+\NC _   \NC braces are mandate and kept \NC \NR
+\NC ^   \NC keep leading spaces \NC \NR
+\ML
+\NC 1-9 \NC an argument \NC \NR
+\NC 0   \NC discard but count the argument \NC \NR
+\ML
+\NC *   \NC ignore spaces \NC \NR
+\NC :   \NC pick up scanning here  \NC \NR
+\NC ;   \NC quit scanning \NC \NR
+\LL
+\stoptabulate
 
 \stopsection
 
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex b/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex
index ba3b903b0..5b8f7558a 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex
@@ -1,4 +1,4 @@
-% language=uk
+    % language=uk
 
 \environment luametatex-style
 
@@ -102,7 +102,8 @@ been removed. There are some new variants that will be discussed later. The \typ
 {\outer} and \type {\long} prefixes are gone as they don't make much sense
 nowadays and them becoming dummies opened the way to something new, again to be
 discussed elsewhere. I don't think that (\CONTEXT) users will notice it. The
-\type {\suppress..} features are now default.
+\type {\suppress..} features are now default and can't be changed so related
+primitives are gone.
 
 The \type {\shipout} primitive does no ship out but just erases the content of
 the box, if that hasn't happened already in another way.
@@ -208,10 +209,17 @@ might be noticed that it is not possible to backtrack or inject something. Of
 course it is no big deal to implement all that in \LUA\ if needed. It removes a
 system dependency and makes for a bit cleaner code.
 
-There are new primitives too as well as some extensions to existing primitive
+There are new primitives as well as some extensions to existing primitive
 functionality. These are described in following chapters but there might be
 hidden treasures in the binary. If you locate them, don't automatically assume
-them to stay, some might be part of experiments!
+them to stay, some might be part of experiments! There are for instance a few
+csname related definers, we have integer and dimension constants, the macro
+argument parser can be brought in tolerant mode, the repertoire of conditionals
+has been extended, some internals can be controlled (think of normalization of
+lines, hyphenation etc.), and macros can be protected against user overload. Not
+all is discussed in detail in this manual but there are introductions in the
+\CONTEXT\ distribution that explain them. But the \TEX\ kernel is of course
+omnipresent.
 
 \startluacode
 
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex b/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex
index ee30683e8..f4dbfb15d 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex
@@ -1110,6 +1110,13 @@ lead to less clutter in tracing.
 
 \stopsubsection
 
+\startsubsection[title={\lpr {ifrelax}}]
+
+This primitive complements \type {\ifdefined}, \type {\ifempty} and \type
+{\ifcsname} so that we have all reasonable tests directly available.
+
+\stopsubsection
+
 \startsubsection[title={\lpr {ifboolean}}]
 
 This primitive tests for non|-|zero, so the next variants are similar
@@ -1151,6 +1158,33 @@ one might evolve).
 
 \stopsubsection
 
+\startsubsection[title={\lpr {ifarguments}, \lpr {ifparameters} and \lpr {ifparameter}}]
+
+These are part of the extended macro argument parsing features. The \type
+{\ifarguments} condition is like an \type {\ifcase} where the number is the
+picked up number of arguments. The number reflects the {\em last} count, so
+successive macro expansions will adapt the value. The \type {\ifparameters}
+counts till the first empty parameter and the \type {\ifparameter} (singular)
+takes a parameter reference (like \type {#2}) and again is an \type {\ifcase}
+where zero means a bad reference, one a non|-|empty argument and two an empty
+one. A typical usage is:
+
+\starttyping
+\def\foo#1#2%
+  {\ifparameter#1\or one\fi
+   \ifparameter#2\or two\fi}
+\stoptyping
+
+No expansion of arguments takes place here but you can use a test like this:
+
+\starttyping
+\def\foo#1#2%
+  {\iftok{#1}{}\else one\fi
+   \iftok{#2}{}\else two\fi}
+\stoptyping
+
+\stopsubsection
+
 \startsubsection[title={\lpr {ifcondition}}]
 
 \topicindex {conditions}
@@ -1321,14 +1355,6 @@ dimensions are calculated.
 
 \stopsubsection
 
-\topicindex {splitting}
-
-The \prm {vsplit} primitive has to be followed by a specification of the required
-height. As alternative for the \type {to} keyword you can use \type {upto} to get
-a split of the given size but result has the natural dimensions then.
-
-\stopsubsection
-
 \startsubsection[title={\prm {vsplit}}]
 
 \topicindex {splitting}
@@ -1931,6 +1957,223 @@ makes sense (for integers).
 
 \stopsection
 
+\startsection[title=Macro arguments]
+
+Again this is experimental and (used and) discussed in document that come with the
+\CONTEXT\ distribution. When defining a macro you can do this:
+
+\starttyping
+\def\foo(#1)#2{...}
+\stoptyping
+
+Here the first argument between parentheses is mandate. But the magic
+prefix \lpr {tolerant} makes that limitation go away:
+
+\starttyping
+\tolerant\def\foo(#1)#2{...}
+\stoptyping
+
+A variant is this:
+
+\starttyping
+\tolerant\def\foo(#1)#*(#2){...}
+\stoptyping
+
+Here we have two optional arguments, possibly be separated by spaces. There are
+more parsing options:
+
+\starttabulate[|T|i2l|]
+\FL
+\NC +   \NC keep the braces \NC \NR
+\NC -   \NC discard and don't count the argument \NC \NR
+\NC /   \NC remove leading an trailing spaces and pars \NC \NR
+\NC =   \NC braces are mandate \NC \NR
+\NC _   \NC braces are mandate and kept \NC \NR
+\NC ^   \NC keep leading spaces \NC \NR
+\ML
+\NC 1-9 \NC an argument \NC \NR
+\NC 0   \NC discard but count the argument \NC \NR
+\ML
+\NC *   \NC ignore spaces \NC \NR
+\NC :   \NC pick up scanning here  \NC \NR
+\NC ;   \NC quit scanning \NC \NR
+\LL
+\stoptabulate
+
+For the moment we leave it to your fantasy what these options do. Most probably
+only make sense when you write a bit more complex macros. Just try to imagine
+what this does:
+
+\starttyping
+\permanent\tolerant\global\protected\def\foo(#1)#*#;[#2]#:#3{...}
+\stoptyping
+
+Of course complex combinations can be confusing because after all \TEX\ is
+parsing for (multi|-|token) delimiters and will happily gobble the whole file if
+you are not careful. You can quit scanning if you want:
+
+\starttyping
+\mymacro 123\ignorearguments
+\stoptyping
+
+which of course only makes sense when used in a nested call where an already
+picked up arguments is processed further. A not (yet) discussed feature of the
+parser is that it will happily skip tokens that have the (probably seldom used)
+ignored characters property.
+
+When you use tracing or see error messages arguments defined using for instance
+\type {#=} will have their usual number in the macro body, so you need to keep
+track of the numbers.
+
+All this is rather easy on the engine and although it might have a little impact
+on performance this has been compensated by some more efficiency in the macro
+parser and engine in general and of course you can gain back some by using these
+features.
+
+\stopsection
+
+\startsection[title=Overload protection]
+
+There is an experimental overload protection mechanism that we will test for a
+while before declaring it stable. The reason for that is that we need to adapt
+the \CONTEXT\ code base in order to test its usefulness. Protection is achieved
+via prefixes. Depending on the value of the \lpr {overloadmode} variable
+warnings or errors will be triggered. Examples of usage can be found in some
+documents that come with \CONTEXT, so here we just stick to the basics.
+
+\starttyping
+\mutable  \def\foo{...}
+\immutable\def\foo{...}
+\permanent\def\foo{...}
+\frozen   \def\foo{...}
+\aliased  \def\foo{...}
+\stoptyping
+
+A \lpr {mutable} macro can always be changed contrary to an \lpr {immutable} one.
+For instance a macro that acts as a variable is normally \lpr {mutable}, while a
+constant can best be immutable. It makes sense to define a public core macro as
+\lpr {permanent}. Primives start out a \lpr {permanent} ones but with a primitive
+property instead.
+
+\startbuffer
+          \let\relaxone  \relax 1: \meaningfull\relaxone
+\aliased  \let\relaxtwo  \relax 2: \meaningfull\relaxtwo
+\permanent\let\relaxthree\relax 3: \meaningfull\relaxthree
+\stopbuffer
+
+\typebuffer
+
+The \lpr {meaningfull} primitive is like \prm {meaning} but report the
+properties too. The \lpr {meaningless} companion reports the body of a macro.
+Anyway, this typesets:
+
+\startlines \tttf \getbuffer \stoplines
+
+So, the \lpr {aliased} prefix copies the properties. Keep in mind that a macro
+package can redefine primitives, but \prm {relax} is an unlikely candidate.
+
+There is an extra prefix \lpr {noaligned} that flags a macro as being valid
+for \prm {noalign} compatible usage (which means that the body must contain that
+one. The idea is that we then can do this:
+
+\starttyping
+\permanent\protected\noaligned\def\foo{\noalign{...}} % \foo is unexpandable
+\stoptyping
+
+that is: we can have protected macros that don't trigger an error in the parser
+where there is a look ahead for \prm {noalign} which is why normally protection
+doesn't work well. So: we have macro flagged as permanent (overload protection),
+being protected (that is, not expandable by default) and a valid equivalent of
+the noalign primitive. Of course we can also apply the \prm {global} and \lpr
+{tolerant} prefixes here. The complete repertoire of extra prefixes is:
+
+\starttabulate
+\HL
+\NC \type {frozen}     \NC a macro that has to be redefined in a managed way \NC \NR
+\NC \type {permanent}  \NC a macro that had better not be redefined \NC \NR
+\NC \type {primitive}  \NC a primitive that normally will not be adapted \NC \NR
+\NC \type {immutable}  \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
+\NC \type {mutable}    \NC a macro that can be changed no matter how well protected it is \NC \NR
+\HL
+\NC \type {instance}   \NC a macro marked as (for instance) be generated by an interface \NC \NR
+\HL
+\NC \type {noaligned}  \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
+\HL
+\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
+\NC \type {enforced}   \NC all is permitted (but only in zero mode or ini mode) \NC \NR
+\NC \type {aliased}    \NC the macro gets the same flags as the original \NC \NR
+\HL
+\stoptabulate
+
+The not yet discussed \lpr {instance} is just a flag with no special meaning
+which can be used as classifier. The \lpr {frozen} also protects against overload
+which brings amount of blockers to four.
+
+To what extent the engine will complain when a property is changed in a way that
+violates the flags depends on the parameter \lpr {overloadmode}. When this
+parameter is set to zero no checking takes place. More interesting are values
+larger than zero. If that is the case, when a control sequence is flagged as
+mutable, it is always permitted to change. When it is set to immutable one can
+never change it. The other flags determine the kind of checking done. Currently
+the following overload values are used:
+
+\starttabulate[|l|l|c|c|c|c|c|]
+    \NC   \NC         \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
+    \NC 1 \NC warning \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
+    \NC 2 \NC error   \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
+    \NC 3 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
+    \NC 4 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
+    \NC 5 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
+    \NC 6 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
+\stoptabulate
+
+The even values (except zero) will abort the run. A value of 255 will freeze this
+parameter. At level five and above the \lpr {instance} flag is also checked but
+no drastic action takes place. We use this to signal to the user that a specific
+instance is redefined (of course the definition macros can check for that too).
+
+The \lpr {overloaded} prefix can be used to overload a frozen macro. The \lpr
+{enforced} is more powerful and forces an overload but that prefix is only
+effective in ini mode or when it's embedded in the body of a macro or token list
+at ini time unless of course at runtime the mode is zero.
+
+So far for a short explanation. More details can be found in the \CONTEXT\
+documentation where we can discuss it in a more relevant perspective. It must be
+noted that this feature only makes sense a controlled situation, that is: user
+modules or macros of unpredictable origin will probably suffer from warnings and
+errors when de mode is set to non zero. In \CONTEXT\ we're okay unless of course
+users redefine instances but there a warning or error is kind of welcome.
+
+\stopsection
+
+\startsection[title=Constants]
+
+It is rather common to store constant values in a register or character
+definition.
+
+\starttyping
+\newcount\MyConstantA \MyConstantA 123
+\newdimen\MyConstantB \MyConstantB 123pt
+\chardef \MyConstantC \MyConstantC 123
+\stoptyping
+
+But in \LUAMETATEX\ we also can do this:
+
+\starttyping
+\integerdef\MyConstantC 456
+\dimendef  \MyConstantD 456pt
+\stoptyping
+
+These two are stored as efficient as a register but don't occupy a register slot.
+They can be set as above, need \prm {the} for serializations and are seen as
+valid number or dimension when needed.
+
+Experiments with constant strings made the engine source more complex than I
+wanted so that features was rejected. Of course we can use the prefixes mentioned
+in a previous section.
+
+\stopsection
+
 \startsection[title=Nodes]
 
 The \ETEX\ primitive \type {\lastnodetype} is not honest in reporting the
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-introduction.tex b/doc/context/sources/general/manuals/luametatex/luametatex-introduction.tex
index 1ad055a83..40c69aeef 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-introduction.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-introduction.tex
@@ -138,30 +138,35 @@ real problem as there's not much demand for that anyway.
 versions which means that \LMTX\ is tested well. We can therefore safely claim
 that end of 2019 the code has become quite stable. There are no complaints about
 performance (on my 2013 laptop this manual compiles at 24.5 pps with \LMTX\
-versus 20.7 pps for the \LUATEX\ manual with \MKIV). Probably no one notices it,
-but memory consumption stepwise got reduced too. And \unknown\ the binary is
-still below 3~MegaBytes on all platforms.
+versus 20.7 pps for the \LUATEX\ manual with \MKIV). After updating some of the
+\CONTEXT\ code to use recently added features by the end of 2020 I could do more
+than 25.5 pps but don't expect spectacular bumps in performance (I need a new
+machine for that to happen). Probably no one notices it, but memory consumption
+stepwise got reduced too. And \unknown\ the binary is still below 3~MegaBytes on
+all platforms.
 
 \stopchapter
 
 \stopcomponent
 
-% I'm not that strict with incrementing numbers, but let's occasionally bump
-% the number. Once we're stable it might happen more systematically. For sure
-% I don't want to end up with these redicoulous 0.99999999 kind of numbers.
-%
+% I'm not that strict with incrementing numbers, but let's occasionally bump the
+% number. Once we're stable it might happen more systematically. For sure I don't
+% want to end up with these 0.99999999 kind of numbers.
+
 % We started with 2.00.0 and kept that number till November 2019, after Alan
-% Braslau and I did the initial beta release at April 1, 2019. After more than
-% a year working on the code base after the \CONTEXT\ 2019 meeting a state was
-% reached where nothing fundamental got added for a while. When \LUATEX\ needs
-% a patch, I check the \LUAMETATEX\ code base as the same patch might be needed
-% there. On the other hand, we don't need a strict compatibility, so some
-% patched in \LUATEX\ are not applied here.
+% Braslau and I did the initial beta release at April 1, 2019. After more than a
+% year working on the code base after the \CONTEXT\ 2019 meeting a state was
+% reached where nothing fundamental got added for a while. When \LUATEX\ needs a
+% patch, I check the \LUAMETATEX\ code base as the same patch might be needed
+% there. On the other hand, we don't need a strict compatibility, so some patched
+% in \LUATEX\ are not applied here.
+%
+% In November 2019 I started wondering if we should bump the number, just for the
+% sake of showing that there's still some progress. So I decided to bump to 2.01.0
+% then. Just as a reminder for myself: it was the day when I watched Jacob Collier
+% perform \LUA\ (feat. MARO) live on YouTube (of course that is not about the
+% language at all, but still a nice coincidence). Just for the fun of it the number
+qbumped a few more times, just to catch up, so end 2019 we're at 2.03.5.
 %
-% In November 2019 I started wondering if we should bump the number, just for
-% the sake of showing that there's still some progress. So I decided to bump to
-% 2.01.0 then. Just as a reminder for myself: it was the day when I watched
-% Jacob Collier perform Lua (feat. MARO) live on YouTube (of course that is not
-% about the language at all, but still a nice coincidence). Just for the fun of
-% it the number bumped a few more times, just to catch up, so end 2019 we're at
-% 2.03.5.
+% Thanks to the patient \CONTEXT\ users we were able to apply the new macro scanner
+% and protection mechanisms that were introduced mid 2020.
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex.tex b/doc/context/sources/general/manuals/luametatex/luametatex.tex
index 99cd6cfa5..e0e7f1575 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex.tex
@@ -1,13 +1,21 @@
-% ------------------------   ------------------------
-% 2019-12-17  32bit  64bit   2020-01-10  32bit  64bit
-% ------------------------   ------------------------
-% freebsd     2270k  2662k   freebsd     2186k  2558k
-% openbsd6.6  2569k  2824k   openbsd6.6  2472k  2722k
-% linux-armhf 2134k          linux-armhf 2063k
-% linux       2927k  2728k   linux       2804k  2613k
-% osx                2821k   osx                2732k
-% mswin       2562k  2555k   mswin       2481k  2471k
-% ------------------------   ------------------------
+% ------------------------   ------------------------   ------------------------
+% 2019-12-17  32bit  64bit   2020-01-10  32bit  64bit   2020-11-30  32bit  64bit
+% ------------------------   ------------------------   ------------------------
+% freebsd     2270k  2662k   freebsd     2186k  2558k   freebsd     2108k  2436k
+% openbsd6.6  2569k  2824k   openbsd6.6  2472k  2722k   openbsd6.8  2411k  2782k
+% linux-armhf 2134k          linux-armhf 2063k          linux-armhf 2138k  2860k
+% linux       2927k  2728k   linux       2804k  2613k   linux   (?) 3314k  2762k
+%                                                       linux-musl  2532k  2686k
+% osx                2821k   osx                2732k   osx                2711k
+% ms mingw    2562k  2555k   ms mingw    2481k  2471k   ms mingw    2754k  2760k
+%                                                       ms intel           2448k
+%                                                       ms arm             3894k
+%                                                       ms clang           2159k
+% ------------------------   ------------------------   ------------------------
+%
+% performance of mingw and native are getting close (small win for mingw) but
+% clang bins are still slower .. quite inconsistent differences between 32 and
+% 64 bit (not all compilers are the same version)
 
 % \loggingall
 % \tracingonline    \zerocount