4 files changed, 418 insertions, 0 deletions
diff --git a/doc/context/documents/general/manuals/lowlevel-grouping.pdf b/doc/context/documents/general/manuals/lowlevel-grouping.pdf
new file mode 100644
index 000000000..5439f29e5
--- /dev/null
+++ b/doc/context/documents/general/manuals/lowlevel-grouping.pdf
diff --git a/doc/context/documents/general/manuals/lowlevel-security.pdf b/doc/context/documents/general/manuals/lowlevel-security.pdf
new file mode 100644
index 000000000..b92f00ce3
--- /dev/null
+++ b/doc/context/documents/general/manuals/lowlevel-security.pdf
diff --git a/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex b/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex
new file mode 100644
index 000000000..eadcca6a9
--- /dev/null
+++ b/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex
@@ -0,0 +1,170 @@
+% language=us
+
+\environment lowlevel-style
+
+\startdocument
+  [title=grouping,
+   color=middlecyan]
+
+\startsection[title=Introduction]
+
+This is a rather short explanation. I decided to write it after presenting the
+other topics at the 2019 \CONTEXT\ meeting where there was a question about
+grouping.
+
+\stopsection
+
+\startsection[title=\PASCAL]
+
+In a language like \PASCAL, the language that \TEX\ has been written in, or
+\MODULA, its successor, there is no concept of grouping like in \TEX. But we can
+find keywords that suggests this:
+
+\starttyping
+for i := 1 to 10 do begin ... end
+\stoptyping
+
+This language probably inspired some of the syntax of \TEX\ and \METAPOST. For
+instance an assignment in \METAPOST\ uses \type {:=} too. However, the \type
+{begin} and \type {end} don't really group but define a block of statements. You
+can have local variables in a procedure or function but the block is just a way
+to pack a sequence of statements.
+
+\stopsection
+
+\startsection[title=\TEX]
+
+In \TEX\ macros (or source code) the following can occur:
+
+\starttyping
+\begingroup
+    ...
+\endgroup
+\stoptyping
+
+as well as:
+
+\starttyping
+\bgroup
+    ...
+\egroup
+\stoptyping
+
+Here we really group in the sense that assignments to variables inside a group
+are forgotten afterwards. All assignments are local to the group unless they are
+explicitly done global:
+
+\starttyping
+\scratchcounter=1
+\def\foo{foo}
+\begingroup
+    \scratchcounter=2
+    \global\globalscratchcounter=2
+    \gdef\foo{FOO}
+\endgroup
+\stoptyping
+
+Here \type {\scratchcounter} is still one after the group is left but its global
+counterpart is now two. The \type {\foo} macro is also changed globally.
+
+Although you can use both sets of commands to group, you cannot mix them, so this
+will trigger an error:
+
+\starttyping
+\bgroup
+\endgroup
+\stoptyping
+
+The bottomline is: if you want a value to persist after the group, you need to
+explicitly change its value globally. This makes a lot of sense in the perspective
+of \TEX.
+
+\stopsection
+
+\startsection[title=\METAPOST]
+
+The \METAPOST\ language also has a concept of grouping but in this case it's more like a
+programming language.
+
+\starttyping
+begingroup ;
+    n := 123 ;
+engroup ;
+\stoptyping
+
+In this case the value of \type {n} is 123 after the group is left, unless you do
+this (for numerics there is actually no need to declare them):
+
+\starttyping
+begingroup ;
+    save n ; numeric n ; n := 123 ;
+engroup ;
+\stoptyping
+
+Given the use of \METAPOST\ (read: \METAFONT) this makes a lot of sense: often
+you use macros to simplify code and you do want variables to change. Grouping in
+this language serves other purposes, like hiding what is between these commands
+and let the last expression become the result. In a \type {vardef} grouping is
+implicit.
+
+So, in \METAPOST\ all assignments are global, unless a variable is explicitly
+saved inside a group.
+
+\stopsection
+
+\startsection[title=\LUA]
+
+In \LUA\ all assignments are global unless a variable is defines local:
+
+\starttyping
+local x = 1
+local y = 1
+for i = 1, 10 do
+    local x = 2
+    y = 2
+end
+\stoptyping
+
+Here the value of \type {x} after the loop is still one but \type {y} is now two.
+As in \LUATEX\ we mix \TEX, \METAPOST\ and \LUA\ you can mix up these concepts.
+Another mixup is using \type {:=}, \type {endfor}, \type {fi} in \LUA\ after done
+some \METAPOST\ coding or using \type {end} instead of \type {endfor} in
+\METAPOST\ which can make the library wait for more without triggering an error.
+Proper syntax highlighting in an editor clearly helps.
+
+\stopsection
+
+\startsection[title=\CCODE]
+
+The \LUA\ language is a mix between \PASCAL\ (which is one reason why I like it)
+and \CCODE.
+
+\starttyping
+int x = 1 ;
+int y = 1 ;
+for (i=1; i<=10;i++) {
+    int x = 2 ;
+    y = 2 ;
+}
+\stoptyping
+
+The semicolon is also used in \PASCAL\ but there it is a separator and not a
+statement end, while in \METAPOST\ it does end a statement (expression).
+
+\stopsection
+
+\stopsection
+
+\startsubject[title=Colofon]
+
+\starttabulate
+\NC Author      \NC Hans Hagen         \NC \NR
+\NC \CONTEXT    \NC \contextversion    \NC \NR
+\NC \LUAMETATEX \NC \texengineversion  \NC \NR
+\NC Support     \NC www.pragma-ade.com \NC \NR
+\NC             \NC contextgarden.net  \NC \NR
+\stoptabulate
+
+\stopsubject
+
+\stopdocument
diff --git a/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex b/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex
new file mode 100644
index 000000000..4bb0c19d8
--- /dev/null
+++ b/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex
@@ -0,0 +1,248 @@
+% language=us
+
+% It took some time to get the right balance of using the overload related features
+% but at some point it started feeling right. Of course it will never be as
+% perfectly timed and integrated as Gavin Harrison performance on "Threatening War"
+% (youtube movie) but that doesn't mean I should not aim for perfection. But as
+% with drumming, it takes practising and that is what I did on a subset of sources
+% when writing the engine code.
+
+\usemodule[system-tokens]
+
+\environment lowlevel-style
+
+\startdocument
+  [title=security,
+   color=middleorange]
+
+\startsection[title=Preamble]
+
+Here I will discuss a moderate security subsystem of \LUAMETATEX\ and therefore
+\CONTEXT\ \LMTX. This is not about security in the sense of the typesetting
+machinery doing harm to your environment, but more about making sure that a user
+doesn't change the behavior of the macro package in ways that introduce
+interference and thereby unwanted side effect. It's all about protecting macros.
+
+This is all very experimental and we need to adapt the \CONTEXT\ source code to
+this. Actually that will happen a few times because experiments trigger that. It
+might take a few years before the security model is finalized and all files are
+updated accordingly. There are lots of files and macros involved. In the process
+the underlying features in the engine might evolve.
+
+\stopsection
+
+\startsection[title=Flags]
+
+Before we go into the security levels we see what flags can be set. The \TEX\
+language has a couple of so called prefixes that can be used when setting values
+and defining macros. Any engine that has traditional \TEX\ with \ETEX\ extensions
+can do this:
+
+\starttyping[option=TEX]
+                 \def\foo{foo}
+\global          \def\foo{foo}
+\global\protected\def\foo{foo}
+\stoptyping
+
+And \LUAMETATEX\ adds another one:
+
+\starttyping[option=TEX]
+       \tolerant          \def\foo{foo}
+\global\tolerant          \def\foo{foo}
+\global\tolerant\protected\def\foo{foo}
+\stoptyping
+
+What these prefixes do is discussed elsewhere. For now is is enough to know that
+the two optional prefixes \type {\protected} and \type {\tolerant} make for four
+distinctive cases of macro calls.
+
+But there are more prefixes:
+
+\starttabulate
+\HL
+\NC \type {frozen}     \NC a macro that has to be redefined in a managed way \NC \NR
+\NC \type {permanent}  \NC a macro that had better not be redefined \NC \NR
+\NC \type {primitive}  \NC a primitive that normally will not be adapted \NC \NR
+\NC \type {immutable}  \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
+\NC \type {mutable}    \NC a macro that can be changed no matter how well protected it is \NC \NR
+\HL
+\NC \type {instance}   \NC a macro marked as (for instance) be generated by an interface \NC \NR
+\HL
+\NC \type {noaligned}  \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
+\HL
+\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
+\NC \type {enforced}   \NC all is permitted (but only in zero mode or ini mode) \NC \NR
+\NC \type {aliased}    \NC the macro gets the same flags as the original \NC \NR
+\HL
+\stoptabulate
+
+These prefixed set flags to the command at hand which can be a macro but
+basically any control sequence.
+
+To what extent the engine will complain when a property is changed in a way that
+violates the above depends on the parameter \type {\overloadmode}. When this
+parameter is set to zero no checking takes place. More interesting are values
+larger than zero. If that is the case, when a control sequence is flagged as
+mutable, it is always permitted to change. When it is set to immutable one can
+never change it. The other flags determine the kind of checking done. Currently
+the following overload values are used:
+
+\starttabulate[|l|l|c|c|c|c|c|]
+    \NC   \NC         \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
+    \NC 1 \NC warning \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
+    \NC 2 \NC error   \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
+    \NC 3 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
+    \NC 4 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
+    \NC 5 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
+    \NC 6 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
+\stoptabulate
+
+The even values (except zero) will abort the run. In \CONTEXT\ we plug in a
+callback that deals with the messages. A value of 255 will freeze this parameter.
+At level five and above the \type {instance} flag is also checked but no drastic
+action takes place. We use this to signal to the user that a specific instance is
+redefined (of course the definition macros can check for that too).
+
+So, how does it work. The following is okay:
+
+\starttyping[option=TEX]
+\def\MacroA{A}
+\def\MacroB{B}
+\let\MyMacro\MacroA
+\let\MyMacro\MacroB
+\stoptyping
+
+The first two macros are ordinary ones, and the last two lines just create an alias. Such
+an alias shares the definition, but when for instance \type {\MacroA} is redefined, its
+new meaning will not be reflected in the alias.
+
+\starttyping[option=TEX]
+\permanent\protected\def\MacroA{A}
+\permanent\protected\def\MacroB{B}
+\let\MyMacro\MacroA
+\let\MyMacro\MacroB
+\stoptyping
+
+This also works, because the \type {\let} will create an alias with the protected
+property but it will not take the \type {permanent} propery along. For that we need
+to say:
+
+\starttyping[option=TEX]
+\permanent\protected\def\MacroA{A}
+\permanent\protected\def\MacroB{B}
+\permanent\let\MyMacro\MacroA
+\permanent\let\MyMacro\MacroB
+\stoptyping
+
+or, when we want to copy all properties:
+
+\starttyping[option=TEX]
+\permanent\protected\def\MacroA{A}
+\permanent\protected\def\MacroB{B}
+\aliased\let\MyMacro\MacroA
+\aliased\let\MyMacro\MacroB
+\stoptyping
+
+However, in \CONTEXT\ we have commands that we like to protect against
+overloading but at the same time have a different meaning depending on the use
+case. An example is the \type {\NC} (next column) command that has a different
+implementation in each of the table mechanisms.
+
+\starttyping[option=TEX]
+\permanent\protected\def\NC_in_table   {...}
+\permanent\protected\def\NC_in_tabulate{...}
+\aliased\let\NC\NC_in_table
+\aliased\let\NC\NC_in_tabulate
+\stoptyping
+
+Here the second aliasing of \type {\NC} fails (assuming of course that we enabled
+overload checking). One can argue that grouping can be used but often no grouping
+takes place when we redefine on the fly. Because \type {frozen} is less restrictive
+than \type {primitive} or \type {permanent}, and of course \type {immutable}, the
+next variant works:
+
+\starttyping[option=TEX]
+\frozen\protected\def\NC_in_table   {...}
+\frozen\protected\def\NC_in_tabulate{...}
+\overloaded\let\NC\NC_in_table
+\overloaded\let\NC\NC_in_tabulate
+\stoptyping
+
+However, in practice, as we want to keep the overload checking, we have to do:
+
+\starttyping[option=TEX]
+\frozen\protected\def\NC_in_table   {...}
+\frozen\protected\def\NC_in_tabulate{...}
+\overloaded\frozen\let\NC\NC_in_table
+\overloaded\frozen\let\NC\NC_in_tabulate
+\stoptyping
+
+or use \type {\aliased}, but there might be conflicting permissions. This is not
+that nice, so there is a kind of dirty trick possible. Consider this:
+
+\starttyping[option=TEX]
+\frozen\protected\def\NC_in_table   {...}
+\frozen\protected\def\NC_in_tabulate{...}
+\def\setNCintable   {\enforced\let\frozen\let\NC\NC_in_table}
+\def\setNCintabulate{\enforced\let\frozen\let\NC\NC_in_tabulate}
+\stoptyping
+
+When we're in so called \type {initex} mode or when the overload mode is zero,
+the \type {\enforced} prefix is internalized in a way that signals that the
+follow up is not limited by the overload mode and permissions. This definition
+time binding mechanism makes it possible to use \type {permanent} macros that
+users cannot redefine, but existing macros can, unless of course they tweak the
+mode parameter.
+
+Now keep in mind that users can always cheat but that is intentional. If you
+really want to avoid that you can set the overload mode to 255 after which it
+cannot be set any more. However, it can be useful to set the mode to zero (or
+some warning level) when foreign macro packages are used.
+
+\stopsection
+
+\startsection[title=Complications]
+
+One side effect of all this is that all those prefixes can lead to more code. On
+the other hand we save some due to the extended macro argument handling features.
+When you take the size of the format file as reference, in the end we get a
+somewhat smaller file. Every token that you add of remove gives a 8~bytes
+difference. The extra overhead that got added to the engine is compensated by the
+fact that some macro implementations can be more efficient. In the end, in spite
+of these new features and the more extensive testing of flags performance is
+about the same. \footnote {And if you wonder about memory, by compacting the used
+(often scattered) token memory before dumping I manages to save some 512K on the
+format file, so often the loss and gain are somewhere else.}
+
+\stopsection
+
+\startsection[title=Introspection]
+
+In case you want to get some details about the properties of a macro, you can
+check its meaning. The full variant shows all of them.
+
+\startbuffer
+% a macro with two optional arguments with optional spacing in between:
+
+\permanent\tolerant\protected\def\MyFoo[#1]#*[#2]{(#1)(#2)}
+
+\meaningless\MyFoo\par
+\meaning    \MyFoo\par
+\meaningfull\MyFoo\par
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+\startpacked \getbuffer \stoppacked
+
+\stopsection
+
+% In \CONTEXT:
+% c! v! s! ??
+% newif newcount ... newconditional etc
+% userinterface (permanent)
+% primitives
+% noaligned
+% frozen is for users
+
+\stopdocument