% language=us runpath=texruns:manuals/lowlevel

% It took some time to get the right balance of using the overload related features
% but at some point it started feeling right. Of course it will never be as
% perfectly timed and integrated as Gavin Harrison performance on "Threatening War"
% (youtube movie) but that doesn't mean I should not aim for perfection. But as
% with drumming, it takes practising and that is what I did on a subset of sources
% when writing the engine code.
%
% It tooks a few weeks but November 21 2020 the last of the core files was turned
% \LMTX, not that the work was done (checking to be done (thanks Wolfgang!), imp
% files to be checked, many \LUA\ files to be updated) but it's a start. This time
% the musical timestamp is listening to Nikola Cvetkovic (piano) and history
% podcasts.

\usemodule[system-tokens]

\environment lowlevel-style

\startdocument
  [title=security,
   color=middleorange]

\startsectionlevel[title=Preamble]

Here I will discuss a moderate security subsystem of \LUAMETATEX\ and therefore
\CONTEXT\ \LMTX. This is not about security in the sense of the typesetting
machinery doing harm to your environment, but more about making sure that a user
doesn't change the behavior of the macro package in ways that introduce
interference and thereby unwanted side effect. It's all about protecting macros.

This is all very experimental and we need to adapt the \CONTEXT\ source code to
this. Actually that will happen a few times because experiments trigger that. It
might take a few years before the security model is finalized and all files are
updated accordingly. There are lots of files and macros involved. In the process
the underlying features in the engine might evolve.

\stopsectionlevel

\startsectionlevel[title=Flags]

Before we go into the security levels we see what flags can be set. The \TEX\
language has a couple of so called prefixes that can be used when setting values
and defining macros. Any engine that has traditional \TEX\ with \ETEX\ extensions
can do this:

\starttyping[option=TEX]
                 \def\foo{foo}
\global          \def\foo{foo}
\global\protected\def\foo{foo}
\stoptyping

And \LUAMETATEX\ adds another one:

\starttyping[option=TEX]
       \tolerant          \def\foo{foo}
\global\tolerant          \def\foo{foo}
\global\tolerant\protected\def\foo{foo}
\stoptyping

What these prefixes do is discussed elsewhere. For now is is enough to know that
the two optional prefixes \type {\protected} and \type {\tolerant} make for four
distinctive cases of macro calls.

But there are more prefixes:

\starttabulate
\HL
\NC \type {frozen}     \NC a macro that has to be redefined in a managed way \NC \NR
\NC \type {permanent}  \NC a macro that had better not be redefined \NC \NR
\NC \type {primitive}  \NC a primitive that normally will not be adapted \NC \NR
\NC \type {immutable}  \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
\NC \type {mutable}    \NC a macro that can be changed no matter how well protected it is \NC \NR
\HL
\NC \type {instance}   \NC a macro marked as (for instance) be generated by an interface \NC \NR
\HL
\NC \type {noaligned}  \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
\HL
\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
\NC \type {enforced}   \NC all is permitted (but only in zero mode or ini mode) \NC \NR
\NC \type {aliased}    \NC the macro gets the same flags as the original \NC \NR
\HL
\stoptabulate

These prefixed set flags to the command at hand which can be a macro but
basically any control sequence.

To what extent the engine will complain when a property is changed in a way that
violates the above depends on the parameter \type {\overloadmode}. When this
parameter is set to zero no checking takes place. More interesting are values
larger than zero. If that is the case, when a control sequence is flagged as
mutable, it is always permitted to change. When it is set to immutable one can
never change it. The other flags determine the kind of checking done. Currently
the following overload values are used:

\starttabulate[|l|l|c|c|c|c|c|]
    \NC   \NC         \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
    \NC 1 \NC warning \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
    \NC 2 \NC error   \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
    \NC 3 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
    \NC 4 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
    \NC 5 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
    \NC 6 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
\stoptabulate

The even values (except zero) will abort the run. In \CONTEXT\ we plug in a
callback that deals with the messages. A value of 255 will freeze this parameter.
At level five and above the \type {instance} flag is also checked but no drastic
action takes place. We use this to signal to the user that a specific instance is
redefined (of course the definition macros can check for that too).

So, how does it work. The following is okay:

\starttyping[option=TEX]
\def\MacroA{A}
\def\MacroB{B}
\let\MyMacro\MacroA
\let\MyMacro\MacroB
\stoptyping

The first two macros are ordinary ones, and the last two lines just create an alias. Such
an alias shares the definition, but when for instance \type {\MacroA} is redefined, its
new meaning will not be reflected in the alias.

\starttyping[option=TEX]
\permanent\protected\def\MacroA{A}
\permanent\protected\def\MacroB{B}
\let\MyMacro\MacroA
\let\MyMacro\MacroB
\stoptyping

This also works, because the \type {\let} will create an alias with the protected
property but it will not take the \type {permanent} propery along. For that we need
to say:

\starttyping[option=TEX]
\permanent\protected\def\MacroA{A}
\permanent\protected\def\MacroB{B}
\permanent\let\MyMacro\MacroA
\permanent\let\MyMacro\MacroB
\stoptyping

or, when we want to copy all properties:

\starttyping[option=TEX]
\permanent\protected\def\MacroA{A}
\permanent\protected\def\MacroB{B}
\aliased\let\MyMacro\MacroA
\aliased\let\MyMacro\MacroB
\stoptyping

However, in \CONTEXT\ we have commands that we like to protect against
overloading but at the same time have a different meaning depending on the use
case. An example is the \type {\NC} (next column) command that has a different
implementation in each of the table mechanisms.

\starttyping[option=TEX]
\permanent\protected\def\NC_in_table   {...}
\permanent\protected\def\NC_in_tabulate{...}
\aliased\let\NC\NC_in_table
\aliased\let\NC\NC_in_tabulate
\stoptyping

Here the second aliasing of \type {\NC} fails (assuming of course that we enabled
overload checking). One can argue that grouping can be used but often no grouping
takes place when we redefine on the fly. Because \type {frozen} is less restrictive
than \type {primitive} or \type {permanent}, and of course \type {immutable}, the
next variant works:

\starttyping[option=TEX]
\frozen\protected\def\NC_in_table   {...}
\frozen\protected\def\NC_in_tabulate{...}
\overloaded\let\NC\NC_in_table
\overloaded\let\NC\NC_in_tabulate
\stoptyping

However, in practice, as we want to keep the overload checking, we have to do:

\starttyping[option=TEX]
\frozen\protected\def\NC_in_table   {...}
\frozen\protected\def\NC_in_tabulate{...}
\overloaded\frozen\let\NC\NC_in_table
\overloaded\frozen\let\NC\NC_in_tabulate
\stoptyping

or use \type {\aliased}, but there might be conflicting permissions. This is not
that nice, so there is a kind of dirty trick possible. Consider this:

\starttyping[option=TEX]
\frozen\protected\def\NC_in_table   {...}
\frozen\protected\def\NC_in_tabulate{...}
\def\setNCintable   {\enforced\let\frozen\let\NC\NC_in_table}
\def\setNCintabulate{\enforced\let\frozen\let\NC\NC_in_tabulate}
\stoptyping

When we're in so called \type {initex} mode or when the overload mode is zero,
the \type {\enforced} prefix is internalized in a way that signals that the
follow up is not limited by the overload mode and permissions. This definition
time binding mechanism makes it possible to use \type {permanent} macros that
users cannot redefine, but existing macros can, unless of course they tweak the
mode parameter.

Now keep in mind that users can always cheat but that is intentional. If you
really want to avoid that you can set the overload mode to 255 after which it
cannot be set any more. However, it can be useful to set the mode to zero (or
some warning level) when foreign macro packages are used.

\stopsectionlevel

\startsectionlevel[title=Complications]

One side effect of all this is that all those prefixes can lead to more code. On
the other hand we save some due to the extended macro argument handling features.
When you take the size of the format file as reference, in the end we get a
somewhat smaller file. Every token that you add of remove gives a 8~bytes
difference. The extra overhead that got added to the engine is compensated by the
fact that some macro implementations can be more efficient. In the end, in spite
of these new features and the more extensive testing of flags performance is
about the same. \footnote {And if you wonder about memory, by compacting the used
(often scattered) token memory before dumping I manages to save some 512K on the
format file, so often the loss and gain are somewhere else.}

\stopsectionlevel

\startsectionlevel[title=Introspection]

In case you want to get some details about the properties of a macro, you can
check its meaning. The full variant shows all of them.

\startbuffer
% a macro with two optional arguments with optional spacing in between:

\permanent\tolerant\protected\def\MyFoo[#1]#*[#2]{(#1)(#2)}

\meaningless\MyFoo\par
\meaning    \MyFoo\par
\meaningfull\MyFoo\par
\stopbuffer

\typebuffer[option=TEX]

\startpacked \getbuffer \stoppacked

\stopsectionlevel

% In \CONTEXT:
% c! v! s! ??
% newif newcount ... newconditional etc
% userinterface (permanent)
% primitives
% noaligned
% frozen is for users

\stopdocument