From 744befce8596a7007e60c1f046da570fadc205bd Mon Sep 17 00:00:00 2001 From: Hans Hagen Date: Mon, 16 Nov 2020 20:16:53 +0100 Subject: 2020-11-16 19:40:00 --- .../general/manuals/lowlevel-grouping.pdf | Bin 0 -> 30758 bytes .../general/manuals/lowlevel-security.pdf | Bin 0 -> 41760 bytes .../general/manuals/lowlevel/lowlevel-grouping.tex | 170 ++++++++++++++ .../general/manuals/lowlevel/lowlevel-security.tex | 248 +++++++++++++++++++++ 4 files changed, 418 insertions(+) create mode 100644 doc/context/documents/general/manuals/lowlevel-grouping.pdf create mode 100644 doc/context/documents/general/manuals/lowlevel-security.pdf create mode 100644 doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex create mode 100644 doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex (limited to 'doc') diff --git a/doc/context/documents/general/manuals/lowlevel-grouping.pdf b/doc/context/documents/general/manuals/lowlevel-grouping.pdf new file mode 100644 index 000000000..5439f29e5 Binary files /dev/null and b/doc/context/documents/general/manuals/lowlevel-grouping.pdf differ diff --git a/doc/context/documents/general/manuals/lowlevel-security.pdf b/doc/context/documents/general/manuals/lowlevel-security.pdf new file mode 100644 index 000000000..b92f00ce3 Binary files /dev/null and b/doc/context/documents/general/manuals/lowlevel-security.pdf differ diff --git a/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex b/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex new file mode 100644 index 000000000..eadcca6a9 --- /dev/null +++ b/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex @@ -0,0 +1,170 @@ +% language=us + +\environment lowlevel-style + +\startdocument + [title=grouping, + color=middlecyan] + +\startsection[title=Introduction] + +This is a rather short explanation. I decided to write it after presenting the +other topics at the 2019 \CONTEXT\ meeting where there was a question about +grouping. + +\stopsection + +\startsection[title=\PASCAL] + +In a language like \PASCAL, the language that \TEX\ has been written in, or +\MODULA, its successor, there is no concept of grouping like in \TEX. But we can +find keywords that suggests this: + +\starttyping +for i := 1 to 10 do begin ... end +\stoptyping + +This language probably inspired some of the syntax of \TEX\ and \METAPOST. For +instance an assignment in \METAPOST\ uses \type {:=} too. However, the \type +{begin} and \type {end} don't really group but define a block of statements. You +can have local variables in a procedure or function but the block is just a way +to pack a sequence of statements. + +\stopsection + +\startsection[title=\TEX] + +In \TEX\ macros (or source code) the following can occur: + +\starttyping +\begingroup + ... +\endgroup +\stoptyping + +as well as: + +\starttyping +\bgroup + ... +\egroup +\stoptyping + +Here we really group in the sense that assignments to variables inside a group +are forgotten afterwards. All assignments are local to the group unless they are +explicitly done global: + +\starttyping +\scratchcounter=1 +\def\foo{foo} +\begingroup + \scratchcounter=2 + \global\globalscratchcounter=2 + \gdef\foo{FOO} +\endgroup +\stoptyping + +Here \type {\scratchcounter} is still one after the group is left but its global +counterpart is now two. The \type {\foo} macro is also changed globally. + +Although you can use both sets of commands to group, you cannot mix them, so this +will trigger an error: + +\starttyping +\bgroup +\endgroup +\stoptyping + +The bottomline is: if you want a value to persist after the group, you need to +explicitly change its value globally. This makes a lot of sense in the perspective +of \TEX. + +\stopsection + +\startsection[title=\METAPOST] + +The \METAPOST\ language also has a concept of grouping but in this case it's more like a +programming language. + +\starttyping +begingroup ; + n := 123 ; +engroup ; +\stoptyping + +In this case the value of \type {n} is 123 after the group is left, unless you do +this (for numerics there is actually no need to declare them): + +\starttyping +begingroup ; + save n ; numeric n ; n := 123 ; +engroup ; +\stoptyping + +Given the use of \METAPOST\ (read: \METAFONT) this makes a lot of sense: often +you use macros to simplify code and you do want variables to change. Grouping in +this language serves other purposes, like hiding what is between these commands +and let the last expression become the result. In a \type {vardef} grouping is +implicit. + +So, in \METAPOST\ all assignments are global, unless a variable is explicitly +saved inside a group. + +\stopsection + +\startsection[title=\LUA] + +In \LUA\ all assignments are global unless a variable is defines local: + +\starttyping +local x = 1 +local y = 1 +for i = 1, 10 do + local x = 2 + y = 2 +end +\stoptyping + +Here the value of \type {x} after the loop is still one but \type {y} is now two. +As in \LUATEX\ we mix \TEX, \METAPOST\ and \LUA\ you can mix up these concepts. +Another mixup is using \type {:=}, \type {endfor}, \type {fi} in \LUA\ after done +some \METAPOST\ coding or using \type {end} instead of \type {endfor} in +\METAPOST\ which can make the library wait for more without triggering an error. +Proper syntax highlighting in an editor clearly helps. + +\stopsection + +\startsection[title=\CCODE] + +The \LUA\ language is a mix between \PASCAL\ (which is one reason why I like it) +and \CCODE. + +\starttyping +int x = 1 ; +int y = 1 ; +for (i=1; i<=10;i++) { + int x = 2 ; + y = 2 ; +} +\stoptyping + +The semicolon is also used in \PASCAL\ but there it is a separator and not a +statement end, while in \METAPOST\ it does end a statement (expression). + +\stopsection + +\stopsection + +\startsubject[title=Colofon] + +\starttabulate +\NC Author \NC Hans Hagen \NC \NR +\NC \CONTEXT \NC \contextversion \NC \NR +\NC \LUAMETATEX \NC \texengineversion \NC \NR +\NC Support \NC www.pragma-ade.com \NC \NR +\NC \NC contextgarden.net \NC \NR +\stoptabulate + +\stopsubject + +\stopdocument diff --git a/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex b/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex new file mode 100644 index 000000000..4bb0c19d8 --- /dev/null +++ b/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex @@ -0,0 +1,248 @@ +% language=us + +% It took some time to get the right balance of using the overload related features +% but at some point it started feeling right. Of course it will never be as +% perfectly timed and integrated as Gavin Harrison performance on "Threatening War" +% (youtube movie) but that doesn't mean I should not aim for perfection. But as +% with drumming, it takes practising and that is what I did on a subset of sources +% when writing the engine code. + +\usemodule[system-tokens] + +\environment lowlevel-style + +\startdocument + [title=security, + color=middleorange] + +\startsection[title=Preamble] + +Here I will discuss a moderate security subsystem of \LUAMETATEX\ and therefore +\CONTEXT\ \LMTX. This is not about security in the sense of the typesetting +machinery doing harm to your environment, but more about making sure that a user +doesn't change the behavior of the macro package in ways that introduce +interference and thereby unwanted side effect. It's all about protecting macros. + +This is all very experimental and we need to adapt the \CONTEXT\ source code to +this. Actually that will happen a few times because experiments trigger that. It +might take a few years before the security model is finalized and all files are +updated accordingly. There are lots of files and macros involved. In the process +the underlying features in the engine might evolve. + +\stopsection + +\startsection[title=Flags] + +Before we go into the security levels we see what flags can be set. The \TEX\ +language has a couple of so called prefixes that can be used when setting values +and defining macros. Any engine that has traditional \TEX\ with \ETEX\ extensions +can do this: + +\starttyping[option=TEX] + \def\foo{foo} +\global \def\foo{foo} +\global\protected\def\foo{foo} +\stoptyping + +And \LUAMETATEX\ adds another one: + +\starttyping[option=TEX] + \tolerant \def\foo{foo} +\global\tolerant \def\foo{foo} +\global\tolerant\protected\def\foo{foo} +\stoptyping + +What these prefixes do is discussed elsewhere. For now is is enough to know that +the two optional prefixes \type {\protected} and \type {\tolerant} make for four +distinctive cases of macro calls. + +But there are more prefixes: + +\starttabulate +\HL +\NC \type {frozen} \NC a macro that has to be redefined in a managed way \NC \NR +\NC \type {permanent} \NC a macro that had better not be redefined \NC \NR +\NC \type {primitive} \NC a primitive that normally will not be adapted \NC \NR +\NC \type {immutable} \NC a macro or quantity that cannot be changed, it is a constant \NC \NR +\NC \type {mutable} \NC a macro that can be changed no matter how well protected it is \NC \NR +\HL +\NC \type {instance} \NC a macro marked as (for instance) be generated by an interface \NC \NR +\HL +\NC \type {noaligned} \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR +\HL +\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR +\NC \type {enforced} \NC all is permitted (but only in zero mode or ini mode) \NC \NR +\NC \type {aliased} \NC the macro gets the same flags as the original \NC \NR +\HL +\stoptabulate + +These prefixed set flags to the command at hand which can be a macro but +basically any control sequence. + +To what extent the engine will complain when a property is changed in a way that +violates the above depends on the parameter \type {\overloadmode}. When this +parameter is set to zero no checking takes place. More interesting are values +larger than zero. If that is the case, when a control sequence is flagged as +mutable, it is always permitted to change. When it is set to immutable one can +never change it. The other flags determine the kind of checking done. Currently +the following overload values are used: + +\starttabulate[|l|l|c|c|c|c|c|] + \NC \NC \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR + \NC 1 \NC warning \NC \star \NC \star \NC \star \NC \NC \NC \NR + \NC 2 \NC error \NC \star \NC \star \NC \star \NC \NC \NC \NR + \NC 3 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \NC \NR + \NC 4 \NC error \NC \star \NC \star \NC \star \NC \star \NC \NC \NR + \NC 5 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR + \NC 6 \NC error \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR +\stoptabulate + +The even values (except zero) will abort the run. In \CONTEXT\ we plug in a +callback that deals with the messages. A value of 255 will freeze this parameter. +At level five and above the \type {instance} flag is also checked but no drastic +action takes place. We use this to signal to the user that a specific instance is +redefined (of course the definition macros can check for that too). + +So, how does it work. The following is okay: + +\starttyping[option=TEX] +\def\MacroA{A} +\def\MacroB{B} +\let\MyMacro\MacroA +\let\MyMacro\MacroB +\stoptyping + +The first two macros are ordinary ones, and the last two lines just create an alias. Such +an alias shares the definition, but when for instance \type {\MacroA} is redefined, its +new meaning will not be reflected in the alias. + +\starttyping[option=TEX] +\permanent\protected\def\MacroA{A} +\permanent\protected\def\MacroB{B} +\let\MyMacro\MacroA +\let\MyMacro\MacroB +\stoptyping + +This also works, because the \type {\let} will create an alias with the protected +property but it will not take the \type {permanent} propery along. For that we need +to say: + +\starttyping[option=TEX] +\permanent\protected\def\MacroA{A} +\permanent\protected\def\MacroB{B} +\permanent\let\MyMacro\MacroA +\permanent\let\MyMacro\MacroB +\stoptyping + +or, when we want to copy all properties: + +\starttyping[option=TEX] +\permanent\protected\def\MacroA{A} +\permanent\protected\def\MacroB{B} +\aliased\let\MyMacro\MacroA +\aliased\let\MyMacro\MacroB +\stoptyping + +However, in \CONTEXT\ we have commands that we like to protect against +overloading but at the same time have a different meaning depending on the use +case. An example is the \type {\NC} (next column) command that has a different +implementation in each of the table mechanisms. + +\starttyping[option=TEX] +\permanent\protected\def\NC_in_table {...} +\permanent\protected\def\NC_in_tabulate{...} +\aliased\let\NC\NC_in_table +\aliased\let\NC\NC_in_tabulate +\stoptyping + +Here the second aliasing of \type {\NC} fails (assuming of course that we enabled +overload checking). One can argue that grouping can be used but often no grouping +takes place when we redefine on the fly. Because \type {frozen} is less restrictive +than \type {primitive} or \type {permanent}, and of course \type {immutable}, the +next variant works: + +\starttyping[option=TEX] +\frozen\protected\def\NC_in_table {...} +\frozen\protected\def\NC_in_tabulate{...} +\overloaded\let\NC\NC_in_table +\overloaded\let\NC\NC_in_tabulate +\stoptyping + +However, in practice, as we want to keep the overload checking, we have to do: + +\starttyping[option=TEX] +\frozen\protected\def\NC_in_table {...} +\frozen\protected\def\NC_in_tabulate{...} +\overloaded\frozen\let\NC\NC_in_table +\overloaded\frozen\let\NC\NC_in_tabulate +\stoptyping + +or use \type {\aliased}, but there might be conflicting permissions. This is not +that nice, so there is a kind of dirty trick possible. Consider this: + +\starttyping[option=TEX] +\frozen\protected\def\NC_in_table {...} +\frozen\protected\def\NC_in_tabulate{...} +\def\setNCintable {\enforced\let\frozen\let\NC\NC_in_table} +\def\setNCintabulate{\enforced\let\frozen\let\NC\NC_in_tabulate} +\stoptyping + +When we're in so called \type {initex} mode or when the overload mode is zero, +the \type {\enforced} prefix is internalized in a way that signals that the +follow up is not limited by the overload mode and permissions. This definition +time binding mechanism makes it possible to use \type {permanent} macros that +users cannot redefine, but existing macros can, unless of course they tweak the +mode parameter. + +Now keep in mind that users can always cheat but that is intentional. If you +really want to avoid that you can set the overload mode to 255 after which it +cannot be set any more. However, it can be useful to set the mode to zero (or +some warning level) when foreign macro packages are used. + +\stopsection + +\startsection[title=Complications] + +One side effect of all this is that all those prefixes can lead to more code. On +the other hand we save some due to the extended macro argument handling features. +When you take the size of the format file as reference, in the end we get a +somewhat smaller file. Every token that you add of remove gives a 8~bytes +difference. The extra overhead that got added to the engine is compensated by the +fact that some macro implementations can be more efficient. In the end, in spite +of these new features and the more extensive testing of flags performance is +about the same. \footnote {And if you wonder about memory, by compacting the used +(often scattered) token memory before dumping I manages to save some 512K on the +format file, so often the loss and gain are somewhere else.} + +\stopsection + +\startsection[title=Introspection] + +In case you want to get some details about the properties of a macro, you can +check its meaning. The full variant shows all of them. + +\startbuffer +% a macro with two optional arguments with optional spacing in between: + +\permanent\tolerant\protected\def\MyFoo[#1]#*[#2]{(#1)(#2)} + +\meaningless\MyFoo\par +\meaning \MyFoo\par +\meaningfull\MyFoo\par +\stopbuffer + +\typebuffer[option=TEX] + +\startpacked \getbuffer \stoppacked + +\stopsection + +% In \CONTEXT: +% c! v! s! ?? +% newif newcount ... newconditional etc +% userinterface (permanent) +% primitives +% noaligned +% frozen is for users + +\stopdocument -- cgit v1.2.3