1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
|
% language=us runpath=texruns:manuals/lowlevel
% It took some time to get the right balance of using the overload related features
% but at some point it started feeling right. Of course it will never be as
% perfectly timed and integrated as Gavin Harrison performance on "Threatening War"
% (youtube movie) but that doesn't mean I should not aim for perfection. But as
% with drumming, it takes practising and that is what I did on a subset of sources
% when writing the engine code.
%
% It tooks a few weeks but November 21 2020 the last of the core files was turned
% \LMTX, not that the work was done (checking to be done (thanks Wolfgang!), imp
% files to be checked, many \LUA\ files to be updated) but it's a start. This time
% the musical timestamp is listening to Nikola Cvetkovic (piano) and history
% podcasts.
\usemodule[system-tokens]
\environment lowlevel-style
\startdocument
[title=security,
color=middleorange]
\startsection[title=Preamble]
Here I will discuss a moderate security subsystem of \LUAMETATEX\ and therefore
\CONTEXT\ \LMTX. This is not about security in the sense of the typesetting
machinery doing harm to your environment, but more about making sure that a user
doesn't change the behavior of the macro package in ways that introduce
interference and thereby unwanted side effect. It's all about protecting macros.
This is all very experimental and we need to adapt the \CONTEXT\ source code to
this. Actually that will happen a few times because experiments trigger that. It
might take a few years before the security model is finalized and all files are
updated accordingly. There are lots of files and macros involved. In the process
the underlying features in the engine might evolve.
\stopsection
\startsection[title=Flags]
Before we go into the security levels we see what flags can be set. The \TEX\
language has a couple of so called prefixes that can be used when setting values
and defining macros. Any engine that has traditional \TEX\ with \ETEX\ extensions
can do this:
\starttyping[option=TEX]
\def\foo{foo}
\global \def\foo{foo}
\global\protected\def\foo{foo}
\stoptyping
And \LUAMETATEX\ adds another one:
\starttyping[option=TEX]
\tolerant \def\foo{foo}
\global\tolerant \def\foo{foo}
\global\tolerant\protected\def\foo{foo}
\stoptyping
What these prefixes do is discussed elsewhere. For now is is enough to know that
the two optional prefixes \type {\protected} and \type {\tolerant} make for four
distinctive cases of macro calls.
But there are more prefixes:
\starttabulate
\HL
\NC \type {frozen} \NC a macro that has to be redefined in a managed way \NC \NR
\NC \type {permanent} \NC a macro that had better not be redefined \NC \NR
\NC \type {primitive} \NC a primitive that normally will not be adapted \NC \NR
\NC \type {immutable} \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
\NC \type {mutable} \NC a macro that can be changed no matter how well protected it is \NC \NR
\HL
\NC \type {instance} \NC a macro marked as (for instance) be generated by an interface \NC \NR
\HL
\NC \type {noaligned} \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
\HL
\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
\NC \type {enforced} \NC all is permitted (but only in zero mode or ini mode) \NC \NR
\NC \type {aliased} \NC the macro gets the same flags as the original \NC \NR
\HL
\stoptabulate
These prefixed set flags to the command at hand which can be a macro but
basically any control sequence.
To what extent the engine will complain when a property is changed in a way that
violates the above depends on the parameter \type {\overloadmode}. When this
parameter is set to zero no checking takes place. More interesting are values
larger than zero. If that is the case, when a control sequence is flagged as
mutable, it is always permitted to change. When it is set to immutable one can
never change it. The other flags determine the kind of checking done. Currently
the following overload values are used:
\starttabulate[|l|l|c|c|c|c|c|]
\NC \NC \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
\NC 1 \NC warning \NC \star \NC \star \NC \star \NC \NC \NC \NR
\NC 2 \NC error \NC \star \NC \star \NC \star \NC \NC \NC \NR
\NC 3 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \NC \NR
\NC 4 \NC error \NC \star \NC \star \NC \star \NC \star \NC \NC \NR
\NC 5 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR
\NC 6 \NC error \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR
\stoptabulate
The even values (except zero) will abort the run. In \CONTEXT\ we plug in a
callback that deals with the messages. A value of 255 will freeze this parameter.
At level five and above the \type {instance} flag is also checked but no drastic
action takes place. We use this to signal to the user that a specific instance is
redefined (of course the definition macros can check for that too).
So, how does it work. The following is okay:
\starttyping[option=TEX]
\def\MacroA{A}
\def\MacroB{B}
\let\MyMacro\MacroA
\let\MyMacro\MacroB
\stoptyping
The first two macros are ordinary ones, and the last two lines just create an alias. Such
an alias shares the definition, but when for instance \type {\MacroA} is redefined, its
new meaning will not be reflected in the alias.
\starttyping[option=TEX]
\permanent\protected\def\MacroA{A}
\permanent\protected\def\MacroB{B}
\let\MyMacro\MacroA
\let\MyMacro\MacroB
\stoptyping
This also works, because the \type {\let} will create an alias with the protected
property but it will not take the \type {permanent} propery along. For that we need
to say:
\starttyping[option=TEX]
\permanent\protected\def\MacroA{A}
\permanent\protected\def\MacroB{B}
\permanent\let\MyMacro\MacroA
\permanent\let\MyMacro\MacroB
\stoptyping
or, when we want to copy all properties:
\starttyping[option=TEX]
\permanent\protected\def\MacroA{A}
\permanent\protected\def\MacroB{B}
\aliased\let\MyMacro\MacroA
\aliased\let\MyMacro\MacroB
\stoptyping
However, in \CONTEXT\ we have commands that we like to protect against
overloading but at the same time have a different meaning depending on the use
case. An example is the \type {\NC} (next column) command that has a different
implementation in each of the table mechanisms.
\starttyping[option=TEX]
\permanent\protected\def\NC_in_table {...}
\permanent\protected\def\NC_in_tabulate{...}
\aliased\let\NC\NC_in_table
\aliased\let\NC\NC_in_tabulate
\stoptyping
Here the second aliasing of \type {\NC} fails (assuming of course that we enabled
overload checking). One can argue that grouping can be used but often no grouping
takes place when we redefine on the fly. Because \type {frozen} is less restrictive
than \type {primitive} or \type {permanent}, and of course \type {immutable}, the
next variant works:
\starttyping[option=TEX]
\frozen\protected\def\NC_in_table {...}
\frozen\protected\def\NC_in_tabulate{...}
\overloaded\let\NC\NC_in_table
\overloaded\let\NC\NC_in_tabulate
\stoptyping
However, in practice, as we want to keep the overload checking, we have to do:
\starttyping[option=TEX]
\frozen\protected\def\NC_in_table {...}
\frozen\protected\def\NC_in_tabulate{...}
\overloaded\frozen\let\NC\NC_in_table
\overloaded\frozen\let\NC\NC_in_tabulate
\stoptyping
or use \type {\aliased}, but there might be conflicting permissions. This is not
that nice, so there is a kind of dirty trick possible. Consider this:
\starttyping[option=TEX]
\frozen\protected\def\NC_in_table {...}
\frozen\protected\def\NC_in_tabulate{...}
\def\setNCintable {\enforced\let\frozen\let\NC\NC_in_table}
\def\setNCintabulate{\enforced\let\frozen\let\NC\NC_in_tabulate}
\stoptyping
When we're in so called \type {initex} mode or when the overload mode is zero,
the \type {\enforced} prefix is internalized in a way that signals that the
follow up is not limited by the overload mode and permissions. This definition
time binding mechanism makes it possible to use \type {permanent} macros that
users cannot redefine, but existing macros can, unless of course they tweak the
mode parameter.
Now keep in mind that users can always cheat but that is intentional. If you
really want to avoid that you can set the overload mode to 255 after which it
cannot be set any more. However, it can be useful to set the mode to zero (or
some warning level) when foreign macro packages are used.
\stopsection
\startsection[title=Complications]
One side effect of all this is that all those prefixes can lead to more code. On
the other hand we save some due to the extended macro argument handling features.
When you take the size of the format file as reference, in the end we get a
somewhat smaller file. Every token that you add of remove gives a 8~bytes
difference. The extra overhead that got added to the engine is compensated by the
fact that some macro implementations can be more efficient. In the end, in spite
of these new features and the more extensive testing of flags performance is
about the same. \footnote {And if you wonder about memory, by compacting the used
(often scattered) token memory before dumping I manages to save some 512K on the
format file, so often the loss and gain are somewhere else.}
\stopsection
\startsection[title=Introspection]
In case you want to get some details about the properties of a macro, you can
check its meaning. The full variant shows all of them.
\startbuffer
% a macro with two optional arguments with optional spacing in between:
\permanent\tolerant\protected\def\MyFoo[#1]#*[#2]{(#1)(#2)}
\meaningless\MyFoo\par
\meaning \MyFoo\par
\meaningfull\MyFoo\par
\stopbuffer
\typebuffer[option=TEX]
\startpacked \getbuffer \stoppacked
\stopsection
% In \CONTEXT:
% c! v! s! ??
% newif newcount ... newconditional etc
% userinterface (permanent)
% primitives
% noaligned
% frozen is for users
\stopdocument
|
