1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
|
% language=us
\startcomponent followingup-stubs
\environment followingup-style
\startchapter[title={Stubs}]
\startsection[title={Bare bone}]
The most barebone way to process a \CONTEXT\ file is something like:
\starttyping
luametatex
--fmt="<cache path to>/luametatex/cont-en"
--lua="<cache path to>/luametatex/cont-en.lui"
--jobname="article"
"cont-yes.mkiv"
\stoptyping
We pas extra options, like:
\starttyping
--c:autopdf
--c:currentrun=1
--c:fulljobname="./article.tex"
--c:input="./article.tex"
--c:kindofrun=1
--c:maxnofruns=9
--c:texmfbinpath="c:/data/develop/tex-context/tex/texmf-win64/bin"
\stoptyping
but for what we are going to discuss here it doesn't really matter. The main point is
that we use a \LUA\ startup file. That one has a minimal amount of code so that the
format can be loaded as we like it. For instance we need to start up with initial
memory settings.
The file \type {cont-yes} sets up the way processing content happens. This can be the
\type {jobname} file but also something different. It is enough to know that this
startup is quite controlled.
I will explore a different approach to format loading but for now this is how it
goes. After al, we need to be compatible with \LUATEX\ and normal \MKIV\ runs, at
least for now.
\stopsection
\startsection[title={Management (some history)}]
In \CONTEXT\ we always had a script: \type {texexec}, originally a \MODULA2
program, later a \PERL\ script, then a \RUBY\ script but now we have \type
{mtxrun}, a \LUA\ script. All take care of making sure that the file is
processed enough times to get the cross references, tables of contents, indexes,
multi|-|pass data stable. It also makes it possible to avoid using these special
binaries (or links) that trick the engine into thinking it is bound to a format:
we never had \type {pdfcontext} or \type {luacontext}, just one \type {context}.
Actually, because we have multiple user interfaces, we would have needed many
stubs instead. Getting this approach accepted was not easy but in the meantime
I've seen management scripts for other packages being mentioned occasionally.
The same is true for scripts: for a long time \CONTEXT\ came with quite some
scripts but when an average \TEX\ distribution started growing, including many
other scripts, we abandoned this approach and stuck to one management script that
also launched auxiliary scripts. That way we could be sure that there were no
clashes in names. If you look at a full \TEX\ installation you see many stubs to
scripts and more keep coming. How that can work out well without unexpected side
effects (name clashes) is not entirely clear to me, as a modern computer can have
large bin paths. Just imagine that all large programs (or ecosystems) would
introduce hundreds of new \quote {binaries}.
Anyway, in the end a \CONTEXT\ installation using \MKIV\ only needs \type {mtxrun}
and as bonus \type {context}. The above call is triggered by:
\starttyping
mtxrun --autogenerate --script context --autopdf article.tex
\stoptyping
from the editor. Here we create formats when none is found, and start or activate
the \PDF\ viewer afterwards, so more minimal is:
\starttyping
mtxrun --script context article.tex
\stoptyping
Normally there is also a \type {context} stub so this also works:
\starttyping
context article.tex
\stoptyping
\stopsection
\startsection[title={The launch process (more history)}]
In \MKII, when we use \PDFTEX, the actual launch of these script is somewhat
complex and a bit different per platform. But, on all platforms \KPSE\ does the
lookup of the script. Already long ago I found out that this startup overhead
could amount to seconds on a complete \TEX Live installation (imagine running
over a network) which is why eventually we came up with the minimals. The reason
is that the file databases have to be loaded: first for looking up, then for the
stub that also needs that information and finally by the actual program. There
were no \SSD's then.
The first hurdle we took was to combine the lookup and the runner. Of course this
is sort of out of our control because an installer can decide to still use a
lookup approach but at least on \MSWINDOWS\ this was achieved quite easy. Sort
of:
\starttyping
texexex -> [lookup] -->
texexec.pl -> [lookup] ->
pdftex + formats ->
[lookup] -> processing
\stoptyping
The first lookup can be avoided by some fast relative lookup, but for more
complex management the second one is always there. Over time this mechanism
became more sophisticated, for instance we use caching, could work over sockets
using a \KPSE\ server, etc.
When \LUATEX\ came around, it was already decided early that it also would serve
as script engine for the \CONTEXT\ runner, this time \type {mtxrun}. The way this
works differs per platform. On \WINDOWS\ there is a small binary, say \type
{runner.exe}. It gets two copies: \type {mtxrun.exe} and \type {context.exe}. If
you find more copies on your system, something might be wrong with your
installation.
\starttyping
mtxrun.exe -> loads mtxrun.lua in same path
context.exe -> idem but runs with --script=context
\stoptyping
The \type {mtxrun.lua} script will load its file database which is very efficient
and fast. It will then load the given script and execute it. In the case of \type
{context.exe} the \type {mtx-context.lua} script is loaded, which lives in the
normal place in the \TEX\ tree (alongside other scripts).
So, a minimal amount of programs and scripts is then:
\starttyping
texmf-win64/bin/luatex.exe
texmf-win64/bin/mtxrun.exe
texmf-win64/bin/mtxrun.lua
texmf-win64/bin/context.exe
\stoptyping
with (we also need to font manager):
\starttyping
texmf-context/scripts/context/lua/mtx-context.lua
texmf-context/scripts/context/lua/mtx-fonts.lua
\stoptyping
But \unknown\ there is a catch here: \LUATEX\ has to be started in script mode in
order to process \type {mtxrun}. So, in fact we see this in distributions.
\starttyping
texmf-win64/bin/luatex.exe
texmf-win64/bin/texlua.exe
texmf-win64/bin/mtxrun.exe
texmf-win64/bin/mtxrun.lua
texmf-win64/bin/context.exe
\stoptyping
The \type {texlua} program is just a copy of \type {luatex} that by its name
knows that is is supposed to run scripts and not process \TEX\ files. The setup
can be different using dynamic libraries (more files but a shared engine part)
but the principles are the same. Nowadays the stub doesn't need the \type
{texlua.exe} binary any more, so this is the real setup:
\starttyping
texmf-win64/bin/luatex.exe large program
texmf-win64/bin/mtxrun.exe small program
texmf-win64/bin/mtxrun.lua large lua file
texmf-win64/bin/context.exe small program
\stoptyping
Just for the record: we cannot really use batch files here because we need to
know the original command, and when run from a script that is normally not known.
It works to some extend but for instance when started indirectly from an editor
it can fail, depending on how that editor is calling programs. Therefore the stub
is the most robust method.
On a \UNIX\ system the situation differs:
\starttyping
texmf-linux-64/bin/luatex large program
texmf-linux-64/bin/texlua symlink to luatex
texmf-linux-64/bin/mtxrun large lua file
texmf-linux-64/bin/context shell script that starts mtxrun
\stoptyping
Here \type {mtxrun.lua} is renamed to \type {mtxrun} with a shebang line that
triggers loading by \type {texlua} which is a symlink to \type {luatex} because
shebang lines don't support the \type {--texlua} argument. As on windows, this
is not really pretty.
\stopsection
\startsection[title={The \LMTX\ way (the present)}]
Now when we move to \LMTX\ we need to make sure that the method that we choose is
acceptable for distributions but also nicely consistent over platforms. We only
have one binary \type {luametatex} with all messy logic removed and no second
face like \type {metaluatex}. When it is copied to another instance (or linked)
it will load the script with its own name when it finds one. So on \WINDOWS\ we
now have:
\starttyping
texmf-win64/bin/luametatex.exe medium program
texmf-win64/bin/mtxrun.exe copy (or link) of luametatex
texmf-win64/bin/mtxrun.lua large lua file
texmf-win64/bin/context.exe copy (or link) of luametatex
texmf-win64/bin/context.lua small lua file
\stoptyping
and in \UNIX:
\starttyping
texmf-linux-64/bin/luametatex mediumprogram
texmf-linux-64/bin/mtxrun copy (or link) of luametatex
texmf-linux-64/bin/mtxrun.lua large lua file
texmf-linux-64/bin/context copy (or link) of luametatex
texmf-linux-64/bin/context.lua small lua file
\stoptyping
So, \type {luametatex[.exe]}, \type {mtxrun[.exe]} and \type {context[.exe]} are
all the same. On both platforms there is \type {mtxrun.lua} (with suffix) and on
both we also use the same runner approach. The \type {context.lua} script is
really small and just sets the script command line argument before loading \type
{mtxrun.lua} from the same path. In the case of copied binaries: keep in mind
that the three copies together are not (much) larger than the \type {luatex} and
\type {texlua} pair (especially when you take additional libraries into account).
The disadvantage of using copies is that one can forget to copy with an update,
but the fact that one can use them might be easier for installers. It's up to
those who create the installers.
One complication is that the \type {mtxrun.lua} script has to deal with the old
and the new setup. But, when we release we will assume that one used either
\LUATEX\ or \LUAMETATEX, not some mix. As \type {mtxrun} and \type {context} know
what got it started they will then trigger the right engine, unless one passes
\typ {--engine=luatex}. In that case the \LUAMETATEX\ launcher will trigger a
\LUATEX\ run. But a mixed installation is unlikely to happen.
\stopsection
\startsection[title={Why not \unknown}]
Technically we could use one call for both the runner and \TEX\ processor but
when multiple runs are needed this would demand an internal engine reset as well
as macro package reset while keeping some (multi|-|pass) data around. A way
in|-|between could be to spawn the next run. In the end the gain would be minimal
(we have now .2 seconds overhead per total run, which can trigger multiple
passes, due to the management script, to basically we can neglect it. (Triggering
the viewer takes more time.)
\stopsection
\stopchapter
\stopcomponent
|
