summaryrefslogtreecommitdiff
path: root/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex
blob: cf8d2dca83c3f4a5542b1fb056dfb186b1d9f56b (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
% language=us runpath=texruns:manuals/luametatex

\environment luametatex-style

\startcomponent luametatex-lua

\startchapter[reference=lua,title={Using \LUAMETATEX}]

\startsection[title={Initialization},reference=init]

\startsubsection[title={A bare bone engine}]

Although the \LUAMETATEX\ engine will start up when you call the program it will
not do much useful. You can compare it to computer hardware without (high level)
operating system with a \TEX\ kernel being the bios. It can interpret \TEX\ code
but for typesetting you need a reasonable setup. You also need to load fonts, and
for output you need a backend, and both can be implemented in \LUA. If you don't
like that and want to get up and running immediately, you will be more happy with
\LUATEX, \PDFTEX\ or \XETEX, combined with your favorite macro package.

If you just want to play around you can install the \CONTEXT\ distribution which
(including manuals and some fonts) is tiny compared to a full \TEXLIVE\
installation and can be run alongside it without problems. If there are issues
you can go to the usual \CONTEXT\ support platforms and seek help where you can
find the people who made \LUATEX\ and \LUAMETATEX.

\stopsubsection

\startsubsection[title={\LUAMETATEX\ as a \LUA\ interpreter}]

\topicindex {initialization}
\topicindex {\LUA+interpreter}

Although \LUAMETATEX\ is primarily meant as a \TEX\ engine, it can also serve as
a stand alone \LUA\ interpreter. There are two ways to make \LUAMETATEX\ behave
like a standalone \LUA\ interpreter. The first method uses the command line
option \type {--luaonly} followed by a filename. The second is more automatic: if
the only non|-|option argument (file) on the commandline has the extension \type
{lmt} or \type {lua}. The \type {luc} extension has been dropped because bytecode
compiled files are not portable and one can always load indirect. The \type {lmt}
suffix is more \CONTEXT\ specific and makes it possible to have files for
\LUATEX\ and \LUAMETATEX\ alongside.

In this mode, it will set \LUA's \type {arg[0]} to the found script name, pushing
preceding options in negative values and the rest of the command line in the
positive values, just like the \LUA\ interpreter does.

\LUAMETATEX\ will exit immediately after executing the specified \LUA\ script and
is, in effect, a somewhat bulky stand alone \LUA\ interpreter with a bunch of
extra preloaded libraries. But we really want to keep the binary small, if
possible below the 3MB which is okay for a script engine.

When no argument is given, \LUAMETATEX\ will look for a \LUA\ file with the same
name as the binary and run that one when present. This makes it possible to use
the engine as a stub. For instance, in \CONTEXT\ a symlink from \type {mtxrun} to
type {luametatex} will run the \type {mtxrun.lmt} or \type {mtxrun.lua} script
when present in the same path as the binary itself. As mentioned before first
checking for (\CONTEXT) \type {lmt} files permits different files for different
engines in the same path.

\stopsubsection

\startsubsection[title={Other commandline processing}]

\topicindex {command line}

When the \LUAMETATEX\ executable starts, it looks for the \type {--lua} command line
option. If there is no \type {--lua} option, the command line is interpreted in a
similar fashion as the other \TEX\ engines. All options are accepted but only some
are understood by \LUAMETATEX\ itself:

\starttabulate[|l|p|]
\DB commandline argument    \BC explanation \NC \NR
\TB
\NC \type{--credits}        \NC display credits and exit \NC \NR
\NC \type{--fmt=FORMAT}     \NC load the format file \type {FORMAT} \NC\NR
\NC \type{--help}           \NC display help and exit \NC\NR
\NC \type{--ini}            \NC be \type {iniluatex}, for dumping formats \NC\NR
\NC \type{--jobname=STRING} \NC set the job name to \type {STRING} \NC \NR
\NC \type{--lua=FILE}       \NC load and execute a \LUA\ initialization script \NC\NR
\NC \type{--version}        \NC display version and exit \NC \NR
\LL
\stoptabulate

There are less options than with \LUATEX, because one has to deal with them in
\LUA\ anyway. So for instance there are no options to enter a safer mode or
control executing programs because this can easily be achieved with a startup
\LUA\ script.

Next the initialization script is loaded and executed. From within the script,
the entire command line is available in the \LUA\ table \type {arg}, beginning
with \type {arg[0]}, containing the name of the executable. As consequence
warnings about unrecognized options are suppressed.

Command line processing happens very early on. So early, in fact, that none of
\TEX's initializations have taken place yet. The \LUA\ libraries that don't deal
with \TEX\ are initialized rather soon so you have these available.

\LUAMETATEX\ allows some of the command line options to be overridden by reading
values from the \type {texconfig} table at the end of script execution (see the
description of the \type {texconfig} table later on in this document for more
details on which ones exactly).

The value to use for \prm {jobname} is decided as follows:

\startitemize
\startitem
    If \type {--jobname} is given on the command line, its argument will be the
    value for \prm {jobname}, without any changes. The argument will not be
    used for actual input so it need not exist. The \type {--jobname} switch only
    controls the \prm {jobname} setting.
\stopitem
\startitem
    Otherwise, \prm {jobname} will be the name of the first file that is read
    from the file system, with any path components and the last extension (the
    part following the last \type {.}) stripped off.
\stopitem
\startitem
    There is an exception to the previous point: if the command line goes into
    interactive mode (by starting with a command) and there are no files input
    via \prm {everyjob} either, then the \prm {jobname} is set to \type
    {texput} as a last resort.
\stopitem
\stopitemize

So let's summarize this. The handling of what is called jobname is a bit complex.
There can be explicit names set on the command line but when not set they can be
taken from the \type {texconfig} table.

\starttabulate[|l|T|T|T|]
\NC startup filename \NC --lua     \NC a \LUA\ file  \NC                      \NC \NR
\NC startup jobname  \NC --jobname \NC a \TEX\ tex   \NC texconfig.jobname    \NC \NR
\NC startup dumpname \NC --fmt     \NC a format file \NC texconfig.formatname \NC \NR
\stoptabulate

These names are initialized according to \type {--luaonly} or the first filename
seen in the list of options. Special treatment of \type {&} and \type {*} as well
as interactive startup is gone but we still enter \TEX\ via an forced \type {\input}
into the input buffer. \footnote {This might change at some point into an explicit
loading triggered via \LUA.}

When we are in \TEX\ mode at some point the engine needs a filename, for instance
for opening a log file. At that moment the set jobname becomes the internal one
and when it has not been set which internalized to jobname but when not set
becomes \type {texput}. When you see a \type {texput.log} file someplace on your
system it normally indicates a bad run.

When running on \MSWINDOWS\ the command line, filenames, environment variable
access etc.\ internally uses the current code page but to the user is exposed as
\UTF8. Normally users won't notice this.

% fileio_state     .jobname         : a tex string (set when a (log) file is opened)
% engine_state     .startup_jobname : handles by option parser
% environment_state.input_name      : temporary interceptor

There is an extra options \type{--permitloadlib} that needs to be given when you
load external libraries via \LUA. Although you could manage this via \LUA\ itself
in a startup script, the reason for having this as option is the wish for
security (at some point that became a demand for \LUATEX), so this might give an
extra feeling of protection.

\stopsubsection

\stopsection

\startsection[title={\LUA\ behaviour}]

\startsubsection[title={The \LUA\ version}]

\topicindex {\LUA+libraries}
\topicindex {\LUA+extensions}

We currently use \LUA\ 5.4 and will follow developments of the language but
normally with some delay. Therefore the user needs to keep an eye on (subtle)
differences in successive versions of the language. Here is an example of one
aspect.

\LUA s \type {tostring} function (and \type {string.format}) may return values in
scientific notation, thereby confusing the \TEX\ end of things when it is used as
the right|-|hand side of an assignment to a \prm {dimen} or \prm {count}. The
output of these serializers also depend on the \LUA\ version, so in \LUA\ 5.3 you
can get different output than from 5.2. It is best not to depend the automatic
cast from string to number and vise versa as this can change in future versions.

\stopsubsection

\startsubsection[title={Locales}]

\index {locales}

In stock \LUA, many things depend on the current locale. In \LUAMETATEX, we can't
do that, because it makes documents non|-|portable. While \LUAMETATEX\ is running
if forces the following locale settings:

\starttyping
LC_CTYPE=C
LC_COLLATE=C
LC_NUMERIC=C
\stoptyping

There is no way to change that as it would interfere badly with the often
language specific conversions needed at the \TEX\ end.

\stopsubsection

\stopsection

\startsection[title={\LUA\ modules}]

\topicindex {\LUA+libraries}
\topicindex {\LUA+modules}

Of course the regular \LUA\ modules are present. In addition we provide the \type
{lpeg} library by Roberto Ierusalimschy, This library is not \UNICODE|-|aware,
but interprets strings on a byte|-|per|-|byte basis. This mainly means that \type
{lpeg.S} cannot be used with \UTF8 characters that need more than one byte, and
thus \type {lpeg.S} will look for one of those two bytes when matching, not the
combination of the two. The same is true for \type {lpeg.R}, although the latter
will display an error message if used with multibyte characters. Therefore \type
{lpeg.R('aä')} results in the message \type {bad argument #1 to 'R' (range must
have two characters)}, since to \type {lpeg}, \type {ä} is two 'characters'
(bytes), so \type {aä} totals three. In practice this is no real issue and with
some care you can deal with \UNICODE\ just fine.

There are some more libraries present. These are discussed on a later chapter.
For instance we embed \type {luasocket} but contrary to \LUATEX\ don't embed the
related \LUA\ code. The \type {luafilesystem} module has been replaced by a more
efficient one that also deals with the \MSWINDOWS\ file and environment
properties better (\UNICODE\ support in \MSWINDOWS\ dates from before \UTF8
became dominant so we need to deal with wide \UNICODE16).

There are more extensive math libraries and there are libraries that deal with
encryption and compression. There are also some optional libraries that we do
interface but that are loaded on demand. The interfaces are as minimal as can be
because we so much in \LUA, which also means that one can tune behaviour to
usage better.

\stopsection

\startsection[title={Files}]

\startsubsection[title={File syntax}]

\topicindex {files+names}

\LUAMETATEX\ will accept a braced argument as a file name:

\starttyping
\input {plain}
\openin 0 {plain}
\stoptyping

This allows for embedded spaces, without the need for double quotes. Macro
expansion takes place inside the argument. Keep in mind that as side effect of
delegating \IO\ to \LUA\ the \tex {openin} primitive is not provided by the
engine and has to be implemented by the macro package. This also means that the
limit on the number of open files is not enforced by the engine.

\stopsubsection

\startsubsection[title={Writing to file}]

\topicindex {files+writing}

Writing to a file in \TEX\ has two forms: delayed and immediate. Delayed writing
means that the to be written text is anchored in the node list and flushed by the
backend. As all \IO\ is delegated to \LUA, this also means that it has to deal
with distinction. In \LUATEX\ the number of open files was already bumped to 127,
but in \LUAMETATEX\ it depends on the macro package. The special meaning of
channel 18 was already dropped in \LUATEX\ because we have \type {os.execute}.

\stopsubsection

\stopsection

\startsection[title={Testing}]

\topicindex {testing}

For development reasons you can influence the used startup date and time. By
setting the \type {start_time} variable in the \type {texconfig} table; as with
other variables we use the internal name there. When Universal Time is needed,
set the entry \type {use_utc_time} in the \type {texconfig} table.

In \CONTEXT\ we provide the command line argument \type {--nodates} that does
a bit more than disabling dates; it avoids time dependent information in the
output file for instance.

\stopsection

\startsection[reference=internals,title={The internals}]

\topicindex{nodes}
\topicindex{boxes}
\topicindex{\LUA}

This is a reference manual and not a tutorial. This means that we discuss changes
relative to traditional \TEX\ and also present new (or extended) functionality.
As a consequence we will refer to concepts that we assume to be known or that
might be explained later. Because the \LUATEX\ and \LUAMETATEX\ engines open up
\TEX\ there's suddenly quite some more to explain, especially about the way a (to
be) typeset stream moves through the machinery. However, discussing all that in
detail makes not much sense, because deep knowledge is only relevant for those
who write code not possible with regular \TEX\ and who are already familiar with
these internals (or willing to spend time on figuring it out).

So, the average user doesn't need to know much about what is in this manual. For
instance fonts and languages are normally dealt with in the macro package that
you use. Messing around with node lists is also often not really needed at the
user level. If you do mess around, you'd better know what you're dealing with.
Reading \quotation {The \TEX\ Book} by Donald Knuth is a good investment of time
then also because it's good to know where it all started. A more summarizing
overview is given by \quotation {\TEX\ by Topic} by Victor Eijkhout. You might
want to peek in \quotation {The \ETEX\ manual} too.

But \unknown\ if you're here because of \LUA, then all you need to know is that
you can call it from within a run. If you want to learn the language, just read
the well written \LUA\ book. The macro package that you use probably will provide
a few wrapper mechanisms but the basic \prm {directlua} command that does the job
is:

\starttyping
\directlua{tex.print("Hi there")}
\stoptyping

You can put code between curly braces but if it's a lot you can also put it in a
file and load that file with the usual \LUA\ commands. If you don't know what
this means, you definitely need to have a look at the \LUA\ book first.

If you still decide to read on, then it's good to know what nodes are, so we do a
quick introduction here. If you input this text:

\starttyping
Hi There ...
\stoptyping

eventually we will get a linked lists of nodes, which in \ASCII\ art looks like:

\starttyping
H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ...
\stoptyping

When we have a paragraph, we actually get something like this, where a \type
{par} node stores some metadata and is followed by a \type {hlist} flagged as
indent box:

\starttyping
[par] <=> [hlist] <=> H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ...
\stoptyping

Each character becomes a so called glyph node, a record with properties like the
current font, the character code and the current language. Spaces become glue
nodes. There are many node types and nodes can have many properties but that will
be discussed later. Each node points back to a previous node or next node, given
that these exist. Sometimes multiple characters are represented by one glyph
(shape), so one can also get:

\starttyping
[par] <=> [hlist] <=> H <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
\stoptyping

And maybe some characters get positioned relative to each other, so we might
see:

\starttyping
[par] <=> [hlist] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
\stoptyping

Actually, the above representation is one view, because in \LUAMETATEX\ we can
choose for this:

\starttyping
[par] <=> [glue] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
\stoptyping

where glue (currently fixed) is used instead of an empty hlist (think of a \type
{\hbox}). Options like this are available because want a certain view on these
lists from the \LUA\ end and the result being predicable is part of that.

It's also good to know beforehand that \TEX\ is basically centered around
creating paragraphs and pages. The par builder takes a list and breaks it into
lines. At some point horizontal blobs are wrapped into vertical ones. Lines are
so called boxes and can be separated by glue, penalties and more. The page
builder accumulates lines and when feasible triggers an output routine that will
take the list so far. Constructing the actual page is not part of \TEX\ but done
using primitives that permit manipulation of boxes. The result is handled back to
\TEX\ and flushed to a (often \PDF) file.

\starttyping
\setbox\scratchbox\vbox\bgroup
    line 1\par line 2
\egroup

\showbox\scratchbox
\stoptyping

The above code produces the next log lines that reveal how the engines sees a
paragraph (wrapped in a \type {\vbox}):

\starttyping[style=small]
1:4: > \box257=
1:4: \vbox[normal][16=1,17=1,47=1], width 483.69687, height 27.58083, depth 0.1416, direction l2r
1:4: .\list
1:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r
1:4: ...\list
1:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt
1:4: ....\glue[left][16=1,17=1,47=1] 0.0pt
1:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt
1:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519
1:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt
1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l
1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i
1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n
1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e
1:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30
1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000031 1
1:4: ....\penalty[line][16=1,17=1,47=1] 10000
1:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil
1:4: ....\glue[right][16=1,17=1,47=1] 0.0pt
1:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt
1:4: ..\glue[par][16=1,17=1,47=1] 5.44995pt plus 1.81665pt minus 1.81665pt
1:4: ..\glue[baseline][16=1,17=1,47=1] 6.79396pt
1:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r
1:4: ...\list
1:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt
1:4: ....\glue[left][16=1,17=1,47=1] 0.0pt
1:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt
1:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519
1:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt
1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l
1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i
1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n
1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e
1:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30
1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000032 2
1:4: ....\penalty[line][16=1,17=1,47=1] 10000
1:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil
1:4: ....\glue[right][16=1,17=1,47=1] 0.0pt
1:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt
\stoptyping

The \LUATEX\ engine provides hooks for \LUA\ code at nearly every reasonable
point in the process: collecting content, hyphenating, applying font features,
breaking into lines, etc. This means that you can overload \TEX's natural
behaviour, which still is the benchmark. When we refer to \quote {callbacks} we
means these hooks. The \TEX\ engine itself is pretty well optimized but when you
kick in much \LUA\ code, you will notices that performance drops. Don't blame and
bother the authors with performance issues. In \CONTEXT\ over 50\% of the time
can be spent in \LUA, but so far we didn't get many complaints about efficiency.
Adding more callbacks makes no sense, also because at some point the performance
hit gets too large. There are plenty of ways to achieve goals. For that reason:
take remarks about \LUATEX, features, potential, performance etc.\ with a natural
grain of salt.

Where plain \TEX\ is basically a basic framework for writing a specific style,
macro packages like \CONTEXT\ and \LATEX\ provide the user a whole lot of
additional tools to make documents look good. They hide the dirty details of font
management, language support, turning structure into typeset results, wrapping
pages, including images, and so on. You should be aware of the fact that when you
hook in your own code to manipulate lists, this can interfere with the macro
package that you use. Each successive step expects a certain result and if you
mess around to much, the engine eventually might bark and quit. It can even
crash, because testing everywhere for what users can do wrong is no real option.

When you read about nodes in the following chapters it's good to keep in mind
what commands relate to them. Here are a few:

\starttabulate[|l|l|p|]
\DB command              \BC node          \BC explanation \NC \NR
\TB
\NC \prm {hbox}          \NC \nod {hlist} \NC horizontal box \NC \NR
\NC \prm {vbox}          \NC \nod {vlist} \NC vertical box with the baseline at the bottom \NC \NR
\NC \prm {vtop}          \NC \nod {vlist} \NC vertical box with the baseline at the top \NC \NR
\NC \prm {hskip}         \NC \nod {glue}  \NC horizontal skip with optional stretch and shrink \NC \NR
\NC \prm {vskip}         \NC \nod {glue}  \NC vertical skip with optional stretch and shrink \NC \NR
\NC \prm {kern}          \NC \nod {kern}  \NC horizontal or vertical fixed skip \NC \NR
\NC \prm {discretionary} \NC \nod {disc}  \NC hyphenation point (pre, post, replace) \NC \NR
\NC \prm {char}          \NC \nod {glyph} \NC a character \NC \NR
\NC \prm {hrule}         \NC \nod {rule}  \NC a horizontal rule \NC \NR
\NC \prm {vrule}         \NC \nod {rule}  \NC a vertical rule \NC \NR
\NC \prm {textdirection} \NC \nod {dir}   \NC a change in text direction \NC \NR
\LL
\stoptabulate

Whatever we feed into \TEX\ at some point becomes a token which is either
interpreted directly or stored in a linked list. A token is just a number that
encodes a specific command (operator) and some value (operand) that further
specifies what that command is supposed to do. In addition to an interface to
nodes, there is an interface to tokens, as later chapters will demonstrate.

Text (interspersed with macros) comes from an input medium. This can be a file,
token list, macro body cq.\ arguments, some internal quantity (like a number),
\LUA, etc. Macros get expanded. In the process \TEX\ can enter a group. Inside
the group, changes to registers get saved on a stack, and restored after leaving
the group. When conditionals are encountered, another kind of nesting happens,
and again there is a stack involved. Tokens, expansion, stacks, input levels are
all terms used in the next chapters. Don't worry, they loose their magic once you
use \TEX\ a lot. You have access to most of the internals and when not, at least
it is possible to query some state we're in or level we're at.

When we talk about pack(ag)ing it can mean two things. When \TEX\ has consumed
some tokens that represent text they are added to the current list. When the text
is put into a so called \prm {hbox} (for instance a line in a paragraph) it
(normally) first gets hyphenated, next ligatures are build, and finally kerns are
added. Each of these stages can be overloaded using \LUA\ code. When these three
stages are finished, the dimension of the content is calculated and the box gets
its width, height and depth. What happens with the box depends on what macros do
with it.

The other thing that can happen is that the text starts a new paragraph. In that
case some information is stored in a leading \type {par} node. Then indentation
is appended and the paragraph ends with some glue. Again the three stages are
applied but this time afterwards, the long line is broken into lines and the
result is either added to the content of a box or to the main vertical list (the
running text so to say). This is called par building. At some point \TEX\ decides
that enough is enough and it will trigger the page builder. So, building is
another concept we will encounter. Another example of a builder is the one that
turns an intermediate math list into something typeset.

Wrapping something in a box is called packing. Adding something to a list is
described in terms of contributing. The more complicated processes are wrapped
into builders. For now this should be enough to enable you to understand the next
chapters. The text is not as enlightening and entertaining as Don Knuths books,
sorry.

\stopsection

\stopchapter

\stopcomponent