1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
|
% language=us runpath=texruns:manuals/lowlevel
\environment lowlevel-style
\startdocument
[title=inserts,
color=middlecyan]
\startsectionlevel[title=Introduction]
This document is a mixed bag. We do discuss inserts but also touch elements of
the page builder because inserts and regular page content are handled there.
Examples of mechanisms that use inserts are footnotes. These have an anchor in
the running text and some content that ends up (normally) at the bottom of the
page. When considering a page break the engine tries to make sure that the anchor
(reference) and the content end up on the same page. When there is too much, it
will distribute (split) the content over pages.
We can discuss page breaks in a (pseudo) scientific way and explore how to
optimize this process, taking into accounts also inserts that contain images but
it doesn't make much sense to do that because in practice we can encounter all
kind of interferences. Theory and practice are too different because a document
can contain a wild mix of text, figures, formulas, notes, have backgrounds and
location dependent processing. It get seven more complex when we are dealing with
columns because \TEX\ doesn't really know that concept.
I will therefore stick to some practical aspects and the main reason for this
document is that I sort of document engine features and at the same time give an
impression of what we deal with. I will do that in the perspective of
\LUAMETATEX, which has a few more options and tracing than other engines.
{\em Currently this document is mostly for myself to keep track of the state of
inserts and the page builder in \LUAMETATEX\ and \CONTEXT\ \LMTX. The text is not
yet corrected and can have errors.}
\stopsectionlevel
\startsectionlevel[title=The page builder]
When your document is processed content eventually gets added to the so called
main vertical list (mvl). Content first get appended to the list of contributions
and at specific moments it will be handed over to the mvl. This process is called
page building. There we can encounter the following elements (nodes):
\starttabulate
\NC \type {glue} \NC a vertical skip \NC \NR
\NC \type {penalty} \NC a vertical penalty \NC \NR
\NC \type {kern} \NC a vertical kern \NC \NR
\NC \type {vlist} \NC a a vertical box \NC \NR
\NC \type {hlist} \NC a horizontal box (often a line) \NC \NR
\NC \type {rule} \NC a horizontal rule \NC \NR
\NC \type {boundary} \NC a boundary node \NC \NR
\NC \type {whatsit} \NC a node that is used by user code (often some extension) \NC \NR
\NC \type {mark} \NC a token list (as used for running headers) \NC \NR
\NC \type {insert} \NC a node list (as used for notes) \NC \NR
\stoptabulate
The engine itself will not insert anything other than this but \LUA\ code can
mess up the contribution list and the mvl and that can trigger an error. Handing
over the contributions is done by the page builder and that one kicks in in
several places:
\startitemize[packed]
\startitem
When a penalty gets inserted it is part of evaluating if the output routine
should be triggered. This triggering can be enforced by values equal or below
10.000 that then can be checked in the set routine.
\stopitem
\startitem
The builder is {\em not} exercised when a glue or kern is injected so there can
be multiple of them before another element triggers the builder.
\stopitem
\startitem
Adding a box triggers the builder as does the result of an alignment which can
be a list of boxes.
\stopitem
\startitem
When the output routine is finished the builder is executed because the
routine can have pushed back content.
\stopitem
\startitem
When a new paragraph is triggered by the \type {\par} command the builder kicks in
but only when the engine was able to enter vertical mode.
\stopitem
\startitem
When the job is finished the builder will make sure that pending content is handled.
\stopitem
\startitem
An insert and vadjust {\em can} trigger the builder but only at the nesting level zero
which normally is not the case (I need an example).
\stopitem
\startitem
At the beginning of a paragraph (like text), before display math is entered,
and when display math ends the builder is also activated.
\stopitem
\stopitemize
At the \TEX\ the builder is triggered automatically in the mentioned cases but at
the \LUA\ end you can use \type {tex.triggerbuildpage()} to flush the pending
contributions.
The properties that relate to the page look like counter and dimension registers ut
they are not. These variables are global and managed differently.
\starttabulate
\NC \type {\pagegoal} \NC the available space \NC \NR
\NC \type {\pagetotal} \NC the accumulated space \NC \NR
\NC \type {\pagestretch} \NC the possible zero order stretch \NC \NR
\NC \type {\pagefilstretch} \NC the possible one order stretch \NC \NR
\NC \type {\pagefillstretch} \NC the possible second order stretch \NC \NR
\NC \type {\pagefilllstretch} \NC the possible third order stretch \NC \NR
\NC \type {\pageshrink} \NC the possible shrink \NC \NR
\NC \type {\pagedepth} \NC the current page depth \NC \NR
\NC \type {\pagevsize} \NC the initial page goal \NC \NR
\stoptabulate
When the first content is added to an empty page the \type {\pagegoal} gets the
value of \type {\vsize} and gets frozen but the value is diminished by the space
needed by left over inserts. These inserts are managed via a separate list so
they don't interfere with the page that itself of course can have additional
inserts. The \type {\pagevsize} is just a (\LUAMETATEX) status variable that hold
the initial \type {\pagegoal} but it might play a role in future extensions.
Another variable is \type {\deadcycles} that registers the number of times the
output routine is called without returning result.
\stopsectionlevel
\startsectionlevel[title=Inserts]
We now come to inserts. In traditional \TEX\ an insert is a data structure that
runs on top of registers: a box, count, dimension and skip. An insert is accessed
by a number so for instance insert 123 will use the four registers of that
number. Because \TEX\ only offers a command alias mechanism for registers (like
\type {\countdef}) a macro package will implement some allocator management
subsystem (like \type {\newcount}). A \type {\newinsert} has to be defined in a
way that the four registers are not clashing with other allocators. When you
start with \TEX\ seeing code that deals with in (in plain \TEX) can be puzzling
but it follows from the way \TEX\ is set up. But inserts are probably not what
you start exploring right away away.
In \LUAMETATEX\ you can set \type {\insertmode} to 1 and that is what we do in
\CONTEXT. In that mode inserts are taken from a pool instead of registers. A side
effect is that like the page properties the insert properties are global too but
that is normally no problem and can be managed well by a macro package (that
probably would assign register the values globally too). The insert pool will
grow dynamically on demand so one can just start at 1; in \CONTEXT\ \MKIV\ we use
the range 127 upto 255 in order to avoid a clash with registers. In \LMTX\ we start
at 1 because there are no clashes.
A consequence of this approach is that we use dedicated commands to set the insert
properties:
\starttabulate[|l|l|p|]
\NC \type {\insertdistance} \NC glue \NC the space before the first instance (on a page) \NC \NR
\NC \type {\insertmultiplier} \NC count \NC a factor that is used to calculate the height used \NC \NR
\NC \type {\insertlimit} \NC dimen \NC the maximum amount of space on a page to be taken \NC \NR
\NC \type {\insertpenalty} \NC count \NC the floating penalty (used when set) \NC \NR
\NC \type {\insertmaxdepth} \NC dimen \NC the maximum split depth (used when set) \NC \NR
\NC \type {\insertstorage} \NC count \NC signals that the insert has to be stored for later \NC \NR
\NC \type {\insertheight} \NC dimen \NC the accumulated height of the inserts so far \NC \NR
\NC \type {\insertdepth} \NC dimen \NC the current depth of the inserts so far \NC \NR
\NC \type {\insertwidth} \NC dimen \NC the width of the inserts \NC \NR
\stoptabulate
These commands take a number and an integer, dimension or glue specification.
They can be set and queried but setting the dimensions can have side effects. The
accumulated height of the inserts is available in \type {\insertheights} (which
can be set too). The \type {\floatingpenalty} variable determines the penalty
applied when a split is needed.
In the output routine the original \TEX\ variable \type {\insertpenalties} is a
counter that keeps the number of insertions that didn't fit on the page while
otherwise if has the accumulated penalties of the split insertions. When \type
{\holdinginserts} is non zero the inserts in the list are not collected for
output, which permits the list to be fed back for reprocessing.
The \LUAMETATEX\ specific storage mode \type {\insertstoring} variable is
explained in the next section.
\stopsectionlevel
\startsectionlevel[title=Storing]
This feature is kind of special and still experimental. When \type
{\insertstoring} is set 1, all inserts that have their storage flag set will be
saved. Think of a multi column setup where inserts have to end up in the last
column. If there are three columns, the first two will store inserts. Then when
the last column is dealt with \type {\insertstoring} can be set to 2 and that
will signal the builder that we will inject the inserts. In both cases, the value
of this register will be set to zero so that it doesn't influence further
processing.
\stopsectionlevel
\startsectionlevel[title=Callbacks]
{\em Todo, nothing new there, so no hurry.}
\stopsectionlevel
\stopdocument
|
