1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
|
% language=uk
\startcomponent mk-tokenspeak
\environment mk-environment
\chapter {Token speak}
\subject{tokenization}
Most \TEX\ users only deal with (keyed in) characters and (produced) output. Some
will play with boxes, skips and kerns or maybe even leaders (repeated sequences
of the former). Others will be grateful that macro package writers take care of
such things.
Macro writers on the other hand deal properties of characters, like catcodes and
a truckload of other codes, with lists made out of boxes, skips, kerns and
penalties but even they cannot look much deeper into \TEX's internals. Their
deeper understanding comes from reading the \TEX book or even looking at the
source code.
When someone enters the magic world of \TEX\ and starts asking around on a bit,
he or she will at some point get confronted with the concept of \quote {tokens}.
A token is what ends up in \TEX\ after characters have entered its machinery.
Sometimes it even seems that one is only considered a qualified macro writer if
one can talk the right token||speak. So what are those magic tokens and how can
\LUATEX\ shed light on this.
In a moment we will show examples of how \LUATEX\ turns characters into tokens,
but when looking at those sequences, you need to keep a few things in mind:
\startitemize[packed]
\startitem
A sequence of characters that starts with an escape symbol (normally this is
the backslash) is looked up in the hash table (which relates those names to
meanings) and replaced by its reference. Such a reference is much faster than
looking up the sequence each time.
\stopitem
\startitem
Characters can have special meanings, for instance a dollar is often used to
enter and exit math mode, and a percent symbol starts a comment and hides
everything following it on the same line. These meanings are determined by
the character's catcode.
\stopitem
\startitem
All the characters that will end up actually typeset have catcode \quote
{letter} or \quote {other} assigned. A sequence of items with catcode
\quote{letter} is considered a word and can potentially become hyphenated.
\stopitem
\stopitemize
\subject{examples}
We will now provide a few examples of how \TEX\ sees your input.
\starttyping
Hi there!
\stoptyping
\starttokens[demo]Hi there!\stoptokens \setups{ShowCollect}
Here we see three kind ot tokens. At this stage a space is still recognizable as
such but later this will become a skip. In our current setup, the exclamation
mark is not a letter.
\starttyping
Hans \& Taco use Lua\TeX \char 33\relax
\stoptyping
\starttokens[demo]Hans \& Taco use Lua\TeX \char 33\relax\stoptokens \setups{ShowCollect}
Here we see a few new tokens, a \quote {char\_given} and a \quote {call}. The
first represents a \type {\chardef} i.e.\ a reference to a character slot in a
font, and the second one a macro that will expand to the \TEX\ logo. Watch how
the space after a control sequence is eaten up. The exclamation mark is a direct
reference to character slot~33.
\starttyping
\noindent {\bf Hans} \par \hbox{Taco} \endgraf
\stoptyping
\starttokens[demo]\noindent {\bf Hans} \par \hbox{Taco} \endgraf\stoptokens \setups{ShowCollect}
As you can see, some primitives and macro's that are bound to them (like \type
{\endgraf}) have an internal representation on top of their name.
\starttyping
before \dimen2=10pt after \the\dimen2
\stoptyping
\starttokens[demo]before \dimen2=10pt after \the\dimen2\stoptokens \setups{ShowCollect}
As you can see, registers are not explicitly named, one needs the associated
register code to determine it's character (a dimension in our case).
\starttyping
before \inframed[width=3cm]{whatever} after
\stoptyping
\starttokens[demo]before \inframed[width=3cm]{whatever} after\stoptokens \setups{ShowCollect}
As you can see, even when control sequences are collapsed into a reference, we
still end up with many tokens, and because each token has three properties (cmd,
chr and id) in practice we end up with more memory used after tokenization.
\starttyping
compound|-|word
\stoptyping
\starttokens[demo]compound|-|word\stoptokens \setups{ShowCollect}
This example uses an active character to handle compound words (a \CONTEXT\
feature).
\starttyping
hm, \directlua 0 { tex.sprint("Hello World") }
\stoptyping
\starttokens[demo]hm, \directlua 0 { tex.sprint("Hello World!") }\stoptokens \setups{ShowCollect}
The previous example shows what happens when we include a bit of \LUA\ code
\unknown\ it is just seen as regular input, but when the string is passed to
\LUA, only the chr property is passed, so we no longer can distinguish between
letters and other characters.
A macro definition converts to tokens as follows.
\starttokens[demo]\def\Test#1#2{[#2][#1]} \Test{A}{B}\stoptokens \setups{ShowCollect}
As we already mentioned, a token has three properties. More details can be found
in the reference manual so we will not go into much detail here.
{\bf The original interceptor for tokens but that one has been replaced by a more
powerful scanning mechanism. The following text is no longer applicable but kept
as historic reference. The new token scanner is discussed in later articles.}
% keep text formatted as it is now:
\starttyping[color=]
A most simple callback is:
\starttyping
callback.register('token_filter', token.get_next)
\stoptyping
In principle you can call \type {token.get_next} anytime you want
to intercept a token. In that case you can feed back tokens into
\TEX\ by using a trick like:
\starttyping
function tex.printlist(data)
callback.register('token_filter', function ()
callback.register('token_filter', nil)
return data
end)
end
\stoptyping
Another example of usage is:
\starttyping
callback.register('token_filter', function ()
local t = token.get_next
local cmd, chr, id = t[1], t[2], t[3]
-- do something with cmd, chr, id
return { cmd, chr, id }
end)
\stoptyping
There is a whole repertoire of related functions, one is \type
{token.create}, which can be used as:
\starttyping
tex.printlist{
token.create("hbox"),
token.create(utf.byte("{"), 1),
token.create(utf.byte("?"), 12),
token.create(utf.byte("}"), 2),
}
\stoptyping
This results in: \ctxlua {
tex.printlist{
token.create("hbox"),
token.create(utf.byte("{"), 1),
token.create(utf.byte("?"), 12),
token.create(utf.byte("}"), 2),
}
}
While playing with this we made a few auxiliary functions that
permit things like:
\starttyping
tex.printlist ( table.unnest ( {
tokens.hbox,
tokens.bgroup,
tokens.letters("12345"),
tokens.egroup,
} ) )
\stoptyping
Unnesting is needed because the result of the \type {letters} call
is a table, and the \type {printlist} function wants a flattened
table.
The result looks like: \ctxlua {
local t = table.unnest {
tokens.hbox,
tokens.bgroup,
tokens.letters("12345"),
tokens.egroup,
}
tex.printlist (t)
tokens.collectors.show(t)
}
In practice, manipulating tokens or constructing lists of tokens
this way is rather cumbersome, but at least we now have some
kind of access, if only for illustrative purposes.
\starttyping
\hbox{12345\hbox{54321}}
\stoptyping
can also be done by saying:
\starttyping
tex.sprint("\\hbox{12345\\hbox{54321}}")
\stoptyping
or under \CONTEXT's basic catcode regime:
\starttyping
tex.sprint(tex.ctxcatcodes, "\\hbox{12345\\hbox{54321}}")
\stoptyping
If you like it the hard way:
\starttyping
tex.printlist ( table.unnest ( {
tokens.hbox,
tokens.bgroup,
tokens.letters("12345"),
tokens.hbox,
tokens.bgroup,
tokens.letters(string.reverse("12345")),
tokens.egroup,
tokens.egroup
} ) )
\stoptyping
This method may attract those who dislike the traditional \TEX\
syntax for doing the same thing. Okay, a careful reader will
notice that reversing the string in \TEX\ takes a bit more
trickery, so \unknown
\stoptyping
% end of verbose text
{\bf The \type {tokens} etc.\ examples shows here make no sense anyway as we have
a more extensive interface to the macro language: \type {context}.}
\stopcomponent
|
