path: root/doc/context/sources/general/manuals/still/still-expanding.tex

% language=uk

\environment still-environment

\starttext

\startchapter[title=Possibly useful extensions]

\startsection[title=Introduction]

While working on \LUATEX, it is tempting to introduce all kinds of new fancy
programming features. Arguments for doing this can be characterized by
descriptions like \quote {handy}, \quote {speedup}, \quote {less code}, \quote
{necessity}. It must be stated that traditional \TEX\ is rather complete, and one
can do quite a lot of macro magic to achieve many goals. So let us look a bit
more at the validity of these arguments.

The \quote {handy} argument is in fact a valid one. Of course, one can always
wrap clumsy code in a macro to hide the dirty tricks, but, still, it would be
nicer to avoid needing to employ extremely dirty tricks. I found myself looking
at old code wondering why something has to be done in such a complex way, only to
realize, after a while, that it comes with the concept; one can get accustomed to
it. After all, every programming language has its stronger and weaker aspects.

The \quote {speedup} argument is theoretically a good one too, but, in practice,
it's hard to prove that a speedup really occurs. Say we save 5\% on a job. This
is nice for multipass on a server where many jobs run at the same time or after
each other, but a little bit of clever macro coding will easily gain much more.
Or, as we often see: sloppy macro or style writing will easily negate those
gains. Another pitfall is that you can measure (say) half a million calls to a
macro can indeed be brought down to a fraction of its runtime thanks to some
helper, but, in practice, you will not see that gain because saving 0.1 seconds
on a 10 second run can be neglected. Furthermore, adding a single page to the
document will already make such a gain invisible to the user as that will itself
increase the runtime. Of course, many small speedups can eventually accumulate to
yield a significant overall gain, but, if the macro package is already quite
optimized, it might not be easy to squeeze out much more. At least in \CONTEXT, I
find it hard to locate bottlenecks that could benefit from extensions, unless one
adds very specific features, which is not what we want.

Of course one can create \quote {less} code by using more wrappers. But this can
definitely have a speed penalty, so this argument should be used with care. An
appropriate extra helper can make wrappers fast and the fewer helpers the better.
The danger is in choosing what helpers. A good criterion is that it should be
hard otherwise in \TEX. Adding more primitives (and overhead) merely because some
macro package would like it would be bad practice. I'm confident that helpers for
\CONTEXT\ would not be that useful for plain \TEX, \LATEX, etc., and vice versa.

The \quote {necessity} argument is a strong one. Many already present extensions
from \ETEX\ fall into this category: fully expandable expressions (although the
implementation is somewhat restricted), better macro protection, expansion
control, and the ability to test for a so|-|called csname (control sequence name)
are examples.

In the end, the only valid argument is \quote {it can't be done otherwise}, which
is a combination of all these arguments with \quote {necessity} being dominant.
This is why in \LUATEX\ there are not that many extensions to the language (nor
will there be). I must admit that even after years of working with \TEX, the
number of wishes for more facilities is not that large.

The extensions in \LUATEX, compared to traditional \TEX, can be summarized as
follows:

\startitemize
    \startitem
        Of course we have the \ETEX\ extensions, and these already have
        a long tradition of proven usage. We did remove the limited directional
        support.
    \stopitem
    \startitem
        From \ALEPH\ (follow-up on \OMEGA), part of the directional support and
        some font support was inherited.
    \stopitem
    \startitem
        From \PDFTEX, we took most of the backend code, but it has been improved
        in the meantime. We also took the protrusion and expansion code, but
        especially the latter has been implemented a bit differently (in the
        frontend as well as in the backend).
    \stopitem
    \startitem
        Some handy extensions from \PDFTEX\ have been generalized; other
        obscure or specialized ones have been removed. So we now have
        frontend support for position tracking, resources (images) and reusable
        content in the core. The backend code has been separated a bit better and
        only a few backend|-|related primitives remain.
    \stopitem
    \startitem
        The input encoding is now \UTF-8, exclusively, but one can easily hook in
        code to preprocess data that enters \TEX's parser using \LUA. The
        characteristic catcode settings for \TEX\ can be grouped and switched
        efficiently.
    \stopitem
    \startitem
        The font machinery has been opened wide so that we can use the embedded
        \LUA\ interpreter to implement any technology that we might want, with
        the usual control that \TEX ies like. Some further limitations have been
        lifted. One interesting point is that one can now construct virtual fonts
        at runtime.
    \stopitem
    \startitem
        Ligature construction, kerning and paragraph building have been separated
        as a side effect of \LUA\ control. There are some extensions in that
        area. For instance, we store the language and min|/|max values in the
        glyph nodes, and we also store penalties with discretionaries. Patterns
        can be loaded at runtime, and character codes that influence
        hyphenation can be manipulated.
    \stopitem
    \startitem
        The math renderer has been upgraded to support \OPENTYPE\ math. This has
        resulted in many new primitives and extensions, not only to define
        characters and spacing, but also to control placement of superscripts and
        subscripts and generally to influence the way things are constructed. A
        couple of mechanisms have gained control options.
    \stopitem
    \startitem
        Several \LUA\ interfaces are available making it possible to manipulate the
        (intermediate) results. One can pipe text to \TEX, write parsers, mess
        with node lists, inspect attributes assigned at the \TEX\ end, etc.
    \stopitem
\stopitemize

Some of the features mentioned above are rather \LUATEX\ specific, such as
catcode tables and attributes. They are present as they permit more advanced
\LUA\ interfacing. Other features, such as \UTF-8\ and \OPENTYPE\ math, are a
side effect of more modern techniques. Bidirectional support is there because it
was one of the original reasons for going forward with \LUATEX. The removal of
backend primitives and thereby separating the code in a better way (see companion
article) comes from the desire to get closer to the traditional core, so that
most documentation by Don Knuth still applies. It's also the reason why we still
speak of \quote {tokens}, \quote {nodes} and \quote {noads}.

In the following sections I will discuss a few new low|-|level primitives. This
is not a complete description (after all, we have reported on much already), and
one can consult the \LUATEX\ manual to get the complete picture. The extensions
described below are also relatively new and date from around version 0.85, the
prelude to the stable version~1 release.

\stopsection

\startsection[title=Rules]

For insiders, it is no secret that \TEX\ has no graphic capabilities, apart from
the ability to draw rules. But with rules you can do quite a lot already. Add to
that the possibility to insert arbitrary graphics or even backend drawing
directives, and the average user won't notice that it's not true core
functionality.

When we started with \LUATEX, we used code from \PDFTEX\ and \OMEGA\ (\ALEPH),
and, as a consequence, we ended up with many whatsits. Normal running text has
characters, kerns, some glue, maybe boxes, all represented by a limited set of
so|-|called nodes. A whatsit is a kind of escape as it can be anything an
extension to \TEX\ needs to wrap up and put in the current list. Examples are (in
traditional \TEX\ already) whatsits that write to file (using \type {\write}) and
whatsits that inject code into the backend (using \type {\special}). The
directional mechanism of \OMEGA\ uses whatsits to indicate direction changes.

For a long time images were also included using whatsits, and basically one had
to reserve the right amount of space and inject a whatsit with a directive for
the backend to inject something there with given dimensions or scale. Of course,
one then needs methods to figure out the image properties, but, in the end, all
of this could be done rather easily.

In \PDFTEX, two new whatsits were introduced: images and reusable so|-|called
forms, and, contrary to other whatsits, these do have dimensions. As a result,
suddenly the \TEX\ code base could no longer just ignore whatsits, but it had to
check for these two when dimensions were important, for instance in the paragraph
builder, packager, and backend.

So what has this to do with rules? Well, in \LUATEX\ all the whatsits are now
back to where they belong, in the backend extension code. Directions are now
first|-|class nodes, and we have native resources and reusable boxes. These
resources and boxes are an abstraction of the \PDFTEX\ images and forms, and,
internally, they are a special kind of rule (i.e.\ a blob with dimensions).
Because checking for rules is part of the (traditional) \TEX\ kernel, we could
simply remove the special whatsit code and let existing rule|-|related code do
the job. This simplified the code a lot.

Because we suddenly had two more types of rules, we took the opportunity to add a
few more.

\starttyping
\nohrule width 10cm height 2cm depth 0cm
\novrule width 10cm height 2cm depth 0cm
\stoptyping

This is a way to reserve space, and it's nearly equivalent to the following
(respectively):

\starttyping
{\setbox0\hbox{}\wd0=10cm\ht0=2cm\dp0=0cm\box0\relax}
{\setbox0\vbox{}\wd0=10cm\ht0=2cm\dp0=0cm\box0\relax}
\stoptyping

There is no real gain in efficiency because keywords also take time to parse, but
the advantage is that no \LUA\ callbacks are triggered. \footnote {I still am
considering adding variants of \type {\hbox} and \type {\vbox} where no callback
would be triggered.} Of course, this variant would not have been introduced had
we still had just rules and no further subtypes; it was just a rather trivial
extension that fit in the repertoire. \footnote {This is one of the things I
wanted to have for a long time but seems less useful today.}

So, while we were at it, yet another rule type was introduced, but this one has
been made available only in \LUA. As this text is about \LUATEX, a bit of \LUA\
code does fit into the discussion, so here we go. The code shown here is rather
generic and looks somewhat different in \CONTEXT, but it does the job.

First, let's create a straightforward rectangle drawing routine. We initialize
some variables first, then scan properties using the token scanner, and, finally,
we construct the rectangle using four rules. The packaged (so|-|called) hlist is
written to \TEX.

\startbuffer
\startluacode
function FramedRule()
    local width     = 0
    local height    = 0
    local depth     = 0
    local linewidth = 0
    --
    while true do
        if token.scan_keyword("width") then
            width = token.scan_dimen()
        elseif token.scan_keyword("height") then
            height = token.scan_dimen()
        elseif token.scan_keyword("depth") then
            depth = token.scan_dimen()
        elseif token.scan_keyword("line") then
            linewidth = token.scan_dimen()
        else
            break
        end
    end
    local doublelinewidth = 2*linewidth
    --
    local left    = node.new("rule")
    local bottom  = node.new("rule")
    local right   = node.new("rule")
    local top     = node.new("rule")
    local back    = node.new("kern")
    local list    = node.new("hlist")
    --
    left.width    = linewidth
    bottom.width  = width - doublelinewidth
    bottom.height = -depth + linewidth
    bottom.depth  = depth
    right.width   = linewidth
    top.width     = width - doublelinewidth
    top.height    = height
    top.depth     = -height + linewidth
    back.kern     = -width + linewidth
    list.list     = left
    list.width    = width
    list.height   = height
    list.depth    = depth
    list.dir      = "TLT"
    --
    node.insert_after(left,left,bottom)
    node.insert_after(left,bottom,right)
    node.insert_after(left,right,back)
    node.insert_after(left,back,top)
    --
    node.write(list)
 end
\stopluacode
\stopbuffer

\typebuffer \getbuffer

This function can be wrapped in a macro:

\startbuffer
\def\FrameRule{\directlua{FramedRule()}}
\stopbuffer

\typebuffer \getbuffer

and the macro can be used as follows:

\startbuffer
\FrameRule width 3cm height 1cm depth 1cm line 2pt
\stopbuffer

\typebuffer

The result is: \inlinebuffer

A different approach follows. Again, we define a rule, but, this time we only set
dimensions and assign some attributes to it. Normally, one would reserve some
attribute numbers for this purpose, but, for our example here, high numbers are
safe enough. Now there is no need to wrap the rule in a box.

\startbuffer
\startluacode
function FramedRule()
    local width     = 0
    local height    = 0
    local depth     = 0
    local linewidth = 0
    local radius    = 0
    local type      = 0
    --
    while true do
        if token.scan_keyword("width") then
            width = token.scan_dimen()
        elseif token.scan_keyword("height") then
            height = token.scan_dimen()
        elseif token.scan_keyword("depth") then
            depth = token.scan_dimen()
        elseif token.scan_keyword("line") then
            linewidth = token.scan_dimen()
        elseif token.scan_keyword("type") then
            type = token.scan_int()
        elseif token.scan_keyword("radius") then
            radius = token.scan_dimen()
        else
            break
        end
    end
    --
    local r   = node.new("rule")
    r.width   = width
    r.height  = height
    r.depth   = depth
    r.subtype = 4 -- user rule
    r[20000]  = type
    r[20001]  = linewidth
    r[20002]  = radius or 0
    node.write(r)
end
\stopluacode
\stopbuffer

\typebuffer \getbuffer

Nodes with subtype~4 (user) are intercepted and passed to a callback function,
when set. Here we show a possible implementation:

\startbuffer
\startluacode
local bpfactor = (7200/7227)/65536

local f_rectangle = "%f w 0 0 %f %f re %s"

local f_radtangle = [[
    %f w %f 0 m
    %f 0 l %f %f %f %f y
    %f %f l %f %f %f %f y
    %f %f l %f %f %f %f y
    %f %f l %f %f %f %f y
    h %s
]]

callback.register("process_rule",function(n,h,v)
    local t = n[20000] == 0 and "f" or "s"
    local l = n[20001] * bpfactor -- linewidth
    local r = n[20002] * bpfactor -- radius
    local w = h * bpfactor
    local h = v * bpfactor
    if r > 0 then
        p = string.format(f_radtangle,
            l, r, w-r, w,0,w,r, w,h-r, w,h,w-r,h,
            r,h, 0,h,0,h-r, 0,r, 0,0,r,0, t)
    else
        p = string.format(f_rectangle, l, w, h, t)
    end
    pdf.print("direct",p)
end)
\stopluacode
\stopbuffer

\typebuffer \getbuffer

We can now also specify a radius and type, where \type {0} is a filled and \type
{1} a stroked shape.

\startbuffer
\FrameRule
    type   1
    width  3cm
    height 1cm
    depth  5mm
    line   0.2mm
    radius 2.5mm
\stopbuffer

\typebuffer

Since we specified a radius we get round corners: \inlinebuffer

The nice thing about these extensions to rules is that the internals of \TEX\ are
not affected much. Rules are just blobs with dimensions and the par builder, for
instance, doesn't care what they are. There is no need for further inspection.
Maybe future versions of \LUATEX\ will provide more useful subtypes.

\stopsection

\startsection[title=Spaces]

Multiple successive spaces in \TEX\ are normally collapsed into one. But, what if
you don't want any spaces at all? It turns out this is rather hard to achieve.
You can, of course, change the catcodes, but that won't work well if you pass
text around as macro arguments. Also, you would not want spaces that separate
macros and text to be ignored, but only those in the typeset text. For such use,
\LUATEX\ introduces \type {\nospaces}.

This new primitive can be used to overrule the usual \type {\spaceskip}|-|related
heuristics when a space character is seen in a text flow. The value~\type{1}
specifies no injection, a value of \type{2} results in injection of a zero skip,
and the default \type{0} gets the standard behavior. Below we see the results for
four characters separated by spaces.

\startlinecorrection \dontcomplain
\startcombination[nx=3,ny=2,distance=1cm]
    {\ruledhbox to 4cm{\vtop{\hsize 10mm\nospaces=0\relax x x x x \par}\hss}} {\type {0 / hsize 10mm}}
    {\ruledhbox to 4cm{\vtop{\hsize 10mm\nospaces=1\relax x x x x \par}\hss}} {\type {1 / hsize 10mm}}
    {\ruledhbox to 4cm{\vtop{\hsize 10mm\nospaces=2\relax x x x x \par}\hss}} {\type {2 / hsize 10mm}}
    {\ruledhbox to 4cm{\vtop{\hsize  1mm\nospaces=0\relax x x x x \par}\hss}} {\type {0 / hsize 1mm}}
    {\ruledhbox to 4cm{\vtop{\hsize  1mm\nospaces=1\relax x x x x \par}\hss}} {\type {1 / hsize 1mm}}
    {\ruledhbox to 4cm{\vtop{\hsize  1mm\nospaces=2\relax x x x x \par}\hss}} {\type {2 / hsize 1mm}}
\stopcombination
\stoplinecorrection

In case you wonder why setting the space related skips to zero is not enough:
even when it is set to zero you will always get something. What gets inserted
depends on \type {\spaceskip}, \type {\xspaceskip}, \type {\spacefactor} and font
dimensions. I must admit that I always have to look up the details, as, normally,
it's wrapped up in a spacing system that you implement once then forget about. In
any case, with \type {\nospaces}, you can completely get rid of even an inserted
zero space.

\stopsection

\startsection[title=Token lists]

The following four primitives are provided because they are more efficient than
macro|-|based variants: \type {\toksapp}, \type {\tokspre}, and \type {\e...}
(expanding) versions of both. They can be used to append or prepend tokens to a
token register.

However, don't overestimate the gain that can be brought in simple situations
with not that many tokens involved (read: there is no need to instantly change
all code that does it the traditional way). The new method avoids saving tokens
in a temporary register. Then, when you combine registers (which is also
possible), the source gets appended to the target and, afterwards, the source is
emptied: we don't copy but combine!

Their use can best be demonstrated by examples. We employ a scratch register
\type {\ToksA}. The examples here show the effects of grouping; in fact, they
were written for testing this effect. Because we don't use the normal assignment
code, we need to initialize a local copy in order to get the original content
outside the group.

\newtoks\ToksA
\newtoks\ToksB

\startbuffer
\ToksA{}
\bgroup
   \ToksA{}
   \bgroup \toksapp\ToksA{!!} [\the\ToksA=!!] \egroup
   [\the\ToksA=]
\egroup
[\the\ToksA=]
\stopbuffer

\typebuffer result: {\nospacing\start\tttf\inlinebuffer\stop}

\startbuffer
\ToksA{}
\bgroup
    \ToksA{A}
    \bgroup \toksapp\ToksA{!!} [\the\ToksA=A!!] \egroup
    [\the\ToksA=A]
\egroup
[\the\ToksA=]
\stopbuffer

\typebuffer result: {\nospacing\start\tttf\inlinebuffer\stop}

\startbuffer
\ToksA{}
\bgroup
    \ToksA{}
    \bgroup
        \ToksA{A} \toksapp\ToksA{!!} [\the\ToksA=A!!]
    \egroup
    [\the\ToksA=]
\egroup
[\the\ToksA=]
\stopbuffer

\typebuffer result: {\nospacing\start\tttf\inlinebuffer\stop}

\startbuffer
\ToksA{}
\bgroup
    \ToksA{A}
    \bgroup
        \ToksA{} \toksapp\ToksA{!!} [\the\ToksA=!!]
    \egroup
    [\the\ToksA=A]
\egroup
[\the\ToksA=]
\stopbuffer

\typebuffer result: {\nospacing\start\tttf\inlinebuffer\stop}


\startbuffer
\ToksA{}
\bgroup
    \ToksA{}
    \bgroup
        \tokspre\ToksA{!!} [\the\ToksA=!!]
    \egroup
   [\the\ToksA=]
\egroup
[\the\ToksA=]
\stopbuffer

\typebuffer result: {\nospacing\start\tttf\inlinebuffer\stop}

\startbuffer
\ToksA{}
\bgroup
    \ToksA{A}
    \bgroup
        \tokspre\ToksA{!!} [\the\ToksA=!!A]
    \egroup
    [\the\ToksA=A]
\egroup
[\the\ToksA=]
\stopbuffer

\typebuffer result: {\nospacing\start\tttf\inlinebuffer\stop}

\startbuffer
\ToksA{}
\bgroup
    \ToksA{}
    \bgroup
        \ToksA{A} \tokspre\ToksA{!!} [\the\ToksA=!!A]
    \egroup
    [\the\ToksA=]
\egroup
[\the\ToksA=]
\stopbuffer

\typebuffer result: {\nospacing\start\tttf\inlinebuffer\stop}

\startbuffer
\ToksA{}
\bgroup
    \ToksA{A}
    \bgroup
        \ToksA{} \tokspre\ToksA{!!} [\the\ToksA=!!]
    \egroup
    [\the\ToksA=A]
\egroup
[\the\ToksA=]
\stopbuffer

\typebuffer result: {\nospacing\start\tttf\inlinebuffer\stop}

Here we used \type {\toksapp} and \type {\tokspre}, but there are two more
primitives, \type {\etoksapp} and \type {\etokspre}; these expand the given
content while it gets added.

The next example demonstrates that you can also append another token list. In
this case the original content is gone after an append or prepend.

\startbuffer
\ToksA{A}
\ToksB{B}
\toksapp\ToksA\ToksB
\toksapp\ToksA\ToksB
[\the\ToksA=AB]
\stopbuffer

\typebuffer result: {\nospacing\start\tttf\inlinebuffer\stop}

This is intended behaviour! The original content of the source is not copied but
really appended or prepended. Of course, grouping works well.

\startbuffer
\ToksA{A}
\ToksB{B}
\bgroup
    \toksapp\ToksA\ToksB
    \toksapp\ToksA\ToksB
    [\the\ToksA=AB]
\egroup
[\the\ToksA=AB]
\stopbuffer

\typebuffer result: {\nospacing\start\tttf\inlinebuffer\stop}

\stopsection

\startsection[title=Active characters]

We now enter an area of very dirty tricks. If you have read the \TEX\ book or
listened to talks by \TEX\ experts, you will, for sure, have run into the term
\quote {active} characters. In short, it boils down to this: each character has a
catcode and there are 16 possible values. For instance, the backslash normally
has catcode zero, braces have values one and two, and normal characters can be 11
or 12. Very special are characters with code 13 as they are \quote {active} and
behave like macros. In Plain \TEX, the tilde is one such active character, and
it's defined to be a \quote {non|-|breakable space}. In \CONTEXT, the vertical
bar is active and used to indicate compound and fence constructs.

Below is an example of a definition:

\starttyping
\catcode`A=13
\def A{B}
\stoptyping

This will make the \type {A} into an active character that will typeset a \type
{B}. Of course, such an example is asking for problems since any \type {A} is
seen that way, so a macro name that uses one will not work. Speaking of macros:

\starttyping
\def\whatever
  {\catcode`A=13
   \def A{B}}
\stoptyping

This won't work out well. When the macro is read it gets tokenized and stored and
at that time the catcode change is not yet done so when this macro is called the
A is frozen with catcode letter (11) and the \type {\def} will not work as
expected (it gives an error). The solution is this:

\starttyping
\bgroup
\catcode`A=13
\gdef\whatever
  {\catcode`A=13
   \def A{B}}
\egroup
\stoptyping

Here we make the \type {A} active before the definition and we use grouping
because we don't want that to be permanent. But still we have a hard|-|coded
solution, while we might want a more general one that can be used like this:

\starttyping
\whatever{A}{B}
\whatever{=}{{\bf =}}
\stoptyping

Here is the definition of \type {whatever}:

\starttyping
\bgroup
\catcode`~=13
\gdef\whatever#1#2%
  {\uccode`~=`#1\relax
   \catcode`#1=13
   \uppercase{\def\tempwhatever{~}}%
   \expandafter\gdef\tempwhatever{#2}}
\egroup
\stoptyping

If you read backwards, you can imagine that \type {\tempwhatever} expands into an
active \type {A} (the first argument). So how did it become one? The trick is in
the \type {\uppercase} (a \type {\lowercase} variant will also work). When casing
an active character, \TEX\ applies the (here) uppercase and makes the result
active too.

We can argue about the beauty of this trick or its weirdness, but it is a fact
that for a novice user this indeed looks more than a little strange. And so, a
new primitive \type {\letcharcode} has been introduced, not so much out of
necessity but simply driven by the fact that, in my opinion, it looks more
natural. Normally the meaning of the active character can be put in its own
macro, say:

\starttyping
\def\MyActiveA{B}
\stoptyping

We can now directly assign this meaning to the active character:

\starttyping
\letcharcode`A=\MyActiveA
\stoptyping

Now, when \type {A} is made active this meaning kicks in.

\starttyping
\def\whatever#1#2%
  {\def\tempwhatever{#2}%
   \letcharcode`#1\tempwhatever
   \catcode`#1=13\relax}
\stoptyping

We end up with less code but, more important, it is easier to explain to a user
and, in my eyes, it looks less obscure, too. Of course, the educational gain here
wins over any practical gain because a macro package hides such details and only
implements such an active character installer once.

\stopsection

\startsection[title=\type {\csname} and friends]

You can check for a macro being defined as follows:

\starttyping
\ifdefined\foo
    do something
\else
    do nothing
\fi
\stoptyping

which, of course, can be obscured to:

\starttyping
do \ifdefined\foo some\else no\fi thing
\stoptyping

A bit more work is needed when a macro is defined using \type {\csname}, in which
case arbitrary characters (like spaces) can be used:

\starttyping
\ifcsname something or nothing\endcsname
    do something
\else
    do nothing
\fi
\stoptyping

Before \ETEX, this was done as follows:

\starttyping
\expandafter\ifx\csname something or nothing\endcsname\relax
    do nothing
\else
    do something
\fi
\stoptyping

The \type {\csname} primitive will do a lookup and create an entry in the hash
for an undefined name that then defaults to \type {\relax}. This can result in
many unwanted entries when checking potential macro names. Thus, \ETEX's \type
{\ifcsname} test primitive can be qualified as a \quote {necessity}.

Now take the following example:

\starttyping
\ifcsname do this\endcsname
    \csname do this\endcsname
\else\ifcsname do that\endcsname
    \csname do that\endcsname
\else
    \csname do nothing\endcsname
\fi\fi
\stoptyping

If \type {do this} is defined, we have two lookups. If it is undefined and \type
{do that} is defined, we have three lookups. So there is always one redundant
lookup. Also, when no match is found, \TEX\ has to skip to the \type {\else} or
\type {\fi}. One can save a bit by uglifying this to:

\starttyping
\csname do%
    \ifcsname do this\endcsname this\else
    \ifcsname do that\endcsname that\else
                            nothing\fi\fi
\endcsname
\stoptyping

This, of course, assumes that there is always a final branch. So let's get back
to:

\starttyping
\ifcsname do this\endcsname
    \csname do this\endcsname
\else\ifcsname do that\endcsname
    \csname do that\endcsname
\fi\fi
\stoptyping

As said, when there is some match, there is always one test too many. In case you
think this might be slowing down \TEX, be warned: it's hard to measure. But as
there can be (m)any character(s) involved, including multi|-|byte \UTF-8\
characters or embedded macros, there is a bit of penalty in terms of parsing
token lists and converting to \UTF\ strings used for the lookup. And, because
\TEX\ has to give an error message in case of troubles, the already|-|seen tokens
are stored too.

So, in order to avoid this somewhat redundant operation of parsing, memory
allocation (for the lookup string) and storing tokens, the new primitive \type
{\lastnamedcs} is now provided:

\starttyping
\ifcsname do this\endcsname
    \lastnamedcs
\else\ifcsname do that\endcsname
    \lastnamedcs
\fi\fi
\stoptyping

In addition to the (in practice, often negligible) speed gain, there are other
advantages: \TEX\ has less to skip, and although skipping is fast, it still isn't
a nice side effect (also useful when tracing). Another benefit is that we don't
have to type the to|-|be|-|looked|-|up text twice. This reduces the chance of
errors. In our example we also save 16 tokens (taking 64 bytes) in the format
file. So, there are enough benefits to gain from this primitive, which is not a
specific feature, but just an extension to an existing mechanism.

It also works in this basic case:

\starttyping
\csname do this\endcsname
\lastnamedcs
\stoptyping

And even this works:

\starttyping
\csname do this\endcsname
\expandafter\let\expandafter\dothis\lastnamedcs
\stoptyping

And after:

\starttyping
\bgroup
\expandafter\def\csname do this\endcsname{or that}
\global\expandafter\let\expandafter\dothis\lastnamedcs
\expandafter\def\csname do that\endcsname{or this}
\global\expandafter\let\expandafter\dothat\lastnamedcs
\egroup
\stoptyping

We can use \type {\dothis} that gives \type {or that} and \type {\dothat} that
gives \type {or this}, so we have the usual freedom to be able to use something
meant to make code clean for the creation of obscure code. % Amen!

A variation on this is the following:

\starttyping
\begincsname do this\endcsname
\stoptyping

This call will check if \type {\do this} is defined, and, if so, will expand it.
However, when \type {\do this} is not found, it does not create a hash entry. It
is equivalent to:

\starttyping
\ifcsname do this\endcsname\lastnamedcs\fi
\stoptyping

but it avoids the \type {\ifcsname}, which is sometimes handy as these tests can
interfere.

I played with variations like \type {\ifbegincsname}, but we then quickly end up
with dirty code due to the fact that we first expand something and then need to
deal with the following \type {\else} and \type {\fi}. The two above|-|mentioned
primitives are non|-|intrusive in the sense that they were relatively easy to add
without obscuring the code base.

As a bonus, \LUATEX\ also provides a variant of \type {\string} that doesn't add
the escape character: \type {\csstring}. There is not much to explain to this:

\starttyping
\string\whatever<>\csstring\whatever
\stoptyping

This gives: \expanded{\type{\string\whatever<>\csstring\whatever}}.

The main advantage of these several new primitives is that a bit less code is
needed and (at least for \CONTEXT) leads to a bit less tracing output. When you
enable \type {\tracingall} for a larger document or example, which is sometimes
needed to figure out a problem, it's not much fun to work with the resulting
megabyte (or sometimes even gigabyte) of output so the more we can get rid of,
the better. This consequence is just an unfortunate side effect of the \CONTEXT\
user interface with its many parameters. As said, there is no real gain in speed.

\stopsection

\startsection[title=Packing]

Deep down in \TEX, horizontal and vertical lists eventually get packed. Packing
of an \type {\hbox} involves:

\startitemize[n,packed]
\startitem ligature building (for traditional \TEX\ fonts), \stopitem
\startitem kerning (for traditional \TEX\ fonts), \stopitem
\startitem calling out to \LUA\ (when enabled) and \stopitem
\startitem wrapping the list in a box and calculating the width. \stopitem
\stopitemize

When a \LUA\ function is called, in most cases, the location where it happens
(group code) is also passed. But say that you try the following:

\starttyping
\hbox{\hbox{\hbox{\hbox foo}}}
\stoptyping

Here we do all four steps, while for the three outer boxes, only the last step
makes any sense. And it's not trivial to avoid the application of the \LUA\
function here. Of course, one can assign an attribute to the boxes and use that
to intercept, but it's kind of clumsy. This is why we now can say:

\starttyping
\hpack{\hpack{\hpack{\hbox foo}}}
\stoptyping

There are also \type {\vpack} for a \type {\vbox} and \type {\tpack} for a \type
{\vtop}. There can be a small gain in speed when many complex manipulations are
done, although in, for instance, \CONTEXT, we already have provisions for that.
It's just that the new primitives are a cleaner way out of a conceptually nasty
problem. Similar functions are available on the \LUA\ side.

\stopsection

\startsection[title=Errors]

We end with a few options that can be convenient to use if you don't care about
exact compatibility.

\starttyping
\suppresslongerror
\suppressmathparerror
\suppressoutererror
\suppressifcsnameerror
\stoptyping

When entering your document on a paper teletype terminal, starting \TEX, and then
going home in order to have a look at the result the next day, it does make sense
to catch runaway cases, like premature ending of a paragraph (using \type {\par}
or equivalent empty lines), or potentially missing \type {$$}s. Nowadays, it's
less important to catch such coding issues (and be more tolerant) because editing
takes place on screen and running (and restarting) \TEX\ is very fast.

The first two flags given above deal with this. If you set the first to any value
greater than zero, macros not defined as \type {\long} (not accepting paragraph
endings) will not complain about \cs{par} tokens in arguments. The second setting
permits and ignores empty lines (also pars) in math without reverting to dirty
tricks. Both are handy when your content comes from places that are outside of
your control. The job will not be aborted (or hang) because of an empty line.

The third setting suppresses the \type {\outer} directive so that macros that
originally can only be used at the outer level can now be used anywhere. It's
hard to explain the concept of outer (and the related error message) to a user
anyway.

The last one is a bit special. Normally, when you use \type {\ifcsname} you will
get an error when \TEX\ sees something unexpandable or that can't be part of a
name. But sometimes you might find it to be quite acceptable and can just
consider the condition as false. When the fourth variable is set to non|-|zero,
\TEX\ will ignore this issue and try to finish the check properly, so basically
you then have an \type {\iffalse}.

\stopsection

\startsection[title=Final remarks]

I mentioned performance a number of times, and it's good to notice that most
changes discussed here will potentially be faster than the alternatives, but this
is not always noticeable, in practice. There are several reasons.

For one thing, \TEX\ is already highly optimized. It has speedy memory management
of tokens and nodes and unnecessary code paths are avoided. However, due to
extensions to the original code, a bit more happens in the engine than in decades
past. For instance, \UNICODE\ fonts demand sparse arrays instead of fixed|-|size,
256|-|slot data structures. Handling \UTF\ involves more testing and construction
of more complex strings. Directional typesetting leads to more testing and
housekeeping in the frontend as well as the backend. More keywords to handle, for
instance \type {\hbox}, result in more parsing and pushing back unmatched tokens.
Some of the penalty has been compensated for through the changing of whatsits
into regular nodes. In recent versions of \LUATEX, scanning of \type {\hbox}
arguments is somewhat more efficient, too.

In any case, any speedup we manage to achieve, as said before, can easily become
noise through inefficient macro coding or user's writing bad styles. And we're
pretty sure that not much more speed can be squeezed out. To achieve higher
performance, it's time to buy a machine with a faster \CPU\ (and a huge cache),
faster memory (lanes), an \SSD, and regularly check your coding.

\stopsection

\stopchapter

\stoptext