path: root/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex

% language=uk

\environment luametatex-style

\startcomponent luametatex-enhancements

\startchapter[reference=enhancements,title={Basic \TEX\ enhancements}]

\startsection[title={Introduction}]

\startsubsection[title={Primitive behaviour}]

From day one, \LUATEX\ has offered extra features compared to the superset of
\PDFTEX, which includes \ETEX, and \ALEPH. This has not been limited to the
possibility to execute \LUA\ code via \prm {directlua}, but \LUATEX\ also adds
functionality via new \TEX|-|side primitives or extensions to existing ones. The
same is true for \LUAMETATEX. Some primitives have \type {luatex} in their name
and there will be no \type {luametatex} variants. This is because we consider
\LUAMETATEX\ to be \LUATEX 2\high{+}.

Contrary to the \LUATEX\ engine \LUAMETATEX\ enables all its primitives. You can
clone (a selection of) primitives with a different prefix, like:

\starttyping
\directlua { tex.enableprimitives('normal',tex.extraprimitives()) }
\stoptyping

The \type {extraprimitives} function returns the whole list or a subset,
specified by one or more keywords \type {core}, \type {tex}, \type {etex} or
\type {luatex}. \footnote {At some point this function might be changed to return
the whole list always}.

But be aware that the curly braces may not have the proper \prm {catcode}
assigned to them at this early time (giving a \quote {Missing number} error), so
it may be needed to put these assignments before the above line:

\starttyping
\catcode `\{=1
\catcode `\}=2
\stoptyping

More fine|-|grained primitives control is possible and you can look up the
details in \in {section} [luaprimitives]. There are only three kinds of
primitives: \type {tex}, \type {etex} and \type {luatex} but a future version
might drop this and no longer make that distinction as it no longer serves
a purpose.

\stopsubsection

\startsubsection[title={Experiments}]

There are a few extensions to the engine regarding the macro machinery. Some are
already well tested but others are (still) experimental. Although they are likely
to stay, their exact behaviour might evolve. Because \LUAMETATEX\ is also used
for experiments, this is not a problem. We can always decide to also add some of
what is discussed here to \LUATEX, but it will happen with a delay.

There are all kinds of small improvements that might find their way into stock
\LUATEX: a few more helpers, some cleanup of code, etc. We'll see. In any case,
if you play with these before they are declared stable, unexpected side effects
are what you have to accept.

\stopsubsection

\startsubsection[title={Version information}]

\startsubsubsection[title={\lpr {luatexbanner}, \lpr {luatexversion} and \lpr {luatexrevision}}]

\topicindex{version}
\topicindex{banner}

There are three primitives to test the version of \LUATEX\ (and \LUAMETATEX):

\unexpanded\def\VersionHack#1% otherwise different luatex and luajittex runs
  {\ctxlua{%
     local banner = "\luatexbanner"
     local banner = string.match(banner,"(.+)\letterpercent(") or banner
     context(string.gsub(banner ,"jit",""))%
  }}

\starttabulate[|l|l|pl|]
\DB primitive             \BC value
                          \BC explanation \NC \NR
\TB
\NC \lpr {luatexbanner}   \NC \VersionHack{\luatexbanner}
                          \NC the banner reported on the command line \NC \NR
\NC \lpr {luatexversion}  \NC \the\luatexversion
                          \NC a combination of major and minor number \NC \NR
\NC \lpr {luatexrevision} \NC \the\luatexrevision
                          \NC the revision number \NC \NR
\LL
\stoptabulate

A version is defined as follows:

\startitemize
\startitem
    The major version is the integer result of \lpr {luatexversion} divided by
    100. The primitive is an \quote {internal variable}, so you may need to prefix
    its use with \prm {the} or \prm {number} depending on the context.
\stopitem
\startitem
    The minor version is a number running from 0 upto 99.
\stopitem
\startitem
    The revision is reported by \lpr {luatexrevision}. Contrary to other engines
    in \LUAMETATEX\ is also a number so one needs to prefix it with \prm {the} or
    \prm {number}. \footnote {In the past it always was good to prefix the
    revision with \prm {number} anyway, just to play safe, although there have
    for instance been times that \PDFTEX\ had funny revision indicators that at
    some point ended up as letters due to the internal conversions.}
\stopitem
\startitem
    The full version number consists of the major version (\type {X}), minor
    version (\type {YY}) and revision (\type {ZZ}), separated by dots, so \type
    {X.YY.ZZ}.
\stopitem
\stopitemize

\stopsubsubsection

The \LUAMETATEX\ version number starts at~2 in order to prevent a clash with
\LUATEX, and the version commands are the same. This is a way to indicate that
these projects are related.

\startsubsubsection[title={\lpr {formatname}}]

\topicindex{format}

The \lpr {formatname} syntax is identical to \prm {jobname}. In \INITEX, the
expansion is empty. Otherwise, the expansion is the value that \prm {jobname} had
during the \INITEX\ run that dumped the currently loaded format. You can use this
token list to provide your own version info.

\stopsubsubsection

\stopsubsection

\stopsection

\startsection[title={\UNICODE\ text support}]

\startsubsection[title={Extended ranges}]

\topicindex{\UNICODE}

Text input and output is now considered to be \UNICODE\ text, so input characters
can use the full range of \UNICODE\ ($2^{20}+2^{16}-1 = \hbox{0x10FFFF}$). Later
chapters will talk of characters and glyphs. Although these are not
interchangeable, they are closely related. During typesetting, a character is
always converted to a suitable graphic representation of that character in a
specific font. However, while processing a list of to|-|be|-|typeset nodes, its
contents may still be seen as a character. Inside the engine there is no clear
separation between the two concepts. Because the subtype of a glyph node can be
changed in \LUA\ it is up to the user. Subtypes larger than 255 indicate that
font processing has happened.

A few primitives are affected by this, all in a similar fashion: each of them has
to accommodate for a larger range of acceptable numbers. For instance, \prm
{char} now accepts values between~0 and $1{,}114{,}111$. This should not be a
problem for well|-|behaved input files, but it could create incompatibilities for
input that would have generated an error when processed by older \TEX|-|based
engines. The affected commands with an altered initial (left of the equal sign)
or secondary (right of the equal sign) value are: \prm {char}, \prm {lccode},
\prm {uccode}, \lpr {hjcode}, \prm {catcode}, \prm {sfcode}, \lpr {efcode}, \lpr
{lpcode}, \lpr {rpcode}, \prm {chardef}.

As far as the core engine is concerned, all input and output to text files is
\UTF-8 encoded. Input files can be pre|-|processed using the \type {reader}
callback. This will be explained in \in {section} [iocallback]. Normalization of
the \UNICODE\ input is on purpose not built|-|in and can be handled by a macro
package during callback processing. We have made some practical choices and the
user has to live with those.

Output in byte|-|sized chunks can be achieved by using characters just outside of
the valid \UNICODE\ range, starting at the value $1{,}114{,}112$ (0x110000). When
the time comes to print a character $c>=1{,}114{,}112$, \LUATEX\ will actually
print the single byte corresponding to $c$ minus 1{,}114{,}112.

Contrary to other \TEX\ engines, the output to the terminal is as|-|is so there
is no escaping with \type {^^}. We operate in a \UTF\ universe.

\stopsubsection

\startsubsection[title={\lpr {Uchar}}]

\topicindex{\UNICODE}

The expandable command \lpr {Uchar} reads a number between~0 and $1{,}114{,}111$
and expands to the associated \UNICODE\ character.

\stopsubsection

\startsubsection[title={Extended tables}]

All traditional \TEX\ and \ETEX\ registers can be 16-bit numbers. The affected
commands are:

\startfourcolumns
\startlines
\prm {count}
\prm {dimen}
\prm {skip}
\prm {muskip}
\prm {marks}
\prm {toks}
\prm {countdef}
\prm {dimendef}
\prm {skipdef}
\prm {muskipdef}
\prm {toksdef}
\prm {insert}
\prm {box}
\prm {unhbox}
\prm {unvbox}
\prm {copy}
\prm {unhcopy}
\prm {unvcopy}
\prm {wd}
\prm {ht}
\prm {dp}
\prm {setbox}
\prm {vsplit}
\stoplines
\stopfourcolumns

Fonts are loaded via \LUA\ and a minimal amount of information is kept at the
\TEX\ end. Sharing resources is up to the loaders. The engine doesn't really care
about what a character (or glyph) number represents (a \UNICODE\ or index) as it
only is interested in dimensions.

\stopsubsection

\stopsection

\startsection[title={Attributes}]

\startsubsection[title={Nodes}]

\topicindex {nodes}

When \TEX\ reads input it will interpret the stream according to the properties
of the characters. Some signal a macro name and trigger expansion, others open
and close groups, trigger math mode, etc. What's left over becomes the typeset
text. Internally we get a linked list of nodes. Characters become \nod {glyph}
nodes that have for instance a \type {font} and \type {char} property and \typ
{\kern 10pt} becomes a \nod {kern} node with a \type {width} property. Spaces are
alien to \TEX\ as they are turned into \nod {glue} nodes. So, a simple paragraph
is mostly a mix of sequences of \nod {glyph} nodes (words) and \nod {glue} nodes
(spaces). A node can have a subtype so that it can be recognized as for instance
a space related glue.

The sequences of characters at some point are extended with \nod {disc} nodes
that relate to hyphenation. After that font logic can be applied and we get a
list where some characters can be replaced, for instance multiple characters can
become one ligature, and font kerns can be injected. This is driven by the
font properties.

Boxes (like \prm {hbox} and \prm {vbox}) become \nod {hlist} or \nod {vlist}
nodes with \type {width}, \type {height}, \type {depth} and \type {shift}
properties and a pointer \type {list} to its actual content. Boxes can be
constructed explicitly or can be the result of subprocesses. For instance, when
lines are broken into paragraphs, the lines are a linked list of \nod {hlist}
nodes, possibly with glue and penalties in between.

Internally nodes have a number. This number is actually an index in the memory
used to store nodes.

So, to summarize: all that you enter as content eventually becomes a node, often
as part of a (nested) list structure. They have a relative small memory footprint
and carry only the minimal amount of information needed. In traditional \TEX\ a
character node only held the font and slot number, in \LUATEX\ we also store some
language related information, the expansion factor, etc. Now that we have access
to these nodes from \LUA\ it makes sense to be able to carry more information
with a node and this is where attributes kick in.

\stopsubsection

\startsubsection[title={Attribute registers}]

\topicindex {attributes}

Attributes are a completely new concept in \LUATEX. Syntactically, they behave a
lot like counters: attributes obey \TEX's nesting stack and can be used after
\prm {the} etc.\ just like the normal \prm {count} registers.

\startsyntax
\attribute <16-bit number> <optional equals> <32-bit number>!crlf
\attributedef <csname> <optional equals> <16-bit number>
\stopsyntax

Conceptually, an attribute is either \quote {set} or \quote {unset}. Unset
attributes have a special negative value to indicate that they are unset, that
value is the lowest legal value: \type {-"7FFFFFFF} in hexadecimal, a.k.a.
$-2147483647$ in decimal. It follows that the value \type {-"7FFFFFFF} cannot be
used as a legal attribute value, but you {\it can\/} assign \type {-"7FFFFFFF} to
\quote {unset} an attribute. All attributes start out in this \quote {unset}
state in \INITEX.

Attributes can be used as extra counter values, but their usefulness comes mostly
from the fact that the numbers and values of all \quote {set} attributes are
attached to all nodes created in their scope. These can then be queried from any
\LUA\ code that deals with node processing. Further information about how to use
attributes for node list processing from \LUA\ is given in~\in {chapter}[nodes].

Attributes are stored in a sorted (sparse) linked list that are shared when
possible. This permits efficient testing and updating. You can define many
thousands of attributes but normally such a large number makes no sense and is
also not that efficient because each node carries a (possibly shared) link to a
list of currently set attributes. But they are a convenient extension and one of
the first extensions we implemented in \LUATEX.

In \LUAMETATEX\ we try to minimize the memory footprint and creation of these
attribute lists more aggressive sharing them. This feature is still somewhat
experimental.

\stopsubsection

\startsubsection[title={Box attributes}]

\topicindex {attributes}
\topicindex {boxes}
\topicindex {vcentering}

Nodes typically receive the list of attributes that is in effect when they are
created. This moment can be quite asynchronous. For example: in paragraph
building, the individual line boxes are created after the \prm {par} command has
been processed, so they will receive the list of attributes that is in effect
then, not the attributes that were in effect in, say, the first or third line of
the paragraph.

Similar situations happen in \LUATEX\ regularly. A few of the more obvious
problematic cases are dealt with: the attributes for nodes that are created
during hyphenation, kerning and ligaturing borrow their attributes from their
surrounding glyphs, and it is possible to influence box attributes directly.

When you assemble a box in a register, the attributes of the nodes contained in
the box are unchanged when such a box is placed, unboxed, or copied. In this
respect attributes act the same as characters that have been converted to
references to glyphs in fonts. For instance, when you use attributes to implement
color support, each node carries information about its eventual color. In that
case, unless you implement mechanisms that deal with it, applying a color to
already boxed material will have no effect. Keep in mind that this
incompatibility is mostly due to the fact that separate specials and literals are
a more unnatural approach to colors than attributes.

It is possible to fine|-|tune the list of attributes that are applied to a \type
{hbox}, \type {vbox} or \type {vtop} by the use of the keyword \type {attr}. The
\type {attr} keyword(s) should come before a \type {to} or \type {spread}, if
that is also specified. An example is:

\startbuffer[tex]
\attribute997=123
\attribute998=456
\setbox0=\hbox {Hello}
\setbox2=\hbox attr 999 = 789 attr 998 = -"7FFFFFFF{Hello}
\stopbuffer

\startbuffer[lua]
  for b=0,2,2 do
    for a=997, 999 do
      tex.sprint("box ", b, " : attr ",a," : ",tostring(tex.box[b]     [a]))
      tex.sprint("\\quad\\quad")
      tex.sprint("list ",b, " : attr ",a," : ",tostring(tex.box[b].list[a]))
      tex.sprint("\\par")
    end
  end
\stopbuffer

\typebuffer[tex]

Box 0 now has attributes 997 and 998 set while box 2 has attributes 997 and 999
set while the nodes inside that box will all have attributes 997 and 998 set.
Assigning the maximum negative value causes an attribute to be ignored.

To give you an idea of what this means at the \LUA\ end, take the following
code:

\typebuffer[lua]

Later we will see that you can access properties of a node. The boxes here are so
called \nod {hlist} nodes that have a field \type {list} that points to the
content. Because the attributes are a list themselves you can access them by
indexing the node (here we do that with \type {[a]}). Running this snippet gives:

\start
    \getbuffer[tex]
    \startpacked \tt
        \ctxluabuffer[lua]
    \stoppacked
\stop

Because some values are not set we need to apply the \type {tostring} function
here so that we get the word \type {nil}.

A special kind of box is \prm {vcenter}. This one also can have attributes. When
one or more are set these plus the currently set attributes are bound to the
resulting box. In regular \TEX\ these centered boxes are only permitted in math
mode, but in \LUAMETATEX\ there is no error message and the box the height and
depth are equally divided. Of course in text mode there is no math axis related
offset applied.

\stopsubsection

\stopsection

\startsection[title={\LUA\ related primitives}]

\startsubsection[title={\prm {directlua}}]

In order to merge \LUA\ code with \TEX\ input, a few new primitives are needed.
The primitive \prm {directlua} is used to execute \LUA\ code immediately. The
syntax is

\startsyntax
\directlua <general text>
\stopsyntax

The \syntax {<general text>} is expanded fully, and then fed into the \LUA\
interpreter. After reading and expansion has been applied to the \syntax
{<general text>}, the resulting token list is converted to a string as if it was
displayed using \type {\the\toks}. On the \LUA\ side, each \prm {directlua} block
is treated as a separate chunk. In such a chunk you can use the \type {local}
directive to keep your variables from interfering with those used by the macro
package.

The conversion to and from a token list means that you normally can not use \LUA\
line comments (starting with \type {--}) within the argument. As there typically
will be only one \quote {line} the first line comment will run on until the end
of the input. You will either need to use \TEX|-|style line comments (starting
with \%), or change the \TEX\ category codes locally. Another possibility is to
say:

\starttyping
\begingroup
\endlinechar=10
\directlua ...
\endgroup
\stoptyping

Then \LUA\ line comments can be used, since \TEX\ does not replace line endings
with spaces. Of course such an approach depends on the macro package that you
use.

The \prm {directlua} command is expandable. Since it passes \LUA\ code to the
\LUA\ interpreter its expansion from the \TEX\ viewpoint is usually empty.
However, there are some \LUA\ functions that produce material to be read by \TEX,
the so called print functions. The most simple use of these is \type
{tex.print(<string> s)}. The characters of the string \type {s} will be placed on
the \TEX\ input buffer, that is, \quote {before \TEX's eyes} to be read by \TEX\
immediately. For example:

\startbuffer
\count10=20
a\directlua{tex.print(tex.count[10]+5)}b
\stopbuffer

\typebuffer

expands to

\getbuffer

Here is another example:

\startbuffer
$\pi = \directlua{tex.print(math.pi)}$
\stopbuffer

\typebuffer

will result in

\getbuffer

Note that the expansion of \prm {directlua} is a sequence of characters, not of
tokens, contrary to all \TEX\ commands. So formally speaking its expansion is
null, but it places material on a pseudo-file to be immediately read by \TEX, as
\ETEX's \prm {scantokens}. For a description of print functions look at \in
{section} [sec:luaprint].

Because the \syntax {<general text>} is a chunk, the normal \LUA\ error handling
is triggered if there is a problem in the included code. The \LUA\ error messages
should be clear enough, but the contextual information is still pretty bad.
Often, you will only see the line number of the right brace at the end of the
code.

While on the subject of errors: some of the things you can do inside \LUA\ code
can break up \LUAMETATEX\ pretty bad. If you are not careful while working with
the node list interface, you may even end up with assertion errors from within
the \TEX\ portion of the executable.

\stopsubsection

\startsubsection[title={\lpr {luaescapestring}}]

\topicindex {escaping}

This primitive converts a \TEX\ token sequence so that it can be safely used as
the contents of a \LUA\ string: embedded backslashes, double and single quotes,
and newlines and carriage returns are escaped. This is done by prepending an
extra token consisting of a backslash with category code~12, and for the line
endings, converting them to \type {n} and \type {r} respectively. The token
sequence is fully expanded.

\startsyntax
\luaescapestring <general text>
\stopsyntax

Most often, this command is not actually the best way to deal with the
differences between \TEX\ and \LUA. In very short bits of \LUA\ code it is often
not needed, and for longer stretches of \LUA\ code it is easier to keep the code
in a separate file and load it using \LUA's \type {dofile}:

\starttyping
\directlua { dofile("mysetups.lua") }
\stoptyping

\stopsubsection

\startsubsection[title={\lpr {luafunction}, \lpr {luafunctioncall} and \lpr {luadef}}]

The \prm {directlua} commands involves tokenization of its argument (after
picking up an optional name or number specification). The tokenlist is then
converted into a string and given to \LUA\ to turn into a function that is
called. The overhead is rather small but when you have millions of calls it can
have some impact. For this reason there is a variant call available: \lpr
{luafunction}. This command is used as follows:

\starttyping
\directlua {
    local t = lua.get_functions_table()
    t[1] = function() tex.print("!") end
    t[2] = function() tex.print("?") end
}

\luafunction1
\luafunction2
\stoptyping

Of course the functions can also be defined in a separate file. There is no limit
on the number of functions apart from normal \LUA\ limitations. Of course there
is the limitation of no arguments but that would involve parsing and thereby give
no gain. The function, when called in fact gets one argument, being the index, so
in the following example the number \type {8} gets typeset.

\starttyping
\directlua {
    local t = lua.get_functions_table()
    t[8] = function(slot) tex.print(slot) end
}
\stoptyping

The \lpr {luafunctioncall} primitive does the same but is unexpandable, for
instance in an \prm {edef}. In addition \LUATEX\ provides a definer:

\starttyping
                 \luadef\MyFunctionA 1
          \global\luadef\MyFunctionB 2
\protected\global\luadef\MyFunctionC 3
\stoptyping

You should really use these commands with care. Some references get stored in
tokens and assume that the function is available when that token expands. On the
other hand, as we have tested this functionality in relative complex situations
normal usage should not give problems.

There are another three (still experimental) primitives that behave like \lpr
{luafunction} but they expect the function to return an integer, dimension (also
an integer) or a gluespec node. The return values gets injected into the input.

\starttyping
\luacountfunction 997 123
\luadimenfunction 998 123pt
\luaskipfunction  999 123pt plus 10pt minus 20pt
\stoptyping

Examples of function 997 in the above lines are:

\starttyping
function() return token.scan_int() end
function() return 1234 end
\stoptyping

This itself is not spectacular so there is more. These functions can be called in
two modes: either \TEX\ is expecting a value, or it is not and just expanding the
call.

\starttyping
local n = 0
function(slot,scanning)
    if scanning then
        return n
    else
        n = token.scan_int()
    end
end
\stoptyping

So, assuming that the function is in slot 997, you can do this:

\starttyping
\luacountfunction 997 123
\count100=\luacountfunction 997
\stoptyping

After which \type {\count 100} has the value \type {123}.

% Also experimental (I need to play with this a bit more when I have time):
%
% The \type {token.set_lua} function already accepts some strings as optional
% arguments (\type {protected} and \type {global}) and now also handles \type
% {count}, \type {dimen} and \type {skip}.

\stopsubsection

\startsubsection[title={\lpr {luabytecode} and \lpr {luabytecodecall}}]

Analogue to the function callers discussed in the previous section we have byte
code callers. Again the call variant is unexpandable.

\starttyping
\directlua {
    lua.bytecode[9998] = function(s)
        tex.sprint(s*token.scan_int())
    end
    lua.bytecode[5555] = function(s)
        tex.sprint(s*token.scan_dimen())
    end
}
\stoptyping

This works with:

\starttyping
\luabytecode    9998 5  \luabytecode    5555 5sp
\luabytecodecall9998 5  \luabytecodecall5555 5sp
\stoptyping

The variable \type {s} in the code is the number of the byte code register that
can be used for diagnostic purposes. The advantage of bytecode registers over
function calls is that they are stored in the format (but without upvalues).

\stopsubsection

\stopsection

\startsection[title={Catcode tables}]

\startsubsection[title={Catcodes}]

\topicindex {catcodes}

Catcode tables are a new feature that allows you to switch to a predefined
catcode regime in a single statement. You can have lots of different tables, but
if you need a dozen you might wonder what you're doing. This subsystem is
backward compatible: if you never use the following commands, your document will
not notice any difference in behaviour compared to traditional \TEX. The contents
of each catcode table is independent from any other catcode table, and its
contents is stored and retrieved from the format file.

\stopsubsection

\startsubsection[title={\lpr {catcodetable}}]

\startsyntax
\catcodetable <15-bit number>
\stopsyntax

The primitive \lpr {catcodetable} switches to a different catcode table. Such a
table has to be previously created using one of the two primitives below, or it
has to be zero. Table zero is initialized by \INITEX.

\stopsubsection

\startsubsection[title={\lpr {initcatcodetable}}]

\startsyntax
\initcatcodetable <15-bit number>
\stopsyntax

The primitive \lpr {initcatcodetable} creates a new table with catcodes
identical to those defined by \INITEX. The new catcode table is allocated
globally: it will not go away after the current group has ended. If the supplied
number is identical to the currently active table, an error is raised. The
initial values are:

\starttabulate[|c|c|l|l|]
\DB catcode \BC character               \BC equivalent \BC category          \NC \NR
\TB
\NC  0 \NC \tttf \letterbackslash       \NC         \NC \type {escape}       \NC \NR
\NC  5 \NC \tttf \letterhat\letterhat M \NC return  \NC \type {car_ret}      \NC \NR
\NC  9 \NC \tttf \letterhat\letterhat @ \NC null    \NC \type {ignore}       \NC \NR
\NC 10 \NC \tttf <space>                \NC space   \NC \type {spacer}       \NC \NR
\NC 11 \NC {\tttf a} \endash\ {\tttf z} \NC         \NC \type {letter}       \NC \NR
\NC 11 \NC {\tttf A} \endash\ {\tttf Z} \NC         \NC \type {letter}       \NC \NR
\NC 12 \NC everything else              \NC         \NC \type {other}        \NC \NR
\NC 14 \NC \tttf \letterpercent         \NC         \NC \type {comment}      \NC \NR
\NC 15 \NC \tttf \letterhat\letterhat ? \NC delete  \NC \type {invalid_char} \NC \NR
\LL
\stoptabulate

\stopsubsection

\startsubsection[title={\lpr {savecatcodetable}}]

\startsyntax
\savecatcodetable <15-bit number>
\stopsyntax

\lpr {savecatcodetable} copies the current set of catcodes to a new table with
the requested number. The definitions in this new table are all treated as if
they were made in the outermost level.

The new table is allocated globally: it will not go away after the current group
has ended. If the supplied number is the currently active table, an error is
raised.

\stopsubsection

\stopsection

\startsection[title={Tokens, commands and strings}]

\startsubsection[title={\lpr {scantextokens}}]

\topicindex {tokens+scanning}

The syntax of \lpr {scantextokens} is identical to \prm {scantokens}. This
primitive is a slightly adapted version of \ETEX's \prm {scantokens}. The
differences are:

\startitemize
\startitem
    The last (and usually only) line does not have a \prm {endlinechar}
    appended.
\stopitem
\startitem
    \lpr {scantextokens} never raises an EOF error, and it does not execute
    \prm {everyeof} tokens.
\stopitem
\startitem
    There are no \quote {\unknown\ while end of file \unknown} error tests
    executed. This allows the expansion to end on a different grouping level or
    while a conditional is still incomplete.
\stopitem
\stopitemize

\stopsubsection

\startsubsection[title={\lpr {toksapp}, \lpr {tokspre}, \lpr {etoksapp}, \lpr {etokspre},
\lpr {gtoksapp}, \lpr {gtokspre}, \lpr {xtoksapp},  \lpr {xtokspre}}]

Instead of:

\starttyping
\toks0\expandafter{\the\toks0 foo}
\stoptyping

you can use:

\starttyping
\etoksapp0{foo}
\stoptyping

The \type {pre} variants prepend instead of append, and the \type {e} variants
expand the passed general text. The \type {g} and \type {x} variants are global.

\stopsubsection

\startsubsection[title={\prm {csstring}, \lpr {begincsname} and \lpr {lastnamedcs}}]

These are somewhat special. The \prm {csstring} primitive is like
\prm {string} but it omits the leading escape character. This can be
somewhat more efficient than stripping it afterwards.

The \lpr {begincsname} primitive is like \prm {csname} but doesn't create
a relaxed equivalent when there is no such name. It is equivalent to

\starttyping
\ifcsname foo\endcsname
  \csname foo\endcsname
\fi
\stoptyping

The advantage is that it saves a lookup (don't expect much speedup) but more
important is that it avoids using the \prm {if} test. The \lpr {lastnamedcs}
is one that should be used with care. The above example could be written as:

\starttyping
\ifcsname foo\endcsname
  \lastnamedcs
\fi
\stoptyping

This is slightly more efficient than constructing the string twice (deep down in
\LUATEX\ this also involves some \UTF8 juggling), but probably more relevant is
that it saves a few tokens and can make code a bit more readable.

\stopsubsection

\startsubsection[title={\lpr {clearmarks}}]

\topicindex {marks}

This primitive complements the \ETEX\ mark primitives and clears a mark class
completely, resetting all three connected mark texts to empty. It is an
immediate command.

\startsyntax
\clearmarks <16-bit number>
\stopsyntax

\stopsubsection

\startsubsection[title={\lpr {alignmark} and \lpr {aligntab}}]

The primitive \lpr {alignmark} duplicates the functionality of \type {#} inside
alignment preambles, while \lpr {aligntab} duplicates the functionality of \type
{&}.

\stopsubsection

\startsubsection[title={\lpr {letcharcode}}]

This primitive can be used to assign a meaning to an active character, as in:

\starttyping
\def\foo{bar} \letcharcode123=\foo
\stoptyping

This can be a bit nicer than using the uppercase tricks (using the property of
\prm {uppercase} that it treats active characters special).

\stopsubsection

\startsubsection[title={\lpr {glet}}]

This primitive is similar to:

\starttyping
\protected\def\glet{\global\let}
\stoptyping

but faster (only measurable with millions of calls) and probably more convenient
(after all we also have \type {\gdef}).

\stopsubsection

\startsubsection[title={\lpr {expanded}, \lpr {immediateassignment} and \lpr {immediateassigned}}]

\topicindex {expansion}

The \lpr {expanded} primitive takes a token list and expands its content which can
come in handy: it avoids a tricky mix of \prm {expandafter} and \prm {noexpand}.
You can compare it with what happens inside the body of an \prm {edef}. But this
kind of expansion still doesn't expand some primitive operations.

\startbuffer
\newcount\NumberOfCalls

\def\TestMe{\advance\NumberOfCalls1 }

\edef\Tested{\TestMe foo:\the\NumberOfCalls}
\edef\Tested{\TestMe foo:\the\NumberOfCalls}
\edef\Tested{\TestMe foo:\the\NumberOfCalls}

\meaning\Tested
\stopbuffer

\typebuffer

The result is a macro that has the not expanded code in its body:

\getbuffer

Instead we can define \tex {TestMe} in a way that expands the assignment
immediately. You need of course to be aware of preventing look ahead interference
by using a space or \tex {relax} (often an expression works better as it doesn't
leave an \tex {relax}).

\startbuffer
\def\TestMe{\immediateassignment\advance\NumberOfCalls1 }

\edef\Tested{\TestMe foo:\the\NumberOfCalls}
\edef\Tested{\TestMe foo:\the\NumberOfCalls}
\edef\Tested{\TestMe foo:\the\NumberOfCalls}

\meaning\Tested
\stopbuffer

\typebuffer

This time the counter gets updates and we don't see interference in the
resulting \tex {Tested} macro:

\getbuffer

Here is a somewhat silly example of expanded comparison:

\startbuffer
\def\expandeddoifelse#1#2#3#4%
  {\immediateassignment\edef\tempa{#1}%
   \immediateassignment\edef\tempb{#2}%
   \ifx\tempa\tempb
     \immediateassignment\def\next{#3}%
   \else
     \immediateassignment\def\next{#4}%
   \fi
   \next}

\edef\Tested
  {(\expandeddoifelse{abc}{def}{yes}{nop}/%
    \expandeddoifelse{abc}{abc}{yes}{nop})}

\meaning\Tested
\stopbuffer

\typebuffer

It gives:

\getbuffer

A variant is:

\starttyping
\def\expandeddoifelse#1#2#3#4%
  {\immediateassigned{
     \edef\tempa{#1}%
     \edef\tempb{#2}%
   }%
   \ifx\tempa\tempb
     \immediateassignment\def\next{#3}%
   \else
     \immediateassignment\def\next{#4}%
   \fi
   \next}
\stoptyping

The possible error messages are the same as using assignments in preambles of
alignments and after the \prm {accent} command. The supported assignments are the
so called prefixed commands (except box assignments).

\stopsubsection

\startsubsection[title={\lpr {ignorepars}}]

This primitive is like \prm {ignorespaces} but also skips paragraph ending
commands (normally \prm {par} and empty lines).

\stopsubsection

\startsubsection[title={\lpr {futureexpand}, \lpr {futureexpandis}, \lpr {futureexpandisap}}]

These commands are used as:

\starttyping
\futureexpand\sometoken\whenfound\whennotfound
\stoptyping

When there is no match and a space was gobbled a space will be put back. The
\type {is} variant doesn't do that while the \type {isap} even skips \type
{\pars}, These characters stand for \quote {ignorespaces} and \quote
{ignorespacesandpars}.

\stopsubsection

\startsubsection[title={\lpr {aftergrouped}}]

There is a new experimental feature that can inject multiple tokens to after the group
ends. An example demonstrate its use:

\startbuffer
{
    \aftergroup A \aftergroup B \aftergroup C
test 1 : }

{
    \aftergrouped{What comes next 1}
    \aftergrouped{What comes next 2}
    \aftergrouped{What comes next 3}
test 2 : }


{
    \aftergroup A \aftergrouped{What comes next 1}
    \aftergroup B \aftergrouped{What comes next 2}
    \aftergroup C \aftergrouped{What comes next 3}
test 3 : }

{
    \aftergrouped{What comes next 1} \aftergroup A
    \aftergrouped{What comes next 2} \aftergroup B
    \aftergrouped{What comes next 3} \aftergroup C
test 4 : }
\stopbuffer

\typebuffer

This gives:

\startpacked\getbuffer\stoppacked

\stopsubsection

\stopsection

\startsection[title=Conditions]

\startsubsection[title={\lpr{ifabsnum} and \lpr {ifabsdim}}]

There are two tests that we took from \PDFTEX:

\startbuffer
\ifabsnum -10 = 10
    the same number
\fi
\ifabsdim -10pt = 10pt
    the same dimension
\fi
\stopbuffer

\typebuffer

This gives

\blank {\tt \getbuffer} \blank

\stopsubsection

\startsubsection[title={\lpr{ifcmpnum}, \lpr {ifcmpdim}, \lpr {ifnumval}, \lpr
{ifdimval}, \lpr {ifchknum} and \lpr {ifchkdim}}]

\topicindex {conditions+numbers}
\topicindex {conditions+dimensions}
\topicindex {numbers}
\topicindex {dimensions}

New are the ones that compare two numbers or dimensions:

\startbuffer
\ifcmpnum 5 8 less \or equal \else more \fi
\ifcmpnum 5 5 less \or equal \else more \fi
\ifcmpnum 8 5 less \or equal \else more \fi
\stopbuffer

\typebuffer \blank {\tt \getbuffer} \blank

and

\startbuffer
\ifcmpdim 5pt 8pt less \or equal \else more \fi
\ifcmpdim 5pt 5pt less \or equal \else more \fi
\ifcmpdim 8pt 5pt less \or equal \else more \fi
\stopbuffer

\typebuffer \blank {\tt \getbuffer} \blank

There are also some number and dimension tests. All four expose the \type {\else}
branch when there is an error, but two also report if the number is less, equal
or more than zero.

\startbuffer
\ifnumval  -123  \or < \or = \or > \or ! \else ? \fi
\ifnumval     0  \or < \or = \or > \or ! \else ? \fi
\ifnumval   123  \or < \or = \or > \or ! \else ? \fi
\ifnumval   abc  \or < \or = \or > \or ! \else ? \fi

\ifdimval -123pt \or < \or = \or > \or ! \else ? \fi
\ifdimval    0pt \or < \or = \or > \or ! \else ? \fi
\ifdimval  123pt \or < \or = \or > \or ! \else ? \fi
\ifdimval  abcpt \or < \or = \or > \or ! \else ? \fi
\stopbuffer

\typebuffer \blank {\tt \getbuffer} \blank

\startbuffer
\ifchknum  -123  \or okay \else bad \fi
\ifchknum     0  \or okay \else bad \fi
\ifchknum   123  \or okay \else bad \fi
\ifchknum   abc  \or okay \else bad \fi

\ifchkdim -123pt \or okay \else bad \fi
\ifchkdim    0pt \or okay \else bad \fi
\ifchkdim  123pt \or okay \else bad \fi
\ifchkdim  abcpt \or okay \else bad \fi
\stopbuffer

\typebuffer \blank {\tt \getbuffer} \blank

\stopsubsection

\startsubsection[title={\lpr {ifboolean}}]

This primitive tests for non|-|zero, so the next variants are similar

\starttyping
       \ifcase   <integer>.F.\else .T.\fi
\unless\ifcase   <integer>.T.\else .F.\fi
       \ifboolean<integer>.T.\else .F.\fi
\stoptyping

\stopsubsection

\startsubsection[title={\lpr {iftok} and \lpr {ifcstok}}]

\topicindex {conditions+tokens}
\topicindex {tokens}

Comparing tokens and macros can be done with \type {\ifx}. Two extra test are
provided in \LUAMETATEX:

\startbuffer
\def\ABC{abc} \def\DEF{def} \def\PQR{abc} \newtoks\XYZ \XYZ {abc}

\iftok{abc}{def}\relax  (same) \else [different] \fi
\iftok{abc}{abc}\relax  [same] \else (different) \fi
\iftok\XYZ {abc}\relax  [same] \else (different) \fi

\ifcstok\ABC \DEF\relax (same) \else [different] \fi
\ifcstok\ABC \PQR\relax [same] \else (different) \fi
\ifcstok{abc}\ABC\relax [same] \else (different) \fi
\stopbuffer

\typebuffer \startpacked[blank] {\tt\nospacing\getbuffer} \stoppacked

You can check if a macro is defined as protected with \type {\ifprotected} while
frozen macros can be tested with \type {\iffrozen}. A provisional \type
{\ifusercmd} tests will check if a command is defined at the user level (and this
one might evolve).

\stopsubsection

\startsubsection[title={\lpr {ifcondition}}]

\topicindex {conditions}

This is a somewhat special one. When you write macros conditions need to be
properly balanced in order to let \TEX's fast branch skipping work well. This new
primitive is basically a no||op flagged as a condition so that the scanner can
recognize it as an if|-|test. However, when a real test takes place the work is
done by what follows, in the next example \tex {something}.

\starttyping
\unexpanded\def\something#1#2%
  {\edef\tempa{#1}%
   \edef\tempb{#2}
   \ifx\tempa\tempb}

\ifcondition\something{a}{b}%
    \ifcondition\something{a}{a}%
        true 1
    \else
        false 1
    \fi
\else
    \ifcondition\something{a}{a}%
        true 2
    \else
        false 2
    \fi
\fi
\stoptyping

If you are familiar with \METAPOST, this is a bit like \type {vardef} where the macro
has a return value. Here the return value is a test.

Experiments with something \type {\ifdef} actually worked ok but were rejected
because in the end it gave no advantage so this generic one has to do. The \type
{\ifcondition} test is basically is a no|-|op except when branches are skipped.
However, when a test is expected, the scanner gobbles it and the next test result
is used. Here is an other example:

\startbuffer
\def\mytest#1%
  {\ifabsdim#1>0pt\else
     \expandafter \unless
   \fi
   \iftrue}

\ifcondition\mytest{10pt}\relax non-zero \else zero \fi
\ifcondition\mytest {0pt}\relax non-zero \else zero \fi
\stopbuffer

\typebuffer \blank {\tt \getbuffer} \blank

The last expansion in a macro like \type {\mytest} has to be a condition and here
we use \type {\unless} to negate the result.

\stopsubsection

\startsubsection[title={\lpr {orelse}}]

Sometimes you have successive tests that, when laid out in the source lead to
deep trees. The \type {\ifcase} test is an exception. Experiments with \type
{\ifcasex} worked out fine but eventually were rejected because we have many
tests so it would add a lot. As \LUAMETATEX\ permitted more experiments,
eventually an alternative was cooked up, one that has some restrictions but is
relative lightweight. It goes like this:

\starttyping
\ifnum\count0<10
    less
\orelse\ifnum\count0=10
    equal
\else
    more
\fi
\stoptyping

The \type {\orelse} has to be followed by one of the if test commands, except
\type {\ifcondition}, and there can be an \type {\unless} in front of such a
command. These restrictions make it possible to stay in the current condition
(read: at the same level). If you need something more complex, using \type
{\orelse} is probably unwise anyway. In case you wonder about performance, there
is a little more checking needed when skipping branches but that can be
neglected. There is some gain due to staying at the same level but that is only
measurable when you runs tens of millions of complex tests and in that case it is
very likely to drown in the real action. It's a convenience mechanism, in the
sense that it can make your code look a bit easier to follow.

There is a nice side effect of this mechanism. When you define:

\starttyping
\def\quitcondition{\orelse\iffalse}
\stoptyping

you can do this:

\starttyping
\ifnum\count0<10
    less
\orelse\ifnum\count0=10
    equal
    \quitcondition
    indeed
\else
    more
\fi
\stoptyping

Of course it is only useful at the right level, so you might end up with cases like

\starttyping
\ifnum\count0<10
    less
\orelse\ifnum\count0=10
    equal
    \ifnum\count2=30
        \expandafter\quitcondition
    \fi
    indeed
\else
    more
\fi
\stoptyping

\stopsubsection

\startsubsection[title={\lpr {ifprotected}, \lpr {frozen}, \lpr {iffrozen} and \lpr {ifusercmd}}]

These checkers deal with control sequences. You can check if a command is a
protected one, that is, defined with the \type {\protected} prefix. A command is
frozen when it has been defined with the \type {\frozen} prefix. Beware: only
macros can be frozen. A user command is a command that is not part of the
predefined set of commands. This is an experimental command.

\stopsubsection

\stopsection

\startsection[title={Boxes, rules and leaders}]

\startsubsection[title={\lpr {outputbox}}]

\topicindex {output}

This integer parameter allows you to alter the number of the box that will be
used to store the page sent to the output routine. Its default value is 255, and
the acceptable range is from 0 to 65535.

\startsyntax
\outputbox = 12345
\stopsyntax

\stopsubsection

\startsubsection[title={\prm {vpack}, \prm {hpack} and \prm {tpack}}]

These three primitives are like \prm {vbox}, \prm {hbox} and \prm {vtop}
but don't apply the related callbacks.

\stopsubsection

\startsubsection[title={\prm {vsplit}}]

\topicindex {splitting}

The \prm {vsplit} primitive has to be followed by a specification of the required
height. As alternative for the \type {to} keyword you can use \type {upto} to get
a split of the given size but result has the natural dimensions then.

\stopsubsection

\startsubsection[title={Images and reused box objects},reference=sec:imagedandforms]

In original \TEX\ image support is dealt with via specials. It's not a native
feature of the engine. All that \TEX\ cares about is dimensions, so in practice
that meant: using a box with known dimensions that wraps a special that instructs
the backend to include an image. The wrapping is needed because a special itself
is a whatsit and as such has no dimensions.

In \PDFTEX\ a special whatsit for images was introduced and that one {\em has}
dimensions. As a consequence, in several places where the engine deals with the
dimensions of nodes, it now has to check the details of whatsits. By inheriting
code from \PDFTEX, the \LUATEX\ engine also had that property. However, at some
point this approach was abandoned and a more natural trick was used: images (and
box resources) became a special kind of rules, and as rules already have
dimensions, the code could be simplified.

When direction nodes and localpar nodes also became first class nodes, whatsits
again became just that: nodes representing whatever you want, but without
dimensions, and therefore they could again be ignored when dimensions mattered.
And, because images were disguised as rules, as mentioned, their dimensions
automatically were taken into account. This seperation between front and backend
cleaned up the code base already quite a bit.

In \LUAMETATEX\ we still have the image specific subtypes for rules, but the
engine never looks at subtypes of rules. That was up to the backend. This means
that image support is not present in \LUAMETATEX. When an image specification was
parsed the special properties, like the filename, or additional attributes, were
stored in the backend and all that \LUATEX\ does is registering a reference to an
image's specification in the rule node. But, having no backend means nothing is
stored, which in turn would make the image inclusion primitives kind of weird.

Therefore you need to realize that contrary to \LUATEX, {\em in \LUAMETATEX\
support for images and box reuse is not built in}! However, we can assume that
an implementation uses rules in a similar fashion as \LUATEX\ does. So, you can
still consider images and box reuse to be core concepts. Here we just mention the
primitives that \LUATEX\ provides. They are not available in the engine but can
of course be implemented in \LUA.

\starttabulate[|l|p|]
\DB command \BC explanation \NC \NR
\TB
\NC \lpr {saveboxresource}             \NC save the box as an object to be included later \NC \NR
\NC \lpr {saveimageresource}           \NC save the image as an object to be included later \NC \NR
\NC \lpr {useboxresource}              \NC include the saved box object here (by index) \NC \NR
\NC \lpr {useimageresource}            \NC include the saved image object here (by index) \NC \NR
\NC \lpr {lastsavedboxresourceindex}   \NC the index of the last saved box object \NC \NR
\NC \lpr {lastsavedimageresourceindex} \NC the index of the last saved image object \NC \NR
\NC \lpr {lastsavedimageresourcepages} \NC the number of pages in the last saved image object \NC \NR
\LL
\stoptabulate

An implementation probably should accept the usual optional dimension parameters
for \type {\use...resource} in the same format as for rules. With images, these
dimensions are then used instead of the ones given to \lpr {useimageresource} but
the original dimensions are not overwritten, so that a \lpr {useimageresource}
without dimensions still provides the image with dimensions defined by \lpr
{saveimageresource}. These optional parameters are not implemented for \lpr
{saveboxresource}.

\starttyping
\useimageresource width 20mm height 10mm depth 5mm \lastsavedimageresourceindex
\useboxresource   width 20mm height 10mm depth 5mm \lastsavedboxresourceindex
\stoptyping

Examples or optional entries are \type {attr} and \type {resources} that accept a
token list, and the \type {type} key. When set to non|-|zero the \type {/Type}
entry is omitted. A value of 1 or 3 still writes a \type {/BBox}, while 2 or 3
will write a \type {/Matrix}. But, as said: this is entirely up to the backend.
Generic macro packages (like \type {tikz}) can use these assumed primitives so
one can best provide them. It is probably, for historic reasons, the only more or
less standardized image inclusion interface one can expect to work in all macro
packages.

\stopsubsection

\startsubsection[title={\lpr {hpack}, \lpr {vpack} and \lpr {tpack}}]

These three primitives are the equivalents of \type {\hbox}, \type {\vbox} and
\type {\vtop} but they don't trigger the packaging related callbacks. Of course
one never know if content needs a treatment so using them should be done with
care.

\stopsubsection

\startsubsection[title={\lpr {nohrule} and \lpr {novrule}}]

\topicindex {rules}

Because introducing a new keyword can cause incompatibilities, two new primitives
were introduced: \lpr {nohrule} and \lpr {novrule}. These can be used to
reserve space. This is often more efficient than creating an empty box with fake
dimensions.

\stopsubsection

\startsubsection[title={\lpr {gleaders}},reference=sec:gleaders]

\topicindex {leaders}

This type of leaders is anchored to the origin of the box to be shipped out. So
they are like normal \prm {leaders} in that they align nicely, except that the
alignment is based on the {\it largest\/} enclosing box instead of the {\it
smallest\/}. The \type {g} stresses this global nature.

\stopsubsection

\stopsection

\startsection[title={Languages}]

\startsubsection[title={\lpr {hyphenationmin}}]

\topicindex {languages}
\topicindex {hyphenation}

This primitive can be used to set the minimal word length, so setting it to a value
of~$5$ means that only words of 6 characters and more will be hyphenated, of course
within the constraints of the \prm {lefthyphenmin} and \prm {righthyphenmin}
values (as stored in the glyph node). This primitive accepts a number and stores
the value with the language.

\stopsubsection

\startsubsection[title={\prm {boundary}, \prm {noboundary}, \prm {protrusionboundary} and \prm {wordboundary}}]

The \prm {noboundary} command is used to inject a whatsit node but now injects a normal
node with type \nod {boundary} and subtype~0. In addition you can say:

\starttyping
x\boundary 123\relax y
\stoptyping

This has the same effect but the subtype is now~1 and the value~123 is stored.
The traditional ligature builder still sees this as a cancel boundary directive
but at the \LUA\ end you can implement different behaviour. The added benefit of
passing this value is a side effect of the generalization. The subtypes~2 and~3
are used to control protrusion and word boundaries in hyphenation and have
related primitives.

\stopsubsection

\stopsection

\startsection[title={Control and debugging}]

\startsubsection[title={Tracing}]

\topicindex {tracing}

If \prm {tracingonline} is larger than~2, the node list display will also print
the node number of the nodes.

\stopsubsection

\startsubsection[title={\lpr {lastnodetype}, \lpr {lastnodesubtype}, \lpr
{currentiftype} and \lpr {internalcodesmode}.}]

The \ETEX\ command \type {\lastnodetype} is limited to some nodes. When the
parameter \type {\internalcodesmode} is set to a non|-|zero value the normal
(internally used) numbers are reported. The same is true for \type
{\currentiftype}, as we have more conditionals and also use a different order.
The \type {\lastnodesubtype} is a bonus.

\stopsubsection

\stopsection

\startsection[title={Files}]

\startsubsection[title={File syntax}]

\topicindex {files+names}

\LUAMETATEX\ will accept a braced argument as a file name:

\starttyping
\input {plain}
\openin 0 {plain}
\stoptyping

This allows for embedded spaces, without the need for double quotes. Macro
expansion takes place inside the argument.

The \lpr {tracingfonts} primitive that has been inherited from \PDFTEX\ has
been adapted to support variants in reporting the font. The reason for this
extension is that a csname not always makes sense. The zero case is the default.

\starttabulate[|l|l|]
\DB value \BC reported \NC \NR
\TB
\NC \type{0} \NC \type{\foo xyz} \NC \NR
\NC \type{1} \NC \type{\foo (bar)} \NC \NR
\NC \type{2} \NC \type{<bar> xyz} \NC \NR
\NC \type{3} \NC \type{<bar @ ..pt> xyz} \NC \NR
\NC \type{4} \NC \type{<id>} \NC \NR
\NC \type{5} \NC \type{<id: bar>} \NC \NR
\NC \type{6} \NC \type{<id: bar @ ..pt> xyz} \NC \NR
\LL
\stoptabulate

\stopsubsection

\startsubsection[title={Writing to file}]

\topicindex {files+writing}

You can now open upto 127 files with \prm {openout}. When no file is open writes
will go to the console and log. The \type {write} related primitives have to be
implemented as part of a backend! As a consequence a system command is no longer
possible but one can use \type {os.execute} to do the same.

\stopsubsection

\stopsection

\startsection[title={Math}]

\topicindex {math}

We will cover math extensions in its own chapter because not only the font
subsystem and spacing model have been enhanced (thereby introducing many new
primitives) but also because some more control has been added to existing
functionality. Much of this relates to the different approaches of traditional
\TEX\ fonts and \OPENTYPE\ math.

\stopsection

\startsection[title={Fonts}]

\topicindex {fonts}

Like math, we will cover fonts extensions in its own chapter. Here we stick to
mentioning that loading fonts is different in \LUAMETATEX. As in \LUATEX\ we have
the extra primitives \type {\fontid} and \type {\setfontid}, \type {\noligs} and
\type {\nokerns}, and \type {\nospaces}. The other new primitives in \LUATEX\
have been dropped.

\stopsection

\startsection[title=Directions]

\topicindex {\OMEGA}
\topicindex {\ALEPH}
\topicindex {directions}

\startsubsection[title={Two directions}]

The directional model in \LUAMETATEX\ is a simplified version the the model used
in \LUATEX. In fact, not much is happening at all: we only register a change in
direction.

\stopsubsection

\startsubsection[title={How it works}]

The approach is that we try to make node lists balanced but also try to avoid
some side effects. What happens is quite intuitive if we forget about spaces
(turned into glue) but even there what happens makes sense if you look at it in
detail. However that logic makes in|-|group switching kind of useless when no
properly nested grouping is used: switching from right to left several times
nested, results in spacing ending up after each other due to nested mirroring. Of
course a sane macro package will manage this for the user but here we are
discussing the low level injection of directional information.

This is what happens:

\starttyping
\textdirection 1 nur {\textdirection 0 run \textdirection 1 NUR} nur
\stoptyping

This becomes stepwise:

\startnarrower
\starttyping
injected: [push 1]nur {[push 0]run [push 1]NUR} nur
balanced: [push 1]nur {[push 0]run [pop 0][push 1]NUR[pop 1]} nur[pop 0]
result  : run {RUNrun } run
\stoptyping
\stopnarrower

And this:

\starttyping
\textdirection 1 nur {nur \textdirection 0 run \textdirection 1 NUR} nur
\stoptyping

becomes:

\startnarrower
\starttyping
injected: [+TRT]nur {nur [+TLT]run [+TRT]NUR} nur
balanced: [+TRT]nur {nur [+TLT]run [-TLT][+TRT]NUR[-TRT]} nur[-TRT]
result  : run {run RUNrun } run
\stoptyping
\stopnarrower

Now, in the following examples watch where we put the braces:

\startbuffer
\textdirection 1 nur {{\textdirection 0 run} {\textdirection 1 NUR}} nur
\stopbuffer

\typebuffer

This becomes:

\startnarrower
\getbuffer
\stopnarrower

Compare this to:

\startbuffer
\textdirection 1 nur {{\textdirection 0 run }{\textdirection 1 NUR}} nur
\stopbuffer

\typebuffer

Which renders as:

\startnarrower
\getbuffer
\stopnarrower

So how do we deal with the next?

\startbuffer
\def\ltr{\textdirection 0\relax}
\def\rtl{\textdirection 1\relax}

run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur}
run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run}
\stopbuffer

\typebuffer

It gets typeset as:

\startnarrower
\startlines
\getbuffer
\stoplines
\stopnarrower

We could define the two helpers to look back, pick up a skip, remove it and
inject it after the dir node. But that way we loose the subtype information that
for some applications can be handy to be kept as|-|is. This is why we now have a
variant of \lpr {textdirection} which injects the balanced node before the skip.
Instead of the previous definition we can use:

\startbuffer[def]
\def\ltr{\linedirection 0\relax}
\def\rtl{\linedirection 1\relax}
\stopbuffer

\typebuffer[def]

and this time:

\startbuffer[txt]
run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur}
run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run}
\stopbuffer

\typebuffer[txt]

comes out as a properly spaced:

\startnarrower
\startlines
\getbuffer[def,txt]
\stoplines
\stopnarrower

Anything more complex that this, like combination of skips and penalties, or
kerns, should be handled in the input or macro package because there is no way we
can predict the expected behaviour. In fact, the \lpr {linedir} is just a
convenience extra which could also have been implemented using node list parsing.

\stopsubsection

\startsubsection[title={Controlling glue with \lpr {breakafterdirmode}}]

Glue after a dir node is ignored in the linebreak decision but you can bypass that
by setting \lpr {breakafterdirmode} to~\type {1}. The following table shows the
difference. Watch your spaces.

\def\ShowSome#1{%
    \BC \type{#1}
    \NC \breakafterdirmode\zerocount\hsize\zeropoint#1
    \NC
    \NC \breakafterdirmode\plusone\hsize\zeropoint#1
    \NC
    \NC \NR
}

\starttabulate[|l|Tp(1pt)|w(5em)|Tp(1pt)|w(5em)|]
    \DB
    \BC \type{0}
    \NC
    \BC \type{1}
    \NC
    \NC \NR
    \TB
    \ShowSome{pre {\textdirection 0 xxx} post}
    \ShowSome{pre {\textdirection 0 xxx }post}
    \ShowSome{pre{ \textdirection 0 xxx} post}
    \ShowSome{pre{ \textdirection 0 xxx }post}
    \ShowSome{pre { \textdirection 0 xxx } post}
    \ShowSome{pre {\textdirection 0\relax\space xxx} post}
    \LL
\stoptabulate

\stopsubsection

\startsubsection[title={Controlling parshapes with \lpr {shapemode}}]

Another adaptation to the \ALEPH\ directional model is control over shapes driven
by \prm {hangindent} and \prm {parshape}. This is controlled by a new parameter
\lpr {shapemode}:

\starttabulate[|c|l|l|]
\DB value    \BC \prm {hangindent} \BC \prm {parshape} \NC \NR
\TB
\BC \type{0} \NC  normal             \NC normal            \NC \NR
\BC \type{1} \NC  mirrored           \NC normal            \NC \NR
\BC \type{2} \NC  normal             \NC mirrored          \NC \NR
\BC \type{3} \NC  mirrored           \NC mirrored          \NC \NR
\LL
\stoptabulate

The value is reset to zero (like \prm {hangindent} and \prm {parshape})
after the paragraph is done with. You can use negative values to prevent
this. In \in {figure} [fig:shapemode] a few examples are given.

\startplacefigure[reference=fig:shapemode,title={The effect of \type {shapemode}.}]
    \startcombination[2*3]
        {\ruledvbox \bgroup \setuptolerance[verytolerant]
            \hsize .45\textwidth \switchtobodyfont[6pt]
                \pardirection 0 \textdirection 0
                \hangindent 40pt \hangafter -3
                \leftskip10pt \input tufte \par
         \egroup} {TLT: hangindent}
        {\ruledvbox \bgroup \setuptolerance[verytolerant]
            \hsize .45\textwidth \switchtobodyfont[6pt]
            \pardirection 0 \textdirection 0
            \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize
            \input tufte \par
         \egroup} {TLT: parshape}
        {\ruledvbox \bgroup \setuptolerance[verytolerant]
            \hsize .45\textwidth \switchtobodyfont[6pt]
            \pardirection 1 \textdirection 1
            \hangindent 40pt \hangafter -3
            \leftskip10pt \input tufte \par
         \egroup} {TRT: hangindent mode 0}
        {\ruledvbox \bgroup \setuptolerance[verytolerant]
            \hsize .45\textwidth \switchtobodyfont[6pt]
            \pardirection 1 \textdirection 1
            \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize
            \input tufte \par
         \egroup} {TRT: parshape mode 0}
        {\ruledvbox \bgroup \setuptolerance[verytolerant]
            \hsize .45\textwidth \switchtobodyfont[6pt]
            \shapemode=3
            \pardirection 1 \textdirection 1
            \hangindent 40pt \hangafter -3
            \leftskip10pt \input tufte \par
         \egroup} {TRT: hangindent mode 1 & 3}
        {\ruledvbox \bgroup \setuptolerance[verytolerant]
            \hsize .45\textwidth \switchtobodyfont[6pt]
            \shapemode=3
            \pardirection 1 \textdirection 1
            \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize
            \input tufte \par
         \egroup} {TRT: parshape mode 2 & 3}
    \stopcombination
\stopplacefigure

We have \type {\pardirection}, \type {\textdirection}, \type {\mathdirection} and
\type {\linedirection} that is like \type {\textdirection} but with some
additional (inline) glue checking.

\stopsubsection

\startsubsection[title=Orientations]

As mentioned, the difference with \LUATEX\ is that we only have numeric
directions and that there are only two: left|-|to|-|right (\type {0}) and
right|-|to|-|left (\type {1}). The direction of a box is set with \type
{direction}.

In addition to that boxes can now have an \type {orientation} keyword followed by
optional \type {xoffset} and|/|or \type {yoffset} keywords. The offsets don't
have consequences for the dimensions. The alternatives \type {xmove} and \type
{ymove} on the contrary are reflected in the dimensions. Just play with them. The
offsets and moves only are accepted when there is also an orientation, so no time
is wasted on testing for these rarely used keywords. There are related primitives
\type {\box...} that set these properties.

As these are experimental it will not be explained here (yet). They are covered
in the descriptions of the development of \LUAMETATEX: articles and|/|or
documents in the \CONTEXT\ distribution. For now it is enough to know that the
orientation can be up, down, left or right (rotated) and that it has some
anchoring variants. Combined with the offsets this permits macro writers to
provide solutions for top|-|down and bottom|-|up writing directions, something
that is rather macro package specific and used for scripts that need
manipulations anyway. The \quote {old} vertical directions were never okay and
therefore not used.

There are a couple of properties in boxes that you can set and query but that
only really take effect when the backend supports them. When usage on \CONTEXT\
shows that is't okay, they will become official, so we just mention them: \type
{\boxdirection}, \type {\boxattr}, \type {\boxorientation}, \type {\boxxoffset},
\type {\boxyoffset}, \type {\boxxmove}, \type {\boxymove} and \type {\boxtotal}.

\stopsubsection

\stopsection

\startsection[title=Expressions]

The \type {*expr} parsers now accept \type {:} as operator for integer division
(the \type {/} operators does rounding. This can be used for division compatible
with \type {\divide}. I'm still wondering if adding a couple of bit operators
makes sense (for integers).

\stopsection

\startsection[title=Nodes]

The \ETEX\ primitive \type {\lastnodetype} is not honest in reporting the
internal numbers as it uses its own values. But you can set \type
{\internalcodesmode} to a non|-|zero value to get the real id's instead. In
addition there is \type {\lastnodesubtype}.

Another last one is \type {\lastnamedcs} which holds the last match but this one
should be used with care because one never knows if in the meantime something
else \quote {last} has been seen.

\stopsection

\stopchapter

\stopcomponent