% language=us % The usual time stamp. This written while listening intermized to Fish (just ran % into), Lazulli (some yt videos too, looking forward to a next live act) and % because this all is boring checking out Sarah Coopers channel for new DT syncs % every few hours. % When you feel unhappy about the lack of detail in this manual, just keep in mind % that you cannot really demand anything from volunteers: just hope for more (or % pay for it). Friendly comments and corrections are of course always welcome. As % we like what we're doing here, it all might eventually evolve to perfection, stay % tuned. % % Hans Hagen | j.hagen @ xs4all . nl | ntg-context @ ntg . nl \usemodule[article-basic] \usemodule[abbreviations-logos] \usemodule[scite] \definecolor[maincolor][darkblue] \definecolor[primcolor][darkblue] \definecolor[nonecolor][darkgray] \setuptyping [option=tex] \setuptype [option=tex] \setuphead [subject] [color=maincolor] \definehead [newprimitive] [section] [color=maincolor] \definehead [oldprimitive] [section] [color=nonecolor] \setuplist [newprimitive] [textcolor=maincolor] \setuplist [oldprimitive] [textcolor=nonecolor] \starttext \startMPpage fill Page withcolor "darkgray" ; draw textext("\sstf {\white new} primitives") xysized (.9bbwidth(Page),bbheight(Page)-2cm) shifted center Page withcolor "maincolor" ; draw textext.ulft("\sstf in luametatex") xysized (.9bbwidth(Page)/3,(bbheight(Page)-2cm)/6) shifted center lrcorner Page shifted (-.1bbwidth(Page),.05bbwidth(Page)) withcolor "white" ; setbounds currentpicture to Page ; \stopMPpage \startsubject[title={Introduction}] Here I will discuss some of the new primitives in \LUATEX\ and \LUAMETATEX, the later being a successor that permits the \CONTEXT\ folks to experiment with new features. The order is arbitrary. When you compare \LUATEX\ with \PDFTEX, there are actually quite some differences. Some primitives that \PDFTEX\ introduced have been dropped in \LUATEX\ because they can be done better in \LUA. Others have been promoted to core primitives that no longer have a \type {pdf} prefix. Then there are lots of new primitives, some introduce new concepts, some are a side effect of for instance new math font technologies, and then there are those that are handy extensions to the macro language. The \LUAMETATEX\ engine drops quite some primitives, like those related to \PDFTEX\ specific font or backend features. It also adds some new primitives, mostly concerning the macro language. We also discuss the primitives that fit into the macro programming scope that are present in traditional \TEX\ and \ETEX\ but there are for sure better of descriptions out there already. Primitives that relate to typesetting, like those controlling math, fonts, boxes, attributes, directions, catcodes, \LUA\ (functions) etc are not discussed here. There are for instance primitives to create aliases to low level registers like counters and dimensions, as well as other (semi|-|numeric) quantities like characters, but normally these are wrapped into high level macros so that definitions can't clash too much. Numbers, dimensions etc can be advanced, multiplied and divided and there is a simple expression mechanism to deal with them. These are not discussed here. \startcolumns \placelist [newprimitive,oldprimitive] [alternative=c] \stopcolumns In this document the section titles that discuss the \color [nonecolor] {original \TEX\ and \ETEX\ primitives} have a different color those explaining the \color [primcolor] {\LUATEX\ and \LUAMETATEX\ primitives}. \stopsubject \page % When writing this manual I also decided to merge some of the condition related % code so that it dealt a bit more natural with the newer features. A usual side % effects if writing manuals. % \startoldprimitive[title={\tex {meaning}}] % \stopoldprimitive % % \startoldprimitive[title={\tex {par}}] % \stopoldprimitive % % \startoldprimitive[title={\tex {linepar}}] % \stopoldprimitive \startoldprimitive[title={\tex {afterassignment}}] The token following \type {\afterassignment}, a traditional \TEX\ primitive, is saved and gets injected (and then expanded) after a following assignment took place. \startbuffer \afterassignment !\def\MyMacro {}\quad \afterassignment !\let\MyMacro ?\quad \afterassignment !\scratchcounter 123\quad \afterassignment !% \afterassignment ?\advance\scratchcounter by 1 \stopbuffer \typebuffer The \type {\afterassignment}s are not accumulated, the last one wins: {\getbuffer} \stopoldprimitive \startnewprimitive[title={\tex {afterassigned}}] The \type {\afterassignment} primitive stores a token to be injected (and thereby expanded) after an assignment has happened. Unlike \type {\aftergroup}, multiple calls are not accumulated, and changing that would be too incompatible. This is why we have \type {\afterassigned}, which can be used to inject a bunch of tokens. But in order to be consistent this one is also not accumulative. \startbuffer \afterassigned{done}% \afterassigned{{\bf done}}% \scratchcounter=123 \stopbuffer \typebuffer results in: \inlinebuffer\ being typeset. \stopnewprimitive \startoldprimitive[title={\tex {aftergroup}}] The traditional \TEX\ \type {\aftergroup} primitive stores the next token and expands that after the group has been closed. \startbuffer before{ ! \aftergroup a\aftergroup f\aftergroup t\aftergroup e\aftergroup r} \stopbuffer Multiple \type {\aftergroup}s are combined: \typebuffer \getbuffer \stopoldprimitive \startnewprimitive[title={\tex {aftergrouped}}] The in itself powerful \type {\aftergroup} primitives works quite well, even if you need to do more than one thing: you can either use it multiple times, or you can define a macro that does multiple things and apply that after the group. However, you can avoid that by using this primitive which takes a list of tokens. \startbuffer regular \bgroup \aftergrouped{regular}% \bf bold \egroup \stopbuffer \typebuffer Because it happens after the group, we're no longer typesetting in bold. {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {atendofgroup}}] The token provided will be injected just before the group ends. Because these tokens are collected, you need to be aware of possible interference between them. However, normally this is managed by the macro package. \startbuffer \bgroup \atendofgroup\unskip \atendofgroup )% (but it works okay \egroup \stopbuffer \typebuffer Of course these effects can also be achieved by combining (extra) grouping with \type {\aftergroup} calls, so this is more a convenience primitives than a real necessity: {\inlinebuffer}, as proven here. \stopnewprimitive \startnewprimitive[title={\tex {atendofgrouped}}] This is the multi token variant of \type {\atendofgroup}. Of course the next example is somewhat naive when it comes to spacing and so, but it shows the purpose. \startbuffer \bgroup \atendofgrouped{\bf QED}% \atendofgrouped{ (indeed)}% This sometimes looks nicer. \egroup \stopbuffer \typebuffer Multiple invocations are accumulated: {\inlinebuffer}. \stopnewprimitive \startnewprimitive[title={\tex {toksapp}}] One way to append something to a token list is the following: \starttyping \scratchtoks\expandafter{\the\scratchtoks more stuff} \stoptyping This works all right, but it involves a copy of what is already in \type {\scratchtoks}. This is seldom a real issue unless we have large token lists and many appends. This is why \LUATEX\ introduced: \starttyping \toksapp\scratchtoks{more stuff} \toksapp\scratchtoksone\scratchtokstwo \stoptyping At some point, when working on \LUAMETATEX, I realized that primitives like this one and the next appenders and prependers to be discussed were always on the radar of Taco and me. Some were even implemented in what we called \type {eetex}: extended \ETEX, and we even found back the prototypes, dating from pre|-|\PDFTEX\ times. \stopnewprimitive \startnewprimitive[title={\tex {etoksapp}}] A variant of \type {\toksapp} is the following: it expands the to be appended content. \starttyping \def\temp{more stuff} \etoksapp\scratchtoks{some \temp} \stoptyping \stopnewprimitive \startnewprimitive[title={\tex {tokspre}}] Where appending something is easy because of the possible \type {\expandafter} trickery a prepend would involve more work, either using temporary token registers and|/|or using a mixture of the (no)expansion added by \ETEX, but all are kind of inefficient and cumbersome. \starttyping \tokspre\scratchtoks{less stuff} \tokspre\scratchtoksone\scratchtokstwo \stoptyping This prepends the token list that is provided. \stopnewprimitive \startnewprimitive[title={\tex {etokspre}}] A variant of \type {\tokspre} is the following: it expands the to be prepended content. \starttyping \def\temp{less stuff} \etokspre\scratchtoks{a bit \temp} \stoptyping \stopnewprimitive \startnewprimitive[title={\tex {gtoksapp}}] This is the global variant of \type {\toksapp}. \stopnewprimitive \startnewprimitive[title={\tex {xtoksapp}}] This is the global variant of \type {\etoksapp}. \stopnewprimitive \startnewprimitive[title={\tex {gtokspre}}] This is the global variant of \type {\tokspre}. \stopnewprimitive \startnewprimitive[title={\tex {xtokspre}}] This is the global variant of \type {\etokspre}. \stopnewprimitive \startoldprimitive[title={\tex {csname}}] This original \TEX\ primitive starts the construction of a control sequence reference. It does a lookup and when no sequence with than name is found, it will create a hash entry and defaults its meaning to \type {\relax}. \starttyping \csname letters and other characters\endcsname \stoptyping \stopoldprimitive \startoldprimitive[title={\tex {endcsname}}] This primitive is used in combination with \type {\csname}, \type {\ifcsname} and \type {\begincsname} where its end the scanning for the to be constructed control sequence token. \stopoldprimitive \startnewprimitive[title={\tex {begincsname}}] The next code creates a control sequence token from the given serialized tokens: \starttyping \csname mymacro\endcsname \stoptyping When \type {\mymacro} is not defined a control sequence will be created with the meaning \type {\relax}. A side effect is that a test for its existence might fail because it now exists. The next sequence will {\em not} create an controil sequence: \starttyping \begincsname mymacro\endcsname \stoptyping This actually is kind of equivalent to: \starttyping \ifcsname mymacro\endcsname \csname mymacro\endcsname \fi \stoptyping \stopnewprimitive \startnewprimitive[title={\tex {lastnamedcs}}] The example code in the previous section has some redundancy, in the sense that there to be looked up control sequence name \type {mymacro} is assembled twice. This is no big deal in a traditional eight bit \TEX\ but in a \UNICODE\ engine multi|-|byte sequences demand some more processing (although it is unlikely that control sequences have many multi|-|byte \UTF8\ characters). \starttyping \ifcsname mymacro\endcsname \csname mymacro\endcsname \fi \stoptyping Instead we can say: \starttyping \ifcsname mymacro\endcsname \lastnamedcs \fi \stoptyping Although there can be some performance benefits another advantage is that it uses less tokens and parsing. It might even look nicer. \stopnewprimitive \startnewprimitive[title={\tex {futureexpand}}] This primitive can be used as an alternative to a \type {\futurelet} approach, which is where the name comes from. \footnote {In the engine primitives that have similar behavior are grouped in commands that are then dealt with together, code wise.} \startbuffer \def\variantone<#1>{(#1)} \def\varianttwo#1{[#1]} \futureexpand<\variantone\varianttwo \futureexpand<\variantone\varianttwo{two} \stopbuffer \typebuffer So, the next token determines which of the two variants is taken: {\getbuffer} Because we look ahead there is some magic involved: spaces are ignored but when we have no match they are pushed back into the input. The next variant demonstrates this: \startbuffer \def\variantone<#1>{(#1)} \def\varianttwo{} \def\temp{\futureexpand<\variantone\varianttwo} [\temp ] [\temp {two}] [\expandafter\temp\space ] [\expandafter\temp\space {two}] \stopbuffer \typebuffer This gives us: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {futureexpandis}}] We assume that the previous section is read. This variant will not push back spaces, which permits a consistent approach i.e.\ the user can assume that macro always gobbles the spaces. \startbuffer \def\variantone<#1>{(#1)} \def\varianttwo{} \def\temp{\futureexpandis<\variantone\varianttwo} [\temp ] [\temp {two}] [\expandafter\temp\space ] [\expandafter\temp\space {two}] \stopbuffer \typebuffer So, here no spaces are pushed back. This \type {is} in the name of this primitive means \quote {ignore spaces}, but having that added to the name would have made the primitive even more verbose (after all, we also don't have \type {\expandeddef} but \type {\edef} and no \type {\globalexpandeddef} but \type {\xdef}. {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {futureexpandisap}}] This primitive is like the one in the previous section but also ignores par tokens, so \type {isap} means \quote {ignore spaces and paragraphs}. \stopnewprimitive \startoldprimitive[title={\tex {expandafter}}] This original \TEX\ primitive stores the next token, does a one level expansion of what follows it, which actually can be an not expandable token, and reinjects the stored token in the input. Like: \starttyping \expandafter\let\csname my weird macro name\endcsname{m w m n} \stoptyping Without \type {\expandafter} the \type {\csname} primitive would have been let to the left brace (effectively then a begin group). Actually in this particular case the control sequence with the weird name is injected and when it didn't yet exist it will get the meaning \type {\relax} so we sort of have two assignments in a row then. \stopoldprimitive \startnewprimitive[title={\tex {expandafterspaces}}] This is a gobbler: the next token is reinjected after following spaces have been read. Here is a simple example: \startbuffer [\expandafterspaces 1 2] [\expandafterspaces 3 4] [\expandafterspaces 5 6] \stopbuffer \typebuffer We get this typeset: \inlinebuffer, because a newline normally is configured to be a space (and leading spaces in a line are normally being ingored anyway). \stopnewprimitive \startnewprimitive[title={\tex {expandafterpars}}] Here is another gobbler: the next token is reinjected after following spaces and par tokens have been read. So: \startbuffer [\expandafterpars 1 2] [\expandafterpars 3 4] [\expandafterpars 5 6] \stopbuffer \typebuffer gives us: \inlinebuffer, because empty lines are like \type {\par} and therefore ignored. \stopnewprimitive \startnewprimitive[title={\tex {expandtoken}}] This primitive creates a token with a specific combination of catcode and character code. Because it assumes some knowledge of \TEX\ we can show it using some \type {\expandafter} magic: \startbuffer \expandafter\let\expandafter\temp\expandtoken 11 `X \meaning\temp\crlf \expandafter\let\expandafter\temp\expandtoken 12 `X \meaning\temp\crlf \stopbuffer \typebuffer The meanings are: {\getbuffer} Using other catcodes is possible but the results of injecting them into the input directly (or here by injecting \type {\temp}) can be unexpected because of what \TEX\ expects. You can get messages you normally won't get, for instance about unexpected alignment interference, which is a side effect of \TEX\ using some catcode|/|character combinations as signals and there is no reason to change those internals. That said: \startbuffer \edef\tempA{\expandtoken 9 `X} \meaning\tempA\crlf \edef\tempB{\expandtoken 10 `X} \meaning\tempB\crlf \edef\tempC{\expandtoken 11 `X} \meaning\tempC\crlf \edef\tempD{\expandtoken 12 `X} \meaning\tempD \stopbuffer \typebuffer are all valid and from the meaning you cannot really deduce what's in there: {\getbuffer} But you can be assured that: \startbuffer [AB: \ifx\tempA\tempB Y\else N\fi] [AC: \ifx\tempA\tempC Y\else N\fi] [AD: \ifx\tempA\tempD Y\else N\fi] [BC: \ifx\tempB\tempC Y\else N\fi] [BD: \ifx\tempB\tempD Y\else N\fi] [CD: \ifx\tempC\tempD Y\else N\fi] \stopbuffer \typebuffer makes clear that they're different: \inlinebuffer, and in case you wonder, the characters with catcode 10 are spaces, while those with code 9 are ignored. \stopnewprimitive \startnewprimitive[title={\tex {expandcstoken}}] The rationale behind this primitive is that when we \type {\let} a single token like a character it is hard to compare that with something similar, stored in a macro. This primitive pushes back a single token alias created by \type {\let} into the input. \startbuffer \let\tempA + \meaning\tempA \crlf \let\tempB X \meaning\tempB \crlf \let\tempC $ \meaning\tempC \par \edef\temp {\tempA} \doifelse{\temp}{+}{Y}{N} \meaning\temp \crlf \edef\temp {\tempB} \doifelse{\temp}{X}{Y}{N} \meaning\temp \crlf \edef\temp {\tempC} \doifelse{\temp}{X}{Y}{N} \meaning\temp \par \edef\temp{\expandcstoken\tempA} \doifelse{\temp}{+}{Y}{N} \meaning\temp \crlf \edef\temp{\expandcstoken\tempB} \doifelse{\temp}{X}{Y}{N} \meaning\temp \crlf \edef\temp{\expandcstoken\tempC} \doifelse{\temp}{$}{Y}{N} \meaning\temp \par \doifelse{\expandcstoken\tempA}{+}{Y}{N} \doifelse{\expandcstoken\tempB}{X}{Y}{N} \doifelse{\expandcstoken\tempC}{$}{Y}{N} \par \stopbuffer \typebuffer The meaning of the \type {\let} macros shows that we have a shortcut to a character with (in this case) catcode letter, other (here \quote {other character} gets abbreviated to \quote {character}), math shift etc. \start \tttf \getbuffer \stop Here we use the \CONTEXT\ macro \type {\doifelse} which can be implemented in different ways, but the only property relevant to the user is that the expanded content of the two arguments is compared. \stopnewprimitive \startnewprimitive[title={\tex {expand}}] Normally a protected macro will not be expanded inside for instance an \type {\edef} but there is a way out: \footnote {This primitive is dedicated to Hans vd Meer who was playing with the unprotected version of \type {\doifelse} and wondered about the reason for it not being expandable in the first place.} \startbuffer \edef\temp {\doifelse{a}{b}{c}{d}} \meaning\temp \crlf \edef\temp{\expand\doifelse{a}{b}{c}{d}} \meaning\temp \stopbuffer \typebuffer In the second case, the \type {\doifelse} command {\em is} expanded, but keep in mind that this only makes sense when the body of such a macro is expandable. This is the case in \CONTEXT\ \LMTX, but not in \MKIV. {\getbuffer} \stopnewprimitive \startoldprimitive[title={\tex {ignorespaces}}] This traditional \TEX\ primitive signals the scanner to ignore the following spaces, if any. We mention it because we show a companion in the next section. \stopoldprimitive \startnewprimitive[title={\tex {ignorepars}}] This is a variant of \type {\ignorespaces}: following spaces {\em and} \type {\par} equivalent tokens are ignored, so for instance: \startbuffer one + \ignorepars two = \ignorepars \par three \stopbuffer \typebuffer renders as: \inlinebuffer. Traditionally \TEX\ has been sensitive to \type {\par} tokens in some of its building blocks. This has to do with the fact that it could indicate a runaway argument which in the times of slower machines and terminals was best to catch early. In \LUAMETATEX\ we no longer have long macros and the mechanisms that are sensitive can be told to accept \type {\par} tokens (and \CONTEXT\ set them such that this is the case). \stopnewprimitive \startnewprimitive[title={\tex {ignorearguments}}] This primitive will quit argument scanning and start expansion of the body of a macro. The number of grabbed arguments can be tested as follows: \startbuffer \def\MyMacro[#1][#2][#3]% {\ifarguments zero\or one\or two\or three \else hm\fi} \MyMacro \ignorearguments \quad \MyMacro [1]\ignorearguments \quad \MyMacro [1][2]\ignorearguments \quad \MyMacro [1][2][3]\ignorearguments \crlf \stopbuffer \typebuffer {\getbuffer} {\em Todo: explain optional delimiters.} \stopnewprimitive \startnewprimitive[title={\tex {lastarguments}}] \startbuffer \def\MyMacro #1{\the\lastarguments (#1) } \MyMacro{1} \crlf \def\MyMacro #1#2{\the\lastarguments (#1) (#2)} \MyMacro{1}{2} \crlf \def\MyMacro#1#2#3{\the\lastarguments (#1) (#2) (#3)} \MyMacro{1}{2}{3} \crlf \def\MyMacro #1{(#1) \the\lastarguments} \MyMacro{1} \crlf \def\MyMacro #1#2{(#1) (#2) \the\lastarguments} \MyMacro{1}{2} \crlf \def\MyMacro#1#2#3{(#1) (#2) (#3) \the\lastarguments} \MyMacro{1}{2}{3} \crlf \stopbuffer \typebuffer The value of \type {\lastarguments} can only be trusted in the expansion until another macro is seen and expanded. For instance in these examples, as soon as a character (like the left parenthesis) is seen, horizontal mode is entered and \type {\everypar} is expanded which in turn can involve macros. You can see that in the second block (that is: unless we changed \type {\everypar} in the meantime). {\getbuffer} \stopnewprimitive \startoldprimitive[title={\tex {scantokens}}] Just forget about this \ETEX\ primnitive, just take the one in the next section. \stopoldprimitive \startnewprimitive[title={\tex {scantextokens}}] This primitive scans the input as if it comes from a file. In the next examples the \type {\detokenize} primitive turns tokenized code into verbatim code that is similar to what is read from a file. \startbuffer \edef\whatever{\detokenize{This is {\bf bold} and this is not.}} \detokenize {This is {\bf bold} and this is not.}\crlf \scantextokens{This is {\bf bold} and this is not.}\crlf \scantextokens{\whatever}\crlf \scantextokens\expandafter{\whatever} \stopbuffer \typebuffer This primitive does not have the end|-|of|-|file side effects of its precursor \type {\scantokens}. {\getbuffer} \stopnewprimitive \startoldprimitive[title={\tex {number}}] This \TEX\ primitive serializes the next token into a number, assuming that it is indeed a number, like \starttyping \number`A \number65 \number\scratchcounter \stoptyping For counters and such the \type {\the} primitive does the same, but when you're not sure if what follows is a verbose number or (for instance) a counter the \type {\number} primitive is a safer bet, because \type {\the 65} will not work. \stopoldprimitive \startoldprimitive[title={\tex {string}}] We mention this original primitive because of the one in the next section. It expands the next token or control sequence as if it was just entered, so normally a control sequence becomes a backslash followed by characters and a space. \stopoldprimitive \startnewprimitive[title={\tex {csstring}}] This primitive returns the name of the control sequence given without the leading escape character (normally a backslash). Of course you could strip that character with a simple helper but this is more natural. \startbuffer \csstring\mymacro \stopbuffer \typebuffer We get the name, not the meaning: {\tt \inlinebuffer}. \stopnewprimitive \startoldprimitive[title={\tex {unexpanded}}] This is an \ETEX\ enhancement. The content will not be expanded in a context where expansion is happening, like in an \type {\edef}. In \CONTEXT\ you need to use \type {\normalunexpanded} because we already had a macro with that name. \startbuffer \def\A{!} \def\B{?} \edef\C{\A\B} \meaning\C \crlf \edef\C{\normalunexpanded{\A}\B} \meaning\C \crlf \stopbuffer \typebuffer {\getbuffer} \stopoldprimitive \startoldprimitive[title={\tex {detokenize}}] This \ETEX\ primitive turns the content of the provides list will become characters, kind of verbatim. \startbuffer \expandafter\let\expandafter\temp\detokenize{1} \meaning\temp \crlf \expandafter\let\expandafter\temp\detokenize{A} \meaning\temp \crlf \stopbuffer \typebuffer {\getbuffer} \stopoldprimitive \startnewprimitive[title={\tex {tokenized}}] Just as \type {\expanded} has a counterpart \type {\unexpanded}, it makes sense to give \type {\detokenize} a companion: \startbuffer \edef\foo{\detokenize{\inframed{foo}}} \edef\oof{\detokenize{\inframed{oof}}} \meaning\foo \crlf \dontleavehmode\foo \edef\foo{\tokenized{\foo\foo}} \meaning\foo \crlf \dontleavehmode\foo \dontleavehmode\tokenized{\foo\oof} \stopbuffer \typebuffer {\getbuffer} This primitive is similar to: \starttyping \def\tokenized#1{\scantextokens\expandafter{\normalexpanded{#1}}} \stoptyping and should be more efficient, not that it matters much as we don't use it that much (if at all). \stopnewprimitive \startnewprimitive[title={\tex {expanded}}] This primitive complements the two expansion related primitives mentioned in the previous two sections. This time the content will be expanded and then pushed back into the input. Protected macros will not be expanded, so you can use this primitive to expand the arguments in a call. In \CONTEXT\ you need to use \type {\normalexpanded} because we already had a macro with that name. We give some examples: \startbuffer \def\A{!} \def\B#1{\string#1} \B{\A} \crlf \def\B#1{\string#1} \normalexpanded{\noexpand\B{\A}} \crlf \protected\def\B#1{\string#1} \B{\A} \crlf \stopbuffer \typebuffer {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {immediateassignment}}] Assignments are not expandable which means that you cannot define fully expandable macros that have assignments. But, there is a way out of this: \startbuffer \scratchcounter = 10 \edef\whatever{% (\the\scratchcounter) \immediateassignment\scratchcounter\numexpr\scratchcounter+10\relax \immediateassignment\advance\scratchcounter -5 (\the\scratchcounter) } \meaning\whatever \stopbuffer \typebuffer Don't expect miracles: you can't mix|-|in content or unexpandable tokens as they will either show up or quit the scanning. {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {immediateassigned}}] This is the multi|-|token variant of the primitive mentioned in the previous section. \startbuffer \scratchcounter = 10 \edef\whatever{% (\the\scratchcounter) \immediateassigned{ \scratchcounter\numexpr\scratchcounter+10\relax \advance\scratchcounter -5 }% (\the\scratchcounter) } \meaning\whatever \stopbuffer \typebuffer The results are the same as in the previous section: {\getbuffer} \stopnewprimitive \startoldprimitive[title={\tex {if}}] This traditional \TEX\ conditional checks if two character codes are the same. In order to understand unexpanded results it is good to know that internally \TEX\ groups primitives in a way that serves the implementation. Each primitive has a command code and a character code, but only for real characters the name character code makes sense. This condition only really tests for character codes when we have a character, in all other cases, the result is true. \startbuffer \def\A{A}\def\B{B} \chardef\C=`C \chardef\D=`D \def\AA{AA} [\if AA YES \else NOP \fi] [\if AB YES \else NOP \fi] [\if \A\B YES \else NOP \fi] [\if \A\A YES \else NOP \fi] [\if \C\D YES \else NOP \fi] [\if \C\C YES \else NOP \fi] [\if \count\dimen YES \else NOP \fi] [\if \AA\A YES \else NOP \fi] \stopbuffer \typebuffer The last example demonstrates that the tokens get expanded, which is why we get the extra \type {A}: {\getbuffer} \stopoldprimitive \startoldprimitive[title={\tex {ifcat}}] Another traditional \TEX\ primitive: what happens with what gets read in depends on the catcode of a character, think of characters marked to start math mode, or alphabetic characters (letters) versus other characters (like punctuation). \startbuffer \def\A{A}\def\B{,} \chardef\C=`C \chardef\D=`, \def\AA{AA} [\ifcat $! YES \else NOP \fi] [\ifcat () YES \else NOP \fi] [\ifcat AA YES \else NOP \fi] [\ifcat AB YES \else NOP \fi] [\ifcat \A\B YES \else NOP \fi] [\ifcat \A\A YES \else NOP \fi] [\ifcat \C\D YES \else NOP \fi] [\ifcat \C\C YES \else NOP \fi] [\ifcat \count\dimen YES \else NOP \fi] [\ifcat \AA\A YES \else NOP \fi] \stopbuffer \typebuffer Close reading is needed here: {\getbuffer} This traditional \TEX\ condition as a well as the one in the previous section are hardly used in \CONTEXT, if only because they expand what follows and we seldom need to compare characters. \stopoldprimitive \startoldprimitive[title={\tex {ifnum}}] This is a frequently used conditional: it compares two numbers where a number is anything that can be seen as such. \startbuffer \scratchcounter=65 \chardef\A=65 \ifnum65=`A YES \else NOP\fi \ifnum\scratchcounter=65 YES \else NOP\fi \ifnum\scratchcounter=\A YES \else NOP\fi \stopbuffer \typebuffer Unless a number is an unexpandable token it ends with a space or \type {\relax}, so when you end up in the true branch, you'd better check if \TEX\ could determine where the number ends. {\getbuffer} \stopoldprimitive \startoldprimitive[title={\tex {ifdim}}] Dimensions can be compared with this traditional \TEX\ primitive. \startbuffer \scratchdimen=1pt \scratchcounter=65536 \ifdim\scratchdimen=\scratchcounter sp YES \else NOP\fi \ifdim\scratchdimen=1 pt YES \else NOP\fi \stopbuffer \typebuffer The units are mandate: {\getbuffer} \stopoldprimitive \startoldprimitive[title={\tex {ifodd}}] One reason for this condition to be around is that in a double sided layout we need test for being on an odd or even page. It scans for a number the same was as other primitives, \startbuffer \ifodd65 YES \else NO\fi & \ifodd`B YES \else NO\fi . \stopbuffer \typebuffer So: {\inlinebuffer} \stopoldprimitive \startoldprimitive[title={\tex {ifvmode}}] This traditional conditional checks we are in (internal) vertical mode. \stopoldprimitive \startoldprimitive[title={\tex {ifhmode}}] This traditional conditional checks we are in (restricted) horizontal mode. \stopoldprimitive \startoldprimitive[title={\tex {ifmmode}}] This traditional conditional checks we are in (inline or display) math mode mode. \stopoldprimitive \startoldprimitive[title={\tex {ifinner}}] This traditional one can be confusing. It is true when we are in restricted horizontal mode (a box), internal vertical mode (a box), or inline math mode. \startbuffer test \ifhmode \ifinner INNER\fi HMODE\fi\crlf \hbox{test \ifhmode \ifinner INNER \fi HMODE\fi} \crlf \ifvmode \ifinner INNER\fi VMODE \fi\crlf \vbox{\ifvmode \ifinner INNER \fi VMODE\fi} \crlf \vbox{\ifinner INNER \ifvmode VMODE \fi \fi} \crlf \stopbuffer \typebuffer Watch the last line: because we typeset \type {INNER} we enter horizontal mode: {\getbuffer} \stopoldprimitive \startoldprimitive[title={\tex {ifvoid}}] This traditional conditional checks if a given box register or internal box variable has any content. \stopoldprimitive \startoldprimitive[title={\tex {ifhbox}}] This traditional conditional checks if a given box register or internal box variable represents a horizontal box, \stopoldprimitive \startoldprimitive[title={\tex {ifvbox}}] This traditional conditional checks if a given box register or internal box variable represents a vertical box, \stopoldprimitive \startoldprimitive[title={\tex {ifx}}] We use this traditional \TEX\ conditional a lot in \CONTEXT. Contrary to \type {\if} the two tokens that are compared are not expanded. This makes it possible to compare the meaning of two macros. Depending on the need, these macros can have their content expanded or not. A different number of parameters results in false. Control sequences are identical when they have the same command code and character code. Because a \type {\let} macro is just a reference, both let macros are the same and equal to \type {\relax}: \starttyping \let\one\relax \let\two\relax \stoptyping The same is true for other definitions that result in the same (primitive) or meaning encoded in the character field (think of \type {\chardef}s and so). \stopoldprimitive \startoldprimitive[title={\tex {ifeof}}] This traditional conditional checks if current pointer into the the file bound to the given index is past the end of file. The read and write channels are not really used in \CONTEXT: in \MKII\ we only had one file for all multi|-|pass data, and in \MKIV\ all file related stuff is dealt with in \LUATEX. \stopoldprimitive \startoldprimitive[title={\tex {iftrue}}] Here we have a traditional \TEX\ conditional that is always true (therefore the same is true for any macro that is \type {\let} to this primitive). \stopoldprimitive \startoldprimitive[title={\tex {iffalse}}] Here we have a traditional \TEX\ conditional that is always false (therefore the same is true for any macro that is \type {\let} to this primitive). \stopoldprimitive \startoldprimitive[title={\tex {ifcase}}] This numeric \TEX\ conditional takes a counter (literal, register, shortcut to a character, internal quantity) and goes to the branch that matches. \startbuffer \ifcase 3 zero\or one\or two\or three\or four\else five or more\fi \stopbuffer \typebuffer Indeed: \inlinebuffer\ equals three. In later sections we will see some \LUAMETATEX\ primitives that behave like an \type {\ifcase}. \stopoldprimitive \startoldprimitive[title={\tex {ifdefined}}] In traditional \TEX\ checking for a macro to exist was a bit tricky and therefore \ETEX\ introduced a convenient conditional. We can do this: \starttyping \ifx\MyMacro\undefined ... \else ... \fi \stoptyping but that assumes that \type {\undefined} is indeed undefined. Another test often seen was this: \starttyping \expandafter\ifx\csname MyMacro\endcsname\relax ... \else ... \fi \stoptyping Instead of comparing with \type {\undefined} we need to check with \type {\relax} because the control sequence is defined when not yet present and defaults to \type {\relax}. This is not pretty. \stopoldprimitive \startoldprimitive[title={\tex {ifcsname}}] This is an \ETEX\ conditional that complements the one on the previous section: \starttyping \expandafter\ifx\csname MyMacro\endcsname\relax ... \else ... \fi \ifcsname MyMacro\endcsname ... \else ... \fi \stoptyping Here the first one has the side effect of defining the macro and defaulting it to \type {\relax}, while the second one doesn't do that. Juts think of checking a few million different names: the first one will deplete the hash table and probably string space too. In \LUAMETATEX\ the construction stops when there is no letter or other character seen (\TEX\ expands on the go so expandable macros are dealt with). Instead of an error message, the match is simply false and all tokens till the \type {\endcsname} are gobbled. \stopoldprimitive \startoldprimitive[title={\tex {iffontchar}}] This is an \ETEX\ conditional. It takes a font identifier and a character number. In modern fonts simply checking could not be enough because complex font features can swap in other ones and their index can be anything. Also, a font mechanism can provide fallback fonts and characters, so don't rely on this one too much. It just reports true when the font passed to the frontend has a slot filled. \stopoldprimitive \startnewprimitive[title={\tex {ifincsname}}] This conditional is sort of obsolete and can be used to check if we're inside a \type {\csname} or \type {\ifcsname} construction. It's not used in \CONTEXT. \stopnewprimitive \startnewprimitive[title={\tex {ifabsnum}}] This test will negate negative numbers before comparison, as in: \startbuffer \def\TestA#1{\ifnum #1<100 too small\orelse\ifnum #1>200 too large\else okay\fi} \def\TestB#1{\ifabsnum#1<100 too small\orelse\ifabsnum#1>200 too large\else okay\fi} \TestA {10}\quad\TestA {150}\quad\TestA {210}\crlf \TestB {10}\quad\TestB {150}\quad\TestB {210}\crlf \TestB{-10}\quad\TestB{-150}\quad\TestB{-210}\crlf \stopbuffer \typebuffer Here we get the same result each time: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {ifabsdim}}] This test will negate negative dimensions before comparison, as in: \startbuffer \def\TestA#1{\ifdim #1<2pt too small\orelse\ifdim #1>4pt too large\else okay\fi} \def\TestB#1{\ifabsdim#1<2pt too small\orelse\ifabsdim#1>4pt too large\else okay\fi} \TestA {1pt}\quad\TestA {3pt}\quad\TestA {5pt}\crlf \TestB {1pt}\quad\TestB {3pt}\quad\TestB {5pt}\crlf \TestB{-1pt}\quad\TestB{-3pt}\quad\TestB{-5pt}\crlf \stopbuffer \typebuffer So we get this: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {ifchknum}}] In \CONTEXT\ there are quite some cases where a variable can have a number or a keyword indicating a symbolic name of a number or maybe even some special treatment. Checking if a valid number is given is possible to some extend, but a native checker makes much sense too. So here is one: \startbuffer \ifchknum oeps\or okay\else error\fi\quad \ifchknum 12 \or okay\else error\fi\quad \ifchknum 12pt\or okay\else error\fi \stopbuffer \typebuffer The result is as expected: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {ifchkdim}}] A variant on the checker in the previous section is a dimension checker: \startbuffer \ifchkdim oeps\or okay\else error\fi\quad \ifchkdim 12 \or okay\else error\fi\quad \ifchkdim 12pt\or okay\else error\fi \stopbuffer \typebuffer We get: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {ifcmpnum}}] This conditional compares two numbers and the resulting \type {\ifcase} reflects their relation: \startbuffer [1 2 : \ifcmpnum 1 2 less\or equal\or more\fi]\quad [1 1 : \ifcmpnum 1 1 less\or equal\or more\fi]\quad [2 1 : \ifcmpnum 2 1 less\or equal\or more\fi] \stopbuffer \typebuffer This gives: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {ifcmpdim}}] This conditional compares two dimensions and the resulting \type {\ifcase} reflects their relation: \startbuffer [1pt 2pt : \ifcmpdim 1pt 2pt less\or equal\or more\fi]\quad [1pt 1pt : \ifcmpdim 1pt 1pt less\or equal\or more\fi]\quad [2pt 1pt : \ifcmpdim 2pt 1pt less\or equal\or more\fi] \stopbuffer \typebuffer This gives: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {ifnumval}}] This conditional is a variant on \type {\ifchknum}. This time we get some more detail about the value: \startbuffer [-12 : \ifnumval -12\or negative\or zero\or positive\else error\fi]\quad [0 : \ifnumval 0\or negative\or zero\or positive\else error\fi]\quad [12 : \ifnumval 12\or negative\or zero\or positive\else error\fi]\quad [oeps : \ifnumval oeps\or negative\or zero\or positive\else error\fi] \stopbuffer \typebuffer This gives: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {ifdimval}}] This conditional is a variant on \type {\ifchkdim} and provides some more detailed information about the value: \startbuffer [-12pt : \ifdimval-12pt\or negative\or zero\or positive\else error\fi]\quad [0pt : \ifdimval 0pt\or negative\or zero\or positive\else error\fi]\quad [12pt : \ifdimval 12pt\or negative\or zero\or positive\else error\fi]\quad [oeps : \ifdimval oeps\or negative\or zero\or positive\else error\fi] \stopbuffer \typebuffer This gives: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {iftok}}] When you want to compare two arguments, the usual way to do this is the following: \starttyping \edef\tempA{#1} \edef\tempb{#2} \ifx\tempA\tempB the same \else different \fi \stoptyping This works quite well but the fact that we need to define two macros can be considered a bit of a nuisance. It also makes macros that use this method to be not so called \quote {fully expandable}. The next one avoids both issues: \starttyping \iftok{#1}{#2} the same \else different \fi \stoptyping Instead of direct list you can also pass registers, so given: \startbuffer[a] \scratchtoks{a}% \toks0{a}% \stopbuffer \typebuffer[a] This: \startbuffer[b] \iftok 0 \scratchtoks Y\else N\fi\space \iftok{a}\scratchtoks Y\else N\fi\space \iftok\scratchtoks\scratchtoks Y\else N\fi \stopbuffer \typebuffer[b] {\getbuffer[a]gives: \inlinebuffer[b].} \stopnewprimitive \startnewprimitive[title={\tex {ifcstok}}] A variant on the primitive mentioned in the previous section is one that operates on lists and macros: \startbuffer[a] \def\a{a} \def\b{b} \def\c{a} \stopbuffer \typebuffer[a] \startbuffer[b] \ifcstok\a\b Y\else N\fi\space \ifcstok\a\c Y\else N\fi\space \ifcstok{\a}\c Y\else N\fi\space \ifcstok{a}\c Y\else N\fi \stopbuffer This: \typebuffer[b] {\getbuffer[a]will give us: \inlinebuffer[b].} \stopnewprimitive \startnewprimitive[title={\tex {ifcondition}}] The conditionals in \TEX\ are hard coded as primitives and although it might look like \type {\newif} creates one, it actually just defined three macros. \startbuffer \newif\ifMyTest \meaning\MyTesttrue \crlf \meaning\MyTestfalse \crlf \meaning\ifMyTest \crlf \MyTesttrue \meaning\ifMyTest \crlf \stopbuffer \typebuffer {\getbuffer} This means that when you say: \starttyping \ifMytest ... \else ... \fi \stoptyping You actually have one of: \starttyping \iftrue ... \else ... \fi \iffalse ... \else ... \fi \stoptyping and because these are proper conditions nesting them like: \starttyping \ifnum\scratchcounter > 0 \ifMyTest A\else B\fi \fi \stoptyping will work out well too. This is not true for macros, so for instance: \starttyping \scratchcounter = 1 \unexpanded\def\ifMyTest{\iftrue} \ifnum\scratchcounter > 0 \ifMyTest A\else B\fi \fi \stoptyping will make a run fail with an error (or simply loop forever, depending on your code). This is where \type {\ifcondition} enters the picture: \starttyping \def\MyTest{\iftrue} \scratchcounter0 \ifnum\scratchcounter > 0 \ifcondition\MyTest A\else B\fi \else x \fi \stoptyping This primitive is seen as a proper condition when \TEX\ is in \quotation {fast skipping unused branches} mode but when it is expanding a branch, it checks if the next expanded token is a proper tests and if so, it deals with that test, otherwise it fails. The main condition here is that the \type {\MyTest} macro expands to a proper true or false test, so, a definition like: \starttyping \def\MyTest{\ifnum\scratchcounter<10 } \stoptyping is also okay. Now, is that neat or not? \stopnewprimitive \startnewprimitive[title={\tex {iffrozen}}] This conditional checks if a control sequence is frozen: \starttyping is \iffrozen\MyMacro \else not \fi frozen \stoptyping \stopnewprimitive \startnewprimitive[title={\tex {ifprotected}}] This conditional checks if a control sequence is protected: \starttyping is \ifprotected\MyMacro \else not \fi protected \stoptyping \stopnewprimitive \startnewprimitive[title={\tex {ifusercmd}}] This conditional checks if a control sequence is not one of the primitives: \starttyping is \ifusercmd\MyMacro \else not \fi a primitive \stoptyping It is not always possible to determine this but it should work okay for regular macros, register allocations and character definitions. \stopnewprimitive \startnewprimitive[title={\tex {ifempty}}] This conditional checks if a control sequence is empty: \starttyping is \ifempty\MyMacro \else not \fi empty \stoptyping It is basically a shortcut of: \starttyping is \ifx\MyMacro\empty \else not \fi empty \stoptyping with: \starttyping \def\empty{} \stoptyping Of course this is not empty at all: \starttyping \def\notempty#1{} \stoptyping \stopnewprimitive \startnewprimitive[title={\tex {ifboolean}}] This tests a number (register or equivalent) and any nonzero value represents \type {true}, which is nicer than using an \type {\unless \ifcase}. \stopnewprimitive \startnewprimitive[title={\tex {ifmathparameter}}] This is an \type {\ifcase} where the value depends on if the given math parameter is zero, (\type {0}), set (\type {1}), or unset (\type {2}). \starttyping \ifmathparameter\Umathpunctclosespacing\displaystyle zero \or nonzero \or unset \fi \stoptyping \stopnewprimitive \startnewprimitive[title={\tex {ifmathstyle}}] This is a variant of \type {\ifcase} were the number is one of the seven possible styles: display, text, cramped text, script, cramped script, script script, cramped script script. \starttyping \ifmathstyle display \or text \or cramped text \else normally smaller than text \fi \stoptyping \stopnewprimitive \startnewprimitive[title={\tex {ifarguments}}] This is a variant of \type {\ifcase} were the selector is the number of arguments picked up. For example: \startbuffer \def\MyMacro#1#2#3{\ifarguments\0\or1\or2\or3\else ?\fi} \MyMacro{A}{B}{C} \def\MyMacro#1#0#3{\ifarguments\0\or1\or2\or3\else ?\fi} \MyMacro{A}{B}{C} \def\MyMacro#1#-#2{\ifarguments\0\or1\or2\or3\else ?\fi} \MyMacro{A}{B}{C}\crlf \stopbuffer \typebuffer Watch the non counted, ignored, argument in the last case. Normally this test will be used in combination with \type {\ignorearguments}. {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {ifhastok}}] This conditional looks for occurrences in token lists where each argument has to be a proper list. \startbuffer \def\scratchtoks{x} \ifhastoks{yz} {xyz} Y\else N\fi\quad \ifhastoks\scratchtoks {xyz} Y\else N\fi \stopbuffer \typebuffer We get: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {ifhastoks}}] This test compares two token lists. When a macro is passed it's meaning gets used. \startbuffer \def\x {x} \def\xyz{xyz} (\ifhastoks {x} {xyz}Y\else N\fi)\quad (\ifhastoks {\x} {xyz}Y\else N\fi)\quad (\ifhastoks \x {xyz}Y\else N\fi)\quad (\ifhastoks {y} {xyz}Y\else N\fi)\quad (\ifhastoks {yz} {xyz}Y\else N\fi)\quad (\ifhastoks {yz} {\xyz}Y\else N\fi) \stopbuffer \typebuffer {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {ifhasxtoks}}] This primitive is like the one in the previous section but this time the given lists are expanded. \startbuffer \def\x {x} \def\xyz{\x yz} (\ifhasxtoks {x} {xyz}Y\else N\fi)\quad (\ifhasxtoks {\x} {xyz}Y\else N\fi)\quad (\ifhastoks \x {xyz}Y\else N\fi)\quad (\ifhasxtoks {y} {xyz}Y\else N\fi)\quad (\ifhasxtoks {yz} {xyz}Y\else N\fi)\quad (\ifhasxtoks {yz} {\xyz}Y\else N\fi) \stopbuffer \typebuffer {\getbuffer} This primitive has some special properties. \startbuffer \edef\+{\expandtoken 9 `+} \ifhasxtoks {xy} {xyz}Y\else N\fi\quad \ifhasxtoks {x\+y} {xyz}Y\else N\fi \stopbuffer \typebuffer Here the first argument has a token that has category code \quote {ignore} which means that such a character will be skipped when seen. So the result is: {\getbuffer} This permits checks like these: \startbuffer \edef\,{\expandtoken 9 `,} \ifhasxtoks{\,x\,} {,x,y,z,}Y\else N\fi\quad \ifhasxtoks{\,y\,} {,x,y,z,}Y\else N\fi\quad \ifhasxtoks{\,z\,} {,x,y,z,}Y\else N\fi\quad \ifhasxtoks{\,x\,} {,xy,z,}Y\else N\fi \stopbuffer \typebuffer I admit that it needs a bit of a twisted mind to come up with this, but it works ok: {\getbuffer} \stopnewprimitive \startoldprimitive[title={\tex {else}}] This traditional primitive is part of the condition testing mechanism. When a condition matches, \TEX\ will continue till it sees an \type {\else} or \type {\or} or \type {\orelse} (to be discussed later). It will then do a fast skipping pass till it sees an \type {\fi}. \stopoldprimitive \startoldprimitive[title={\tex {or}}] This traditional primitive is part of the condition testing mechanism and relates to an \type {\ifcase} test (or a similar test to be introduced in later sections). Depending on the value, \TEX\ will do a fast scanning till the right \type {\or} is seen, then it will continue expanding till it sees a \type {\or} or \type {\else} or \type {\orelse} (to be discussed later). It will then do a fast skipping pass till it sees an \type {\fi}. \stopoldprimitive \startoldprimitive[title={\tex {fi}}] This traditional primitive is part of the condition testing mechanism and ends a test. So, we have: \starttyping \ifsomething ... \else ... \fi \ifsomething ... \or ... \or ... \else ... \fi \ifsomething ... \orelse \ifsometing ... \else ... \fi \ifsomething ... \or ... \orelse \ifsometing ... \else ... \fi \stoptyping The \type {\orelse} is new in \LUAMETATEX\ and a continuation like we find in other programming languages (see later section). \stopoldprimitive \startoldprimitive[title={\tex {unless}}] This \ETEX\ prefix will negate the test (when applicable). \starttyping \ifx\one\two YES\else NO\fi \unless\ifx\one\two NO\else YES\fi \stoptyping This primitive is hardly used in \CONTEXT\ and we probably could get rid of these few cases. \stopoldprimitive \startnewprimitive[title={\tex {orelse}}] This primitive provides a convenient way to flatten your conditional tests. So instead of \starttyping \ifnum\scratchcounter<-10 too small \else\ifnum\scratchcounter>10 too large \else just right \fi\fi \stoptyping You can say this: \starttyping \ifnum\scratchcounter<-10 too small \orelse\ifnum\scratchcounter>10 too large \else just right \fi \stoptyping You can mix tests and even the case variants will work in most cases \footnote {I just play safe because there are corner cases that might not work yet.} \starttyping \ifcase\scratchcounter zero \or one \or two \orelse\ifnum\scratchcounter<10 less than ten \else ten or more \fi \stoptyping Performance wise there are no real benefits although in principle there is a bit less housekeeping involved than with nested checks. However you might like this: \starttyping \ifnum\scratchcounter<-10 \expandafter\toosmall \orelse\ifnum\scratchcounter>10 \expandafter\toolarge \else \expandafter\justright \fi \stoptyping over: \starttyping \ifnum\scratchcounter<-10 \expandafter\toosmall \else\ifnum\scratchcounter>10 \expandafter\expandafter\expandafter\toolarge \else \expandafter\expandafter\expandafter\justright \fi\fi \stoptyping or the more \CONTEXT\ specific: \starttyping \ifnum\scratchcounter<-10 \expandafter\toosmall \else\ifnum\scratchcounter>10 \doubleexpandafter\toolarge \else \doubleexpandafter\justright \fi\fi \stoptyping But then, some \TEX ies like complex and obscure code and throwing away working old code that took ages to perfect and get working and also showed that one masters \TEX\ might hurt. \stopnewprimitive \startoldprimitive[title={\tex {futurelet}}] The original \TEX\ primitive \type {\futurelet} can be used to create an alias to a next token, push it back into the input and then expand a given token. \startbuffer \let\MySpecialToken[ \def\DoWhatever{\ifx\NextToken\MySpecialToken YES\else NOP\fi : } \futurelet\NextToken\DoWhatever [A]\crlf \futurelet\NextToken\DoWhatever (A) \stopbuffer \typebuffer This is typically the kind of primitive that most users will never use because it expects a sane follow up handler (here \type {\DoWhatever}) and therefore is related to user interfacing. {\getbuffer} \stopoldprimitive \startnewprimitive[title={\tex {futuredef}}] We elaborate on the example of using \type {\futurelet} in the previous section. Compare that one with the next: \startbuffer \def\MySpecialToken{[} \def\DoWhatever{\ifx\NextToken\MySpecialToken YES\else NOP\fi : } \futurelet\NextToken\DoWhatever [A]\crlf \futurelet\NextToken\DoWhatever (A) \stopbuffer \typebuffer This time we get: {\getbuffer} It is for that reason that we now also have \type {\futuredef}: \startbuffer \def\MySpecialToken{[} \def\DoWhatever{\ifx\NextToken\MySpecialToken YES\else NOP\fi : } \futuredef\NextToken\DoWhatever [A]\crlf \futuredef\NextToken\DoWhatever (A) \stopbuffer \typebuffer So we're back to what we want: {\getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {letcharcode}}] Assigning a meaning to an active character can sometimes be a bit cumbersome; think of using some documented uppercase magic that one tends to forget as it's used only a few times and then never looked at again. So we have this: \startbuffer {\letcharcode 65 1 \catcode 65 13 A : \meaning A}\crlf {\letcharcode 65 2 \catcode 65 13 A : \meaning A}\crlf \stopbuffer \typebuffer here we define \type {A} as an active charcter with meaning \type {1} in the first line and \type {2} in the second. {\getbuffer} Normally one will assign a control sequence: \startbuffer {\letcharcode 66 \bf \catcode 66 13 {B bold}: \meaning B}\crlf {\letcharcode 73 \it \catcode 73 13 {I italic}: \meaning I}\crlf \stopbuffer \typebuffer Of course \type {\bf} and \type {\it} are \CONTEXT\ specific commands: {\getbuffer} \stopnewprimitive \startoldprimitive[title={\tex {global}}] This is one of the original prefixes that can be used when we define a macro of change some register. \starttyping \bgroup \def\MyMacroA{a} \global\def\MyMacroB{a} \gdef\MyMacroC{a} \egroup \stoptyping The macro defined in the first line is forgotten when the groups is left. The second and third definition are both global and these definitions are retained. \stopoldprimitive \startoldprimitive[title={\tex {long}}] This original prefix gave the macro being defined the property that it could not have \type {\par} (or the often equivalent empty lines) in its arguments. It was mostly a protection against a forgotten right curly brace, resulting in a so called run|-|away argument. That mattered on a paper terminal or slow system where such a situation should be catched early. In \LUATEX\ it was already optional, and in \LUAMETATEX\ we dropped this feature completely (so that we could introduce others). \stopoldprimitive \startoldprimitive[title={\tex {outer}}] An outer macro is one that can only be used at the outer level. This property is no longer supported. Like \type {\long}, the \type {\outer} prefix is now an no|-|op (and we don't expect this to have unfortunate side effects). \stopoldprimitive \startoldprimitive[title={\tex {protected}}] A protected macro is one that doesn't get expanded unless it is time to do so. For instance, inside an \type {\edef} it just stays what it is. It often makes sense to pass macros as|-|is to (multi|-|pass) file (for tables of contents). In \CONTEXT\ we use either \type {\protected} or \type {\unexpanded} because the later was the command we used to achieve the same results before \ETEX\ introduced this protection primitive. Originally the \type {\protected} macro was also defined but it has been dropped. \stopoldprimitive \startnewprimitive[title={\tex {frozen}}] You can define a macro as being frozen: \starttyping \frozen\def\MyMacro{...} \stoptyping When you redefine this macro you get an error: \starttyping ! You can't redefine a frozen macro. \stoptyping This is a prefix like \type {\global} and it can be combined with other prefixes. \footnote {The \type {\outer} and \type {\long} prefixes are no|-|ops in \LUAMETATEX\ and \LUATEX\ can be configured to ignore them.} \stopnewprimitive \startnewprimitive[title={\tex {letfrozen}}] You can explicitly freeze an unfrozen macro: \starttyping \def\MyMacro{...} \letfrozen\MyMacro \stoptyping A redefinition will now give: \starttyping ! You can't redefine a frozen macro. \stoptyping \stopnewprimitive \startnewprimitive[title={\tex {unletfrozen}}] A frozen macro cannot be redefined: you get an error. But as nothing in \TEX\ is set in stone, you can do this: \starttyping \frozen\def\MyMacro{...} \unletfrozen\MyMacro \stoptyping and \type {\MyMacro} is no longer protected from overloading. It is still undecided to what extend \CONTEXT\ will use this feature. \stopnewprimitive \startnewprimitive[title={\tex {letprotected}}] Say that you have these definitions: \startbuffer \def \MyMacroA{alpha} \protected \def \MyMacroB{beta} \edef \MyMacroC{\MyMacroA\MyMacroB} \letprotected \MyMacroA \edef \MyMacroD{\MyMacroA\MyMacroB} \meaning \MyMacroC\crlf \meaning \MyMacroD \stopbuffer \typebuffer The typeset meaning in this example is: {\tttf \getbuffer} \stopnewprimitive \startnewprimitive[title={\tex {unletprotected}}] The complementary operation of \type {\letprotected} can be used to unprotect a macro, so that it gets expandable. \startbuffer \def \MyMacroA{alpha} \protected \def \MyMacroB{beta} \edef \MyMacroC{\MyMacroA\MyMacroB} \unletprotected \MyMacroB \edef \MyMacroD{\MyMacroA\MyMacroB} \meaning \MyMacroC\crlf \meaning \MyMacroD \stopbuffer \typebuffer Compare this with the example in the previous section: {\tttf \getbuffer} \stopnewprimitive % \startnewprimitive[title={\tex {letdatacode}}] % {\em Todo.} % \stopnewprimitive \startnewprimitive[title={\tex {beginlocalcontrol}}] Once \TEX\ is initialized it will enter the main loop. In there certain commands trigger a function that itself can trigger further scanning and functions. In \LUAMETATEX\ we can have local main loops and we can either enter it from the \LUA\ end (which we don't discuss here) or at the \TEX\ end using this primitive. \startbuffer \scratchcounter100 \edef\whatever{ a \beginlocalcontrol \advance\scratchcounter 10 b \endlocalcontrol \beginlocalcontrol c \endlocalcontrol d \advance\scratchcounter 10 } \the\scratchcounter \whatever \the\scratchcounter \stopbuffer \typebuffer A bit of close reading probably gives an impression of what happens here: {\getbuffer} The local loop can actually result in material being injected in the current node list. However, where normally assignments are not taking place in an \type {\edef}, here they are applied just fine. Basically we have a local \TEX\ job, be it that it shares all variables with the parent loop. \stopnewprimitive \startnewprimitive[title={\tex {endlocalcontrol}}] See previous section. \stopnewprimitive \startnewprimitive[title={\tex {alignmark}}] When you have the \type{#} not set up as macro parameter character cq.\ align mark, you can use this primitive instead. The same rules apply with respect to multiple such tokens in (nested) macros and alignments. \stopnewprimitive \startnewprimitive[title={\tex {aligntab}}] When you have the \type{&} not set up as align tab, you can use this primitive instead. The same rules apply with respect to multiple such tokens in (nested) macros and alignments. \stopnewprimitive \page % % It doesn't make sense to typeset this, also because it makes me feel old. % % \startsubject[title=A few notes on extensions] % ,placeholder=todo] % % This is a companion to the regular \LUAMETATEX\ reference manual, which is mostly % a concise summary of the program and its features. They don't replace each other, % and none of them claims completeness. There might be more manuals that discuss % specific kind of extensions in the future. First some comments on extensions. % % The starting point of all \TEX\ engines is \TEX. The first follow up was \TEX\ % with support for 8 bit and languages. After that it took some time, but then two % projects started that extended \TEX: \ETEX\ and \OMEGA. In the end the first % brought some extensions to the macro machinery, more registers, a simple right to % left typesetting feature, some more tracing, etc. The second was more ambitious % and has input translation mechanisms, larger fonts, and multi directional % typesetting. By the time \ETEX\ became stable, the \PDFTEX\ engine had showed up % and at some point it integrated \ETEX. But \PDFTEX\ itself also extended the % several components that make up \TEX. The \NTS\ project was started as follow up % on \ETEX\ but although an engine written in \JAVA\ was the result it never was % used for extensions; this project was fully funded by the german language user % group. % % The \CONTEXT\ macro package was an early adopter of the \ETEX\ and \PDFTEX\ % extensions. Probably the most significant effect was that we got more registers % (some other features were already kind of present in macro form). In between % these engines we played with \type {eetex} (extended \ETEX) because we had some % wishes of our own. We also explored extensions to the \DVI\ format but in the end % \PDF\ won that race. An example of a new mechanism that we introduced in % \CONTEXT\ was position tracking: marking positions that can be saved when the % output is created and used in a second run. This started as a \DVI\ postprocessor % in \PERL\ written by me, later turned into a \CLANGUAGE\ program by Taco, and % eventually integrated in \PDFTEX\ by Thanh (\PDFTEX\ was a phd project). At some % point \XETEX\ was developed, funded and driven by an organization that did high % end multi lingual typesetting; it was based on \ETEX\ and uses a \DVI\ to \PDF\ % backend processor. Both \PDFTEX\ and \XETEX\ are supported by \CONTEXT\ \MKII, % and both engines are basically stable and frozen. % % At some point Hartmut and I started playing with \LUA\ in \PDFTEX\ but soon Taco, % Hartmut and I decided to start a follow up project. All the work on \LUATEX\ (and % later \LUAMETATEX) is done whenever there is time and without financial % compensation, so we have a slow but steady development track. Early in the % \LUATEX\ development there has been some funding for the initial transition (by % Taco) from \PDFTEX\ to what became the early versions of \LUATEX . You can read % more about the oriental \TEX\ project in other documents and articles in user % group journals. There has been some funded development of a library subsystem % (which for some reason never took off) as well as \LUAJIT\ integration (by % Luigi). The initial \METAPOST\ library (also done by Taco) was funded by a couple % of user groups. Then there are the (ongoing) font projects by GUST that got % funded by user groups as these were much needed for the \UNICODE\ engines. % % After the jump start, most work was and is still done in the usual \TEX\ spirit, % on a voluntary basis, by folks from the \CONTEXT\ community, and after a decades % we reached the stable version 1.00. It's one of the engines in \TEXLIVE\ and % Luigi makes sure it integrates well in there. We did continue and around 1.10 the % more of less final version was reached and \LUAMETATEX\ took off. In \LUATEX\ % only bugs get fixed, occasionally some helpers can get added, and we might port % some of \LUAMETATEX\ back to its parent, when it doesn't harm compatibility. % % Already early in development some primitives were added that enhance the macro % language. More were added later. It's these extensions that are discussed in this % document. There are several documents in the \CONTEXT\ distribution that discuss % the (ongoing) development, right from the start, and these often contain % examples. For instance some of the new primitives have been introduced there, % complete with a rationale and examples of usage. % % Just for the record: the \CONTEXT\ group runs the build farm that is used to % generate binaries for all sorts of platforms. We make sure that there are always % versions that can be used for real production jobs. You can expect regular % updates as long as there are developments (of course, eventually we're done). % % \stopsubject \startsubject[title=Rationale] % ,placeholder=todo] Some words about the why and how it came. One of the early adopters of \CONTEXT\ was Taco Hoekwater and we spent numerous trips to \TEX\ meetings all over the globe. He was also the only one I knew who had read the \TEX\ sources. Because \CONTEXT\ has always been on the edge of what is possible and at that time we both used it for rather advanced rendering, we also ran into the limitations. I'm not talking of \TEX\ features here. Naturally old school \TEX\ is not really geared for dealing with images of all kind, colors in all kind of color spaces, highly interactive documents, input methods like \XML, etc. The nice thing is that it offers some escapes, like specials and writes and later execution of programs that opened up lots of possibilities, so in practice there were no real limitations to what one could do. But coming up with a consistent and extensible (multi lingual) user interface was non trivial, because it had an impact in memory usage and performance. A lot could be done given some programming, as \CONTEXT\ \MKII\ proves, but it was not always pretty under the hood. The move to \LUATEX\ and \MKIV\ transferred some action to \LUA, and because \LUATEX\ effectively was a \CONTEXT\ related project, we could easily keep them in sync. Our traveling together, meeting several times per year, and eventually email and intense \LUATEX\ developments (lots of Skype sessions) for a couple of years, gave us enough opportunity to discuss all kind of nice features not present in the engine. The previous century we discussed lots of them, rejected some, stayed with others, and I admit that forgot about most of the arguments already. Some that we did was already explored in \type {eetex}, some of those ended up in \LUATEX, and eventually what we have in \LUAMETATEX\ can been seen as the result of years of programming in \TEX, improving macros, getting more performance and efficiency out of existing \CONTEXT\ code and inspiration that we got out of the \CONTEXT\ community, a demanding lot, always willing to experiment with us. Once I decided to work on \LUAMETATEX\ and bind its source to the \CONTEXT\ distribution so that we can be sure that it won't get messed up and might interfere with the \CONTEXT\ expectations, some more primitives saw their way into it. It is very easy to come up with all kind of bells and whistles but it is equally easy to hurt performance of an engine and what might go unnoticed in simple tests can really affect a macro package that depends on stability. So, what I did was mostly looking at the \CONTEXT\ code and wondering how to make some of the low level macros look more natural, also because I know that there are users who look into these sources. We spend a lot of time making them look consistent and nice and the nicer the better. Getting a better performance was seldom an argument because much is already as fast as can be so there is not that much to gain, but less clutter in tracing was an argument for some new primitives. Also, the fact that we soon might need to fall back on our phones to use \TEX\ a smaller memory footprint and less byte shuffling also was a consideration. The \LUAMETATEX\ memory footprint is somewhat smaller than the \LUATEX\ footprint. By binding \LUAMETATEX\ to \CONTEXT\ we can also guarantee that the combinations works as expected. I'm aware of the fact that \CONTEXT\ is in a somewhat unique position. First of all it has always been kind of cutting edge so its users are willing to experiment. There are users who immediately update and run tests, so bugs can and will be fixed fast. Already for a long time the community has an convenient infrastructure for updating and the build farm for generating binaries (also for other engines) is running smoothly. Then there is the \CONTEXT\ user interface that is quite consistent and permits extensions with staying backward compatible. Sometimes users run into old manuals or examples and then complain that \CONTEXT\ is not compatible but that then involves obsolete technology: we no longer need font and input encodings and font definitions are different for \OPENTYPE\ fonts. We always had an abstract backend model, but nowadays \PDF\ is kind of dominant and drives a lot of expectations. So, some of the \MKII\ commands are gone and \MKIV\ has some more. Also, as \METAPOST\ evolved that department in \CONTEXT\ also evolved. Think of it like cars: soon all are electric so one cannot expect a hole to poor in some fluid but gets a (often incompatible) plug instead. And buttons became touch panels. There is no need to use much force to steer or brake. Navigation is different, as are many controls. And do we need to steer ourselves a decade from now? So, just look at \TEX\ and \CONTEXT\ in the same way. A system from the nineties in the previous century differs from one three decades later. Demands differ, input differs, resources change, editing and processing moves on, and so on. Manuals, although still being written are seldom read from cover to cover because online searching replaced them. And who buys books about programming? So \LUAMETATEX, while still being \TEX\ also moves on, as do the way we do our low level coding. This makes sense because the original \TEX\ ecosystem was not made with a huge and complex macro package in mind, that just happened. An author was supposed to make a style for each document. An often used argument for using another macro package over \CONTEXT\ was that the later evolved and other macro packages would work the same forever and not change from the perspective of the user. In retrospect those arguments were somewhat strange because the world, computers, users etc.\ do change. Standards come and go, as do software politics and preferences. In many aspects the \TEX\ community is not different from other large software projects, operating system wars, library devotees, programming language addicts, paradigm shifts. But, don't worry, if you don't like \LUAMETATEX\ and its new primitives, just forget about them. The other engines will be there forever and are a safe bet, although \LUATEX\ already stirred up the pot I guess. But keep in mind that new features in the latest greatest \CONTEXT\ version will more and more rely on \LUAMETATEX\ being used; after all that is where it's made for. And this manual might help understand its users why, where and how the low level code differs between \MKII, \MKIV\ and \LMTX. Can we expect more new primitives than the ones introduced here? Given the amount of time I spend on experimenting and considering what made sense and what not, the answer probably is \quotation {no}, or at least \quotation {not that much}. As in the past no user ever requested the kind of primitives that were added, I don't expect users to come up with requests in the future either. Of course, those more closely related to \CONTEXT\ development look at it from the other end. Because it's there where the low level action really is, demands might still evolve. Hans Hagen \crlf Hasselt NL \stopsubject \stoptext