diff options
| author | Hans Hagen <pragma@wxs.nl> | 2022-08-01 20:41:35 +0200 |
|---|---|---|
| committer | Context Git Mirror Bot <phg@phi-gamma.net> | 2022-08-01 20:41:35 +0200 |
| commit | 1e83b1b27a30fc8df44607a37103b6c65efec538 (patch) | |
| tree | 4031d6d4a36ab96f39eee03cc1326a42b89197c0 /doc | |
| parent | 659d787cc8a329d01ff920c7e1a4659dc66b7daa (diff) | |
| download | context-1e83b1b27a30fc8df44607a37103b6c65efec538.tar.gz | |
2022-08-01 20:31:00
Diffstat (limited to 'doc')
14 files changed, 2730 insertions, 1746 deletions
diff --git a/doc/context/documents/general/manuals/luametatex.pdf b/doc/context/documents/general/manuals/luametatex.pdf Binary files differindex ab8f8b347..e7c0b89f0 100644 --- a/doc/context/documents/general/manuals/luametatex.pdf +++ b/doc/context/documents/general/manuals/luametatex.pdf diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-building.tex b/doc/context/sources/general/manuals/luametatex/luametatex-building.tex index 357e2ed41..62b1be668 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex-building.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex-building.tex @@ -4,7 +4,7 @@ \startcomponent luametatex-building -\startchapter[reference=building,title={Building paragraphs and pages}] +\startchapter[reference=building,title={Boxes, paragraphs and pages}] \startsection[title={Introduction}] @@ -20,12 +20,849 @@ other chapters. These enhancements are still somewhat experimental. \stopsection +\startsection[title=Directions] + +\topicindex {\OMEGA} +\topicindex {\ALEPH} +\topicindex {directions} + +\startsubsection[title={Two directions}] + +The directional model in \LUAMETATEX\ is a simplified version the the model used +in \LUATEX. In fact, not much is happening at all: we only register a change in +direction. + +\stopsubsection + +\startsubsection[title={How it works}] + +The approach is that we try to make node lists balanced but also try to avoid +some side effects. What happens is quite intuitive if we forget about spaces +(turned into glue) but even there what happens makes sense if you look at it in +detail. However that logic makes in|-|group switching kind of useless when no +properly nested grouping is used: switching from right to left several times +nested, results in spacing ending up after each other due to nested mirroring. Of +course a sane macro package will manage this for the user but here we are +discussing the low level injection of directional information. + +This is what happens: + +\starttyping +\textdirection 1 nur {\textdirection 0 run \textdirection 1 NUR} nur +\stoptyping + +This becomes stepwise: + +\startnarrower +\starttyping +injected: [push 1]nur {[push 0]run [push 1]NUR} nur +balanced: [push 1]nur {[push 0]run [pop 0][push 1]NUR[pop 1]} nur[pop 0] +result : run {RUNrun } run +\stoptyping +\stopnarrower + +And this: + +\starttyping +\textdirection 1 nur {nur \textdirection 0 run \textdirection 1 NUR} nur +\stoptyping + +becomes: + +\startnarrower +\starttyping +injected: [+TRT]nur {nur [+TLT]run [+TRT]NUR} nur +balanced: [+TRT]nur {nur [+TLT]run [-TLT][+TRT]NUR[-TRT]} nur[-TRT] +result : run {run RUNrun } run +\stoptyping +\stopnarrower + +Now, in the following examples watch where we put the braces: + +\startbuffer +\textdirection 1 nur {{\textdirection 0 run} {\textdirection 1 NUR}} nur +\stopbuffer + +\typebuffer + +This becomes: + +\startnarrower +\getbuffer +\stopnarrower + +Compare this to: + +\startbuffer +\textdirection 1 nur {{\textdirection 0 run }{\textdirection 1 NUR}} nur +\stopbuffer + +\typebuffer + +Which renders as: + +\startnarrower +\getbuffer +\stopnarrower + +So how do we deal with the next? + +\startbuffer +\def\ltr{\textdirection 0\relax} +\def\rtl{\textdirection 1\relax} + +run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur} +run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run} +\stopbuffer + +\typebuffer + +It gets typeset as: + +\startnarrower +\startlines +\getbuffer +\stoplines +\stopnarrower + +We could define the two helpers to look back, pick up a skip, remove it and +inject it after the dir node. But that way we loose the subtype information that +for some applications can be handy to be kept as|-|is. This is why we now have a +variant of \prm {textdirection} which injects the balanced node before the skip. +Instead of the previous definition we can use: + +\startbuffer[def] +\def\ltr{\linedirection 0\relax} +\def\rtl{\linedirection 1\relax} +\stopbuffer + +\typebuffer[def] + +and this time: + +\startbuffer[txt] +run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur} +run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run} +\stopbuffer + +\typebuffer[txt] + +comes out as a properly spaced: + +\startnarrower +\startlines +\getbuffer[def,txt] +\stoplines +\stopnarrower + +Anything more complex that this, like combination of skips and penalties, or +kerns, should be handled in the input or macro package because there is no way we +can predict the expected behaviour. In fact, the \prm {linedirection} is just a +convenience extra which could also have been implemented using node list parsing. + +Directions are complicated by the fact that they often need to work over groups +so a separate grouping related stack is used. A side effect is that there can be +paragraphs with only a local par node followed by direction synchronization +nodes. Paragraphs like that are seen as empty paragraphs and therefore ignored. +Because \prm {noindent} doesn't inject anything but a \prm {indent} injects +an box, paragraphs with only an indent and directions are handles and paragraphs +with content. When indentation is normalized a paragraph with an indentation +skip is seen as content. + +\stopsubsection + +\startsubsection[title={Normalizing lines}] + +The original \TEX\ machinery was never meant to be opened up. As a consequence a +constructed line can have different layouts. There can be left- and/or right +skips and hanging indentation or parshape can result in a shift and adapted +width. In \LUATEX\ glue got subtypes so we can recognize the left-, right and +parfill skips, but still there is no hundred percent certainty about the shape. + +In \LUAMETATEX\ lines can be normalized. This is optional because we want to +preserve the original (for comparison) and is controlled by \prm +{normalizelinemode}. That variable actually drives some more. An earlier version +provided a few more granular options (for instance: does a leftskip comes before +or after a left hanging indentation) but in the end that was dropped. Because +this normalization only is seen at the \LUA\ end there is no need to go into much +detail here. + +At this moment a line has this pattern: left parfill, left hang, left skip, +indentation, content, right hang, right skip, right parfill. Of course the +indentation and fill skips are not present in every line. + +Control over normalization happens via the mentioned mode variable and here is +what the engine provides right now. We use a bitmap: + +\starttabulate[|l|l|] +\DB value \BC reported \NC \NR +\TB +\NC \type{0x0001} \NC normalize line as described above \NC \NR +\NC \type{0x0002} \NC use a skip for parindent instead of a box \NC \NR +\NC \type{0x0004} \NC swap hangindent in l2r mode \NC \NR +\NC \type{0x0008} \NC swap parshape in l2r mode \NC \NR +\NC \type{0x0010} \NC put breaks after dir in l2r mode \NC \NR +\NC \type{0x0020} \NC remove margin kerns (\PDFTEX\ left-over) \NC \NR +\NC \type{0x0040} \NC if needed clip width and use correction kern \NC \NR +\LL +\stoptabulate + +Setting the bit enables the related normalization. More features might be added +in future releases. + +% Swapping shapes +% +% Another adaptation to the \ALEPH\ directional model is control over shapes driven +% by \prm {hangindent} and \prm {parshape}. This is controlled by a new parameter +% \prm {shapemode}: +% +% \starttabulate[|c|l|l|] +% \DB value \BC \prm {hangindent} \BC \prm {parshape} \NC \NR +% \TB +% \BC \type{0} \NC normal \NC normal \NC \NR +% \BC \type{1} \NC mirrored \NC normal \NC \NR +% \BC \type{2} \NC normal \NC mirrored \NC \NR +% \BC \type{3} \NC mirrored \NC mirrored \NC \NR +% \LL +% \stoptabulate +% +% The value is reset to zero (like \prm {hangindent} and \prm {parshape}) +% after the paragraph is done with. You can use negative values to prevent +% this. In \in {figure} [fig:shapemode] a few examples are given. +% +% \startplacefigure[reference=fig:shapemode,title={The effect of \type {shapemode}.}] +% \startcombination[2*3] +% {\ruledvbox \bgroup \setuptolerance[verytolerant] +% \hsize .45\textwidth \switchtobodyfont[6pt] +% \pardirection 0 \textdirection 0 +% \hangindent 40pt \hangafter -3 +% \leftskip10pt \input tufte \par +% \egroup} {TLT: hangindent} +% {\ruledvbox \bgroup \setuptolerance[verytolerant] +% \hsize .45\textwidth \switchtobodyfont[6pt] +% \pardirection 0 \textdirection 0 +% \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize +% \input tufte \par +% \egroup} {TLT: parshape} +% {\ruledvbox \bgroup \setuptolerance[verytolerant] +% \hsize .45\textwidth \switchtobodyfont[6pt] +% \pardirection 1 \textdirection 1 +% \hangindent 40pt \hangafter -3 +% \leftskip10pt \input tufte \par +% \egroup} {TRT: hangindent mode 0} +% {\ruledvbox \bgroup \setuptolerance[verytolerant] +% \hsize .45\textwidth \switchtobodyfont[6pt] +% \pardirection 1 \textdirection 1 +% \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize +% \input tufte \par +% \egroup} {TRT: parshape mode 0} +% {\ruledvbox \bgroup \setuptolerance[verytolerant] +% \hsize .45\textwidth \switchtobodyfont[6pt] +% \shapemode=3 +% \pardirection 1 \textdirection 1 +% \hangindent 40pt \hangafter -3 +% \leftskip10pt \input tufte \par +% \egroup} {TRT: hangindent mode 1 & 3} +% {\ruledvbox \bgroup \setuptolerance[verytolerant] +% \hsize .45\textwidth \switchtobodyfont[6pt] +% \shapemode=3 +% \pardirection 1 \textdirection 1 +% \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize +% \input tufte \par +% \egroup} {TRT: parshape mode 2 & 3} +% \stopcombination +% \stopplacefigure +% +% We have \type {\pardirection}, \type {\textdirection}, \type {\mathdirection} and +% \type {\linedirection} that is like \type {\textdirection} but with some +% additional (inline) glue checking. + +% Controlling glue with \prm {breakafterdirmode} +% +% Glue after a dir node is ignored in the linebreak decision but you can bypass that +% by setting \prm {breakafterdirmode} to~\type {1}. The following table shows the +% difference. Watch your spaces. +% +% \def\ShowSome#1{% +% \BC \type{#1} +% \NC \breakafterdirmode\zerocount\hsize\zeropoint#1 +% \NC +% \NC \breakafterdirmode\plusone\hsize\zeropoint#1 +% \NC +% \NC \NR +% } +% +% \starttabulate[|l|Tp(1pt)|w(5em)|Tp(1pt)|w(5em)|] +% \DB +% \BC \type{0} +% \NC +% \BC \type{1} +% \NC +% \NC \NR +% \TB +% \ShowSome{pre {\textdirection 0 xxx} post} +% \ShowSome{pre {\textdirection 0 xxx }post} +% \ShowSome{pre{ \textdirection 0 xxx} post} +% \ShowSome{pre{ \textdirection 0 xxx }post} +% \ShowSome{pre { \textdirection 0 xxx } post} +% \ShowSome{pre {\textdirection 0\relax\space xxx} post} +% \LL +% \stoptabulate + +\stopsubsection + +\startsubsection[title=Orientations] + +\topicindex {boxes+orientations} + +As mentioned, the difference with \LUATEX\ is that we only have numeric +directions and that there are only two: left|-|to|-|right (\type {0}) and +right|-|to|-|left (\type {1}). The direction of a box is set with \type +{direction}. + +In addition to that boxes can now have an \type {orientation} keyword followed by +optional \type {xoffset} and|/|or \type {yoffset} keywords. The offsets don't +have consequences for the dimensions. The alternatives \type {xmove} and \type +{ymove} on the contrary are reflected in the dimensions. Just play with them. The +offsets and moves only are accepted when there is also an orientation, so no time +is wasted on testing for these rarely used keywords. There are related primitives +\type {\box...} that set these properties. + +As these are experimental it will not be explained here (yet). They are covered +in the descriptions of the development of \LUAMETATEX: articles and|/|or +documents in the \CONTEXT\ distribution. For now it is enough to know that the +orientation can be up, down, left or right (rotated) and that it has some +anchoring variants. Combined with the offsets this permits macro writers to +provide solutions for top|-|down and bottom|-|up writing directions, something +that is rather macro package specific and used for scripts that need +manipulations anyway. The \quote {old} vertical directions were never okay and +therefore not used. + +There are a couple of properties in boxes that you can set and query but that +only really take effect when the backend supports them. When usage on \CONTEXT\ +shows that is't okay, they will become official, so we just mention them: \prm +{boxdirection}, \prm {boxattribute}, \prm {boxorientation}, \prm {boxxoffset}, +\prm {boxyoffset}, \prm {boxxmove}, \prm {boxymove} and \prm {boxtotal}. + +{\em This is still somewhat experimental and will be documented in more detail +when I've used it more in \CONTEXT\ and the specification is frozen. This might +take some time (and user input).} + +\stopsubsection + +\stopsection + +\startsection[title={Boxes, rules and leaders}] + +\startsubsection[title={\prm {outputbox}}] + +\topicindex {output} + +This integer parameter allows you to alter the number of the box that will be +used to store the page sent to the output routine. Its default value is 255, and +the acceptable range is from 0 to 65535. + +\startsyntax +\outputbox = 12345 +\stopsyntax + +\stopsubsection + +\startsubsection[title={\prm {hrule}, \prm {vrule}, \prm {srule}, \prm {nohrule} and \prm {novrule}}] + +\topicindex {rules} + +Both rule drawing commands take an optional \type {xoffset} and \type {yoffset} +parameter. The displacement is virtual and not taken into account when the +dimensions are calculated. A rule is specified in the usual way: + +\obeydepth + +\startbuffer +\blue \vrule + height 2ex depth 1ex width 10cm +\relax +\stopbuffer + +\startlinecorrection +\getbuffer +\stoplinecorrection + +There is however a catch. The keyword scanners in \LUAMETATEX\ are implemented +slightly different. When \TEX\ scans a keyword it will (case insensitive) scan +for a whole keyword. So, it scans for \type {height} and when it doesn't find it +it will scan for \type {depth} etc. When it does find a keyword in this case it +expects a dimension next. When that criterium is not met it will issue an error +message. + +In order to avoid look ahead failures like that it is recommended to end the +specification with \type {\relax}. A glue specification is an other example where +a \type {\relax} makes sense when look ahead issues are expected and actually +there in traditional scanning the order of keywords can also matter. In any case, +when no valid keyword is seen the characters scanned so far are pushed back in +the input. + +The main reason for using an adapted scanner is that we always permit repetition +(consistency) and accept an arbitrary order. Because we have more keywords to +process the scanner quits at a partial failure. This prevents some push back and +also gives an earlier warning. Interesting is that some \CONTEXT\ users ran into +error messages due to a missing \type {\relax} and found out that their style has +a potential flaw with respect to look ahead. One can be lucky for years. + +Back to rules, there are some extra keywords, two deal with an offset, and four +provide margins. The margins are a bit special because \type {left} and \type +{top} are the same as are \type {right} and \type {bottom}. They influence the +edges and these depend on it being a horizontal or vertical rule. + +\obeydepth + +\startbuffer +\blue \vrule + height 2.0ex depth 1.0ex width 10cm +\relax +\white \vrule + height 1.0ex depth 0.5ex width 9cm + xoffset -9.5cm yoffset .25ex +\relax +\blue \vrule + height .5ex depth 0.25ex width 8cm + xoffset -18cm yoffset .375ex top 1pt +\relax +\stopbuffer + +\startlinecorrection +\getbuffer +\stoplinecorrection + +Two new primitives were introduced: \prm {nohrule} and \prm {novrule}. These can +be used to reserve space. This is often more efficient than creating an empty box +with fake dimensions. Of course this assumes that the backend implements them +being invisible but still taking space. + +An \prm {srule} is sort of special. In text mode it is just a convenience (we +could do without it for ages) but in math mode it comes in handy when we want to +enforce consistency. \footnote {In \CONTEXT\ there is a lot of focus on +consistent vertical spacing, something that doesn't naturally comes with \TEX\ +(you have to pay attention!) and therefore for decades now you can find plenty of +documents with bad spacing of a nature that has seem to have become accepted as +quality. This probably makes these \prm {srule}'s one of the few primitives that +actually targets at \CONTEXT.} + +As with all rules, the backend will makes rules span the width or height and +depth of the encapsulating box. An \prm {srule} is just a \prm {vrule} but is set +up such that it can adapt itself: + +\startbuffer +\hbox to 3cm {x\leaders\hrule\hfil x} +\hbox{x \vrule width 4cm \relax x} +\hbox{x \srule width 4cm \relax x} +\hbox{x \vrule font \font char `( width 4cm \relax x} +\hbox{x \srule font \font char `( width 4cm \relax x} +\hbox{$x \srule fam \fam char `( width 4cm \relax x$} +\hbox{$x \vrule fam \fam char `( width 4cm \relax x$} +\stopbuffer + +\typebuffer + +You can hard code the height and depth or get it from a font|/|family|/|character +combination. This is especially important in math mode where then can adapt to +(stylistic) circumstances. + +\startlines +\showboxes\getbuffer +\stoplines + +Because this kind of rules has a dedicated subtype you can intercept it in the backend +if needed. + +\stopsubsection + +\startsubsection[title={\prm {vsplit}}] + +\topicindex {splitting} + +The \prm {vsplit} primitive has to be followed by a specification of the required +height. As alternative for the \type {to} keyword you can use \type {upto} to get +a split of the given size but result has the natural dimensions then. + +\starttyping +\vsplit 123 to 10cm % final box has the required height +\vsplit 123 upto 10cm % final box has its natural height +\stoptyping + +\stopsubsection + +\startsubsection[title={\prm {boxxoffset}, \prm {boxyoffset}, \prm {boxxmove}, \prm {boxymove}, +\prm{boxorientation} and \prm{boxgeometry}}] + +This repertoire of primitives can be used to do relative positioning. The offsets +are virtual while the moves adapt the dimensions. The orientation bitset can be +used to rotate the box over 90, 180 and 270 degrees. It also influences the +corner, midpoint or baseline. + +{\em There is information in the \CONTEXT\ low level manuals and in due time I +will add a few examples here. This feature needs support in the backend when used +(as in \CONTEXT) so it might influence performance.} + +\stopsubsection + +\startsubsection[title={\prm {boxtotal}}] + +The \prm {boxtotal} primitive returns the sum of the height and depth and is less +useful as setter: it just sets the height and depth to half of the given value. + +\stopsubsection + +\startsubsection[title={\prm {boxshift}}] + +In traditional \TEX\ a box has height, depth, width and a shift where the later +relates to \prm {raise}, \prm {lower}, \prm {moveleft} and \prm {moveright}. This +primitive can be used to query and set this property. + +\startbuffer +\setbox0\hbox{test test test} +\setbox2\hbox{test test test} \boxshift2 -10pt +\ruledhbox{x \raise10pt\box0\ x} +\ruledhbox{x \box2\ x} +\stopbuffer + +\typebuffer + +\stopsubsection + +\startsubsection[title={\prm {boxanchor}, \prm {boxanchors}, \prm {boxsource} and \prm {boxtarget}}] + +{\em These are experimental.} + +\stopsubsection + +\startsubsection[title={\prm {boxfreeze}, \prm {boxadapt} and \prm {boxrepack}}] + +\topicindex {boxes+postprocessing} + +This operation will freeze the glue in the given box, something that normally is +delayed and delegated to the backend. + +\startbuffer +\setbox 0 \hbox to 5cm {\hss test} +\setbox 2 \hbox to 5cm {\hss test} +\boxfreeze 2 0 +\ruledhbox{\unhbox 0} +\ruledhbox{\unhbox 2} +\stopbuffer + +\typebuffer + +The second parameter to \prm {boxfreeze} determines recursion. Here we just +freeze the outer level: + +\getbuffer + +Repacking will take the content of an existing box and add or subtract from it: + +\startbuffer +\setbox 0 \hbox {test test test} +\setbox 2 \hbox {\red test test test} \boxrepack0 +.2em +\setbox 4 \hbox {\green test test test} \boxrepack0 -.2em +\ruledhbox{\box0} \vskip-\lineheight +\ruledhbox{\box0} \vskip-\lineheight +\ruledhbox{\box0} +\stopbuffer + +\typebuffer + +\getbuffer + +We can use this primitive to check the natural dimensions: + +\startbuffer +\setbox 0 \hbox spread 10pt {test test test} +\ruledhbox{\box0} (\the\boxrepack0,\the\wd0) +\stopbuffer + +\typebuffer + +\getbuffer + +Adapting will recalculate the dimensions with a scale factor for the glue: + +\startbuffer +\setbox 0 \hbox {test test test} +\setbox 2 \hbox {\red test test test} \boxadapt 0 200 +\setbox 4 \hbox {\blue test test test} \boxadapt 0 -200 +\ruledhbox{\box0} \vskip-\lineheight +\ruledhbox{\box0} \vskip-\lineheight +\ruledhbox{\box0} +\stopbuffer + +\typebuffer + +\getbuffer + +\stopsubsection + +\startsubsection[title={Overshooting dimensions}] + +\topicindex {boxes+overfull} + +The \prm {overshoot} primitive reports the most recent amount of overshoot when a +box is packages. It relates to overfull boxes and the then set \prm {badness} of +1000000. + +\startbuffer +\hbox to 2cm {does it fit} \the\overshoot +\hbox to 2cm {does it fit in here} \the\overshoot +\hbox to 2cm {how much does fit in here} \the\overshoot +\stopbuffer + +\typebuffer + +This global state variables reports a dimension: + +\startlines +\getbuffer +\stoplines + +\stopsubsection + +\startsubsection[title={Images and reused box objects},reference=sec:imagesandforms] + +\topicindex {images} + +In original \TEX\ image support is dealt with via specials. It's not a native +feature of the engine. All that \TEX\ cares about is dimensions, so in practice +that meant: using a box with known dimensions that wraps a special that instructs +the backend to include an image. The wrapping is needed because a special itself +is a whatsit and as such has no dimensions. + +In \PDFTEX\ a special whatsit for images was introduced and that one {\em has} +dimensions. As a consequence, in several places where the engine deals with the +dimensions of nodes, it now has to check the details of whatsits. By inheriting +code from \PDFTEX, the \LUATEX\ engine also had that property. However, at some +point this approach was abandoned and a more natural trick was used: images (and +box resources) became a special kind of rules, and as rules already have +dimensions, the code could be simplified. + +When direction nodes and (formerly local) par nodes also became first class +nodes, whatsits again became just that: nodes representing whatever you want, but +without dimensions, and therefore they could again be ignored when dimensions +mattered. And, because images were disguised as rules, as mentioned, their +dimensions automatically were taken into account. This separation between front +and backend cleaned up the code base already quite a bit. + +In \LUAMETATEX\ we still have the image specific subtypes for rules, but the +engine never looks at subtypes of rules. That was up to the backend. This means +that image support is not present in \LUAMETATEX. When an image specification was +parsed the special properties, like the filename, or additional attributes, were +stored in the backend and all that \LUATEX\ does is registering a reference to an +image's specification in the rule node. But, having no backend means nothing is +stored, which in turn would make the image inclusion primitives kind of weird. + +Therefore you need to realize that contrary to \LUATEX, {\em in \LUAMETATEX\ +support for images and box reuse is not built in}! However, we can assume that +an implementation uses rules in a similar fashion as \LUATEX\ does. So, you can +still consider images and box reuse to be core concepts. Here we just mention the +primitives that \LUATEX\ provides. They are not available in the engine but can +of course be implemented in \LUA. + +\starttabulate[|l|p|] +\DB command \BC explanation \NC \NR +\TB +\NC \tex {saveboxresource} \NC save the box as an object to be included later \NC \NR +\NC \tex {saveimageresource} \NC save the image as an object to be included later \NC \NR +\NC \tex {useboxresource} \NC include the saved box object here (by index) \NC \NR +\NC \tex {useimageresource} \NC include the saved image object here (by index) \NC \NR +\NC \tex {lastsavedboxresourceindex} \NC the index of the last saved box object \NC \NR +\NC \tex {lastsavedimageresourceindex} \NC the index of the last saved image object \NC \NR +\NC \tex {lastsavedimageresourcepages} \NC the number of pages in the last saved image object \NC \NR +\LL +\stoptabulate + +An implementation probably should accept the usual optional dimension parameters +for \type {\use...resource} in the same format as for rules. With images, these +dimensions are then used instead of the ones given to \tex {useimageresource} but +the original dimensions are not overwritten, so that a \tex {useimageresource} +without dimensions still provides the image with dimensions defined by \tex +{saveimageresource}. These optional parameters are not implemented for \tex +{saveboxresource}. + +\starttyping +\useimageresource width 20mm height 10mm depth 5mm \lastsavedimageresourceindex +\useboxresource width 20mm height 10mm depth 5mm \lastsavedboxresourceindex +\stoptyping + +Examples or optional entries are \type {attr} and \type {resources} that accept a +token list, and the \type {type} key. When set to non|-|zero the \type {/Type} +entry is omitted. A value of 1 or 3 still writes a \type {/BBox}, while 2 or 3 +will write a \type {/Matrix}. But, as said: this is entirely up to the backend. +Generic macro packages (like \type {tikz}) can use these assumed primitives so +one can best provide them. It is probably, for historic reasons, the only more or +less standardized image inclusion interface one can expect to work in all macro +packages. + +\stopsubsection + +\startsubsection[title={\prm {hpack}, \prm {vpack} and \prm {tpack}}] + +\topicindex {packing} + +These three primitives are the equivalents of \prm {hbox}, \prm {vbox} and +\prm {vtop} but they don't trigger the packaging related callbacks. Of course +one never know if content needs a treatment so using them should be done with +care. Apart from accepting more keywords (and therefore options) the normal +box behave the same as before. The \prm {vcenter} builder also works in text +mode. + +\stopsubsection + +\startsubsection[title={\prm {unhpack}, \prm {unvpack}}] + +\topicindex {unpacking} + +These two are somewhat experimental. They ignore the accumulated pre- and +postmigrated material bound to a box. I needed it for some experiment so the +functionality might change when I really need it. + +\stopsubsection + +\startsubsection[title={\prm {gleaders} and \prm {uleaders}},reference=sec:gleaders] + +\topicindex {leaders} + +This type of leaders is anchored to the origin of the box to be shipped out. So +they are like normal \prm {leaders} in that they align nicely, except that the +alignment is based on the {\it largest\/} enclosing box instead of the {\it +smallest\/}. The \type {g} stresses this global nature. The \prm {uleaders} are +used for flexible boxes and are discussed elsewhere. + +\stopsubsection + +\stopsection + \startsection[title={Paragraphs}] -{\em This section will describe \prm {autoparagraphmode}, \prm -{shapingpenaltiesmode}, \prm {shapingpenaltymode}, \prm {everybeforepar}, \prm -{snapshotpar}, \prm {wrapuppar}, \prm {orphanpenalties}, \prm {orphanpenalty}, -enz. For the moment the manuals that come with \CONTEXT\ have to do.} +\startsubsection[title=Freezing] + +In \LUAMETATEX\ we store quite some properties with a paragraph. Where in traditional +\TEX\ the properties that are set when the paragraph broken into lines are used, here +we can freeze them. + +{\em At some point this section will describe \prm {autoparagraphmode}, \prm +{everybeforepar}, \prm {snapshotpar}, \prm {wrapuppar}, etc. For the moment the +manuals that come with \CONTEXT\ have to do.} + +\stopsubsection + +\startsubsection[title=Penalties] + +In addition to the penalties introduced in \ETEX, we also provide \prm +{orphanpenalty} and \prm {orphanpenalties}. When we're shaping a paragraph +an additional \prm {shapingpenalty} can be injected. This penalty gets +injected instead of the usual penalties when the following bits are set in +\prm {shapingpenaltiesmode}: + +\starttabulate[|l|l|p|] +\DB value \BC ignored \NC \NR +\TB +\NC \type {0x01} \NC interlinepenalty \NC \NR +\NC \type {0x02} \NC widowpenalty \NC \NR +\NC \type {0x04} \NC clubpenalty \NC \NR +\NC \type {0x08} \NC brokenpenalty \NC \NR +\LL +\stoptabulate + +When none of these is set the shaping penalty will be added. That way one can +prevent a page break inside a shape. + +\stopsubsection + +\startsubsection[title=Criteria] + +The linebreak algorithm uses some heuristics for determining the badness of a +line. In most cases that works quite well. Of course one can run into a bad +result when one has a large document of weird (extreme) constraints and it can be +tempting to mess around with parameters which then of course can lead to bad +results in other places. A solution is is to locally tweak penalties or looseness +but one can also just accept the occasional less optimal result (after all there +are plenty occasions to make a document look bad otherwise so best focus on the +average first). That said, it is tempting to see if changing the hard codes +criteria makes a difference. Experiments with this demonstrated the usual: when +asked what looks best contradictions mix with expectations and being triggered by +events that one related to \TEX, like successive hyphenated lines. + +The \prm {linebreakcriterium} parameter can be set to a value made from four bytes. We're +not going to explain the magic numbers because they come from and are discussed in original +\TEX. It is enough to know that we have four criteria: + +\starttabulate[|l|l|p|] +\DB magic \BC bound to \NC bytes \NC \NR +\TB +\NC 12 \NC semi tight \NC 0x7F...... \NC \NR +\NC 12 \NC decent \NC 0x..7F.... \NC \NR +\NC 12 \NC semi loose \NC 0x....7F.. \NC \NR +\NC 99 \NC loose \NC 0x......7F \NC \NR +\LL +\stoptabulate + +These four values can be changed according to the above pattern and are limited +to the range 1\endash127 which is plenty especially when one keeps in mind that +the actual useful values sit around the 12 anyway. Values outside the range (and +therefore an all|-|over zero assignment) makes the defaults kick in. + +The original decisions are made in the following way: + +\starttyping +function loose(badness) + if badness > loose_criterium then + return very_loose_fit + elseif badness > decent_criterium then + return loose_fit + else { + return decent_fit + end +end + +function tight(badness) + if badness > decent_criterium then + return tight_fit + else { + return decent_fit + end +end +\stoptyping + +while in \LUAMETATEX\ we use (again in \LUA speak): + +\starttyping +function loose(badness) + if badness > loose then + return very_loose_fit + elseif badness > semi_loose then + return semi_loose_fit + elseif badness > decent then + return loose_fit + else + return decent_fit + end +end + +function tight(badness) + if badness > semi_tight then + return semi_tight_fit + else if badness > decent then + return tight_fit + else + return decent_fit + end +end +\stoptyping + +So we have a few more steps to play with. But don't be disappointed when it +doesn't work out as you expect. Don Knuth did a good job on the heuristics and +after many decades there is no real need to change something. Consider it a +playground. + +\stopsubsection \stopsection @@ -59,9 +896,15 @@ very early in the macro package loading process. \NC \prm {insertpenalty} \NC \prm {insertpenalties} \NC the floating penalty (used when set) \NC \NR \NC \prm {insertmaxdepth} \NC \prm {maxdepth} \NC the maximum split depth (used when set) \NC \NR \NC \prm {insertstorage} \NC \NC signals that the insert has to be stored for later \NC \NR -\NC \prm {insertheight} \NC \prm {ht} box \NC the accumulated height of the inserts so far \NC \NR -\NC \prm {insertdepth} \NC \prm {dp} box \NC the current depth of the inserts so far \NC \NR -\NC \prm {insertwidth} \NC \prm {wd} box \NC the width of the inserts \NC \NR +\NC \prm {insertheight} \NC \prm {ht} box / index \NC the accumulated height of the inserts so far \NC \NR +\NC \prm {insertdepth} \NC \prm {dp} box / index \NC the current depth of the inserts so far \NC \NR +\NC \prm {insertwidth} \NC \prm {wd} box / index \NC the width of the inserts \NC \NR +\NC \prm {insertbox} \NC box / index \NC the boxed content \NC \NR +\NC \prm {insertcopy} \NC box / index \NC a copy of the boxed content \NC \NR +\NC \prm {insertunbox} \NC box / index \NC the unboxed content \NC \NR +\NC \prm {insertuncopy} \NC box / index \NC a copy of the unboxed content \NC \NR +\NC \prm {insertuncopy} \NC box / index \NC a copy of the unboxed content \NC \NR +\NC \prm {insertprogress} \NC box / index \NC the currently accumulated height \NC \NR \LL \stoptabulate @@ -88,6 +931,8 @@ The \LUAMETATEX\ engine has some tracing built in that is enabled by setting \pr \startsection[title={Marks}] +\topicindex {marks} + Marks are kind of signal nodes in the list that refer to stored token lists. When a page has been split off and is handed over to the output routine these signals are resolved into first, top and bottom mark references that can (for instance) @@ -98,6 +943,13 @@ In \ETEX\ the standard \TEX\ primitives \prm {mark}, \prm {firstmark}, \prm been extended with plural forms that accent a number before the token list. That number indicates a mark class. +In addition to the mark fetch commands, we also have access to the last set +mark in the given class with \prm {currentmarks}: + +\startsyntax +\currentmarks <16-bit number> +\stopsyntax + A problem with marks is that one cannot really reset them. Mark states are kept in the node lists and only periodically the state is snapshot into the global state variables. The \LUATEX\ engine can reset these global states with \prm @@ -106,12 +958,28 @@ state variables. The \LUATEX\ engine can reset these global states with \prm This permits implementing controlled resets of specific marks at the cost of a possible interfering mode, but that can normally be dealt with rather well. +The \prm {clearmarks} primitive complements the \ETEX\ mark primitives and clears +a mark class completely, resetting all three connected mark texts to empty. It is +an immediate command (no synchronization node is used). + +\startsyntax +\clearmarks <16-bit number> +\stopsyntax + +The \prm {flushmarks} variant is delayed but puts a (mark) node in the list as +signal (we could have gone for a keyword to \prm {marks} instead). + +\startsyntax +\flushmarks <16-bit number> +\stopsyntax + Another problem with marks is that when they are buried too deep, a property they share with inserts, they become invisible. This can be dealt with by the migration feature described in the next section. The \LUAMETATEX\ engine has some tracing built in that is enabled by setting \prm -{tracingmarks} to a positive value. +{tracingmarks} to a positive value. When set to~1 the page builder shows the set +values, and when set to a higher value details about collecting them are shown. \stopsection @@ -144,6 +1012,16 @@ which is why the value has to be at least 2 in order to be compatible with other \startsection[title={Migration}] +There are a few injected node types that are used to track information: marks, +inserts and adjusts (see previous sections). Marks are token lists that can be +used to register states like section numbers and titles they are synchronized in +the page builder when a page is shipped out. Inserts are node lists that get +rendered and relate to specific locations and these are flushed with the main +vertical list which also means that in calculating page breaks they need to be +taken into account. An Adjust is material that gets injected before or after a +line. Strictly spoke local boxes also in this repertoire but they are dealt with +in the par builder. + A new primitive \prm {automigrationmode} can be used to let deeply burried marks and inserts bubble up to the outer level. @@ -162,6 +1040,22 @@ If you want to migrate marks and inserts you need to set all these flags. Migrat marks and inserts end up as post|-|box properties and will be handled in the page builder as such. At the \LUA\ end you can add pre- and post|-|box material too. +The primitive register \prm {holdingmigrations} is a bitset that can be used to temporarily +disable migrations. It is a generalization of \prm {holdinginserts}. + +\starttabulate[|cT|p|] +\DB value \BC explanation \NC \NR +\TB +\NC 0x01 \NC marks \NC \NR +\NC 0x02 \NC inserts \NC \NR +\NC 0x04 \NC adjusts \NC \NR +\LL +\stoptabulate + +Migrates material is bound to boxes so boxed material gets unboxed it is taken +into account, but you should be aware of potential side effects. But then, marks, +inserts and adjusts always demanded care. + \stopsection \startsection[title={Pages}] @@ -170,6 +1064,57 @@ The page builder can be triggered by (for instance) a penalty but you can also use \prm {pageboundary}. This will trigger the page builder but not leave anything behind. +{\em In due time we will discuss \prm {pagevsize}, \prm {pageextragoal} and \prm +{lastpageextra} but for now we treat them as very experimental and they will be +tested in \CONTEXT, also in discussion with users.} + +\stopsection + +\startsection[title={Paragraphs}] + +The numeric primitive \prm {lastparcontext} inspector reports the current context +in which a paragraph triggering commands happened. The numbers can be queried +with \type {tex.getparcontextvalues()} and currently are: \showvaluelist +{tex.getparcontextvalues()}. As with the other \type {\last...} primitives this +variable is global. + +Traditional \TEX\ has the \prm {parfillskip} parameter that determines the way +the last line is filled. In \LUAMETATEX\ we also have \prm {parfillleftskip}. The +counterparts for the first line are \prm {parinitleftskip} and \prm +{parinitrightskip}. + +\startbuffer +\leftskip 2em +\rightskip \leftskip +\parfillskip \zeropoint plus 1 fill +\parfillleftskip \parfillskip +\parinitleftskip \parfillleftskip +\parinitrightskip\parfillleftskip +\input ward +\stopbuffer + +\typebuffer This results in: \par \start \em \getbuffer \par \stop + +An additional tracing primitive \prm {tracingfullboxes} reports details about the +encountered overfull boxes. This can be rather verbose! + +Normally \TEX\ will insert an empty hbox when paragraph indentation is requested +but when the second bit in \prm {normalizelinemode} has been set \LUAMETATEX\ +will in a glue node instead. You can zero the set value with \prm {undent} unless +of course some more has been inserted already. + +\startbuffer +\parinitleftskip1cm \parindent 1cm \indent test \par +\parinitleftskip1cm \parindent 1cm \undent test \par +\parinitleftskip1cm \parindent 1cm \indent \undent test \par +\parinitleftskip1cm \parindent 1cm \indent \strut \undent test \par +\stopbuffer + +\typebuffer \startpacked \getbuffer \stoppacked + +By setting \prm {tracingpenalties} to a positive value penalties related to +windows, clubs, lines etc.\ get reported to the output channels. + \stopsection \startsection[title={Local boxes}] @@ -207,6 +1152,140 @@ The commands: \prm {localleftboxbox}, \prm {localrightboxbox} and \prm \stopsection +\startsection[title={Leaders}] + +Leaders are flexible content that are basically just seen as glue and it is up to +the backend to apply the effective glue to the result as seen in the backend +(like a rule of box). This means that the frontend doesn't do anything with the +fact that we have a regular \prm {leaders}, a \prm {gleaders}, \prm {xleaders} or +\prm {cleaders}. The \prm {uleaders} that has been added in \LUAMETATEX\ is just +that: an extra leader category. The main difference is that the width of the +given box is added to the glue. That way we create a stretchable box. + +\startbuffer +\unexpandedloop 1 30 1 {x \hbox{1 2 3} x } +\unexpandedloop 1 30 1 {x {\uleaders \hbox{1 2 3}\hskip 0pt plus 10pt minus 10pt\relax} x } +\unexpandedloop 1 30 1 {x {\uleaders \hbox{1 2 3}\hskip 0pt plus \interwordstretch minus \interwordshrink} x } +\unexpandedloop 1 30 1 {x {\uleaders \hbox{1 2 3}\hskip 0pt plus 2\interwordstretch minus 2\interwordshrink} x } +\stopbuffer + +\typebuffer + +Here are some examples: + +\startlines +\getbuffer +\stoplines + +So the flexibility fo the box plays a role in the line break calculations. But in +the end the backend has to do the work. + +\startbuffer[a] +{\green \hrule width \hsize} \par \vskip2pt +\vbox to 40pt { + {\red\hrule width \hsize} \par \vskip2pt + \vbox { + \vskip2pt {\blue\hrule width \hsize} \par + \vskip 10pt plus 10pt minus 10pt + {\blue\hrule width \hsize} \par \vskip2pt + } + \vskip2pt {\red\hrule width \hsize} \par +} +\vskip2pt {\green \hrule width \hsize} \par +\stopbuffer + +\startbuffer[b] +{\green \hrule width \hsize} \par \vskip2pt +\vbox to 40pt { + {\red\hrule width \hsize} \par \vskip2pt + \uleaders\vbox { + \vskip2pt {\blue\hrule width \hsize} \par + \vskip 10pt plus 10pt minus 10pt + {\blue\hrule width \hsize} \par \vskip2pt + }\vskip 0pt plus 10pt minus 10pt + \vskip2pt {\red\hrule width \hsize} \par +} +\vskip2pt {\green \hrule width \hsize} \par +\stopbuffer + +\typebuffer[a] + +with + +\typebuffer[b] + +In the first case we get the this: + +\startlinecorrection +\getbuffer[a] +\stoplinecorrection + +but with \prm {uleaders} we get: + +\startlinecorrection +\normalizeparmode\zerocount +\getbuffer[b] +\stoplinecorrection + +or this: + +\startlinecorrection +\normalizeparmode"FF +\getbuffer[b] +\stoplinecorrection + +In the second case we flatten the leaders in the engine by setting the second bit +in the \prm {normalizeparmode} parameter (\type {0x2}). We actually do the same +with \prm {normalizelinemode} where bit 10 is set (\type {0x200}). The \type +{delay} keyword can be passed with a box to prevent flattening. If we don't do +this in the engine, the backend has to take care of it. In principle this permits +implementing variants in a macro package. Eventually there will be plenty examples in +the \CONTEXT\ code base and documentation. Till then, consider this experimental. + +\stopsection + +\startsection[title=Alignments] + +The primitive \prm {alignmark} duplicates the functionality of \type {#} inside +alignment preambles, while \prm {aligntab} duplicates the functionality of \type +{&}. The \prm {aligncontent} primitive directly refers to an entry so that one +does not get repeated. + +Alignments can be traced with \prm {tracingalignments}. When set to~1 basics +usage is shown, for instance of \prm {noalign} but more interesting is~2 or more: +you then get the preambles reported. + +The \prm {halign} (tested) and \prm {valign} (yet untested) primitives accept a +few keywords in addition to \type {to} and \type {spread}: + +\starttabulate[|l|p|] +\DB keyword \BC explanation \NC \NR +\TB +\NC \type {attr} \NC set the given attribute to the given value \NC \NR +\NC \type {callback} \NC trigger the \type {alignment_filter} callback \NC \NR +\NC \type {discard} \NC discard zero \prm {tabskip}'s \NC \NR +\NC \type {noskips} \NC don't even process zero \prm {tabskip}'s \NC \NR +\NC \type {reverse} \NC reverse the final rows \NC \NR +\LL +\stoptabulate + +In the preamble the \prm {tabsize} primitive can be used to set the width of a +column. By doing so one can avoid using a box in the preamble which, combined +with the sparse tabskip features, is a bit easier on memory when you produce +tables that span hundreds of pages and have a dozen columns. + +The \prm {everytab} complements the \prm {everycr} token register but is sort of +experimental as it might become more selective and powerful some day. + +The two primitives \prm {alignmentcellsource} and \prm {alignmentwrapsource} that +associate a source id (integer) to the current cell and row (line). Sources and +targets are experimental and are being explored in \CONTEXT\ so we'll see where +that ends up in. + +{\em todo: callbacks} + +\stopsection + \stopchapter \stopcomponent diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex b/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex deleted file mode 100644 index 403f1a029..000000000 --- a/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex +++ /dev/null @@ -1,411 +0,0 @@ -% language=us runpath=texruns:manuals/luametatex - -\environment luametatex-style - -\startcomponent luametatex-differences - -\startchapter[reference=differences,title={Differences with \LUATEX}] - -As \LUAMETATEX\ is a leaner and meaner \LUATEX. This means that substantial parts and -dependencies are gone: quite some font code, all backend code with related frontend -code and of course image and font inclusion. There is also new functionality which -makes for less lean but in the end we still have less, also in terms of dependencies. -This chapter will discuss what is gone. We start with the primitives that were dropped. - -\starttabulate[|l|pl|] -\BC fonts \NC \type {\letterspacefont} - \type {\copyfont} - \type {\expandglyphsinfont} - \type {\ignoreligaturesinfont} - \type {\tagcode} - \type {\leftghost} - \type {\rightghost} - \NC \NR -\BC backend \NC \type {\dviextension} - \type {\dvivariable } - \type {\dvifeedback} - \type {\pdfextension} - \type {\pdfvariable } - \type {\pdffeedback} - \type {\dviextension} - \type {\draftmode} - \type {\outputmode} - \NC \NR -\BC dimensions \NC \type {\pageleftoffset} - \type {\pagerightoffset} - \type {\pagetopoffset} - \type {\pagebottomoffset} - \type {\pageheight} - \type {\pagewidth} - \NC \NR -\BC resources \NC \type {\saveboxresource} - \type {\useboxresource} - \type {\lastsavedboxresourceindex} - \type {\saveimageresource} - \type {\useimageresource} - \type {\lastsavedimageresourceindex} - \type {\lastsavedimageresourcepages} - \NC \NR -\BC positioning \NC \type {\savepos} - \type {\lastxpos} - \type {\lastypos} - \NC \NR -\BC directions \NC \type {\textdir} - \type {\linedir} - \type {\mathdir} - \type {\pardir} - \type {\pagedir} - \type {\bodydir} - \type {\pagedirection} - \type {\bodydirection} - \NC \NR -\BC randomizer \NC \type {\randomseed} - \type {\setrandomseed} - \type {\normaldeviate} - \type {\uniformdeviate} - \NC \NR -\BC utilities \NC \type {\synctex} - \NC \NR -\BC extensions \NC \type {\latelua} - \type {\lateluafunction} - \type {\openout} - \type {\write} - \type {\closeout} - \type {\openin} - \type {\read} - \type {\readline} - \type {\closein} - \type {\ifeof} - \NC \NR -\BC control \NC \type {\suppressfontnotfounderror} - \type {\suppresslongerror} - \type {\suppressprimitiveerror} - \type {\suppressmathparerror} - \type {\suppressifcsnameerror} - \type {\suppressoutererror} - \type {\mathoption} - \NC \NR -\BC system \NC \type {\primitive} - \type {\ifprimitive} - \type {\formatname} - \NC \NR -\BC ignored \NC \type {\long} - \type {\outer} - \type {\mag} - \NC \NR -\stoptabulate - -The math machinery has been overhauled stepwise. In the process detailed control -has been added but later some of that got removed or replaced. The engine now -assumes that \OPENTYPE\ fonts are used but you do need to set up the engine -properly, something that has to be done with respect to fonts anyway. By enabling -and|/|disabling certain features you can emulate the traditional engine. Font -parameters no longer are taken from the traditional parameters when they are not -set. We just assume properly passed so called math constants and quite a few new -ones have been added. - -The resources and positioning primitives are actually useful but can be defined -as macros that (via \LUA) inject nodes in the input that suit the macro package -and backend. The three||letter direction primitives are gone and the numeric -variants are now leading. There is no need for page and body related directions -and they don't work well in \LUATEX\ anyway. We only have two directions left. -Because we can hook in \LUA\ functions that get information about what is expected -(consumer or provider) there are plenty possibilities for adding functionality -using this scripting language. - -The primitive related extensions were not that useful and reliable so they have -been removed. There are some new variants that will be discussed later. The \type -{\outer} and \type {\long} prefixes are gone as they don't make much sense -nowadays and them becoming dummies opened the way to something new: control -sequence properties that permit protection against as well as controlled -overloading of definitions. I don't think that (\CONTEXT) users will notice these -prefixes being gone. The definition and parsing related \type {\suppress..} -features are now default and can't be changed so related primitives are gone. - -The \type {\shipout} primitive does no ship out but just erases the content of -the box unless of course that has happened already in another way. A macro -package should implement its own backend and related shipout. Talking of backend, -the extension primitives that relate to backends can be implemented as part of a -backend design using generic whatsits. There is only one type of whatsit now. In -fact we're now closer to original \TEX\ with respect to the extensions. - -The \type {img} library has been removed as it's rather bound to the backend. The -\type {slunicode} library is also gone. There are some helpers in the string -library that can be used instead and one can write additional \LUA\ code if -needed. There is no longer a \type {pdf} backend library but we have an up to -date \PDF\ parsing library on board. - -In the \type {node}, \type {tex} and \type {status} library we no longer have -helpers and variables that relate to the backend. The \LUAMETATEX\ engine is in -principle \DVI\ and \PDF\ unaware. There are, as mentioned, only generic whatsit -nodes that can be used for some management related tasks. For instance you can -use them to implement user nodes. More extensive status information is provided -in the overhauled status library. All libraries have additional functionality and -names of functions have been normalized (for as far as possible). - -The margin kern nodes are gone and we now use regular kern nodes for them. As a -consequence there are two extra subtypes indicating the injected left or right -kern. The glyph field served no real purpose so there was no reason for a special -kind of node. - -The \KPSE\ library is no longer built|-|in, but one can use an external \KPSE\ -library, assuming that it is present on the system, because the engine has a so -called optional library interface to it. Because there is no backend, quite some -file related callbacks could go away. The following file related callbacks -remained (till now): - -\starttyping -find_write_file find_format_file open_data_file -\stoptyping - -The callbacks related to errors are changed: - -\starttyping -intercept_tex_error intercept_lua_error -show_error_message show_warning_message -\stoptyping - -There is a hook that gets called when one of the fundamental memory structures -gets reallocated. - -\starttyping -trace_memory -\stoptyping - -When you use the overload protect mechanisms, a callback can be plugged in to handle -exceptions: - -\starttyping -handle_overload -\stoptyping - -The (job) management hooks are kept: - -\starttyping -process_jobname -start_run stop_run wrapup_run -pre_dump -start_file stop_file -\stoptyping - -Because we use a more generic whatsit model, there is a new callback: - -\starttyping -show_whatsit -\stoptyping - -Because tracing boxes now reports a lot more information, we have a plug in for -detail: - -\starttyping -get_attribute -\stoptyping - -Being the core of extensibility, the typesetting callbacks of course stayed. This -is what we ended up with: - -\startalign[flushleft,nothyphenated] -\tt \cldcontext{table.concat(table.sortedkeys(callbacks.list), ", ")} -\stopalign - -As in \LUATEX\ font loading happens with the following callback. This time it -really needs to be set because there is no built|-|in font loader. - -\starttyping -define_font -\stoptyping - -There are all kinds of subtle differences in the implementation, for instance we -no longer intercept \type {*} and \type {&} as these were already replaced long -ago in \TEX\ engines by command line options. Talking of options, only a few are -left. All input goes via \LUA, even the console. One can program a terminal if -needed. - -We took our time for reaching a stable state in \LUATEX. Among the reasons is the -fact that most was experimented with in \CONTEXT, which we can adapt to the -engine as we go. It took many years to decide what to keep and how to do things. -Of course there are places when things can be improved but that most likely only -happens in \LUAMETATEX. Contrary to what is sometimes suggested, the -\LUATEX|-|\CONTEXT\ \MKIV\ combination (assuming matched versions) has been quite -stable. It made no sense otherwise. Most \CONTEXT\ functionality didn't change -much at the user level. Of course there have been issues, as is natural with -everything new and beta, but we have a fast update cycle. - -The same is true for \LUAMETATEX\ and \CONTEXT\ \LMTX: it can be used for -production as usual and in practice \CONTEXT\ users tend to use the beta -releases, which proves this. Of course, if you use low level features that are -experimental you're on your own. Also, as with \LUATEX\ it might take many years -before a long term stable is defined. The good news is that, when the source code -has become part of the \CONTEXT\ distribution, there is always a properly -working, more or less long term stable, snapshot. - -The error reporting subsystem has been redone quite a bit but is still -fundamentally the same. We don't really assume interactive usage but if someone -uses it, it might be noticed that it is not possible to backtrack or inject -something. Of course it is no big deal to implement all that in \LUA\ if needed. -It removes a system dependency and makes for a bit cleaner code. In \CONTEXT\ we -quit on an error simply because one has to fix source anyway and runs are fast -enough. Logging provides more detail and new primitives can be used to prevent -clutter in tracing (the more complex a macro package becomes, the more extreme -tracing becomes). - -There are new primitives as well as some extensions to existing primitive -functionality. These are described in following chapters but there might be -hidden treasures in the binary. If you locate them, don't automatically assume -them to stay, some might be part of experiments! There are for instance a few -csname related definers, we have integer and dimension constants, the macro -argument parser can be brought in tolerant mode, the repertoire of conditionals -has been extended, some internals can be controlled (think of normalization of -lines, hyphenation etc.), and macros can be protected against user overload. Not -all is discussed in detail in this manual but there are introductions in the -\CONTEXT\ distribution that explain them. But the \TEX\ kernel is of course -omnipresent. - -\startluacode - -local luametatex = tex.primitives() -local luatex = table.load("luatex-primitives.lua") - -if not luatex then - local tex = "\\starttext \\ctxlua {table.save(tex.jobname .. '.lua',tex.primitives())} \\stoptext" - - io.savedata("luatex-primitives.tex", tex) - - os.execute("context --luatex --once luatex-primitives") - - luatex = table.load("luatex-primitives.lua") -end - - -if luatex and luametatex then - - local match = string.match - - local found = { } - - local function collect(index) - if index then - local data = index.entries - for i=1,#data do - found[match(data[i].list[1][1],"\\tex%s*{(.-)}") or ""] = true - end - -- inspect(found) - end - end - - collect(structures.registers.collected and structures.registers.collected.texindex) - collect(structures.registers.collected and structures.registers.collected.luatexindex) - - luatex = table.tohash(luatex) - luametatex = table.tohash(luametatex) - - -- context.page() - - context("The following primitives are available in \\LUATEX\\ but not in \\LUAMETATEX. ") - context("Some of these are emulated in \\CONTEXT.") - - context.blank() - context.startcolumns { n = 2 } - for k, v in table.sortedhash(luatex) do - if not luametatex[k] then - if not found[k] then - context.dontleavehmode() - end - context.type(k) - context.crlf() - end - end - context.stopcolumns() - context.blank() - - -- context.page() - - context("The following primitives are available in \\LUAMETATEX\\ only. In the meantime ") - context("the \\LUAMETATEX\\ code base is so different from \\LUATEX\\ that backporting ") - context("is no longer reasonable.") - - context.blank() - context.startcolumns { n = 2 } - for k, v in table.sortedhash(luametatex) do - if not luatex[k] then - if not found[k] then - context.dontleavehmode() - context.llap("\\infofont[todo] ") - end - context.type(k) - context.crlf() - end - end - context.stopcolumns() - context.blank() - -end - -\stopluacode - -As part of a bit more consistency some function names also changed. Names with an -\type {_} got that removed (as that was the minority). It's easy to provide a -back mapping if needed (just alias the functions). - -{\em Todo: only mention the \LUATEX\ ones.} - -\starttabulate[|l|l|l|l|] -\DB library \BC old name \BC new name \BC comment \NC \NR -\TB -\NC language \NC clear_patterns \NC clearpatterns \NC \NR -\NC \NC clear_hyphenation \NC clearhyphenation \NC \NR -\NC mplib \NC italcor \NC italic \NC \NR -\NC \NC pen_info \NC peninfo \NC \NR -\NC \NC solve_path \NC solvepath \NC \NR -\NC texio \NC write_nl \NC writenl \NC old name stays \NC \NR -\NC node \NC protect_glyph \NC protectglyph \NC \NR -\NC \NC protect_glyphs \NC protectglyphs \NC \NR -\NC \NC unprotect_glyph \NC unprotectglyph \NC \NR -\NC \NC unprotect_glyphs \NC unprotectglyphs \NC \NR -\NC \NC end_of_math \NC endofmath \NC \NR -\NC \NC mlist_to_hlist \NC mlisttohlist \NC \NR -\NC \NC effective_glue \NC effectiveglue \NC \NR -\NC \NC has_glyph \NC hasglyph \NC \NR -\NC \NC first_glyph \NC firstglyph \NC \NR -\NC \NC has_field \NC hasfield \NC \NR -\NC \NC copy_list \NC copylist \NC \NR -\NC \NC flush_node \NC flushnode \NC \NR -\NC \NC flush_list \NC flushlist \NC \NR -\NC \NC insert_before \NC insertbefore \NC \NR -\NC \NC insert_after \NC insertafter \NC \NR -\NC \NC last_node \NC lastnode \NC \NR -\NC \NC is_zero_glue \NC iszeroglue \NC \NR -\NC \NC make_extensible \NC makeextensible \NC \NR -\NC \NC uses_font \NC usesfont \NC \NR -\NC \NC is_char \NC ischar \NC \NR -\NC \NC is_direct \NC isdirect \NC \NR -\NC \NC is_glyph \NC isglyph \NC \NR -\NC \NC is_node \NC isnode \NC \NR -\NC token \NC scan_keyword \NC scankeyword \NC \NR -\NC \NC scan_keywordcs \NC scankeywordcs \NC \NR -\NC \NC scan_int \NC scanint \NC \NR -\NC \NC scan_real \NC scanreal \NC \NR -\NC \NC scan_float \NC scanfloat \NC \NR -\NC \NC scan_dimen \NC scandimen \NC \NR -\NC \NC scan_glue \NC scanglue \NC \NR -\NC \NC scan_toks \NC scantoks \NC \NR -\NC \NC scan_code \NC scancode \NC \NR -\NC \NC scan_string \NC scanstring \NC \NR -\NC \NC scan_argument \NC scanargument \NC \NR -\NC \NC scan_word \NC scanword \NC \NR -\NC \NC scan_csname \NC scancsname \NC \NR -\NC \NC scan_list \NC scanlist \NC \NR -\NC \NC scan_box \NC scanbox \NC \NR -\LL -\stoptabulate - -It's all part of trying to make the code base consistent but it is sometimes a -bit annoying. However, that's why we develop this engine independent of the -\LUATEX\ code base. It's anyway a change that has been on my todo list for quite -a while because those inconsistencies annoyed me. It might take some years to -get all done. - -\stopchapter - -\stopcomponent - diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex b/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex index ad80e18b9..ca3759926 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex @@ -1,10 +1,12 @@ % language=us runpath=texruns:manuals/luametatex +% todo: move some to elsewhere (e.g. builders / paragraphs + \environment luametatex-style \startcomponent luametatex-enhancements -\startchapter[reference=enhancements,title={Basic \TEX\ enhancements}] +\startchapter[reference=enhancements,title={Enhancements}] \startsection[title={Introduction}] @@ -545,6 +547,9 @@ You can set attributes of the current paragraph specification node with \prm \startsubsection[title={\prm {directlua}}] +\topicindex{scripting} +\topicindex{lua+direct} + In order to merge \LUA\ code with \TEX\ input, a few new primitives are needed. The primitive \prm {directlua} is used to execute \LUA\ code immediately. The syntax is @@ -659,6 +664,9 @@ in a separate file and load it using \LUA's \type {dofile}: \startsubsection[title={\prm {luafunction}, \prm {luafunctioncall} and \prm {luadef}}] +\topicindex{functions} +\topicindex{lua+functions} + The \prm {directlua} commands involves tokenization of its argument (after picking up an optional name or number specification). The tokenlist is then converted into a string and given to \LUA\ to turn into a function that is @@ -710,6 +718,9 @@ normal usage should not give problems. \startsubsection[title={\prm {luabytecode} and \prm {luabytecodecall}}] +\topicindex{lua+bytecode} +\topicindex{bytecode} + Analogue to the function callers discussed in the previous section we have byte code callers. Again the call variant is unexpandable. @@ -812,9 +823,22 @@ the currently active table, an error is raised. \stopsubsection +\startsubsection[title={\prm {letcharcode}}] + +This primitive can be used to assign a meaning to an active character, as in: + +\starttyping +\def\foo{bar} \letcharcode123=\foo +\stoptyping + +This can be a bit nicer than using the uppercase tricks (using the property of +\prm {uppercase} that it treats active characters special). + +\stopsubsection + \stopsection -\startsection[title={Tokens, commands and strings}] +\startsection[title={Tokens and expansion}] \startsubsection[title={\prm {scantextokens}, \prm {tokenized} and \prm {retokenized}}] @@ -884,222 +908,7 @@ their content first. The \type {x} variant does a global assignment. \stopsubsection -\startsubsection[title={\prm {csstring}, \prm {begincsname} and \prm {lastnamedcs}}] - -These are somewhat special. The \prm {csstring} primitive is like -\prm {string} but it omits the leading escape character. This can be -somewhat more efficient than stripping it afterwards. - -The \prm {begincsname} primitive is like \prm {csname} but doesn't create -a relaxed equivalent when there is no such name. It is equivalent to - -\starttyping -\ifcsname foo\endcsname - \csname foo\endcsname -\fi -\stoptyping - -The advantage is that it saves a lookup (don't expect much speedup) but more -important is that it avoids using the \prm {if} test. The \prm {lastnamedcs} -is one that should be used with care. The above example could be written as: - -\starttyping -\ifcsname foo\endcsname - \lastnamedcs -\fi -\stoptyping - -This is slightly more efficient than constructing the string twice (deep down in -\LUATEX\ this also involves some \UTF8 juggling), but probably more relevant is -that it saves a few tokens and can make code a bit more readable. - -\stopsubsection - -\startsubsection[title={\prm {clearmarks}, \prm {flushmarks}, \prm {currentmarks}}] - -\topicindex {marks} - -The \prm {clearmarks} primitive complements the \ETEX\ mark primitives and clears -a mark class completely, resetting all three connected mark texts to empty. It is -an immediate command (no synchronization node is used). - -\startsyntax -\clearmarks <16-bit number> -\stopsyntax - -The \prm {flushmarks} variant is delayed but puts a (mark) node in the list as -signal (we could have gone for a keyword to \prm {marks} instead). - -\startsyntax -\flushmarks <16-bit number> -\stopsyntax - -In addition to the three mark fetch commands, we also have access to the last set -mark in the given class. - -\startsyntax -\currentmarks <16-bit number> -\stopsyntax - -Marks can be traced with \prm {tracingmarks}. When set to~1 the page builder -shows the set values, and when set to a higher value details about collecting -them are shown. - -\stopsubsection - -\startsubsection[title={\prm {alignmark}, \prm {aligntab}, \prm {aligncontent}, \prm {tabsize} and \prm {everytab}}] - -The primitive \prm {alignmark} duplicates the functionality of \type {#} inside -alignment preambles, while \prm {aligntab} duplicates the functionality of \type -{&}. The \prm {aligncontent} primitive directly refers to an entry so that one -does not get repeated. - -Alignments can be traced with \prm {tracingalignments}. When set to~1 basics -usage is shown, for instance of \prm {noalign} but more interesting is~2 or more: -you then get the preambles reported. - -The \prm {halign} (tested) and \prm {valign} (yet untested) primitives accept a -few keywords in addition to \type {to} and \type {spread}: - -\starttabulate[|l|p|] -\DB keyword \BC explanation \NC \NR -\TB -\NC \type {attr} \NC set the given attribute to the given value \NC \NR -\NC \type {callback} \NC trigger the \type {alignment_filter} callback \NC \NR -\NC \type {discard} \NC discard zero \prm {tabskip}'s \NC \NR -\NC \type {noskips} \NC don't even process zero \prm {tabskip}'s \NC \NR -\NC \type {reverse} \NC reverse the final rows \NC \NR -\LL -\stoptabulate - -In the preamble the \prm {tabsize} primitive can be used to set the width of a -column. By doing so one can avoid using a box in the preamble which, combined -with the sparse tabskip features, is a bit easier on memory when you produce -tables that span hundreds of pages and have a dozen columns. - -The \prm {everytab} complements the \prm {everycr} token register but is sort of -experimental as it might become more selective and powerful some day. - -\stopsubsection - -\startsubsection[title={\prm {letcharcode}}] - -This primitive can be used to assign a meaning to an active character, as in: - -\starttyping -\def\foo{bar} \letcharcode123=\foo -\stoptyping - -This can be a bit nicer than using the uppercase tricks (using the property of -\prm {uppercase} that it treats active characters special). - -\stopsubsection - -\startsubsection[title={\prm {lettonothing} and \prm {glettonothing}}] - -This primitive is equivalent to: - -\starttyping -\protected\def\lettonothing#1{\def#1{}} -\stoptyping - -and although it might feel faster (only measurable with millions of calls) it's -mostly there because it is easier on tracing (less clutter). An advantage over -letting to an empty predefined macro is also that in tracing we keep seeing the -name (relaxing would show the relax equivalent). - -\stopsubsection - -\startsubsection[title={\prm {glet}}] - -This primitive is similar to: - -\starttyping -\protected\def\glet{\global\let} -\stoptyping - -but faster (only measurable with millions of calls) and probably more convenient -(after all we also have \type {\gdef}). - -\stopsubsection - -\startsubsection[title={\prm {defcsname}, \prm {edefcsname}, \prm {gdefcsname} and \prm {xdefcsname}}] - -Although we can implement these primitives easily using macros it makes sense, -given the popularity of \prm {csname} to have these as primitives. It also saves -some \prm {expandafter} usage and it looks a bit better in the source. - -\starttyping -\gdefcsname foo\endcsname{oof} -\stoptyping - -\stopsubsection - -\startsubsection[title={\prm {letcsname} and \prm {gletcsname}}] - -These can also be implemented using macros but again they are natively provided -by the engine for the same reasons: less code and less tracing clutter. - -\starttyping -\gletcsname foo\endcsname \relax -\stoptyping - -\stopsubsection - -\startsubsection[title={\prm {futuredef} and \prm {futurecsname}}] - -This is just the definition variant of \prm {futurelet} and a simple example -shows the difference: - -\startbuffer -\def\whatever{[\next:\meaning\next]} -\futurelet\next\whatever A -\futuredef\next\whatever B -\stopbuffer - -\typebuffer - -\getbuffer - -The next one was more an experiment that then stayed around, just to see what -surprising abuse of this primitive will happen: - -\startbuffer -\def\whateveryes{[YES]} -\def\whatevernop{[NOP]} -\let\whatever\undefined -\futurecsname\whatevernop whatever\endcsname -\futurecsname\whatevernop whateveryes\endcsname -\stopbuffer - -\typebuffer - -When the assembles control sequence is undefined the given one will be expanded, -a weird one, right? I will probably apply it some day in cases where I want less -tracing and a more direct expansion of an assembled name. - -\getbuffer - -Here is a usage example: - -\starttyping -\xdef\Whatever{\futurecsname\whatevernop whatever\endcsname} -\xdef\Whatever{\futurecsname\whateveryes whateveryes\endcsname} -\xdef\Whatever{\ifcsname whatever\endcsname\lastnamedcs\else\whatevernop\fi} -\xdef\Whatever{\ifcsname whateveryes\endcsname\lastnamedcs\else\whatevernop\fi} -\xdef\Whatever{\ifcsname whatever\endcsname\csname whatever\endcsname\else\whatevernop\fi} -\xdef\Whatever{\ifcsname whateveryes\endcsname\csname whateveryes\endcsname\else\whatevernop\fi} -\stoptyping - -The timings for one million times defining each of these definitions are 0.277, -0.313, 0.310, 0.359, 0.352 and 0.573 seconds (on a 2018 Dell 7250 Precision -laptop with mobile E3-1505M v6 processor), so there is a little gain here, but of -course in practice no one will notice that because not that many such macros are -defined (or used). - -\stopsubsection - -\startsubsection[title={\prm {expanded}, \prm {localcontrol}, \prm +\startsubsection[title={\prm {expanded}, \prm {expandedafter}, \prm {localcontrol}, \prm {localcontrolled}, \prm {beginlocalcontrol} and \prm {endlocalcontrol}}] \topicindex {expansion} @@ -1154,6 +963,9 @@ These local control primitives are a bit tricky and error message can be confusing. Future versions might have a bit better recovery but in practice it works as expected. +An \prm {expandedafter} primitive is also provided as an variant on \prm +{expandafter} that takes a token list instead of a single token. + \stopsubsection \startsubsection[title={\prm {semiprotected}, \prm {semiexpanded}, \prm {expand} and @@ -1202,54 +1014,106 @@ protect mechanism we have in \MKII. \stopsubsection -\startsubsection[title={\prm {norelax}}] +\startsubsection[title={Going ahead with \prm {expandafterpars} and \prm {expandafterspaces}}] -There are a few cases where the \TEX\ scanned skips over spaces and \prm {relax} as -well as quits on a \prm {relax} in which case it gets pushed back. An example is -given below: +\topicindex{expansion+after} + +Here are again some convenience primitives that simplify coding, remove the need +to show off with multi|-|step macros and are nicely expandable. They fit in the +repertoire of additional primitives that make macro code look somewhat easier. +Here are a few examples: \startbuffer -\edef\TestA{\ifnum1=1\relax Y\else N\fi} \meaningasis\TestA -\edef\TestB{\ifnum1=1\norelax Y\else N\fi} \meaningasis\TestB +\def\foo{!!} [\expandafterpars \foo \par test] +\def\foo{!!} [\expandafterspaces\foo test] + +\def\foo{!!} \def\oof{\foo} [{\oof} test] +\def\foo{!!} \def\oof{\expandafterspaces\foo} [{\oof}test] \stopbuffer \typebuffer -The second line also contains a sentinel but this time we use \prm {norelax} -which will not be pushed back. So, this feature is just a trick to get rid of (in -itself reasonable) side effects. +These are typically used when building high level interfaces so not many users +will see them in document sources. -\startlines\getbuffer \stoplines +\startlines +\getbuffer +\stoplines \stopsubsection -\startsubsection[title={\prm {ignorepars}}] +\startsubsection[title={\prm {afterassigned}}] -This primitive is like \prm {ignorespaces} but also skips paragraph ending -commands (normally \prm {par} and empty lines). +\topicindex{assignments+after} + +This primitive is a multiple token variant of \prm {afterassignment} and it takes +a token list. It might look better in some cases than multiple single token +\quote {calls}. \stopsubsection -\startsubsection[title={\prm {futureexpand}, \prm {futureexpandis}, \prm {futureexpandisap}}] +\startsubsection[title={\prm {detokenized}}] -These commands are used as: +\topicindex{serializing} -\starttyping -\futureexpand\sometoken\whenfound\whennotfound -\stoptyping +The \prm {string} primitive serializes what comes next, a control sequence or +something more primitive string representation or just the (\UTF) character so it +does look at what it sees next in some detail. This can give confusing results +when the next token is for instance a new line. The \prm {detokenized} is less +picky and just serializes the token, so in the next examples an empty lines is +what we normally expect it to become: a serialized par token. -When there is no match and a space was gobbled a space will be put back. The -\type {is} variant doesn't do that while the \type {isap} even skips \type -{\pars}, These characters stand for \quote {ignorespaces} and \quote -{ignorespacesandpars}. +\def\oof{s\expandafter\foo\string} +\def\ofo{d\expandafter\foo\detokenized} +\def\foo#1{:[#1]} + +\startbuffer +\oof test +\ofo test +\oof \relax +\ofo \relax +\oof \par +\ofo \par + +\oof + +\ofo + +done +\stopbuffer + +\typebuffer + +We need the empty lines and \quote {done} to make sure we see the effect: + +{\tttf \getbuffer} \stopsubsection -\startsubsection[title={\prm {afterassigned}}] +\startsubsection[title={\prm {expandtoken} and \prm {expandcstoken}}] -This primitive is a multiple token variant of \prm {afterassignment} and it takes -a token list. It might look better in some cases than multiple single token -\quote {calls}. +\topicindex{expansion+tokens} + +These two are not really needed but can make code look less weird (and +impressive) because there are no catcode changes involved. The next example +illustrates what they do: + +\startbuffer +\edef\foo{\expandtoken 12 123 } \meaning\foo +\edef\oof{\bgroup \egroup} \meaning\oof +\edef\oof{\expandcstoken \bgroup\expandcstoken \egroup} \meaning\oof +\edef\oof{\expandcstoken \foo } \meaning\oof +\stopbuffer + +\typebuffer + +So \prm {expandtoken} expects two arguments: a catcode and a character number. +The \prm {expandcstoken} will only look at control sequences representing a +character. + +\startlines +\getbuffer +\stoplines \stopsubsection @@ -1259,6 +1123,8 @@ a token list. It might look better in some cases than multiple single token \startsubsection[title={\prm {endsimplegroup}}] +\topicindex{grouping+ending} + This feature might look somewhat weird so just ignore that it is there. It is one of these features that might never make it in a engine when discussed in committee but it comes in handy in \CONTEXT, so: @@ -1288,6 +1154,8 @@ when it has been changed in the group. \startsubsection[title={\prm {aftergrouped}}] +\topicindex{grouping+after} + There is a new experimental feature that can inject multiple tokens to after the group ends. An example demonstrate its use: @@ -1326,6 +1194,8 @@ This gives: \startsubsection[title={\prm {atendofgroup} and \prm {atendofgrouped}}] +\topicindex{grouping+ending} + These are variants of \prm {aftergroup} and \prm {aftergrouped} but they happen {\em before} the groups is closed. It is one of these primitives that is not really needed but that can make code (and tracing) cleaner, which is one of the @@ -1339,6 +1209,8 @@ objectives (at least for \CONTEXT). \startsubsection[title={\prm{ifabsnum} and \prm {ifabsdim}}] +\topicindex{conditions} + There are two tests that we took from \PDFTEX: \startbuffer @@ -1536,12 +1408,12 @@ nested balanced \quote {lists}, as in: \startsubsection[title={\prm {ifarguments}, \prm {ifparameters} and \prm {ifparameter}}] -These are part of the extended macro argument parsing features. The \type -{\ifarguments} condition is like an \type {\ifcase} where the number is the +These are part of the extended macro argument parsing features. The \prm +{ifarguments} condition is like an \prm {ifcase} where the number is the picked up number of arguments. The number reflects the {\em last} count, so -successive macro expansions will adapt the value. The \type {\ifparameters} -counts till the first empty parameter and the \type {\ifparameter} (singular) -takes a parameter reference (like \type {#2}) and again is an \type {\ifcase} +successive macro expansions will adapt the value. The \prm {ifparameters} +counts till the first empty parameter and the \prm {ifparameter} (singular) +takes a parameter reference (like \type {#2}) and again is an \prm {ifcase} where zero means a bad reference, one a non|-|empty argument and two an empty one. A typical usage is: @@ -1559,6 +1431,7 @@ No expansion of arguments takes place here but you can use a test like this: \iftok{#2}{}\else two\fi} \stoptyping + \stopsubsection \startsubsection[title={\prm {ifcondition}}] @@ -1706,359 +1579,6 @@ be queried with \typ {tex.getflagvalues}. \stopsection -\startsection[title={Boxes, rules and leaders}] - -\startsubsection[title={\prm {outputbox}}] - -\topicindex {output} - -This integer parameter allows you to alter the number of the box that will be -used to store the page sent to the output routine. Its default value is 255, and -the acceptable range is from 0 to 65535. - -\startsyntax -\outputbox = 12345 -\stopsyntax - -\stopsubsection - -\startsubsection[title={\prm {hrule}, \prm {vrule}, \prm {nohrule} and \prm {novrule}}] - -\topicindex {rules} - -Both rule drawing commands take an optional \type {xoffset} and \type {yoffset} -parameter. The displacement is virtual and not taken into account when the -dimensions are calculated. A rule is specified in the usual way: - -\obeydepth - -\startbuffer -\blue \vrule - height 2ex depth 1ex width 10cm -\relax -\stopbuffer - -\startlinecorrection -\getbuffer -\stoplinecorrection - -There is however a catch. The keyword scanners in \LUAMETATEX\ are implemented -slightly different. When \TEX\ scans a keyword it will (case insensitive) scan -for a whole keyword. So, it scans for \type {height} and when it doesn't find it -it will scan for \type {depth} etc. When it does find a keyword in this case it -expects a dimension next. When that criterium is not met it will issue an error -message. - -In order to avoid look ahead failures like that it is recommended to end the -specification with \type {\relax}. A glue specification is an other example where -a \type {\relax} makes sense when look ahead issues are expected and actually -there in traditional scanning the order of keywords can also matter. In any case, -when no valid keyword is seen the characters scanned so far are pushed back in -the input. - -The main reason for using an adapted scanner is that we always permit repetition -(consistency) and accept an arbitrary order. Because we have more keywords to -process the scanner quits at a partial failure. This prevents some push back and -also gives an earlier warning. Interesting is that some \CONTEXT\ users ran into -error messages due to a missing \type {\relax} and found out that their style has -a potential flaw with respect to look ahead. One can be lucky for years. - -Back to rules, there are some extra keywords, two deal with an offset, and four -provide margins. The margins are a bit special because \type {left} and \type -{top} are the same as are \type {right} and \type {bottom}. They influence the -edges and these depend on it being a horizontal or vertical rule. - -\obeydepth - -\startbuffer -\blue \vrule - height 2.0ex depth 1.0ex width 10cm -\relax -\white \vrule - height 1.0ex depth 0.5ex width 9cm - xoffset -9.5cm yoffset .25ex -\relax -\blue \vrule - height .5ex depth 0.25ex width 8cm - xoffset -18cm yoffset .375ex top 1pt -\relax -\stopbuffer - -\startlinecorrection -\getbuffer -\stoplinecorrection - -Two new primitives were introduced: \prm {nohrule} and \prm {novrule}. These can -be used to reserve space. This is often more efficient than creating an empty box -with fake dimensions. Of course this assumes that the backend implements them -being invisible but still taking space. - -\stopsubsection - -\startsubsection[title={\prm {vsplit}}] - -\topicindex {splitting} - -The \prm {vsplit} primitive has to be followed by a specification of the required -height. As alternative for the \type {to} keyword you can use \type {upto} to get -a split of the given size but result has the natural dimensions then. - -\starttyping -\vsplit 123 to 10cm % final box has the required height -\vsplit 123 upto 10cm % final box has its natural height -\stoptyping - -\stopsubsection - -\startsubsection[title={\prm {boxxoffset}, \prm {boxyoffset}, \prm {boxxmove}, \prm {boxymove}, -\prm{boxorientation} and \prm{boxgeometry}}] - -This repertoire of primitives can be used to do relative positioning. The offsets -are virtual while the moves adapt the dimensions. The orientation bitset can be -used to rotate the box over 90, 180 and 270 degrees. It also influences the -corner, midpoint or baseline. - -{\em There is information in the \CONTEXT\ low level manuals and in due time I -will add a few examples here. This feature needs support in the backend when used -(as in \CONTEXT) so it might influence performance.} - -\stopsubsection - -\startsubsection[title={\prm {boxtotal}}] - -The \prm {boxtotal} primitive returns the sum of the height and depth and is less -useful as setter: it just sets the height and depth to half of the given value. - -\stopsubsection - -\startsubsection[title={\prm {boxshift}}] - -In traditional \TEX\ a box has height, depth, width and a shift where the later -relates to \prm {raise}, \prm {lower}, \prm {moveleft} and \prm {moveright}. This -primitive can be used to query and set this property. - -\startbuffer -\setbox0\hbox{test test test} -\setbox2\hbox{test test test} \boxshift2 -10pt -\ruledhbox{x \raise10pt\box0\ x} -\ruledhbox{x \box2\ x} -\stopbuffer - -\typebuffer - -\stopsubsection - -\startsubsection[title={\prm {boxanchor}, \prm {boxanchors}, \prm {boxsource} and \prm {boxtarget}}] - -{\em These are experimental.} - -\stopsubsection - -\startsubsection[title={\prm {boxfreeze}, \prm {boxadapt} and \prm {boxrepack}}] - -This operation will freeze the glue in the given box, something that normally is -delayed and delegated to the backend. - -\startbuffer -\setbox 0 \hbox to 5cm {\hss test} -\setbox 2 \hbox to 5cm {\hss test} -\boxfreeze 2 0 -\ruledhbox{\unhbox 0} -\ruledhbox{\unhbox 2} -\stopbuffer - -\typebuffer - -The second parameter to \prm {boxfreeze} determines recursion. Here we just -freeze the outer level: - -\getbuffer - -Repacking will take the content of an existing box and add or subtract from it: - -\startbuffer -\setbox 0 \hbox {test test test} -\setbox 2 \hbox {\red test test test} \boxrepack0 +.2em -\setbox 4 \hbox {\green test test test} \boxrepack0 -.2em -\ruledhbox{\box0} \vskip-\lineheight -\ruledhbox{\box0} \vskip-\lineheight -\ruledhbox{\box0} -\stopbuffer - -\typebuffer - -\getbuffer - -We can use this primitive to check the natural dimensions: - -\startbuffer -\setbox 0 \hbox spread 10pt {test test test} -\ruledhbox{\box0} (\the\boxrepack0,\the\wd0) -\stopbuffer - -\typebuffer - -\getbuffer - -Adapting will recalculate the dimensions with a scale factor for the glue: - -\startbuffer -\setbox 0 \hbox {test test test} -\setbox 2 \hbox {\red test test test} \boxadapt 0 200 -\setbox 4 \hbox {\blue test test test} \boxadapt 0 -200 -\ruledhbox{\box0} \vskip-\lineheight -\ruledhbox{\box0} \vskip-\lineheight -\ruledhbox{\box0} -\stopbuffer - -\typebuffer - -\getbuffer - -\stopsubsection - -\startsubsection[title={Images and reused box objects},reference=sec:imagesandforms] - -In original \TEX\ image support is dealt with via specials. It's not a native -feature of the engine. All that \TEX\ cares about is dimensions, so in practice -that meant: using a box with known dimensions that wraps a special that instructs -the backend to include an image. The wrapping is needed because a special itself -is a whatsit and as such has no dimensions. - -In \PDFTEX\ a special whatsit for images was introduced and that one {\em has} -dimensions. As a consequence, in several places where the engine deals with the -dimensions of nodes, it now has to check the details of whatsits. By inheriting -code from \PDFTEX, the \LUATEX\ engine also had that property. However, at some -point this approach was abandoned and a more natural trick was used: images (and -box resources) became a special kind of rules, and as rules already have -dimensions, the code could be simplified. - -When direction nodes and (formerly local) par nodes also became first class -nodes, whatsits again became just that: nodes representing whatever you want, but -without dimensions, and therefore they could again be ignored when dimensions -mattered. And, because images were disguised as rules, as mentioned, their -dimensions automatically were taken into account. This separation between front -and backend cleaned up the code base already quite a bit. - -In \LUAMETATEX\ we still have the image specific subtypes for rules, but the -engine never looks at subtypes of rules. That was up to the backend. This means -that image support is not present in \LUAMETATEX. When an image specification was -parsed the special properties, like the filename, or additional attributes, were -stored in the backend and all that \LUATEX\ does is registering a reference to an -image's specification in the rule node. But, having no backend means nothing is -stored, which in turn would make the image inclusion primitives kind of weird. - -Therefore you need to realize that contrary to \LUATEX, {\em in \LUAMETATEX\ -support for images and box reuse is not built in}! However, we can assume that -an implementation uses rules in a similar fashion as \LUATEX\ does. So, you can -still consider images and box reuse to be core concepts. Here we just mention the -primitives that \LUATEX\ provides. They are not available in the engine but can -of course be implemented in \LUA. - -\starttabulate[|l|p|] -\DB command \BC explanation \NC \NR -\TB -\NC \tex {saveboxresource} \NC save the box as an object to be included later \NC \NR -\NC \tex {saveimageresource} \NC save the image as an object to be included later \NC \NR -\NC \tex {useboxresource} \NC include the saved box object here (by index) \NC \NR -\NC \tex {useimageresource} \NC include the saved image object here (by index) \NC \NR -\NC \tex {lastsavedboxresourceindex} \NC the index of the last saved box object \NC \NR -\NC \tex {lastsavedimageresourceindex} \NC the index of the last saved image object \NC \NR -\NC \tex {lastsavedimageresourcepages} \NC the number of pages in the last saved image object \NC \NR -\LL -\stoptabulate - -An implementation probably should accept the usual optional dimension parameters -for \type {\use...resource} in the same format as for rules. With images, these -dimensions are then used instead of the ones given to \tex {useimageresource} but -the original dimensions are not overwritten, so that a \tex {useimageresource} -without dimensions still provides the image with dimensions defined by \tex -{saveimageresource}. These optional parameters are not implemented for \tex -{saveboxresource}. - -\starttyping -\useimageresource width 20mm height 10mm depth 5mm \lastsavedimageresourceindex -\useboxresource width 20mm height 10mm depth 5mm \lastsavedboxresourceindex -\stoptyping - -Examples or optional entries are \type {attr} and \type {resources} that accept a -token list, and the \type {type} key. When set to non|-|zero the \type {/Type} -entry is omitted. A value of 1 or 3 still writes a \type {/BBox}, while 2 or 3 -will write a \type {/Matrix}. But, as said: this is entirely up to the backend. -Generic macro packages (like \type {tikz}) can use these assumed primitives so -one can best provide them. It is probably, for historic reasons, the only more or -less standardized image inclusion interface one can expect to work in all macro -packages. - -\stopsubsection - -\startsubsection[title={\prm {hpack}, \prm {vpack} and \prm {tpack}}] - -These three primitives are the equivalents of \prm {hbox}, \prm {vbox} and -\prm {vtop} but they don't trigger the packaging related callbacks. Of course -one never know if content needs a treatment so using them should be done with -care. Apart from accepting more keywords (and therefore options) the normal -box behave the same as before. The \prm {vcenter} builder also works in text -mode. - -\stopsubsection - -\startsubsection[title={\prm {unhpack}, \prm {unvpack}}] - -These two are somewhat experimental. They ignore the accumulated pre- and -postmigrated material bound to a box. I needed it for some experiment so the -functionality might change when I really need it. - -\stopsubsection - -\startsubsection[title={\prm {gleaders}},reference=sec:gleaders] - -\topicindex {leaders} - -This type of leaders is anchored to the origin of the box to be shipped out. So -they are like normal \prm {leaders} in that they align nicely, except that the -alignment is based on the {\it largest\/} enclosing box instead of the {\it -smallest\/}. The \type {g} stresses this global nature. - -\stopsubsection - -\stopsection - -\startsection[title={Languages}] - -\startsubsection[title={\prm {hyphenationmin}}] - -\topicindex {languages} -\topicindex {hyphenation} - -This primitive can be used to set the minimal word length, so setting it to a value -of~$5$ means that only words of 6 characters and more will be hyphenated, of course -within the constraints of the \prm {lefthyphenmin} and \prm {righthyphenmin} -values (as stored in the glyph node). This primitive accepts a number and stores -the value with the language. - -\stopsubsection - -\startsubsection[title={\prm {boundary}, \prm {noboundary}, \prm {protrusionboundary} and \prm {wordboundary}}] - -The \prm {noboundary} command is used to inject a whatsit node but now injects a normal -node with type \nod {boundary} and subtype~0. In addition you can say: - -\starttyping -x\boundary 123\relax y -\stoptyping - -This has the same effect but the subtype is now~1 and the value~123 is stored. -The traditional ligature builder still sees this as a cancel boundary directive -but at the \LUA\ end you can implement different behaviour. The added benefit of -passing this value is a side effect of the generalization. The subtypes~2 and~3 -are used to control protrusion and word boundaries in hyphenation and have -related primitives. - -\stopsubsection - -\stopsection - \startsection[title={Control and debugging}] \startsubsection[title={Tracing}] @@ -2074,17 +1594,12 @@ and for nodes that have sublists (like discretionaries) these are also shown. Al that could have been delegated to \LUA\ but it felt wrong to not made that a core engine feature. -When bit~1 of \prm {tracinglevels} is set the current level is prepended to -tracing lines in the log and when bit~2 is set the input level is prepended. You -can set both bits and get both numbers prepended. In \CONTEXT\ we default to -the value~3, so you get prefixes like \type {3:4:} followed by a space. +The \prm {tracingpenalties} parameter triggers the line break routine to report +the applied interline penalties to the output. When \prm {tracingcommands} is larger than 3 the mode switch will be not be prefixed to the \type {{command}} but get its own \type {[line]}. -When \prm {tracinglevels} variable is set to 3 the group and input level are -shown, a value of 1 or 2 shows only one of them (in \CONTEXT\ we default to 3). - When \prm {tracinghyphenation} is set to 1 duplicate patterns are reported (in \CONTEXT\ we default to that) and higher values will also show details about the \LUA\ hyphenation (exception) feedback loop discussed elsewhere. @@ -2093,6 +1608,40 @@ When set to 1 the \prm {tracingmath} variable triggers the reporting of the mode (inline or display) an mlist is processed. Other new tracing commands are discussed where the mechanisms that they relate to are introduced. +The \prm {tracingnodes} variable makes that when a node list is reported the node +numbers are also shown. This is only useful when you have callbacks that access +nodes. + +\starttabulate[|l|p|] +\DB value \BC effect \NC \NR +\TB +\NC 1 \NC show node numbers in lists \NC \NR +\NC 2 \NC also show numbers of attribute nodes \NC \NR +\NC 3 \NC also show glue spec node numbers \NC \NR +\LL +\stoptabulate + +When the \prm {shownodedetails} variable is set to a value larger than zero and a +node is shown (in a list) then more details will be revealed. This can be rather +verbose because in \LUAMETATEX\ node carry more properties than in traditional +\TEX\ and \LUATEX. A value larger than one will also show details of attributes +that are bound to nodes. + +The \prm {tracinglevels} variable is a bitset and offers the following features: + +\starttabulate[|l|p|] +\DB value \BC effect \NC \NR +\TB +\NC 1 \NC show group level \NC \NR +\NC 2 \NC show input level \NC \NR +\NC 4 \NC show catcode regime \NC \NR +\LL +\stoptabulate + +So a value of~7 shows them all. In \CONTEXT\ we set this variable to~3 which +gives a rather verbose log when tracing is on but in the end its'not that bad +because using some of the newer programming related primitive can save tracing. + Because in \LUATEX\ the saving and restoring of locally redefined macros and set variables is optimized a bit in order to prevent redundant stack usage, there will be less tracing visible. @@ -2120,13 +1669,13 @@ alternative). \startsubsection[title={\prm {lastnodetype}, \prm {lastnodesubtype}, \prm {currentiftype}}] -The \ETEX\ command \type {\lastnodetype} returns the node codes as used in the +The \ETEX\ command \prm {lastnodetype} returns the node codes as used in the engine. You can query the numbers at the \LUA\ end if you need the actual values. The parameter \type {\internalcodesmode} is no longer provided as compatibility switch because \LUATEX\ has more cq. some different nodes and it makes no sense -to be incompatible with the \LUA\ end of the engine. The same is true for \type -{\currentiftype}, as we have more conditionals and also use a different order. -The \type {\lastnodesubtype} is a bonus and again reports the codes used +to be incompatible with the \LUA\ end of the engine. The same is true for \prm +{currentiftype}, as we have more conditionals and also use a different order. +The \prm {lastnodesubtype} is a bonus and again reports the codes used internally. During development these might occasionally change, but eventually they will be stable. @@ -2141,473 +1690,258 @@ assigned to a user boundary node. This means that we also have a \prm \stopsubsection -\stopsection - -\startsection[title={Files}] +\startsubsection[title=Nodes] -\startsubsection[title={File syntax}] +\topicindex {nodes} -\topicindex {files+names} +The \ETEX\ primitive \prm {lastnodetype} is not honest in reporting the +internal numbers as it uses its own values. But you can set \type +{\internalcodesmode} to a non|-|zero value to get the real id's instead. In +addition there is \prm {lastnodesubtype}. -\LUAMETATEX\ will accept a braced argument as a file name: +Another last one is \prm {lastnamedcs} which holds the last match but this one +should be used with care because one never knows if in the meantime something +else \quote {last} has been seen. -\starttyping -\input {plain} -\openin 0 {plain} -\stoptyping +\stopsubsection -This allows for embedded spaces, without the need for double quotes. Macro -expansion takes place inside the argument. Keep in mind that as side effect of -delegating \IO\ to \LUA\ the \tex {openin} primitive is not provided by the -engine and has to be implemented by the macro package. This also means that the -limit on the number of open files is not enforced by the engine. +\stopsection -The \prm {tracingfonts} primitive that has been inherited from \PDFTEX\ has -been adapted to support variants in reporting the font. The reason for this -extension is that a csname not always makes sense. The zero case is the default. +\startsection[title=Scanning] -\starttabulate[|l|l|] -\DB value \BC reported \NC \NR -\TB -\NC \type{0} \NC \type{\foo xyz} \NC \NR -\NC \type{1} \NC \type{\foo (bar)} \NC \NR -\NC \type{2} \NC \type{<bar> xyz} \NC \NR -\NC \type{3} \NC \type{<bar @ ..pt> xyz} \NC \NR -\NC \type{4} \NC \type{<id>} \NC \NR -\NC \type{5} \NC \type{<id: bar>} \NC \NR -\NC \type{6} \NC \type{<id: bar @ ..pt> xyz} \NC \NR -\LL -\stoptabulate +\startsubsection[title=Keywords] -\stopsubsection +\topicindex {keywords} +\topicindex {scanning+keywords} -\startsubsection[title={Writing to file}] +Some primitives accept one or more keywords and \LUAMETATEX\ adds some more. In +order to deal with this efficiently the keyword scanner has been optimized, where +even the context was taken into account. As a result the scanner was quite a bit +faster. This kind of optimization was a graduate process the eventually ended up +in what we have now. In traditional \TEX\ (and also \LUATEX) the order of +keywords is sometimes mixed and sometimes prescribed. In most cases only one +occurrence is permitted. So, for instance, this is valid in \LUATEX: -\topicindex {files+writing} +\starttyping +\hbox attr 123 456 attr 123 456 spread 10cm { } +\hrule width 10cm depth 3mm +\hskip 3pt plus 2pt minus 1pt +\stoptyping -Writing to a file in \TEX\ has two forms: delayed and immediate. Delayed writing -means that the to be written text is anchored in the node list and flushed by the -backend. As all \IO\ is delegated to \LUA, this also means that it has to deal -with distinction. In \LUATEX\ the number of open files was already bumped to 127, -but in \LUAMETATEX\ it depends on the macro package. The special meaning of -channel 18 was already dropped in \LUATEX\ because we have \type {os.execute}. +The \type {attr} comes before the \type {spread}, rules can have multiple mixed +dimension specifiers, and in glue the optional \type {minus} part always comes +last. The last two commands are famous for look ahead side effects which is why +macro packages will end them with something not keyword, like \type {\relax}, +when needed. -\stopsubsection +In \LUAMETATEX\ the following is okay. Watch the few more keywords in box and +rule specifications. -\stopsection +\starttyping +\hbox reverse to 10cm attr 123 456 orientation 4 xoffset 10pt spread 10cm { } +\hrule xoffset 10pt width 10cm depth 3mm +\hskip 3pt minus 1pt plus 2pt +\stoptyping -\startsection[title={Math}] +Here the order is not prescribed and, as demonstrated with the box specifier, for +instance dimensions (specified by \type {to} or \type {spread} can be overloaded +by later settings. In case you wonder if that breaks compatibility: in some way +it does but bad or sloppy keyword usage breaks a run anyway. For instance \type +{minuscule} results in \type {minus} with no dimension being seen. So, in the end +the user should not noticed it and when a user does, the macro package already +had an issue that had to be fixed. -\topicindex {math} +\stopsubsection -We will cover math extensions in its own chapter because not only the font -subsystem and spacing model have been enhanced (thereby introducing many new -primitives) but also because some more control has been added to existing -functionality. Much of this relates to the different approaches of traditional -\TEX\ fonts and \OPENTYPE\ math. +\startsubsection[title={\prm {norelax}}] -\stopsection +\topicindex{relaxing} -\startsection[title={Fonts}] +There are a few cases where the \TEX\ scanned skips over spaces and \prm {relax} as +well as quits on a \prm {relax} in which case it gets pushed back. An example is +given below: -\topicindex {fonts} +\startbuffer +\edef\TestA{\ifnum1=1\relax Y\else N\fi} \meaningasis\TestA +\edef\TestB{\ifnum1=1\norelax Y\else N\fi} \meaningasis\TestB +\stopbuffer -Like math, we will cover fonts extensions in its own chapter. Here we stick to -mentioning that loading fonts is different in \LUAMETATEX. As in \LUATEX\ we have -the extra primitives \type {\fontid} and \type {\setfontid}, \type {\noligs} and -\type {\nokerns}, and \type {\nospaces}. The other new primitives in \LUATEX\ -have been dropped. +\typebuffer -\stopsection +The second line also contains a sentinel but this time we use \prm {norelax} +which will not be pushed back. So, this feature is just a trick to get rid of (in +itself reasonable) side effects. -\startsection[title=Directions] +\startlines\getbuffer \stoplines -\topicindex {\OMEGA} -\topicindex {\ALEPH} -\topicindex {directions} +\stopsubsection -\startsubsection[title={Two directions}] +\startsubsection[title={\prm {ignorepars}}] -The directional model in \LUAMETATEX\ is a simplified version the the model used -in \LUATEX. In fact, not much is happening at all: we only register a change in -direction. +This primitive is like \prm {ignorespaces} but also skips paragraph ending +commands (normally \prm {par} and empty lines). \stopsubsection -\startsubsection[title={How it works}] +\startsubsection[title={\prm {futureexpand}, \prm {futureexpandis}, \prm {futureexpandisap}}] -The approach is that we try to make node lists balanced but also try to avoid -some side effects. What happens is quite intuitive if we forget about spaces -(turned into glue) but even there what happens makes sense if you look at it in -detail. However that logic makes in|-|group switching kind of useless when no -properly nested grouping is used: switching from right to left several times -nested, results in spacing ending up after each other due to nested mirroring. Of -course a sane macro package will manage this for the user but here we are -discussing the low level injection of directional information. +\topicindex{expansion+future} -This is what happens: +These commands are used as: \starttyping -\textdirection 1 nur {\textdirection 0 run \textdirection 1 NUR} nur +\futureexpand\sometoken\whenfound\whennotfound \stoptyping -This becomes stepwise: +When there is no match and a space was gobbled a space will be put back. The +\type {is} variant doesn't do that while the \type {isap} even skips \type +{\pars}, These characters stand for \quote {ignorespaces} and \quote +{ignorespacesandpars}. -\startnarrower -\starttyping -injected: [push 1]nur {[push 0]run [push 1]NUR} nur -balanced: [push 1]nur {[push 0]run [pop 0][push 1]NUR[pop 1]} nur[pop 0] -result : run {RUNrun } run -\stoptyping -\stopnarrower +\stopsubsection -And this: +\stopsection -\starttyping -\textdirection 1 nur {nur \textdirection 0 run \textdirection 1 NUR} nur -\stoptyping +\startsection[title=Macros] + +\startsubsection[title={\prm {lettonothing} and \prm {glettonothing}}] -becomes: +This primitive is equivalent to: -\startnarrower \starttyping -injected: [+TRT]nur {nur [+TLT]run [+TRT]NUR} nur -balanced: [+TRT]nur {nur [+TLT]run [-TLT][+TRT]NUR[-TRT]} nur[-TRT] -result : run {run RUNrun } run +\protected\def\lettonothing#1{\def#1{}} \stoptyping -\stopnarrower - -Now, in the following examples watch where we put the braces: - -\startbuffer -\textdirection 1 nur {{\textdirection 0 run} {\textdirection 1 NUR}} nur -\stopbuffer - -\typebuffer -This becomes: - -\startnarrower -\getbuffer -\stopnarrower +and although it might feel faster (only measurable with millions of calls) it's +mostly there because it is easier on tracing (less clutter). An advantage over +letting to an empty predefined macro is also that in tracing we keep seeing the +name (relaxing would show the relax equivalent). -Compare this to: +\stopsubsection -\startbuffer -\textdirection 1 nur {{\textdirection 0 run }{\textdirection 1 NUR}} nur -\stopbuffer +\startsubsection[title={\prm {glet}}] -\typebuffer +This primitive is similar to: -Which renders as: +\starttyping +\protected\def\glet{\global\let} +\stoptyping -\startnarrower -\getbuffer -\stopnarrower +but faster (only measurable with millions of calls) and probably more convenient +(after all we also have \type {\gdef}). -So how do we deal with the next? +\stopsubsection -\startbuffer -\def\ltr{\textdirection 0\relax} -\def\rtl{\textdirection 1\relax} +\startsubsection[title={\prm {defcsname}, \prm {edefcsname}, \prm {gdefcsname} and \prm {xdefcsname}}] -run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur} -run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run} -\stopbuffer +Although we can implement these primitives easily using macros it makes sense, +given the popularity of \prm {csname} to have these as primitives. It also saves +some \prm {expandafter} usage and it looks a bit better in the source. -\typebuffer +\starttyping +\gdefcsname foo\endcsname{oof} +\stoptyping -It gets typeset as: +\stopsubsection -\startnarrower -\startlines -\getbuffer -\stoplines -\stopnarrower +\startsubsection[title={\prm {letcsname} and \prm {gletcsname}}] -We could define the two helpers to look back, pick up a skip, remove it and -inject it after the dir node. But that way we loose the subtype information that -for some applications can be handy to be kept as|-|is. This is why we now have a -variant of \prm {textdirection} which injects the balanced node before the skip. -Instead of the previous definition we can use: +These can also be implemented using macros but again they are natively provided +by the engine for the same reasons: less code and less tracing clutter. -\startbuffer[def] -\def\ltr{\linedirection 0\relax} -\def\rtl{\linedirection 1\relax} -\stopbuffer +\starttyping +\gletcsname foo\endcsname \relax +\stoptyping -\typebuffer[def] +\stopsubsection -and this time: +\startsubsection[title={\prm {csstring}, \prm {begincsname} and \prm {lastnamedcs}}] -\startbuffer[txt] -run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur} -run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run} -\stopbuffer +These are somewhat special. The \prm {csstring} primitive is like +\prm {string} but it omits the leading escape character. This can be +somewhat more efficient than stripping it afterwards. -\typebuffer[txt] +The \prm {begincsname} primitive is like \prm {csname} but doesn't create +a relaxed equivalent when there is no such name. It is equivalent to -comes out as a properly spaced: +\starttyping +\ifcsname foo\endcsname + \csname foo\endcsname +\fi +\stoptyping -\startnarrower -\startlines -\getbuffer[def,txt] -\stoplines -\stopnarrower +The advantage is that it saves a lookup (don't expect much speedup) but more +important is that it avoids using the \prm {if} test. The \prm {lastnamedcs} +is one that should be used with care. The above example could be written as: -Anything more complex that this, like combination of skips and penalties, or -kerns, should be handled in the input or macro package because there is no way we -can predict the expected behaviour. In fact, the \prm {linedirection} is just a -convenience extra which could also have been implemented using node list parsing. +\starttyping +\ifcsname foo\endcsname + \lastnamedcs +\fi +\stoptyping -Directions are complicated by the fact that they often need to work over groups -so a separate grouping related stack is used. A side effect is that there can be -paragraphs with only a local par node followed by direction synchronization -nodes. Paragraphs like that are seen as empty paragraphs and therefore ignored. -Because \type {\noindent} doesn't inject anything but a \type {\indent} injects -an box, paragraphs with only an indent and directions are handles and paragraphs -with content. When indentation is normalized a paragraph with an indentation -skip is seen as content. +This is slightly more efficient than constructing the string twice (deep down in +\LUATEX\ this also involves some \UTF8 juggling), but probably more relevant is +that it saves a few tokens and can make code a bit more readable. \stopsubsection -\startsubsection[title={Normalizing lines}] - -The original \TEX\ machinery was never meant to be opened up. As a consequence a -constructed line can have different layouts. There can be left- and/or right -skips and hanging indentation or parshape can result in a shift and adapted -width. In \LUATEX\ glue got subtypes so we can recognize the left-, right and -parfill skips, but still there is no hundred percent certainty about the shape. - -In \LUAMETATEX\ lines can be normalized. This is optional because we want to -preserve the original (for comparison) and is controlled by \prm -{normalizelinemode}. That variable actually drives some more. An earlier version -provided a few more granular options (for instance: does a leftskip comes before -or after a left hanging indentation) but in the end that was dropped. Because -this normalization only is seen at the \LUA\ end there is no need to go into much -detail here. - -At this moment a line has this pattern: left parfill, left hang, left skip, -indentation, content, right hang, right skip, right parfill. Of course the -indentation and fill skips are not present in every line. - -Control over normalization happens via the mentioned mode variable and here is -what the engine provides right now. We use a bitmap: - -\starttabulate[|l|l|] -\DB value \BC reported \NC \NR -\TB -\NC \type{0x0001} \NC normalize line as described above \NC \NR -\NC \type{0x0002} \NC use a skip for parindent instead of a box \NC \NR -\NC \type{0x0004} \NC swap hangindent in l2r mode \NC \NR -\NC \type{0x0008} \NC swap parshape in l2r mode \NC \NR -\NC \type{0x0010} \NC put breaks after dir in l2r mode \NC \NR -\NC \type{0x0020} \NC remove margin kerns (\PDFTEX\ left-over) \NC \NR -\NC \type{0x0040} \NC if needed clip width and use correction kern \NC \NR -\LL -\stoptabulate +\startsubsection[title={\prm {futuredef} and \prm {futurecsname}}] -Setting the bit enables the related normalization. More features might be added -in future releases. +This is just the definition variant of \prm {futurelet} and a simple example +shows the difference: -% Swapping shapes -% -% Another adaptation to the \ALEPH\ directional model is control over shapes driven -% by \prm {hangindent} and \prm {parshape}. This is controlled by a new parameter -% \prm {shapemode}: -% -% \starttabulate[|c|l|l|] -% \DB value \BC \prm {hangindent} \BC \prm {parshape} \NC \NR -% \TB -% \BC \type{0} \NC normal \NC normal \NC \NR -% \BC \type{1} \NC mirrored \NC normal \NC \NR -% \BC \type{2} \NC normal \NC mirrored \NC \NR -% \BC \type{3} \NC mirrored \NC mirrored \NC \NR -% \LL -% \stoptabulate -% -% The value is reset to zero (like \prm {hangindent} and \prm {parshape}) -% after the paragraph is done with. You can use negative values to prevent -% this. In \in {figure} [fig:shapemode] a few examples are given. -% -% \startplacefigure[reference=fig:shapemode,title={The effect of \type {shapemode}.}] -% \startcombination[2*3] -% {\ruledvbox \bgroup \setuptolerance[verytolerant] -% \hsize .45\textwidth \switchtobodyfont[6pt] -% \pardirection 0 \textdirection 0 -% \hangindent 40pt \hangafter -3 -% \leftskip10pt \input tufte \par -% \egroup} {TLT: hangindent} -% {\ruledvbox \bgroup \setuptolerance[verytolerant] -% \hsize .45\textwidth \switchtobodyfont[6pt] -% \pardirection 0 \textdirection 0 -% \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize -% \input tufte \par -% \egroup} {TLT: parshape} -% {\ruledvbox \bgroup \setuptolerance[verytolerant] -% \hsize .45\textwidth \switchtobodyfont[6pt] -% \pardirection 1 \textdirection 1 -% \hangindent 40pt \hangafter -3 -% \leftskip10pt \input tufte \par -% \egroup} {TRT: hangindent mode 0} -% {\ruledvbox \bgroup \setuptolerance[verytolerant] -% \hsize .45\textwidth \switchtobodyfont[6pt] -% \pardirection 1 \textdirection 1 -% \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize -% \input tufte \par -% \egroup} {TRT: parshape mode 0} -% {\ruledvbox \bgroup \setuptolerance[verytolerant] -% \hsize .45\textwidth \switchtobodyfont[6pt] -% \shapemode=3 -% \pardirection 1 \textdirection 1 -% \hangindent 40pt \hangafter -3 -% \leftskip10pt \input tufte \par -% \egroup} {TRT: hangindent mode 1 & 3} -% {\ruledvbox \bgroup \setuptolerance[verytolerant] -% \hsize .45\textwidth \switchtobodyfont[6pt] -% \shapemode=3 -% \pardirection 1 \textdirection 1 -% \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize -% \input tufte \par -% \egroup} {TRT: parshape mode 2 & 3} -% \stopcombination -% \stopplacefigure -% -% We have \type {\pardirection}, \type {\textdirection}, \type {\mathdirection} and -% \type {\linedirection} that is like \type {\textdirection} but with some -% additional (inline) glue checking. +\startbuffer +\def\whatever{[\next:\meaning\next]} +\futurelet\next\whatever A +\futuredef\next\whatever B +\stopbuffer -% Controlling glue with \prm {breakafterdirmode} -% -% Glue after a dir node is ignored in the linebreak decision but you can bypass that -% by setting \prm {breakafterdirmode} to~\type {1}. The following table shows the -% difference. Watch your spaces. -% -% \def\ShowSome#1{% -% \BC \type{#1} -% \NC \breakafterdirmode\zerocount\hsize\zeropoint#1 -% \NC -% \NC \breakafterdirmode\plusone\hsize\zeropoint#1 -% \NC -% \NC \NR -% } -% -% \starttabulate[|l|Tp(1pt)|w(5em)|Tp(1pt)|w(5em)|] -% \DB -% \BC \type{0} -% \NC -% \BC \type{1} -% \NC -% \NC \NR -% \TB -% \ShowSome{pre {\textdirection 0 xxx} post} -% \ShowSome{pre {\textdirection 0 xxx }post} -% \ShowSome{pre{ \textdirection 0 xxx} post} -% \ShowSome{pre{ \textdirection 0 xxx }post} -% \ShowSome{pre { \textdirection 0 xxx } post} -% \ShowSome{pre {\textdirection 0\relax\space xxx} post} -% \LL -% \stoptabulate - -\stopsubsection - -\startsubsection[title=Orientations] - -As mentioned, the difference with \LUATEX\ is that we only have numeric -directions and that there are only two: left|-|to|-|right (\type {0}) and -right|-|to|-|left (\type {1}). The direction of a box is set with \type -{direction}. - -In addition to that boxes can now have an \type {orientation} keyword followed by -optional \type {xoffset} and|/|or \type {yoffset} keywords. The offsets don't -have consequences for the dimensions. The alternatives \type {xmove} and \type -{ymove} on the contrary are reflected in the dimensions. Just play with them. The -offsets and moves only are accepted when there is also an orientation, so no time -is wasted on testing for these rarely used keywords. There are related primitives -\type {\box...} that set these properties. - -As these are experimental it will not be explained here (yet). They are covered -in the descriptions of the development of \LUAMETATEX: articles and|/|or -documents in the \CONTEXT\ distribution. For now it is enough to know that the -orientation can be up, down, left or right (rotated) and that it has some -anchoring variants. Combined with the offsets this permits macro writers to -provide solutions for top|-|down and bottom|-|up writing directions, something -that is rather macro package specific and used for scripts that need -manipulations anyway. The \quote {old} vertical directions were never okay and -therefore not used. - -There are a couple of properties in boxes that you can set and query but that -only really take effect when the backend supports them. When usage on \CONTEXT\ -shows that is't okay, they will become official, so we just mention them: \type -{\boxdirection}, \type {\boxattr}, \type {\boxorientation}, \type {\boxxoffset}, -\type {\boxyoffset}, \type {\boxxmove}, \type {\boxymove} and \type {\boxtotal}. - -{\em This is still somewhat experimental and will be documented in more detail -when I've used it more in \CONTEXT\ and the specification is frozen. This might -take some time (and user input).} +\typebuffer -\stopsubsection +\getbuffer -\stopsection +The next one was more an experiment that then stayed around, just to see what +surprising abuse of this primitive will happen: -\startsection[title=Keywords] +\startbuffer +\def\whateveryes{[YES]} +\def\whatevernop{[NOP]} +\let\whatever\undefined +\futurecsname\whatevernop whatever\endcsname +\futurecsname\whatevernop whateveryes\endcsname +\stopbuffer -Some primitives accept one or more keywords and \LUAMETATEX\ adds some more. In -order to deal with this efficiently the keyword scanner has been optimized, where -even the context was taken into account. As a result the scanner was quite a bit -faster. This kind of optimization was a graduate process the eventually ended up -in what we have now. In traditional \TEX\ (and also \LUATEX) the order of -keywords is sometimes mixed and sometimes prescribed. In most cases only one -occurrence is permitted. So, for instance, this is valid in \LUATEX: +\typebuffer -\starttyping -\hbox attr 123 456 attr 123 456 spread 10cm { } -\hrule width 10cm depth 3mm -\hskip 3pt plus 2pt minus 1pt -\stoptyping +When the assembles control sequence is undefined the given one will be expanded, +a weird one, right? I will probably apply it some day in cases where I want less +tracing and a more direct expansion of an assembled name. -The \type {attr} comes before the \type {spread}, rules can have multiple mixed -dimension specifiers, and in glue the optional \type {minus} part always comes -last. The last two commands are famous for look ahead side effects which is why -macro packages will end them with something not keyword, like \type {\relax}, -when needed. +\getbuffer -In \LUAMETATEX\ the following is okay. Watch the few more keywords in box and -rule specifications. +Here is a usage example: \starttyping -\hbox reverse to 10cm attr 123 456 orientation 4 xoffset 10pt spread 10cm { } -\hrule xoffset 10pt width 10cm depth 3mm -\hskip 3pt minus 1pt plus 2pt +\xdef\Whatever{\futurecsname\whatevernop whatever\endcsname} +\xdef\Whatever{\futurecsname\whateveryes whateveryes\endcsname} +\xdef\Whatever{\ifcsname whatever\endcsname\lastnamedcs\else\whatevernop\fi} +\xdef\Whatever{\ifcsname whateveryes\endcsname\lastnamedcs\else\whatevernop\fi} +\xdef\Whatever{\ifcsname whatever\endcsname\csname whatever\endcsname\else\whatevernop\fi} +\xdef\Whatever{\ifcsname whateveryes\endcsname\csname whateveryes\endcsname\else\whatevernop\fi} \stoptyping -Here the order is not prescribed and, as demonstrated with the box specifier, for -instance dimensions (specified by \type {to} or \type {spread} can be overloaded -by later settings. In case you wonder if that breaks compatibility: in some way -it does but bad or sloppy keyword usage breaks a run anyway. For instance \type -{minuscule} results in \type {minus} with no dimension being seen. So, in the end -the user should not noticed it and when a user does, the macro package already -had an issue that had to be fixed. - -\stopsection - -\startsection[title=Expressions and \prm {numericscale}] - -The \type {*expr} parsers now accept \type {:} as operator for integer division -(the \type {/} operators does rounding. This can be used for division compatible -with \type {\divide}. I'm still wondering if adding a couple of bit operators -makes sense (for integers). +The timings for one million times defining each of these definitions are 0.277, +0.313, 0.310, 0.359, 0.352 and 0.573 seconds (on a 2018 Dell 7250 Precision +laptop with mobile E3-1505M v6 processor), so there is a little gain here, but of +course in practice no one will notice that because not that many such macros are +defined (or used). -The \prm{numericscale} parser is kind of special (and might evolve). For now it -converts a following number in a scale value as often used in \TEX, where 1000 -means scaling by~1.0. The trick is in the presence of a digit (or comma): 1.234 -becomes 1234 but 1234 stays 1234 and from this you can deduce that 12.34 becomes -123400. Internally \TEX\ calculates with integers, but this permits the macro -package to provide an efficient mix. +\stopsubsection -\stopsection +\startsubsection[title=Arguments] -\startsection[title=Macro arguments] +\topicindex {macros+arguments} Again this is experimental and (used and) discussed in document that come with the \CONTEXT\ distribution. When defining a macro you can do this: @@ -2663,7 +1997,8 @@ what this does: Of course complex combinations can be confusing because after all \TEX\ is parsing for (multi|-|token) delimiters and will happily gobble the whole file if -you are not careful. You can quit scanning if you want: +you are not careful. You can quit scanning with \prm {ignorearguments} if you +want: \starttyping \mymacro 123\ignorearguments @@ -2683,9 +2018,66 @@ on performance this has been compensated by some more efficiency in the macro parser and engine in general and of course you can gain back some by using these features. -\stopsection +\stopsubsection + +\startsubsection[title={\prm {parametermark}}] + +The meaning of primitive \prm {parametermark} is equivalent to \type {#} in a macro +definition, just like \prm {alignmark} is in an alignment. It can be used to circumvent +catcode issues. The normal \quotation {duplicate them when nesting} rules apply. + +\startbuffer +\def\foo\parametermark1% + {\def\oof\parametermark\parametermark1% + {[\parametermark1:\parametermark\parametermark1]}} +\stopbuffer + +\typebuffer \getbuffer + +Here \type {\foo{X}\oof{Y}} gives: \foo{X}\oof{Y}. + +\stopsubsection + +\startsubsection[title={\prm {lastarguments} and \prm {parametercount}}] -\startsection[title=Overload protection] +\topicindex{arguments+numberof} + +There are two state variables that refer to the number of read arguments. An +example can show the difference: + +\startbuffer +\tolerant\def\foo[#1]#*[#2]{[\the\lastarguments,\the\parametercount]} + +\foo[1][2] +\foo[1] +\foo + +x: \foo[1][2] +x: \foo[1] +x: \foo +\stopbuffer + +\typebuffer + +What you get actually depends on the macro package. When for instance \prm +{everypar} has some value that results in a macro being expanded, the numbers +reported can refer to the most recent macro because serializing the number can +result in entering horizontal mode. + +\startlines +\getbuffer +\stoplines + +The \prm {lastarguments} returns the most recent global state variable as with +any \type {\last...} primitives. Because it actually looks at the parameter stack +of the currently expanded macro \prm {parametercount} is more reliable but also +less efficient. + +\stopsubsection + +\startsubsection[title=Overload protection] + +\topicindex {macros+overloading} There is an experimental overload protection mechanism that we will test for a while before declaring it stable. The reason for that is that we need to adapt @@ -2803,9 +2195,51 @@ There is an extra prefix \prm {untraced} that will suppress the meaning when tracing so that the macro looks more like a primitive. It is still somewhat experimental so what gets displayed might change. +The \prm {letfrozen}, \prm {unletfrozen}, \prm {letprotected} and \prm +{unletprotected} primitives do as their names advertise. Of course the \prm +{overloadmode} must be set so that it is permitted. + +\stopsubsection + +\startsubsection[title={Swapping meaning}] + +\topicindex {macros+swapping} + +The \prm {swapcsvalues} will swap the values of two control sequences of the same +type. This is a somewhat tricky features because it can interfere with grouping. + +\startbuffer +\scratchcounterone 1 \scratchcountertwo 2 +(\the\scratchcounterone,\the\scratchcountertwo) +\swapcsvalues \scratchcounterone \scratchcountertwo +(\the\scratchcounterone,\the\scratchcountertwo) +\swapcsvalues \scratchcounterone \scratchcountertwo +(\the\scratchcounterone,\the\scratchcountertwo) + +\scratchcounterone 3 \scratchcountertwo 4 +(\the\scratchcounterone,\the\scratchcountertwo) +\bgroup +\swapcsvalues \scratchcounterone \scratchcountertwo +(\the\scratchcounterone,\the\scratchcountertwo) +\egroup +(\the\scratchcounterone,\the\scratchcountertwo) +\stopbuffer + +\typebuffer + +We get similar results: + +\startlines +\getbuffer +\stoplines + +\stopsubsection + \stopsection -\startsection[title={Constants with \prm{integerdef}, \prm {dimensiondef}, +\startsection[title=Quantities] + +\startsubsection[title={Constants with \prm{integerdef}, \prm {dimensiondef}, \prm {gluespecdef} and \prm {mugluespecdef}}] It is rather common to store constant values in a register or character @@ -2834,9 +2268,9 @@ Experiments with constant strings made the engine source more complex than I wanted so that features was rejected. Of course we can use the prefixes mentioned in a previous section. -\stopsection +\stopsubsection -\startsection[title={Getting internal indices with \prm {indexofcharacter} and \prm {indexofregister}}] +\startsubsection[title={Getting internal indices with \prm {indexofcharacter} and \prm {indexofregister}}] When you have defined a register with one of the \tex {...def} primitives but for some reasons needs to know the register index you can query that: @@ -2871,9 +2305,9 @@ A similar primitive gives us the (normally \UNICODE) value of a character: The result is equivalent to \type {\number `A} but avoids the back quote: \inlinebuffer. -\stopsection +\stopsubsection -\startsection[title={Serialization with \prm {todimension}, \prm {toscaled}, \prm {tohexadecimal} and \prm {tointeger}}] +\startsubsection[title={Serialization with \prm {todimension}, \prm {toscaled}, \prm {tohexadecimal} and \prm {tointeger}}] These serializers take a verbose or symbolic quantity: @@ -2893,9 +2327,11 @@ when a value is stored in a macro. Using \type {\the} could fail there while: is often overkill and gives more noise in a trace. -\stopsection +\stopsubsection -\startsection[title={Serialization with \prm {thewithoutunit}, \prm {tosparsedimension} and \prm {tosparsescaled}}] +\startsubsection[title={Serialization with \prm {thewithoutunit}, \prm {tosparsedimension} and \prm {tosparsescaled}}] + +\topicindex {units} By default \TEX\ lets \type {1pt} come out as \type {1.0pt} which is why we also have two sparse variants: @@ -2916,9 +2352,33 @@ This time trailing zeros (and a trailing period) will be dropped: The \prm {thewithoutunit} primitive is like \prm {the} on a dimension but it omits the unit. +\stopsubsection + \stopsection -\startsection[title=Expressions with \prm {numexpression}, \prm {dimexpression}] +\startsection[title=Expressions] + +\startsubsection[title={Rounding and scaling}] + +\topicindex {expressions+traditional} + +The \type {*expr} parsers now accept \type {:} as operator for integer division +(the \type {/} operators does rounding. This can be used for division compatible +with \type {\divide}. I'm still wondering if adding a couple of bit operators +makes sense (for integers). + +The \prm{numericscale} parser is kind of special (and might evolve). For now it +converts a following number in a scale value as often used in \TEX, where 1000 +means scaling by~1.0. The trick is in the presence of a digit (or comma): 1.234 +becomes 1234 but 1234 stays 1234 and from this you can deduce that 12.34 becomes +123400. Internally \TEX\ calculates with integers, but this permits the macro +package to provide an efficient mix. + +\stopsubsection + +\startsubsection[title={Enhanced expressions}] + +\topicindex {expressions+enhanced} The \ETEX\ expression primitives are handy but have some limitations. Although the parsers have been rewritten in \LUAMETATEX\ and somewhat more efficient the @@ -3000,18 +2460,108 @@ The if|-|test shown before can be done using the new primitives \prm {ifdimexpression} and \prm {ifnumexpression} which are boolean tests with zero being \type {false}. +\stopsubsection + \stopsection -\startsection[title=Nodes] +\startsection[title=Loops] -The \ETEX\ primitive \type {\lastnodetype} is not honest in reporting the -internal numbers as it uses its own values. But you can set \type -{\internalcodesmode} to a non|-|zero value to get the real id's instead. In -addition there is \type {\lastnodesubtype}. +\topicindex {loops} -Another last one is \type {\lastnamedcs} which holds the last match but this one -should be used with care because one never knows if in the meantime something -else \quote {last} has been seen. +There is actually not that much to tell about the three loop primitives \prm +{expandedloop}, \prm {unexpandedloop} and \prm {localcontrolledloop}. They are +used like: + +\startbuffer +\unexpandedloop 1 10 1 { + [!] +} +\stopbuffer + +\typebuffer + +This will give 10 snippets. + +\getbuffer + +So what will the next give? + +\startbuffer +\edef\TestA{\unexpandedloop 1 10 1 {!}}\meaning\TestA +\edef\TestB{\expandedloop 1 10 1 {!}}\meaning\TestB +\stopbuffer + +\typebuffer + +We see no difference in results between the two loops: + +\startlines \tt +\getbuffer +\stoplines + +But the the next variant shows that they do: + +\startbuffer +\edef\TestA{\unexpandedloop 1 10 1 {\the\currentloopiterator}}\meaning\TestA +\edef\TestB{\expandedloop 1 10 1 {\the\currentloopiterator}}\meaning\TestB +\stopbuffer + +\typebuffer + +The unexpanded variants sort of delays: + +\startlines \tt +\getbuffer +\stoplines + +You can nest loops and query the nesting level: + +\startbuffer +\expandedloop 1 10 1 {% + \ifodd\currentloopiterator\else + [\expandedloop 1 \currentloopiterator 1 {% + \the\currentloopnesting + }] + \fi +} +\stopbuffer + +\typebuffer + +Here we use the two numeric state primitives \prm {currentloopiterator} and \prm +{currentloopnesting}. This results in: + +\getbuffer + +The \prm {quitloop} primitive makes it possible to prematurely exit a loop (at +the next step), although of course in the next case one can just adapt the final +iterator value instead. Here we step by 2: + +\startbuffer +\expandedloop 1 20 2 {% + \ifnum\currentloopiterator>10 + \quitloop + \else + [!] + \fi +} +\stopbuffer + +\typebuffer + +This results in: + +\getbuffer + +The \prm {lastloopiterator} primitive keeps the last iterator value and is a global +one as all \type {\last...} primitives. The loops also work with negative values. + +A special case is \prm {localcontrolledloop} which fits into the repertoire of +local control primitives. In that case the loop body gets expanded in a nested +main loop which can come in handy in tricky cases where full expansion is mixed +with for instance assignments but of course users should then be aware of +out|-|of|-|order side effects when you push back something in the input. Consider +it a playground. \stopsection diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-fonts.tex b/doc/context/sources/general/manuals/luametatex/luametatex-fonts.tex index d92bb7b55..686d1728f 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex-fonts.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex-fonts.tex @@ -457,7 +457,7 @@ expansion takes place inside the argument. \stopsubsection -\startsubsection[title={\prm {fontid} and \prm {setfontid}}] +\startsubsection[title={\prm {fontid}, \prm {setfontid} and \prm {mathstylefontid}}] \startsyntax \fontid\font @@ -465,7 +465,7 @@ expansion takes place inside the argument. This primitive expands into a number. The currently used font id is \number\fontid\font. Here are some more: \footnote {Contrary to \LUATEX\ this is -now a number so you need to use \type {\number} or \type {\the}. The same is true +now a number so you need to use \prm {number} or \prm {the}. The same is true for some other numbers and dimensions that for some reason ended up in the serializer that produced a sequence of tokens.} @@ -488,6 +488,15 @@ be bound to a real font, after all it's just a number. The primitive \prm {setfontid} can be used to enable a font with the given id, which of course needs to be a valid one. +In math mode the font id depends on the style because there we have a family of +three related fonts. In this document we get the following identifiers: + +\starttabulate[|l|c|] +\NC \type {$ \the \mathstylefontid \scriptscriptstyle \fam$} \NC $\the\mathstylefontid\scriptscriptstyle \fam$ \NC \NR +\NC \type {$ \the \mathstylefontid \scriptstyle \fam$} \NC $\the\mathstylefontid\scriptstyle \fam$ \NC \NR +\NC \type {$ \the \mathstylefontid \textstyle \fam$} \NC $\the\mathstylefontid\textstyle \fam$ \NC \NR +\stoptabulate + \stopsubsection \startsubsection[title={\prm {fontspecifiedname} and \prm {fontspecifiedsize}}] @@ -526,6 +535,8 @@ features but in \LUAMETATEX\ these primitives are gone. They are replace by a mo generic control primitive \prm {glyphoptions}. This numerical parameter is a bitset with the following fields: +% todo: use values + \starttabulate[|l|pl|] \DB value \BC effect \NC\NR \TB @@ -556,6 +567,62 @@ query a font dimensions with \prm {fontdimen} you get the raw value but with \pr \stopsubsection +\startsubsection[title={\prm {glyphxscaled}, \prm {glyphyscaled}}] + +These two relate to the previous one: + +\startbuffer +{\glyphxscale 1500 \the\glyphxscaled 100pt} and +{\glyphyscale 750 \the\glyphyscaled 100pt} +\stopbuffer + +\typebuffer + +We get: \inlinebuffer. + +\stopsubsection + +\startsubsection[title={Scaled fontdimensions}] + +When you use \prm {glyphscale}, \prm {glyphxscale} and|/|or \prm {glyphyscale} the font +dimensions + +\starttabulate[|l|c|c|c|] +\DB dimension \BC scale \NC xscale \NC yscale \NC\NR +\TB +\NC \prm {scaledemwidth} \NC \star \NC \star \NC \NC\NR +\NC \prm {scaledexheight} \NC \star \NC \NC \star \NC\NR +\NC \prm {scaledextraspace} \NC \star \NC \star \NC \NC\NR +\NC \prm {scaledinterwordshrink} \NC \star \NC \star \NC \NC\NR +\NC \prm {scaledinterwordspace} \NC \star \NC \star \NC \NC\NR +\NC \prm {scaledinterwordstretch} \NC \star \NC \star \NC \NC\NR +\NC \prm {scaledslantperpoint} \NC \star \NC \star \NC \NC\NR +\LL +\stoptabulate + +The next table shows the effective sized when we scale by 2000. The last two +columns scale twice: the shared scale and the x or y scale. + +\def\ShowThem#1% + {\normalexpanded{ + \BC \small \prm {\csstring\parametermark1} + \NC {\localcontrolled{\glyphscale2000 \glyphxscale1000 \glyphyscale 1000} \withoutpt\parametermark1} + \NC {\localcontrolled{\glyphscale1000 \glyphxscale2000 \glyphyscale 1000} \withoutpt\parametermark1} + \NC {\localcontrolled{\glyphscale1000 \glyphxscale1000 \glyphyscale 2000} \withoutpt\parametermark1} + \NC {\localcontrolled{\glyphscale2000 \glyphxscale2000 \glyphyscale 1000} \withoutpt\parametermark1} + \NC {\localcontrolled{\glyphscale2000 \glyphxscale1000 \glyphyscale 2000} \withoutpt\parametermark1} + \NC \NR}} + +\starttabulate[|l|c{.}|c{.}|c{.}|c{.}|c{.}|] + \ShowThem\scaledemwidth + \ShowThem\scaledexheight + \ShowThem\scaledextraspace + \ShowThem\scaledinterwordshrink + \ShowThem\scaledinterwordspace + \ShowThem\scaledinterwordstretch + \ShowThem\scaledslantperpoint +\stoptabulate + \startsubsection[title={\prm {fontspecdef}, \prm {fontspecid}, \prm {fontspecscale}, , \prm {fontspecxscale}, \prm {fontspecyscale}}] Because we have three scale related primitives \prm {glyphscale}, \prm @@ -702,6 +769,18 @@ instance content moved into the margin: \stopsubsection +\startsubsection[title={\prm{fontcharta}}] + +\topicindex {glyphs + properties} + +The \prm {fontcharwd}, \prm {fontcharht}, \prm {fontchardp} and \prm {fontcharit} +give access to character properties. To this repertoire \LUAMETATEX\ adds the top +accent accessor \prm {fontcharta} which came in handy for tracing. You pass a +font reference and character code. Normally only \OPENTYPE\ math fonts have this +property. + +\stopsubsection + \stopsection \startsection[title={The \LUA\ font library}][library=font] @@ -803,8 +882,8 @@ whatever purpose their use in \CONTEXT\ is fixed. More details about fonts in math mode can be found in the chapter about math so here we just mention a few primitives. The internal \prm {glyphtextscale}, \prm -{glyphscriptscriptscale} and \prm {glyphscriptscriptscale} registers can be set -to enforce additional scaling of math, like this: +{glyphscriptscale} and \prm {glyphscriptscriptscale} registers can be set to +enforce additional scaling of math, like this: \startbuffer $ a = b^2 = c^{d^2}$ @@ -845,6 +924,27 @@ values only makes sense in the perspective of the used macro package. \stopsubsection +\startsubsection[title={Tracing}] + +The \prm {tracingfonts} primitive that has been inherited from \PDFTEX\ has +been adapted to support variants in reporting the font. The reason for this +extension is that a csname not always makes sense. The zero case is the default. + +\starttabulate[|l|l|] +\DB value \BC reported \NC \NR +\TB +\NC \type{0} \NC \type{\foo xyz} \NC \NR +\NC \type{1} \NC \type{\foo (bar)} \NC \NR +\NC \type{2} \NC \type{<bar> xyz} \NC \NR +\NC \type{3} \NC \type{<bar @ ..pt> xyz} \NC \NR +\NC \type{4} \NC \type{<id>} \NC \NR +\NC \type{5} \NC \type{<id: bar>} \NC \NR +\NC \type{6} \NC \type{<id: bar @ ..pt> xyz} \NC \NR +\LL +\stoptabulate + +\stopsubsection + \stopsection \stopchapter diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-internals.tex b/doc/context/sources/general/manuals/luametatex/luametatex-internals.tex deleted file mode 100644 index d71ce7ab8..000000000 --- a/doc/context/sources/general/manuals/luametatex/luametatex-internals.tex +++ /dev/null @@ -1,244 +0,0 @@ -% language=us runpath=texruns:manuals/luametatex - -\environment luametatex-style - -\startcomponent luametatex-internals - -\startchapter[reference=internals,title={The internals}] - -\topicindex{nodes} -\topicindex{boxes} -\topicindex{\LUA} - -This is a reference manual and not a tutorial. This means that we discuss changes -relative to traditional \TEX\ and also present new (or extended) functionality. -As a consequence we will refer to concepts that we assume to be known or that -might be explained later. Because the \LUATEX\ and \LUAMETATEX\ engines open up -\TEX\ there's suddenly quite some more to explain, especially about the way a (to -be) typeset stream moves through the machinery. However, discussing all that in -detail makes not much sense, because deep knowledge is only relevant for those -who write code not possible with regular \TEX\ and who are already familiar with -these internals (or willing to spend time on figuring it out). - -So, the average user doesn't need to know much about what is in this manual. For -instance fonts and languages are normally dealt with in the macro package that -you use. Messing around with node lists is also often not really needed at the -user level. If you do mess around, you'd better know what you're dealing with. -Reading \quotation {The \TEX\ Book} by Donald Knuth is a good investment of time -then also because it's good to know where it all started. A more summarizing -overview is given by \quotation {\TEX\ by Topic} by Victor Eijkhout. You might -want to peek in \quotation {The \ETEX\ manual} too. - -But \unknown\ if you're here because of \LUA, then all you need to know is that -you can call it from within a run. If you want to learn the language, just read -the well written \LUA\ book. The macro package that you use probably will provide -a few wrapper mechanisms but the basic \prm {directlua} command that does the job -is: - -\starttyping -\directlua{tex.print("Hi there")} -\stoptyping - -You can put code between curly braces but if it's a lot you can also put it in a -file and load that file with the usual \LUA\ commands. If you don't know what -this means, you definitely need to have a look at the \LUA\ book first. - -If you still decide to read on, then it's good to know what nodes are, so we do a -quick introduction here. If you input this text: - -\starttyping -Hi There ... -\stoptyping - -eventually we will get a linked lists of nodes, which in \ASCII\ art looks like: - -\starttyping -H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ... -\stoptyping - -When we have a paragraph, we actually get something like this, where a \type -{par} node stores some metadata and is followed by a \type {hlist} flagged as -indent box: - -\starttyping -[par] <=> [hlist] <=> H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ... -\stoptyping - -Each character becomes a so called glyph node, a record with properties like the -current font, the character code and the current language. Spaces become glue -nodes. There are many node types and nodes can have many properties but that will -be discussed later. Each node points back to a previous node or next node, given -that these exist. Sometimes multiple characters are represented by one glyph -(shape), so one can also get: - -\starttyping -[par] <=> [hlist] <=> H <=> i <=> [glue] <=> Th <=> e <=> r <=> e ... -\stoptyping - -And maybe some characters get positioned relative to each other, so we might -see: - -\starttyping -[par] <=> [hlist] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ... -\stoptyping - -Actually, the above representation is one view, because in \LUAMETATEX\ we can -choose for this: - -\starttyping -[par] <=> [glue] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ... -\stoptyping - -where glue (currently fixed) is used instead of an empty hlist (think of a \type -{\hbox}). Options like this are available because want a certain view on these -lists from the \LUA\ end and the result being predicable is part of that. - -It's also good to know beforehand that \TEX\ is basically centered around -creating paragraphs and pages. The par builder takes a list and breaks it into -lines. At some point horizontal blobs are wrapped into vertical ones. Lines are -so called boxes and can be separated by glue, penalties and more. The page -builder accumulates lines and when feasible triggers an output routine that will -take the list so far. Constructing the actual page is not part of \TEX\ but done -using primitives that permit manipulation of boxes. The result is handled back to -\TEX\ and flushed to a (often \PDF) file. - -\starttyping -\setbox\scratchbox\vbox\bgroup - line 1\par line 2 -\egroup - -\showbox\scratchbox -\stoptyping - -The above code produces the next log lines that reveal how the engines sees a -paragraph (wrapped in a \type {\vbox}): - -\starttyping[style=small] -1:4: > \box257= -1:4: \vbox[normal][16=1,17=1,47=1], width 483.69687, height 27.58083, depth 0.1416, direction l2r -1:4: .\list -1:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r -1:4: ...\list -1:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt -1:4: ....\glue[left][16=1,17=1,47=1] 0.0pt -1:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt -1:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519 -1:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt -1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l -1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i -1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n -1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e -1:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30 -1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000031 1 -1:4: ....\penalty[line][16=1,17=1,47=1] 10000 -1:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil -1:4: ....\glue[right][16=1,17=1,47=1] 0.0pt -1:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt -1:4: ..\glue[par][16=1,17=1,47=1] 5.44995pt plus 1.81665pt minus 1.81665pt -1:4: ..\glue[baseline][16=1,17=1,47=1] 6.79396pt -1:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r -1:4: ...\list -1:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt -1:4: ....\glue[left][16=1,17=1,47=1] 0.0pt -1:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt -1:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519 -1:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt -1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l -1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i -1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n -1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e -1:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30 -1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000032 2 -1:4: ....\penalty[line][16=1,17=1,47=1] 10000 -1:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil -1:4: ....\glue[right][16=1,17=1,47=1] 0.0pt -1:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt -\stoptyping - -The \LUATEX\ engine provides hooks for \LUA\ code at nearly every reasonable -point in the process: collecting content, hyphenating, applying font features, -breaking into lines, etc. This means that you can overload \TEX's natural -behaviour, which still is the benchmark. When we refer to \quote {callbacks} we -means these hooks. The \TEX\ engine itself is pretty well optimized but when you -kick in much \LUA\ code, you will notices that performance drops. Don't blame and -bother the authors with performance issues. In \CONTEXT\ over 50\% of the time -can be spent in \LUA, but so far we didn't get many complaints about efficiency. -Adding more callbacks makes no sense, also because at some point the performance -hit gets too large. There are plenty of ways to achieve goals. For that reason: -take remarks about \LUATEX, features, potential, performance etc.\ with a natural -grain of salt. - -Where plain \TEX\ is basically a basic framework for writing a specific style, -macro packages like \CONTEXT\ and \LATEX\ provide the user a whole lot of -additional tools to make documents look good. They hide the dirty details of font -management, language support, turning structure into typeset results, wrapping -pages, including images, and so on. You should be aware of the fact that when you -hook in your own code to manipulate lists, this can interfere with the macro -package that you use. Each successive step expects a certain result and if you -mess around to much, the engine eventually might bark and quit. It can even -crash, because testing everywhere for what users can do wrong is no real option. - -When you read about nodes in the following chapters it's good to keep in mind -what commands relate to them. Here are a few: - -\starttabulate[|l|l|p|] -\DB command \BC node \BC explanation \NC \NR -\TB -\NC \prm {hbox} \NC \nod {hlist} \NC horizontal box \NC \NR -\NC \prm {vbox} \NC \nod {vlist} \NC vertical box with the baseline at the bottom \NC \NR -\NC \prm {vtop} \NC \nod {vlist} \NC vertical box with the baseline at the top \NC \NR -\NC \prm {hskip} \NC \nod {glue} \NC horizontal skip with optional stretch and shrink \NC \NR -\NC \prm {vskip} \NC \nod {glue} \NC vertical skip with optional stretch and shrink \NC \NR -\NC \prm {kern} \NC \nod {kern} \NC horizontal or vertical fixed skip \NC \NR -\NC \prm {discretionary} \NC \nod {disc} \NC hyphenation point (pre, post, replace) \NC \NR -\NC \prm {char} \NC \nod {glyph} \NC a character \NC \NR -\NC \prm {hrule} \NC \nod {rule} \NC a horizontal rule \NC \NR -\NC \prm {vrule} \NC \nod {rule} \NC a vertical rule \NC \NR -\NC \prm {textdirection} \NC \nod {dir} \NC a change in text direction \NC \NR -\LL -\stoptabulate - -Whatever we feed into \TEX\ at some point becomes a token which is either -interpreted directly or stored in a linked list. A token is just a number that -encodes a specific command (operator) and some value (operand) that further -specifies what that command is supposed to do. In addition to an interface to -nodes, there is an interface to tokens, as later chapters will demonstrate. - -Text (interspersed with macros) comes from an input medium. This can be a file, -token list, macro body cq.\ arguments, some internal quantity (like a number), -\LUA, etc. Macros get expanded. In the process \TEX\ can enter a group. Inside -the group, changes to registers get saved on a stack, and restored after leaving -the group. When conditionals are encountered, another kind of nesting happens, -and again there is a stack involved. Tokens, expansion, stacks, input levels are -all terms used in the next chapters. Don't worry, they loose their magic once you -use \TEX\ a lot. You have access to most of the internals and when not, at least -it is possible to query some state we're in or level we're at. - -When we talk about pack(ag)ing it can mean two things. When \TEX\ has consumed -some tokens that represent text they are added to the current list. When the text -is put into a so called \type {\hbox} (for instance a line in a paragraph) it -(normally) first gets hyphenated, next ligatures are build, and finally kerns are -added. Each of these stages can be overloaded using \LUA\ code. When these three -stages are finished, the dimension of the content is calculated and the box gets -its width, height and depth. What happens with the box depends on what macros do -with it. - -The other thing that can happen is that the text starts a new paragraph. In that -case some information is stored in a leading \type {par} node. Then indentation -is appended and the paragraph ends with some glue. Again the three stages are -applied but this time afterwards, the long line is broken into lines and the -result is either added to the content of a box or to the main vertical list (the -running text so to say). This is called par building. At some point \TEX\ decides -that enough is enough and it will trigger the page builder. So, building is -another concept we will encounter. Another example of a builder is the one that -turns an intermediate math list into something typeset. - -Wrapping something in a box is called packing. Adding something to a list is -described in terms of contributing. The more complicated processes are wrapped -into builders. For now this should be enough to enable you to understand the next -chapters. The text is not as enlightening and entertaining as Don Knuths books, -sorry. - -\stopchapter - -\stopcomponent diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-languages.tex b/doc/context/sources/general/manuals/luametatex/luametatex-languages.tex index 61ad57296..83e19dcc9 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex-languages.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex-languages.tex @@ -341,6 +341,41 @@ side effects have been explored. You can find some experimental use cases in \stopsection +\startsection[title={Controlling hyphenation}] + +\startsubsection[title={\prm {hyphenationmin}}] + +\topicindex {languages} +\topicindex {hyphenation} + +This primitive can be used to set the minimal word length, so setting it to a value +of~$5$ means that only words of 6 characters and more will be hyphenated, of course +within the constraints of the \prm {lefthyphenmin} and \prm {righthyphenmin} +values (as stored in the glyph node). This primitive accepts a number and stores +the value with the language. + +\stopsubsection + +\startsubsection[title={\prm {boundary}, \prm {noboundary}, \prm {protrusionboundary} and \prm {wordboundary}}] + +The \prm {noboundary} command is used to inject a whatsit node but now injects a normal +node with type \nod {boundary} and subtype~0. In addition you can say: + +\starttyping +x\boundary 123\relax y +\stoptyping + +This has the same effect but the subtype is now~1 and the value~123 is stored. +The traditional ligature builder still sees this as a cancel boundary directive +but at the \LUA\ end you can implement different behaviour. The added benefit of +passing this value is a side effect of the generalization. The subtypes~2 and~3 +are used to control protrusion and word boundaries in hyphenation and have +related primitives. + +\stopsubsection + +\stopsection + \startsection[title={The main control loop}] \topicindex {main loop} @@ -771,7 +806,7 @@ ligatures are used. Of course kerning also complicates matters here. \startsection[title={The \type {language} library}][library=lang] -\subsection {\type {new} and \type {id}} +\startsubsection[title={\type {new} and \type {id}}] \topicindex {languages+library} @@ -800,7 +835,9 @@ to the internal language with that id number. The number returned is the internal \prm {language} id number this object refers to. -\subsection {\type {hyphenation}} +\stopsubsection + +\startsubsection[title={\type {hyphenation}}] \libindex {hyphenation} @@ -814,7 +851,9 @@ language.hyphenation(<language> l, <string> n) When no string is given (the first example) a string with all exceptions is returned. -\subsection {\type {clearhyphenation} and \type {clean}} +\stopsubsection + +\startsubsection[title={\type {clearhyphenation} and \type {clean}}] \libindex {clearhyphenation} \libindex {clean} @@ -839,7 +878,9 @@ syntax of the argument string is explained in \in {section} [patternsexceptions] This function is useful if you want to do something else based on the words in a dictionary file, like spell|-|checking. -\subsection {\type {patterns} and \type {clearpatterns}} +\stopsubsection + +\startsubsection[title={\type {patterns} and \type {clearpatterns}}] \libindex {patterns} \libindex {clearpatterns} @@ -859,7 +900,9 @@ language.clearpatterns(<language> l) This can be used to clear the pattern dictionary for a language. -\subsection {\type {hyphenationmin}} +\stopsubsection + +\startsubsection[title={\type {hyphenationmin}}] \libindex {hyphenationmin} @@ -871,7 +914,9 @@ n = language.hyphenationmin(<language> l) language.hyphenationmin(<language> l, <number> n) \stopfunctioncall -\subsection {\type {[pre|post][ex|]hyphenchar}} +\stopsubsection + +\startsubsection[title={\type {[pre|post][ex|]hyphenchar}}] \libindex {prehyphenchar} \libindex {posthyphenchar} @@ -902,7 +947,9 @@ These gets or set the \quote {pre|-|break} and \quote {post|-|break} hyphen characters for explicit hyphenation in this language. Both are initially decimal~0 (indicating emptiness). -\subsection {\type {hyphenate}} +\stopsubsection + +\startsubsection[title={\type {hyphenate}}] \libindex {hyphenate} @@ -921,7 +968,9 @@ nodes with the node subtype having the value \type {1}. Glyph modes with different subtypes are not processed. See \in {section} [charsandglyphs] for more details. -\subsection {\type {[set|get]hjcode}} +\stopsubsection + +\startsubsection[title={\type {[set|get]hjcode}}] \libindex {sethjcode} \libindex {gethjcode} @@ -936,6 +985,35 @@ language.sethjcode(<language> l, <number> char, <number> usedchar) When you set a hjcode the current sets get initialized unless the set was already initialized due to \prm {savinghyphcodes} being larger than zero. +\subsection{\prm {hccode} and \type {[set|get]hccode}} + +A character can be set to non zero to indicate that it should be regarded as +value visible hyphenation point. These examples show how that works (it si the +second bit in \prm {hyphenationmode} that does the magic but we set them all +here): + +\startbuffer +{\hsize 1mm \hccode"2014 \zerocount \hyphenationmode "0000000 xxx\emdash xxx \par} +{\hsize 1mm \hccode"2014 "2014\relax \hyphenationmode "0000000 xxx\emdash xxx \par} + +{\hsize 1mm \hccode"2014 \zerocount \hyphenationmode "FFFFFFF xxx\emdash xxx \par} +{\hsize 1mm \hccode"2014 "2014\relax \hyphenationmode "FFFFFFF xxx\emdash xxx \par} + +{\hyphenationmode "0000000 xxx--xxx---xxx \par} +{\hyphenationmode "FFFFFFF xxx--xxx---xxx \par} +\stopbuffer + +\typebuffer + +Here we assign the code point because who knows what future extensions will +bring. As with the other codes you can also set them from \LUA. The feature is +experimental and might evolve when \CONTEXT\ users come up with reasonable +demands. + +\startpacked \getbuffer \stoppacked + +\stopsubsection + \stopsection \stopchapter diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex b/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex index 11e8b352d..cf8d2dca8 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex @@ -189,8 +189,8 @@ cast from string to number and vise versa as this can change in future versions. \index {locales} In stock \LUA, many things depend on the current locale. In \LUAMETATEX, we can't -do that, because it makes documents unportable. While \LUAMETATEX\ is running if -forces the following locale settings: +do that, because it makes documents non|-|portable. While \LUAMETATEX\ is running +if forces the following locale settings: \starttyping LC_CTYPE=C @@ -237,6 +237,42 @@ usage better. \stopsection +\startsection[title={Files}] + +\startsubsection[title={File syntax}] + +\topicindex {files+names} + +\LUAMETATEX\ will accept a braced argument as a file name: + +\starttyping +\input {plain} +\openin 0 {plain} +\stoptyping + +This allows for embedded spaces, without the need for double quotes. Macro +expansion takes place inside the argument. Keep in mind that as side effect of +delegating \IO\ to \LUA\ the \tex {openin} primitive is not provided by the +engine and has to be implemented by the macro package. This also means that the +limit on the number of open files is not enforced by the engine. + +\stopsubsection + +\startsubsection[title={Writing to file}] + +\topicindex {files+writing} + +Writing to a file in \TEX\ has two forms: delayed and immediate. Delayed writing +means that the to be written text is anchored in the node list and flushed by the +backend. As all \IO\ is delegated to \LUA, this also means that it has to deal +with distinction. In \LUATEX\ the number of open files was already bumped to 127, +but in \LUAMETATEX\ it depends on the macro package. The special meaning of +channel 18 was already dropped in \LUATEX\ because we have \type {os.execute}. + +\stopsubsection + +\stopsection + \startsection[title={Testing}] \topicindex {testing} @@ -252,6 +288,243 @@ output file for instance. \stopsection +\startsection[reference=internals,title={The internals}] + +\topicindex{nodes} +\topicindex{boxes} +\topicindex{\LUA} + +This is a reference manual and not a tutorial. This means that we discuss changes +relative to traditional \TEX\ and also present new (or extended) functionality. +As a consequence we will refer to concepts that we assume to be known or that +might be explained later. Because the \LUATEX\ and \LUAMETATEX\ engines open up +\TEX\ there's suddenly quite some more to explain, especially about the way a (to +be) typeset stream moves through the machinery. However, discussing all that in +detail makes not much sense, because deep knowledge is only relevant for those +who write code not possible with regular \TEX\ and who are already familiar with +these internals (or willing to spend time on figuring it out). + +So, the average user doesn't need to know much about what is in this manual. For +instance fonts and languages are normally dealt with in the macro package that +you use. Messing around with node lists is also often not really needed at the +user level. If you do mess around, you'd better know what you're dealing with. +Reading \quotation {The \TEX\ Book} by Donald Knuth is a good investment of time +then also because it's good to know where it all started. A more summarizing +overview is given by \quotation {\TEX\ by Topic} by Victor Eijkhout. You might +want to peek in \quotation {The \ETEX\ manual} too. + +But \unknown\ if you're here because of \LUA, then all you need to know is that +you can call it from within a run. If you want to learn the language, just read +the well written \LUA\ book. The macro package that you use probably will provide +a few wrapper mechanisms but the basic \prm {directlua} command that does the job +is: + +\starttyping +\directlua{tex.print("Hi there")} +\stoptyping + +You can put code between curly braces but if it's a lot you can also put it in a +file and load that file with the usual \LUA\ commands. If you don't know what +this means, you definitely need to have a look at the \LUA\ book first. + +If you still decide to read on, then it's good to know what nodes are, so we do a +quick introduction here. If you input this text: + +\starttyping +Hi There ... +\stoptyping + +eventually we will get a linked lists of nodes, which in \ASCII\ art looks like: + +\starttyping +H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ... +\stoptyping + +When we have a paragraph, we actually get something like this, where a \type +{par} node stores some metadata and is followed by a \type {hlist} flagged as +indent box: + +\starttyping +[par] <=> [hlist] <=> H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ... +\stoptyping + +Each character becomes a so called glyph node, a record with properties like the +current font, the character code and the current language. Spaces become glue +nodes. There are many node types and nodes can have many properties but that will +be discussed later. Each node points back to a previous node or next node, given +that these exist. Sometimes multiple characters are represented by one glyph +(shape), so one can also get: + +\starttyping +[par] <=> [hlist] <=> H <=> i <=> [glue] <=> Th <=> e <=> r <=> e ... +\stoptyping + +And maybe some characters get positioned relative to each other, so we might +see: + +\starttyping +[par] <=> [hlist] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ... +\stoptyping + +Actually, the above representation is one view, because in \LUAMETATEX\ we can +choose for this: + +\starttyping +[par] <=> [glue] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ... +\stoptyping + +where glue (currently fixed) is used instead of an empty hlist (think of a \type +{\hbox}). Options like this are available because want a certain view on these +lists from the \LUA\ end and the result being predicable is part of that. + +It's also good to know beforehand that \TEX\ is basically centered around +creating paragraphs and pages. The par builder takes a list and breaks it into +lines. At some point horizontal blobs are wrapped into vertical ones. Lines are +so called boxes and can be separated by glue, penalties and more. The page +builder accumulates lines and when feasible triggers an output routine that will +take the list so far. Constructing the actual page is not part of \TEX\ but done +using primitives that permit manipulation of boxes. The result is handled back to +\TEX\ and flushed to a (often \PDF) file. + +\starttyping +\setbox\scratchbox\vbox\bgroup + line 1\par line 2 +\egroup + +\showbox\scratchbox +\stoptyping + +The above code produces the next log lines that reveal how the engines sees a +paragraph (wrapped in a \type {\vbox}): + +\starttyping[style=small] +1:4: > \box257= +1:4: \vbox[normal][16=1,17=1,47=1], width 483.69687, height 27.58083, depth 0.1416, direction l2r +1:4: .\list +1:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r +1:4: ...\list +1:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt +1:4: ....\glue[left][16=1,17=1,47=1] 0.0pt +1:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt +1:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519 +1:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt +1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l +1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i +1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n +1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e +1:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30 +1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000031 1 +1:4: ....\penalty[line][16=1,17=1,47=1] 10000 +1:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil +1:4: ....\glue[right][16=1,17=1,47=1] 0.0pt +1:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt +1:4: ..\glue[par][16=1,17=1,47=1] 5.44995pt plus 1.81665pt minus 1.81665pt +1:4: ..\glue[baseline][16=1,17=1,47=1] 6.79396pt +1:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r +1:4: ...\list +1:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt +1:4: ....\glue[left][16=1,17=1,47=1] 0.0pt +1:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt +1:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519 +1:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt +1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l +1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i +1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n +1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e +1:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30 +1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000032 2 +1:4: ....\penalty[line][16=1,17=1,47=1] 10000 +1:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil +1:4: ....\glue[right][16=1,17=1,47=1] 0.0pt +1:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt +\stoptyping + +The \LUATEX\ engine provides hooks for \LUA\ code at nearly every reasonable +point in the process: collecting content, hyphenating, applying font features, +breaking into lines, etc. This means that you can overload \TEX's natural +behaviour, which still is the benchmark. When we refer to \quote {callbacks} we +means these hooks. The \TEX\ engine itself is pretty well optimized but when you +kick in much \LUA\ code, you will notices that performance drops. Don't blame and +bother the authors with performance issues. In \CONTEXT\ over 50\% of the time +can be spent in \LUA, but so far we didn't get many complaints about efficiency. +Adding more callbacks makes no sense, also because at some point the performance +hit gets too large. There are plenty of ways to achieve goals. For that reason: +take remarks about \LUATEX, features, potential, performance etc.\ with a natural +grain of salt. + +Where plain \TEX\ is basically a basic framework for writing a specific style, +macro packages like \CONTEXT\ and \LATEX\ provide the user a whole lot of +additional tools to make documents look good. They hide the dirty details of font +management, language support, turning structure into typeset results, wrapping +pages, including images, and so on. You should be aware of the fact that when you +hook in your own code to manipulate lists, this can interfere with the macro +package that you use. Each successive step expects a certain result and if you +mess around to much, the engine eventually might bark and quit. It can even +crash, because testing everywhere for what users can do wrong is no real option. + +When you read about nodes in the following chapters it's good to keep in mind +what commands relate to them. Here are a few: + +\starttabulate[|l|l|p|] +\DB command \BC node \BC explanation \NC \NR +\TB +\NC \prm {hbox} \NC \nod {hlist} \NC horizontal box \NC \NR +\NC \prm {vbox} \NC \nod {vlist} \NC vertical box with the baseline at the bottom \NC \NR +\NC \prm {vtop} \NC \nod {vlist} \NC vertical box with the baseline at the top \NC \NR +\NC \prm {hskip} \NC \nod {glue} \NC horizontal skip with optional stretch and shrink \NC \NR +\NC \prm {vskip} \NC \nod {glue} \NC vertical skip with optional stretch and shrink \NC \NR +\NC \prm {kern} \NC \nod {kern} \NC horizontal or vertical fixed skip \NC \NR +\NC \prm {discretionary} \NC \nod {disc} \NC hyphenation point (pre, post, replace) \NC \NR +\NC \prm {char} \NC \nod {glyph} \NC a character \NC \NR +\NC \prm {hrule} \NC \nod {rule} \NC a horizontal rule \NC \NR +\NC \prm {vrule} \NC \nod {rule} \NC a vertical rule \NC \NR +\NC \prm {textdirection} \NC \nod {dir} \NC a change in text direction \NC \NR +\LL +\stoptabulate + +Whatever we feed into \TEX\ at some point becomes a token which is either +interpreted directly or stored in a linked list. A token is just a number that +encodes a specific command (operator) and some value (operand) that further +specifies what that command is supposed to do. In addition to an interface to +nodes, there is an interface to tokens, as later chapters will demonstrate. + +Text (interspersed with macros) comes from an input medium. This can be a file, +token list, macro body cq.\ arguments, some internal quantity (like a number), +\LUA, etc. Macros get expanded. In the process \TEX\ can enter a group. Inside +the group, changes to registers get saved on a stack, and restored after leaving +the group. When conditionals are encountered, another kind of nesting happens, +and again there is a stack involved. Tokens, expansion, stacks, input levels are +all terms used in the next chapters. Don't worry, they loose their magic once you +use \TEX\ a lot. You have access to most of the internals and when not, at least +it is possible to query some state we're in or level we're at. + +When we talk about pack(ag)ing it can mean two things. When \TEX\ has consumed +some tokens that represent text they are added to the current list. When the text +is put into a so called \prm {hbox} (for instance a line in a paragraph) it +(normally) first gets hyphenated, next ligatures are build, and finally kerns are +added. Each of these stages can be overloaded using \LUA\ code. When these three +stages are finished, the dimension of the content is calculated and the box gets +its width, height and depth. What happens with the box depends on what macros do +with it. + +The other thing that can happen is that the text starts a new paragraph. In that +case some information is stored in a leading \type {par} node. Then indentation +is appended and the paragraph ends with some glue. Again the three stages are +applied but this time afterwards, the long line is broken into lines and the +result is either added to the content of a box or to the main vertical list (the +running text so to say). This is called par building. At some point \TEX\ decides +that enough is enough and it will trigger the page builder. So, building is +another concept we will encounter. Another example of a builder is the one that +turns an intermediate math list into something typeset. + +Wrapping something in a box is called packing. Adding something to a list is +described in terms of contributing. The more complicated processes are wrapped +into builders. For now this should be enough to enable you to understand the next +chapters. The text is not as enlightening and entertaining as Don Knuths books, +sorry. + +\stopsection + \stopchapter \stopcomponent diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-math.tex b/doc/context/sources/general/manuals/luametatex/luametatex-math.tex index 9a9fb9eb1..b114dbba3 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex-math.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex-math.tex @@ -848,6 +848,12 @@ feature. These additional commands are not all that valuable on their own, but they come in handy as arguments to the math parameter settings that will be added shortly. +Because internally the eight styles are represented as numbers some of the new +primnitives that relate to them also work with numbers and often you can use them +mixed. The \prm {tomathstyle} prefix converts a symbolic style into a number so +\typ {\number \tomathstyle \crampedscriptstyle} gives~\number \tomathstyle +\crampedscriptstyle. + In Eijkhouts \quotation {\TEX\ by Topic} the rules for handling styles in scripts are described as follows: @@ -1004,19 +1010,18 @@ undefined will get presets, quite likely zero): \DB primitive name \BC description \NC \NR \TB \NC \prm {Umathconnectoroverlapmin} \NC \NC \NR -\NC \prm {Umathsubscriptsuperscriptshiftdown} \NC \NC \NR -\NC \prm {Umathfractiondelsize} \NC \NC \NR +\NC \prm {Umathsubsupshiftdown} \NC \NC \NR \NC \prm {Umathfractiondelsize} \NC \NC \NR \NC \prm {Umathnolimitsupfactor} \NC a multiplier for the way limits are shifted up and down \NC \NR \NC \prm {Umathnolimitsubfactor} \NC a multiplier for the way limits are shifted up and down \NC \NR -\NC \prm {Umathaccentbasedepth} \NC the complement of \prm {UmathAccentBaseHeight} \NC \NR -\NC \prm {Umathflattenedaccentbasedepth} \NC the complement of \prm {UmathFlattenedAccentBaseHeight} \NC \NR +\NC \prm {Umathaccentbasedepth} \NC the complement of \prm {Umathaccentbaseheight} \NC \NR +\NC \prm {Umathflattenedaccentbasedepth} \NC the complement of \prm {Umathflattenedaccentbaseheight} \NC \NR \NC \prm {Umathspacebeforescript} \NC \NC \NR \NC \prm {Umathraisepercent} \NC the percentage that the vertical position is scaled \NC \NR -\NC \prm {Umathprimeshiftup} \NC the prime variant of \prm {SuperscriptShiftUp} \NC \NR -\NC \prm {Umathprimeshiftupcramped} \NC the prime variant of \prm {SuperscriptShiftUpCramped} \NC \NR -\NC \prm {Umathprimespaceafter} \NC the prescript variant of \prm {UmathSpaceAfterScript} \NC \NR -\NC \prm {Umathprimebaselinedropmax} \NC the prime variant of \prm {SuperscriptBaselineDropMax} \NC \NR +\NC \prm {Umathprimeshiftup} \NC the prime variant of \prm {Umathsupshiftup} \NC \NR +\NC \prm {Umathprimeshiftupcramped} \NC the prime variant of \prm {Umathsupshiftup} \NC \NR +\NC \prm {Umathprimespaceafter} \NC the prescript variant of \prm {Umathspaceafterscript} \NC \NR +\NC \prm {Umathprimebaselinedropmax} \NC the prime variant of \prm {Umathsupshiftdrop} \NC \NR \NC \prm {Umathprimewidthpercent} \NC the percentage of width that gets added \NC \NR \NC \prm {Umathskeweddelimitertolerance} \NC \NC \NR \NC \prm {Umathaccenttopshiftup} \NC the amount that a top accent is shifted up \NC \NR @@ -1038,7 +1043,7 @@ defaults and tweak them in the goodie files: \DB font parameter \BC primitive name \BC default \NC \NR \TB \NC MinConnectorOverlap \NC \prm {Umathconnectoroverlapmin} \NC 0 \NC \NR -\NC SubscriptShiftDownWithSuperscript \NC \prm {Umathsubscriptsuperscriptshiftdown} \NC inherited \NC \NR +\NC SubscriptShiftDownWithSuperscript \NC \prm {Umathsubsupshiftdown} \NC inherited \NC \NR \NC FractionDelimiterSize \NC \prm {Umathfractiondelsize} \NC undefined \NC \NR \NC FractionDelimiterDisplayStyleSize \NC \prm {Umathfractiondelsize} \NC undefined \NC \NR \NC NoLimitSubFactor \NC \prm {Umathnolimitsupfactor} \NC 0 \NC \NR @@ -1276,6 +1281,16 @@ As this mostly refers to \LUATEX\ there is more to tell about how \LUAMETATEX\ deals with it. However, it is enough to know that much more behavior is configurable. +You can let the engine ignore parameter with \prm {setmathignore}, like: + +\starttyping +\setmathignore \Umathspacebeforescript 1 +\setmathignore \Umathspaceafterscript 1 +\stoptyping + +Be aware of the fact that a global setting can get unnoticed by users because +there is no warning that some parameter is ignored. + \stopsubsection \stopsection @@ -1565,6 +1580,23 @@ keywords: To what extend the options kick in depends on the class as well where and how the atom is used. +The original \TEX\ engines has three atom modifiers: \prm {displaylimits}, \prm +{limits}, and \prm {nolimits}. These look back to the last atom and set a limit +related signal. Just to be consistent we have some more of that: \prm +{Umathadapttoleft}, \prm {Umathadapttoright}, \prm {Umathuseaxis}, \prm +{Umathnoaxis}, \prm {Umathphantom}, \prm {Umathvoid}, \prm {Umathsource}, \prm +{Umathopenupheight}, \prm {Umathopenupdepth}, \prm {Umathlimits}, \prm +{Umathnolimits}. The last two are equivalent to the lowercase ones with the +similar names. Al these modifiers are cheap primitives and one can wonder if they +are needed but that also now also applies to the original three. We could stick +to one modifier that takes an integer but let's not diverge too much from the +original concept. + +The \prm {nonscript} primitive injects a glue node that signals that the next +glue is to be ignored when we are in script or scriptscript mode. The \prm +{noatomruling} does the same but this time the signal is that no inter|-|atom +rules need to be applied. + \stopsubsection \startsubsection[title={Checking a state with \prm {ifmathparameter}}] @@ -1701,6 +1733,10 @@ In typeset form this looks like: \getbuffer +Normally fences need to be matched, that is: when a left fence is seen, there has +to be a right fence. When you set \prm {mathcheckfencesmode} to non|-|zero the +scanner silently recovers from this. + \stopsubsection \startsubsection[title={Accent handling with \prm {Umathaccent}},reference=mathacc] @@ -1963,6 +1999,10 @@ For Latin Modern the result looks as follows: \stoptabulate \stop +For good old times we also have \prm {skewed} and \prm {skewedwithdelims} +although it is unlikely that one will notice as it only makes sense in the +updated approach. + The \type {\over} and related primitives have the form: \starttyping @@ -2040,7 +2080,7 @@ keeping the spacing compatible. \stopsubsection -\startsubsection[title={Delimiters: \type{\Uleft}, \prm {Umiddle} and \prm {Uright}}] +\startsubsection[title={Delimiters: \prm {Uleft}, \prm {Umiddle} and \prm {Uright}}] \topicindex {math+delimiters} @@ -2273,11 +2313,13 @@ $} \typebuffer -The question is: are these double super and subscript triggers the way to go? -Anyway, you need to have them either being active (which in \CONTEXT\ then boils -down to them being other characters), or say \type {\supmarkmode = 1} to disable -the normal multiple \type {^} treatment (a value larger than 1 will also disable -that in text mode). +The problem with the circumflex is that it is also used for escaping character +input. Normally that only happens in a format file so you can safely disable +that. Alternatives are using active characters that adapt. In \CONTEXT\ we make +them regular (other) characters in text mode and set \prm {supmarkmode} to~1 to +disable the normal multiple \type {^} treatment (a value larger than 1 will also +disable that in text mode). In math mode we make them active and behave as +expected. \blank \getbuffer \blank @@ -2450,6 +2492,15 @@ and get: \inlinebuffer. \stopsubsection +\startsubsection[title={Default math codes}] + +The probably not that useful primitive (but who knows) \prm {setdefaultmathcodes} +initializes the mathcodes of digits to family zero and the lowercase and +uppercase letters to family one, just as standard \TEX\ does. Don't do this in in +\CONTEXT. + +\stopsubsection + \stopsection \startsection[title={Modes}] @@ -2630,7 +2681,7 @@ experimenting. \stopsubsection -\startsubsection[title={Some spacing control with \prm {mathsurroundmode} and \prm {mathspacingmode}}] +\startsubsection[title={Some spacing control with \prm {mathsurroundmode}, \prm {mathspacingmode} and \prm {mathgluemode}}] See \in {section} [spacing:surround] for more about inline spacing. The \prm {mathsurroundmode} parameter just permits the glue variant to kick in and indeed @@ -2640,12 +2691,17 @@ The \prm {mathspacingmode} parameter is for tracing: normally zero inter atom glue is not injected but when this parameter is set to non|-|zero even zero spacing will show up. This permits us to check the applied inter atom spacing. -% \mathcheckfencesmode -% \mathpenaltiesmode -% \mathscriptboxmode -% \mathrulesmode - % \supmarkmode -% \mathgluemode +The \prm {mathgluemode} bitset controls if glue can stretch and|/|or shrink. It +is used in some of the upgraded \CONTEXT\ high level math alignment command so +probably more qualifies as a feature specific for that usage. + +\starttabulate[|c|l|] +\DB value \BC meaning \NC \NR +\TB +\NC 0x01 \NC obey stretch component \NC \NR +\NC 0x02 \NC obey shrink component \NC \NR +\LL +\stoptabulate \stopsection @@ -2677,6 +2733,26 @@ effect and use a 20\percent\ scaling: \getbuffer +\startsubsection[title={Scaling with \prm {scaledmathstyle}}] + +Because styles put a style switching node in the stream we have a +scaling primitive that uses such a style node to signal dynamic +scaling. Thisis still somewhat experimental. + +\startbuffer +$ + {\scaledmathstyle 500 x + x}\quad + {\scaledmathstyle 1000 x + x}\quad + {\scaledmathstyle 1500 x + x} +$ +\stopbuffer + +\typebuffer + +You get differently sized math but of course you then probably also need to +handle spacing differently, although for small deviations from 1000 is should +come out acceptable. + \stopsubsection \startsubsection[title={Spreading math with \prm {maththreshold}}] @@ -2688,6 +2764,67 @@ calculations. The \prm {maththreshold} primitive is a regular glue parameter. \stopsubsection +\startsubsection[title={\prm {everymathatom} and \prm {lastatomclass}}] + +Just for completeness we have \prm {everymathatom} as companion for \prm {everyhbox} +and friends and it is probably just as useful. The next example shows how it works: + +\startbuffer +\everymathatom + {\begingroup + \scratchcounter\lastatomclass + \everymathatom{}% + \mathghost{\hbox to 0pt yoffset -1ex{\smallinfofont \setstrut\strut \the\scratchcounter\hss}}% + \endgroup} + +$ a = \mathatom class 4 {b} + \mathatom class 5 {c} $ +\stopbuffer + +\typebuffer + +We get a formula with open- and close atom spacing applied to~$b$ and~$c$: + +{\getbuffer} + +This example shows bit of all: we want the number to be invisible to the math +machinery so we ghost it. So, we need to make sure we don't get recursion due to +nested injection and expansion of \prm {everymathatom} and of course we need to +store the number. The \prm {lastatomclass} state variable is only meaningful +inside an explicit atom wrapper like \prm {mathatom} and \prm {mathatom}. + +\stopsubsection + +\startsubsection[title={\prm {postinlinepenalty} and \prm {preinlinepenalty}}] + +In horizontal lists math is embedded in a begin and end math node. These nodes +also carry information about the surrounding space, and the in \LUAMETATEX\ +optional glue. We also store a penalty so that we can let that play a role in the +decisions to be made; these two internal integer registers control this. Just +like the mentioned spacing they are not visible as nodes in the list. + +\stopsubsection + +\startsubsection[title={\prm {mathforwardpenalties} and \prm {mathbackwardpenalties}}] + +These penalties are experimental and deltas added to the regular penalties +between atoms. Here is an example, as with other primitives that take more +arguments the first number indicates how much follows. + +\startbuffer +$ a + b + c + d + e + f + g + h = x $\par +\mathforwardpenalties 3 300 200 100 +\mathbackwardpenalties 3 250 150 50 +$ a + b + c + d + e + f + g + h = x $\par +\stopbuffer + +\typebuffer + +You'll notice that we apply more severe penalties at the edges: + +{\showmakeup[penalty]\multiply\glyphscale\plustwo \getbuffer} + +\stopsubsection + \stopsection \stopchapter diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex b/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex index 054eec421..3122a5431 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex @@ -496,13 +496,13 @@ When it became unavoidable that we output more detail, it also became clear that it made no sense to stay log and trace compatible. Some is controlled by parameters in order to stay close the original, but \CONTEXT\ is configured such that we benefit from the new possibilities. Examples are that in addition to -\type {\meaning} we have \type {\meaningfull} that also exposes macro properties, -and \type {\meaningless} that only exposes the body. The \type {\untraced} prefix -will suppress some in the log, and we set \type {\tracinglevels} to 3 in order to +\prm {meaning} we have \prm {meaningfull} that also exposes macro properties, +and \prm {meaningless} that only exposes the body. The \prm {untraced} prefix +will suppress some in the log, and we set \prm {tracinglevels} to 3 in order to get details about the input and grouping level. When there's less shown than expected keep in mind that \LUAMETATEX\ has a somewhat optimized saving and restoring of meanings so less can happen which is reflected in tracing. When node -lists are serialized (as with \type {\showbox}) some nodes, like discretionaries +lists are serialized (as with \prm {showbox}) some nodes, like discretionaries report more detail. The compact serializer, used for instance to signal overfull boxes, also shows a bit more detail with respect to non|-|content nodes. I math more is shown if only because we have more control and additional mechanisms. @@ -511,7 +511,7 @@ more is shown if only because we have more control and additional mechanisms. \startsubsection[title=Parsing] -Token parsers have been upgraded for the sake of \LUA, \type {\csname} handling +Token parsers have been upgraded for the sake of \LUA, \prm {csname} handling has been extended, macro definitions can be more flexible so there code was adapted, more conditionals also brought some changes. But we build upon the (reorganized) \TEX\ foundation so the basics can definitely be recognized. @@ -525,10 +525,10 @@ The reworked \LUAMETATEX\ engine is substantially faster than the \LUATEX\ predecessor. The handling of conditionals has been adapted so that we can have flatter -branches (\type {\orelse} cum suis). This again has some consequences for +branches (\prm {orelse} cum suis). This again has some consequences for parsing. Because parsing alignments is rather interwoven in general parsing and expansion the handling of related primitives has been slightly adapted (also for -the sake of \LUA\ interfacing) and dealing with \type {\noalign} situations is a +the sake of \LUA\ interfacing) and dealing with \prm {noalign} situations is a bit more convenient. This are just a few of the adaptations and most of this happened stepwise with @@ -548,6 +548,419 @@ errors are still errors. \stopsubsection +\startsection[reference=differences,title={Differences with \LUATEX}] + +\startsubsection[title=Dropped primitives] + +As \LUAMETATEX\ is a leaner and meaner \LUATEX. This means that substantial parts and +dependencies are gone: quite some font code, all backend code with related frontend +code and of course image and font inclusion. There is also new functionality which +makes for less lean but in the end we still have less, also in terms of dependencies. +This chapter will discuss what is gone. We start with the primitives that were dropped. + +\starttabulate[|l|pl|] +\BC fonts \NC \type {\letterspacefont} + \type {\copyfont} + \type {\expandglyphsinfont} + \type {\ignoreligaturesinfont} + \type {\tagcode} + \type {\leftghost} + \type {\rightghost} + \NC \NR +\BC backend \NC \type {\dviextension} + \type {\dvivariable } + \type {\dvifeedback} + \type {\pdfextension} + \type {\pdfvariable } + \type {\pdffeedback} + \type {\dviextension} + \type {\draftmode} + \type {\outputmode} + \NC \NR +\BC dimensions \NC \type {\pageleftoffset} + \type {\pagerightoffset} + \type {\pagetopoffset} + \type {\pagebottomoffset} + \type {\pageheight} + \type {\pagewidth} + \NC \NR +\BC resources \NC \type {\saveboxresource} + \type {\useboxresource} + \type {\lastsavedboxresourceindex} + \type {\saveimageresource} + \type {\useimageresource} + \type {\lastsavedimageresourceindex} + \type {\lastsavedimageresourcepages} + \NC \NR +\BC positioning \NC \type {\savepos} + \type {\lastxpos} + \type {\lastypos} + \NC \NR +\BC directions \NC \type {\textdir} + \type {\linedir} + \type {\mathdir} + \type {\pardir} + \type {\pagedir} + \type {\bodydir} + \type {\pagedirection} + \type {\bodydirection} + \NC \NR +\BC randomizer \NC \type {\randomseed} + \type {\setrandomseed} + \type {\normaldeviate} + \type {\uniformdeviate} + \NC \NR +\BC utilities \NC \type {\synctex} + \NC \NR +\BC extensions \NC \type {\latelua} + \type {\lateluafunction} + \type {\openout} + \type {\write} + \type {\closeout} + \type {\openin} + \type {\read} + \type {\readline} + \type {\closein} + \type {\ifeof} + \NC \NR +\BC control \NC \type {\suppressfontnotfounderror} + \type {\suppresslongerror} + \type {\suppressprimitiveerror} + \type {\suppressmathparerror} + \type {\suppressifcsnameerror} + \type {\suppressoutererror} + \type {\mathoption} + \NC \NR +\BC system \NC \type {\primitive} + \type {\ifprimitive} + \type {\formatname} + \NC \NR +\BC ignored \NC \type {\long} + \type {\outer} + \type {\mag} + \NC \NR +\stoptabulate + +The math machinery has been overhauled stepwise. In the process detailed control +has been added but later some of that got removed or replaced. The engine now +assumes that \OPENTYPE\ fonts are used but you do need to set up the engine +properly, something that has to be done with respect to fonts anyway. By enabling +and|/|disabling certain features you can emulate the traditional engine. Font +parameters no longer are taken from the traditional parameters when they are not +set. We just assume properly passed so called math constants and quite a few new +ones have been added. + +The resources and positioning primitives are actually useful but can be defined +as macros that (via \LUA) inject nodes in the input that suit the macro package +and backend. The three||letter direction primitives are gone and the numeric +variants are now leading. There is no need for page and body related directions +and they don't work well in \LUATEX\ anyway. We only have two directions left. +Because we can hook in \LUA\ functions that get information about what is expected +(consumer or provider) there are plenty possibilities for adding functionality +using this scripting language. + +The primitive related extensions were not that useful and reliable so they have +been removed. There are some new variants that will be discussed later. The \prm +{outer} and \prm {long} prefixes are gone as they don't make much sense +nowadays and them becoming dummies opened the way to something new: control +sequence properties that permit protection against as well as controlled +overloading of definitions. I don't think that (\CONTEXT) users will notice these +prefixes being gone. The definition and parsing related \type {\suppress..} +features are now default and can't be changed so related primitives are gone. + +The \prm {shipout} primitive does no ship out but just erases the content of +the box unless of course that has happened already in another way. A macro +package should implement its own backend and related shipout. Talking of backend, +the extension primitives that relate to backends can be implemented as part of a +backend design using generic whatsits. There is only one type of whatsit now. In +fact we're now closer to original \TEX\ with respect to the extensions. + +The \type {img} library has been removed as it's rather bound to the backend. The +\type {slunicode} library is also gone. There are some helpers in the string +library that can be used instead and one can write additional \LUA\ code if +needed. There is no longer a \type {pdf} backend library but we have an up to +date \PDF\ parsing library on board. + +In the \type {node}, \type {tex} and \type {status} library we no longer have +helpers and variables that relate to the backend. The \LUAMETATEX\ engine is in +principle \DVI\ and \PDF\ unaware. There are, as mentioned, only generic whatsit +nodes that can be used for some management related tasks. For instance you can +use them to implement user nodes. More extensive status information is provided +in the overhauled status library. All libraries have additional functionality and +names of functions have been normalized (for as far as possible). + +The margin kern nodes are gone and we now use regular kern nodes for them. As a +consequence there are two extra subtypes indicating the injected left or right +kern. The glyph field served no real purpose so there was no reason for a special +kind of node. + +The \KPSE\ library is no longer built|-|in, but one can use an external \KPSE\ +library, assuming that it is present on the system, because the engine has a so +called optional library interface to it. Because there is no backend, quite some +file related callbacks could go away. The following file related callbacks +remained (till now): + +\starttyping +find_write_file find_format_file open_data_file +\stoptyping + +The callbacks related to errors are changed: + +\starttyping +intercept_tex_error intercept_lua_error +show_error_message show_warning_message +\stoptyping + +There is a hook that gets called when one of the fundamental memory structures +gets reallocated. + +\starttyping +trace_memory +\stoptyping + +When you use the overload protect mechanisms, a callback can be plugged in to handle +exceptions: + +\starttyping +handle_overload +\stoptyping + +The (job) management hooks are kept: + +\starttyping +process_jobname +start_run stop_run wrapup_run +pre_dump +start_file stop_file +\stoptyping + +Because we use a more generic whatsit model, there is a new callback: + +\starttyping +show_whatsit +\stoptyping + +Because tracing boxes now reports a lot more information, we have a plug in for +detail: + +\starttyping +get_attribute +\stoptyping + +Being the core of extensibility, the typesetting callbacks of course stayed. This +is what we ended up with: + +\startalign[flushleft,nothyphenated] +\tt \cldcontext{table.concat(table.sortedkeys(callbacks.list), ", ")} +\stopalign + +As in \LUATEX\ font loading happens with the following callback. This time it +really needs to be set because there is no built|-|in font loader. + +\starttyping +define_font +\stoptyping + +There are all kinds of subtle differences in the implementation, for instance we +no longer intercept \type {*} and \type {&} as these were already replaced long +ago in \TEX\ engines by command line options. Talking of options, only a few are +left. All input goes via \LUA, even the console. One can program a terminal if +needed. + +We took our time for reaching a stable state in \LUATEX. Among the reasons is the +fact that most was experimented with in \CONTEXT, which we can adapt to the +engine as we go. It took many years to decide what to keep and how to do things. +Of course there are places when things can be improved but that most likely only +happens in \LUAMETATEX. Contrary to what is sometimes suggested, the +\LUATEX|-|\CONTEXT\ \MKIV\ combination (assuming matched versions) has been quite +stable. It made no sense otherwise. Most \CONTEXT\ functionality didn't change +much at the user level. Of course there have been issues, as is natural with +everything new and beta, but we have a fast update cycle. + +The same is true for \LUAMETATEX\ and \CONTEXT\ \LMTX: it can be used for +production as usual and in practice \CONTEXT\ users tend to use the beta +releases, which proves this. Of course, if you use low level features that are +experimental you're on your own. Also, as with \LUATEX\ it might take many years +before a long term stable is defined. The good news is that, when the source code +has become part of the \CONTEXT\ distribution, there is always a properly +working, more or less long term stable, snapshot. + +The error reporting subsystem has been redone quite a bit but is still +fundamentally the same. We don't really assume interactive usage but if someone +uses it, it might be noticed that it is not possible to backtrack or inject +something. Of course it is no big deal to implement all that in \LUA\ if needed. +It removes a system dependency and makes for a bit cleaner code. In \CONTEXT\ we +quit on an error simply because one has to fix source anyway and runs are fast +enough. Logging provides more detail and new primitives can be used to prevent +clutter in tracing (the more complex a macro package becomes, the more extreme +tracing becomes). + +\stopsubsection + +\startsubsection[title=New primitives] + +There are new primitives as well as some extensions to existing primitive +functionality. These are described in following chapters but there might be +hidden treasures in the binary. If you locate them, don't automatically assume +them to stay, some might be part of experiments! There are for instance a few +csname related definers, we have integer and dimension constants, the macro +argument parser can be brought in tolerant mode, the repertoire of conditionals +has been extended, some internals can be controlled (think of normalization of +lines, hyphenation etc.), and macros can be protected against user overload. Not +all is discussed in detail in this manual but there are introductions in the +\CONTEXT\ distribution that explain them. But the \TEX\ kernel is of course +omnipresent. + +\startluacode + +local luametatex = tex.primitives() +local luatex = table.load("luatex-primitives.lua") + +if not luatex then + local tex = "\\starttext \\ctxlua {table.save(tex.jobname .. '.lua',tex.primitives())} \\stoptext" + + io.savedata("luatex-primitives.tex", tex) + + os.execute("context --luatex --once luatex-primitives") + + luatex = table.load("luatex-primitives.lua") +end + + +if luatex and luametatex then + + local match = string.match + + local found = { } + + local function collect(index) + if index then + local data = index.entries + for i=1,#data do + found[match(data[i].list[1][1],"\\tex%s*{(.-)}") or ""] = true + end + -- inspect(found) + end + end + + collect(structures.registers.collected and structures.registers.collected.texindex) + collect(structures.registers.collected and structures.registers.collected.luatexindex) + + luatex = table.tohash(luatex) + luametatex = table.tohash(luametatex) + + -- context.page() + + context("The following primitives are available in \\LUATEX\\ but not in \\LUAMETATEX. ") + context("Some of these are emulated in \\CONTEXT.") + + context.blank() + context.startcolumns { n = 2 } + for k, v in table.sortedhash(luatex) do + if not luametatex[k] then + if not found[k] then + context.dontleavehmode() + end + context.type(k) + context.crlf() + end + end + context.stopcolumns() + context.blank() + + -- context.page() + + context("The following primitives are available in \\LUAMETATEX\\ only. In the meantime ") + context("the \\LUAMETATEX\\ code base is so different from \\LUATEX\\ that backporting ") + context("is no longer reasonable.") + + context.blank() + context.startcolumns { n = 2 } + for k, v in table.sortedhash(luametatex) do + if not luatex[k] then + if not found[k] then + context.dontleavehmode() + context.llap("\\infofont[todo] ") + end + context.type(k) + context.crlf() + end + end + context.stopcolumns() + context.blank() + +end + +\stopluacode + +\stopsubsection + +\startsubsection[title=Changed function names] + +As part of a bit more consistency some function names also changed. Names with an +\type {_} got that removed (as that was the minority). It's easy to provide a +back mapping if needed (just alias the functions). + +{\em Todo: only mention the \LUATEX\ ones.} + +\starttabulate[|l|l|l|l|] +\DB library \BC old name \BC new name \BC comment \NC \NR +\TB +\NC language \NC clear_patterns \NC clearpatterns \NC \NR +\NC \NC clear_hyphenation \NC clearhyphenation \NC \NR +\NC mplib \NC italcor \NC italic \NC \NR +\NC \NC pen_info \NC peninfo \NC \NR +\NC \NC solve_path \NC solvepath \NC \NR +\NC texio \NC write_nl \NC writenl \NC old name stays \NC \NR +\NC node \NC protect_glyph \NC protectglyph \NC \NR +\NC \NC protect_glyphs \NC protectglyphs \NC \NR +\NC \NC unprotect_glyph \NC unprotectglyph \NC \NR +\NC \NC unprotect_glyphs \NC unprotectglyphs \NC \NR +\NC \NC end_of_math \NC endofmath \NC \NR +\NC \NC mlist_to_hlist \NC mlisttohlist \NC \NR +\NC \NC effective_glue \NC effectiveglue \NC \NR +\NC \NC has_glyph \NC hasglyph \NC \NR +\NC \NC first_glyph \NC firstglyph \NC \NR +\NC \NC has_field \NC hasfield \NC \NR +\NC \NC copy_list \NC copylist \NC \NR +\NC \NC flush_node \NC flushnode \NC \NR +\NC \NC flush_list \NC flushlist \NC \NR +\NC \NC insert_before \NC insertbefore \NC \NR +\NC \NC insert_after \NC insertafter \NC \NR +\NC \NC last_node \NC lastnode \NC \NR +\NC \NC is_zero_glue \NC iszeroglue \NC \NR +\NC \NC make_extensible \NC makeextensible \NC \NR +\NC \NC uses_font \NC usesfont \NC \NR +\NC \NC is_char \NC ischar \NC \NR +\NC \NC is_direct \NC isdirect \NC \NR +\NC \NC is_glyph \NC isglyph \NC \NR +\NC \NC is_node \NC isnode \NC \NR +\NC token \NC scan_keyword \NC scankeyword \NC \NR +\NC \NC scan_keywordcs \NC scankeywordcs \NC \NR +\NC \NC scan_int \NC scanint \NC \NR +\NC \NC scan_real \NC scanreal \NC \NR +\NC \NC scan_float \NC scanfloat \NC \NR +\NC \NC scan_dimen \NC scandimen \NC \NR +\NC \NC scan_glue \NC scanglue \NC \NR +\NC \NC scan_toks \NC scantoks \NC \NR +\NC \NC scan_code \NC scancode \NC \NR +\NC \NC scan_string \NC scanstring \NC \NR +\NC \NC scan_argument \NC scanargument \NC \NR +\NC \NC scan_word \NC scanword \NC \NR +\NC \NC scan_csname \NC scancsname \NC \NR +\NC \NC scan_list \NC scanlist \NC \NR +\NC \NC scan_box \NC scanbox \NC \NR +\LL +\stoptabulate + +It's all part of trying to make the code base consistent but it is sometimes a +bit annoying. However, that's why we develop this engine independent of the +\LUATEX\ code base. It's anyway a change that has been on my todo list for quite +a while because those inconsistencies annoyed me. It might take some years to +get all done. + +\stopsubsection + \stopsection \stopchapter diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-registers.tex b/doc/context/sources/general/manuals/luametatex/luametatex-registers.tex index 931a57efe..a4ef2cb71 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex-registers.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex-registers.tex @@ -16,7 +16,7 @@ are of course many more primitives. The \ETEX\ and \LUATEX\ primitives are typeset in bold (some originate in \PDFTEX). -% \placeregister[primitiveindex][indicator=no] + %\placeregister[primitiveindex][indicator=no] \placeregister[texindex,etexindex,luatexindex][indicator=no] \stopchapter diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-style.tex b/doc/context/sources/general/manuals/luametatex/luametatex-style.tex index e60b6977d..6586e6fc1 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex-style.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex-style.tex @@ -117,11 +117,12 @@ \stopluacode -\protected\def\showfields #1{\ctxlua{document.functions.showfields("#1")}} -\protected\def\showid #1{\ctxlua{document.functions.showid("#1")}} -\protected\def\showsubtypes#1{\ctxlua{document.functions.showsubtypes("#1")}} -\protected\def\showvalues #1{\ctxlua{document.functions.showvalues(node.values("#1"))}} -\protected\def\showtypes {\ctxlua{document.functions.showvalues(node.types())}} +\protected\def\showfields #1{\ctxlua{document.functions.showfields("#1")}} +\protected\def\showid #1{\ctxlua{document.functions.showid("#1")}} +\protected\def\showsubtypes #1{\ctxlua{document.functions.showsubtypes("#1")}} +\protected\def\showvalues #1{\ctxlua{document.functions.showvalues(node.values("#1"))}} +\protected\def\showtypes {\ctxlua{document.functions.showvalues(node.types())}} +\protected\def\showvaluelist#1{\ctxlua{document.functions.showvalues(#1)}} \definecolor[blue] [b=.5] @@ -387,12 +388,21 @@ \setuplist [chapter] [style=bold, - before={\testpage[4]\blank}, color=keptcolor] \setuplist - [section] - [before={\testpage[3]}] + [chapter] + [before={\testpage[4]\blank}, +% after={\blank[none,samepage]}] + after={\blank[samepage]}] + +% \setuplist +% [section] +% [before={\testpage[3]}] % \ifnum \structurelistrawnumber{section} > 2 ... \fi + +\setuplist + [section,subsection] + [before={\ifnum \structurelistrawnumber{section}=1 \blank[samepage] \fi}] \setuplist [subsection,subsubsection] diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-tex.tex b/doc/context/sources/general/manuals/luametatex/luametatex-tex.tex index 089a8202d..837cc6e01 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex-tex.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex-tex.tex @@ -4,16 +4,6 @@ \startcomponent luametatex-tex -% { "getbytecode", lualib_get_bytecode }, -% { "setbytecode", lualib_set_bytecode }, -% { "callbytecode", lualib_call_bytecode }, -% { "getfunctionstable", lualib_get_functions_table }, -% { "getstartupfile", lualib_get_startupfile }, -% { "getversion", lualib_get_version }, -% { "setexitcode", lualib_set_exitcode }, -% { NULL, NULL }, - - \startchapter[reference=tex,title={The \TEX\ related libraries}] \startsection[title={The \type {lua} library}][library=lua] @@ -341,6 +331,8 @@ stack. \subsubsection{Integer parameters} +\topicindex{parameters+integer} + The integer parameters accept and return \LUA\ integers. In some cases the values are checked, trigger other settings or result in some immediate change of behaviour: \ctxlua {document.filteredprimitives ("internal_int")}. @@ -353,6 +345,8 @@ some internal integer register but to an engine property: \typ {deadcycles}, \subsubsection{Dimension parameters} +\topicindex{parameters+dimension} + The dimension parameters accept \LUA\ numbers (signifying scaled points) or strings (with included dimension). The result is always a number in scaled points. These are read|-|write: \ctxlua {document.filteredprimitives @@ -364,6 +358,8 @@ These are read|-|only: \typ {pagedepth}, \typ {pagefilllstretch}, \typ \subsubsection{Direction parameters} +\topicindex{parameters+direction} + The direction states can be queried with: \typ {gettextdir}, \typ {getlinedir}, \typ {getmathdir} and \typ {getpardir}. You can set them with \typ {settextdir}, \typ {setlinedir}, \typ {setmathdir} and \typ {setpardir}, @@ -378,16 +374,22 @@ tex.textdirection = 1 \subsubsection{Glue parameters} +\topicindex{parameters+glue} + The internal glue parameters accept and return a userdata object that represents a \nod {glue_spec} node: \ctxlua {document.filteredprimitives ("internal_glue")}. \subsubsection{Muglue parameters} +\topicindex{parameters+muglue} + All muglue parameters are to be used read|-|only and return a \LUA\ string \ctxlua {document.filteredprimitives ("internal_mu_glue")}. \subsubsection{Tokenlist parameters} +\topicindex{parameters+tokens} + The tokenlist parameters accept and return \LUA\ strings. \LUA\ strings are converted to and from token lists using \prm {the} \prm {toks} style expansion: all category codes are either space (10) or other (12). It follows that assigning diff --git a/doc/context/sources/general/manuals/luametatex/luametatex.tex b/doc/context/sources/general/manuals/luametatex/luametatex.tex index f2becf0b7..bb4613dab 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex.tex @@ -138,8 +138,6 @@ \stopfrontmatter \startbodymatter - \component luametatex-internals - \component luametatex-differences \component luametatex-modifications \component luametatex-lua \component luametatex-enhancements @@ -154,11 +152,10 @@ \component luametatex-pdf \component luametatex-libraries \component luametatex-primitives % this generates a list - \component luametatex-rejected + % \component luametatex-rejected % local file \stopbodymatter \startbackmatter - % \component luametatex-rejected % local file \component luametatex-codes \component luametatex-registers \component luametatex-statistics |