diff options
Diffstat (limited to 'doc/context/sources/general/manuals/luatex/luatex-modifications.tex')
| -rw-r--r-- | doc/context/sources/general/manuals/luatex/luatex-modifications.tex | 668 |
1 files changed, 382 insertions, 286 deletions
diff --git a/doc/context/sources/general/manuals/luatex/luatex-modifications.tex b/doc/context/sources/general/manuals/luatex/luatex-modifications.tex index b5d8f2750..50d23fb1b 100644 --- a/doc/context/sources/general/manuals/luatex/luatex-modifications.tex +++ b/doc/context/sources/general/manuals/luatex/luatex-modifications.tex @@ -1,7 +1,6 @@ % language=uk \environment luatex-style -\environment luatex-logos \startcomponent luatex-modifications @@ -11,6 +10,9 @@ \startsubsection[title=The need for change] +\topicindex {engines} +\topicindex {history} + The first version of \LUATEX\ only had a few extra primitives and it was largely the same as \PDFTEX. Then we merged substantial parts of \ALEPH\ into the code and got more primitives. When we got more stable the decision was made to clean @@ -22,13 +24,15 @@ most of the adapted ones. Besides the expected changes caused by new functionality, there are a number of not|-|so|-|expected changes. These are sometimes a side|-|effect of a new -(conflicting) feature, or, more often than not, a change neccessary to clean up +(conflicting) feature, or, more often than not, a change necessary to clean up the internal interfaces. These will also be mentioned. \stopsubsection \startsubsection[title=Changes from \TEX\ 3.1415926] +\topicindex {\TEX} + Of course it all starts with traditional \TEX. Even if we started with \PDFTEX, most still comes from the original. But we divert a bit. @@ -38,7 +42,7 @@ most still comes from the original. But we divert a bit. The current code base is written in \CCODE, not \PASCAL. We use \CWEB\ when possible. As a consequence instead of one large file plus change files, we now have multiple files organized in categories like \type {tex}, \type - {pdf}, \type {lang}, \type {font}, \type {lua}, etc. There are some artefacts + {pdf}, \type {lang}, \type {font}, \type {lua}, etc. There are some artifacts of the conversion to \CCODE, but in due time we will clean up the source code and make sure that the documentation is done right. Many files are in the \CWEB\ format, but others, like those interfacing to \LUA, are \CCODE\ files. @@ -63,21 +67,21 @@ most still comes from the original. But we divert a bit. \stopitem \startitem - The upper limit to \type {\endlinechar} and \type {\newlinechar} is 127. + The upper limit to \prm {endlinechar} and \prm {newlinechar} is 127. \stopitem \startitem - Magnification (\type {\mag}) is only supported in \DVI\ output mode. You can + Magnification (\prm {mag}) is only supported in \DVI\ output mode. You can set this parameter and it even works with \type {true} units till you switch to \PDF\ output mode. When you use \PDF\ output you can best not touch the - \type {\mag} variable. This fuzzy behaviour is not much different from using + \prm {mag} variable. This fuzzy behaviour is not much different from using \PDF\ backend related functionality while eventually \DVI\ output is required. After the output mode has been frozen (normally that happens when the first page is shipped out) or when \PDF\ output is enabled, the \type {true} specification is ignored. When you preload a plain format adapted to - \LUATEX\ it can be that the \type {\mag} parameter already has been set. + \LUATEX\ it can be that the \prm {mag} parameter already has been set. \stopitem \stopitemize @@ -86,6 +90,8 @@ most still comes from the original. But we divert a bit. \startsubsection[title=Changes from \ETEX\ 2.2] +\topicindex {\ETEX} + Being the de factor standard extension of course we provide the \ETEX\ functionality, but with a few small adaptations. @@ -99,18 +105,19 @@ functionality, but with a few small adaptations. \startitem The \TEXXET\ extension is not present, so the primitives \type {\TeXXeTstate}, \type {\beginR}, \type {\beginL}, \type {\endR} and \type - {\endL} are missing. Instead we use the \OMEGA\ approach to directionality. + {\endL} are missing. Instead we used the \OMEGA/\ALEPH\ approach to + directionality as starting point. \stopitem \startitem - Some of the tracing information that is output by \ETEX's \type - {\tracingassigns} and \type {\tracingrestores} is not there. + Some of the tracing information that is output by \ETEX's \prm + {tracingassigns} and \prm {tracingrestores} is not there. \stopitem \startitem - Register management in \LUATEX\ uses the \ALEPH\ model, so the maximum value - is 65535 and the implementation uses a flat array instead of the mixed - flat|\&|sparse model from \ETEX. + Register management in \LUATEX\ uses the \OMEGA/\ALEPH\ model, so the maximum + value is 65535 and the implementation uses a flat array instead of the mixed + flat & sparse model from \ETEX. \stopitem \startitem @@ -127,6 +134,8 @@ functionality, but with a few small adaptations. \startsubsection[title=Changes from \PDFTEX\ 1.40] +\topicindex {\PDFTEX} + Because we want to produce \PDF\ the most natural starting point was the popular \PDFTEX\ program. We inherit the stable features, dropped most of the experimental code and promoted some functionality to core \LUATEX\ functionality @@ -134,76 +143,73 @@ which in turn triggered renaming primitives. For compatibility reasons we still refer to \type {\pdf...} commands but \LUATEX\ has a different backend interface. Instead of these primitives there are three -interfacing primitives: \type {\pdfextension}, \type {\pdfvariable} and -\type {\pdffeedback} that take keywords and optional further arguments. This way -we can extend the features when needed but don't need to adapt the core engine. -The front- and backend are decoupled as much as possible. +interfacing primitives: \lpr {pdfextension}, \lpr {pdfvariable} and \lpr +{pdffeedback} that take keywords and optional further arguments (below we will +still use the \tex {pdf} prefix names as reference). This way we can extend the +features when needed but don't need to adapt the core engine. The front- and +backend are decoupled as much as possible. \startitemize \startitem The (experimental) support for snap nodes has been removed, because it is much more natural to build this functionality on top of node processing and - attributes. The associated primitives that are now gone are: \type - {\pdfsnaprefpoint}, \type {\pdfsnapy}, and \type {\pdfsnapycomp}. + attributes. The associated primitives that are gone are: \orm + {pdfsnaprefpoint}, \orm {pdfsnapy}, and \orm {pdfsnapycomp}. \stopitem \startitem The (experimental) support for specialized spacing around nodes has also been - removed. The associated primitives that are now gone are: \type - {\pdfadjustinterwordglue}, \type {\pdfprependkern}, and \type - {\pdfappendkern}, as well as the five supporting primitives \type - {\knbscode}, \type {\stbscode}, \type {\shbscode}, \type {\knbccode}, and - \type {\knaccode}. + removed. The associated primitives that are gone are: \orm + {pdfadjustinterwordglue}, \orm {pdfprependkern}, and \orm {pdfappendkern}, as + well as the five supporting primitives \orm {knbscode}, \orm {stbscode}, \orm + {shbscode}, \orm {knbccode}, and \orm {knaccode}. \stopitem \startitem A number of \quote {\PDFTEX\ primitives} have been removed as they can be - implemented using \LUA: - - \start \raggedright - \type {\pdfelapsedtime}, \type {\pdfescapehex}, \type {\pdfescapename}, \type - {\pdfescapestring}, \type {\pdffiledump}, \type {\pdffilemoddate}, \type - {\pdffilesize}, \type {\pdfforcepagebox}, \type {\pdflastmatch}, \type - {\pdfmatch}, \type {\pdfmdfivesum}, \type {\pdfmovechars}, \type - {\pdfoptionalwaysusepdfpagebox}, \type {\pdfoptionpdfinclusionerrorlevel}, - \type {\pdfresettimer}, \type {\pdfshellescape}, \type {\pdfstrcmp} and \type - {\pdfunescapehex} - \par \stop + implemented using \LUA: \orm {pdfelapsedtime}, \orm {pdfescapehex}, \orm + {pdfescapename}, \orm {pdfescapestring}, \orm {pdffiledump}, \orm + {pdffilemoddate}, \orm {pdffilesize}, \orm {pdfforcepagebox}, \orm + {pdflastmatch}, \orm {pdfmatch}, \orm {pdfmdfivesum}, \orm {pdfmovechars}, + \orm {pdfoptionalwaysusepdfpagebox}, \orm {pdfoptionpdfinclusionerrorlevel}, + \orm {pdfresettimer}, \orm {pdfshellescape}, \orm {pdfstrcmp} and \orm + {pdfunescapehex}. \stopitem \startitem - The version related primitives \type {\pdftexbanner}, \type {\pdftexversion} - and \type {\pdftexrevision} are no longer present as there is no longer a + The version related primitives \orm {pdftexbanner}, \orm {pdftexversion} + and \orm {pdftexrevision} are no longer present as there is no longer a relationship with \PDFTEX\ development. \stopitem \startitem The experimental snapper mechanism has been removed and therefore also the - primitives: - - \start \raggedright - \type {\pdfignoreddimen}, \type {\pdffirstlineheight}, \type - {\pdfeachlineheight}, \type {\pdfeachlinedepth} and \type - {\pdflastlinedepth} - \par \stop + primitives \orm {pdfignoreddimen}, \orm {pdffirstlineheight}, \orm + {pdfeachlineheight}, \orm {pdfeachlinedepth} and \orm {pdflastlinedepth}. \stopitem \startitem - The experimental primitives \type {\primitive}, \type {\ifprimitive}, \type - {\ifabsnum} and \type {\ifabsdim} are promoted to core primitives. The \type + The experimental primitives \lpr {primitive}, \lpr {ifprimitive}, \lpr + {ifabsnum} and \lpr {ifabsdim} are promoted to core primitives. The \type {\pdf*} prefixed originals are not available. \stopitem \startitem - The \PNG\ transparency fix from 1.40.6 is not applied as high|-|level support - is pending. Because \LUATEX\ has a different subsystem for managing images, - more diversion from its ancestor happened in the meantime. + Because \LUATEX\ has a different subsystem for managing images, more + diversion from its ancestor happened in the meantime. We don't adapt to + changes in \PDFTEX. +\stopitem + +\startitem + Two extra token lists are provided, \orm {pdfxformresources} and \orm + {pdfxformattr}, as an alternative to \orm {pdfxform} keywords. \stopitem \startitem - Two extra token lists are provides, \type {\pdfxformresources} and \type - {\pdfxformattr}, as an alternative to \type {\pdfxform} keywords. + Image specifications also support \type {visiblefilename}, \type + {userpassword} and \type {ownerpassword}. The password options are only + relevant for encrypted \PDF\ files. \stopitem \startitem @@ -214,27 +220,27 @@ The front- and backend are decoupled as much as possible. \stopitem \startitem - The primitives \type {\pdfpagewidth} and \type {\pdfpageheight} have been removed - because \type {\pagewidth} and \type {\pageheight} have that purpose. + The primitives \orm {pdfpagewidth} and \orm {pdfpageheight} have been removed + because \lpr {pagewidth} and \lpr {pageheight} have that purpose. \stopitem \startitem - The primitives \type {\pdfnormaldeviate}, \type {\pdfuniformdeviate}, \type - {\pdfsetrandomseed} and \type {\pdfrandomseed} have been promoted to core + The primitives \orm {pdfnormaldeviate}, \orm {pdfuniformdeviate}, \orm + {pdfsetrandomseed} and \orm {pdfrandomseed} have been promoted to core primitives without \type {pdf} prefix so the original commands are no longer recognized. \stopitem \startitem - The primitives \type {\ifincsname}, \type {\expanded} and \type {\quitvmode} + The primitives \lpr {ifincsname}, \lpr {expanded} and \lpr {quitvmode} are now core primitives. \stopitem \startitem As the hz and protrusion mechanism are part of the core the related - primitives \type {\lpcode}, \type {\rpcode}, \type {\efcode}, \type - {\leftmarginkern}, \type {\rightmarginkern} are promoted to core primitives. The - two commands \type {\protrudechars} and \type {\adjustspacing} replace their + primitives \lpr {lpcode}, \lpr {rpcode}, \lpr {efcode}, \lpr + {leftmarginkern}, \lpr {rightmarginkern} are promoted to core primitives. The + two commands \lpr {protrudechars} and \lpr {adjustspacing} replace their prefixed with \type {\pdf} originals. \stopitem @@ -245,36 +251,36 @@ The front- and backend are decoupled as much as possible. \stopitem \startitem - When \type {\adjustspacing} has value~2, hz optimization will be applied to + When \lpr {adjustspacing} has value~2, hz optimization will be applied to glyphs and kerns. When the value is~3, only glyphs will be treated. A value smaller than~2 disables this feature. \stopitem \startitem - The \type {\tagcode} primitive is promoted to core primitive. + The \lpr {tagcode} primitive is promoted to core primitive. \stopitem \startitem - The \type {\letterspacefont} feature is now part of the core but will not be + The \lpr {letterspacefont} feature is now part of the core but will not be changed (improved). We just provide it for legacy use. \stopitem \startitem - The \type {\pdfnoligatures} primitive is now \type {\ignoreligaturesinfont}. + The \orm {pdfnoligatures} primitive is now \lpr {ignoreligaturesinfont}. \stopitem \startitem - The \type {\pdfcopyfont} primitive is now \type {\copyfont}. + The \orm {pdfcopyfont} primitive is now \lpr {copyfont}. \stopitem \startitem - The \type {\pdffontexpand} primitive is now \type {\expandglyphsinfont}. + The \orm {pdffontexpand} primitive is now \lpr {expandglyphsinfont}. \stopitem \startitem - Because position tracking is also available in \DVI\ mode the \type - {\savepos}, \type {\lastxpos} and \type {\lastypos} commands now replace - their \type {pdf} prefixed originals. + Because position tracking is also available in \DVI\ mode the \lpr {savepos}, + \lpr {lastxpos} and \lpr {lastypos} commands now replace their \type {pdf} + prefixed originals. \stopitem \startitem @@ -284,27 +290,31 @@ The front- and backend are decoupled as much as possible. \stopitem \startitem - The initializers \type {\pdfoutput} has been replaced by \type {\outputmode} and - \type {\pdfdraftmode} is now \type {\draftmode}. + The initializers \orm {pdfoutput} has been replaced by \lpr {outputmode} and + \orm {pdfdraftmode} is now \lpr {draftmode}. \stopitem \startitem - The pixel multiplier dimension \type {\pdfpxdimen} lots its prefix and is now calles - \type {\pxdimen}. + The pixel multiplier dimension \orm {pdfpxdimen} lost its prefix and is now + called \lpr {pxdimen}. \stopitem \startitem - An extra \type {\pdfimageaddfilename} option has been added that can be used to block - writing the filename to the \PDF\ file. + An extra \orm {pdfimageaddfilename} option has been added that can be used to + block writing the filename to the \PDF\ file. \stopitem \startitem - The primitive \type {\pdftracingfonts} is now \type {\tracingfonts} as it + The primitive \orm {pdftracingfonts} is now \lpr {tracingfonts} as it doesn't relate to the backend. \stopitem \startitem - The experimental primitive \type {\pdfinsertht} is kept as \type {\insertht}. + The experimental primitive \orm {pdfinsertht} is kept as \lpr {insertht}. +\stopitem + +\startitem + There is some more control over what metadata goes into the \PDF\ file. \stopitem \startitem @@ -318,25 +328,28 @@ The front- and backend are decoupled as much as possible. One change involves the so called xforms and ximages. In \PDFTEX\ these are implemented as so called whatsits. But contrary to other whatsits they have dimensions that need to be taken into account when for instance calculating -optimal line breaks. In \LUATEX\ these are now promoted to normal nodes, which -simplifies code that needs those dimensions. +optimal line breaks. In \LUATEX\ these are now promoted to a special type of rule +nodes, which simplifies code that needs those dimensions. Another reason for promotion is that these are useful concepts. Backends can -provide the ability to use content that has been rendered in several places, -and images are also common. For that reason we also changed the names: +provide the ability to use content that has been rendered in several places, and +images are also common. As already mentioned in \in {section} +[sec:imagedandforms], we now have: \starttabulate[|l|l|] -\BC new name \BC old name \NC \NR -\NC \type {\saveboxresource} \NC \type {\pdfxform} \NC \NR -\NC \type {\saveimageresource} \NC \type {\pdfximage} \NC \NR -\NC \type {\useboxresource} \NC \type {\pdfrefxform} \NC \NR -\NC \type {\useimageresource} \NC \type {\pdfrefximage} \NC \NR -\NC \type {\lastsavedboxresourceindex} \NC \type {\pdflastxform} \NC \NR -\NC \type {\lastsavedimageresourceindex} \NC \type {\pdflastximage} \NC \NR -\NC \type {\lastsavedimageresourcepages} \NC \type {\pdflastximagepages} \NC \NR +\DB \LUATEX \BC \PDFTEX \NC \NR +\TB +\NC \lpr {saveboxresource} \NC \orm {pdfxform} \NC \NR +\NC \lpr {saveimageresource} \NC \orm {pdfximage} \NC \NR +\NC \lpr {useboxresource} \NC \orm {pdfrefxform} \NC \NR +\NC \lpr {useimageresource} \NC \orm {pdfrefximage} \NC \NR +\NC \lpr {lastsavedboxresourceindex} \NC \orm {pdflastxform} \NC \NR +\NC \lpr {lastsavedimageresourceindex} \NC \orm {pdflastximage} \NC \NR +\NC \lpr {lastsavedimageresourcepages} \NC \orm {pdflastximagepages} \NC \NR +\LL \stoptabulate -There are a few \type {\pdffeedback} features that relate to this but these are +There are a few \lpr {pdffeedback} features that relate to this but these are typical backend specific ones. The index that gets returned is to be considered as \quote {just a number} and although it still has the same meaning (object related) as before, you should not depend on that. @@ -344,7 +357,7 @@ related) as before, you should not depend on that. The protrusion detection mechanism is enhanced a bit to enable a bit more complex situations. When protrusion characters are identified some nodes are skipped: -\startitemize[packed] +\startitemize[packed,columns,two] \startitem zero glue \stopitem \startitem penalties \stopitem \startitem empty discretionaries \stopitem @@ -374,6 +387,8 @@ instance content moved into the margin: \startsubsection[title=Changes from \ALEPH\ RC4] +\topicindex {\ALEPH} + Because we wanted proper directional typesetting the \ALEPH\ mechanisms looked most attractive. These are rather close to the ones provided by \OMEGA, so what we say next applies to both these programs. @@ -381,46 +396,40 @@ we say next applies to both these programs. \startitemize \startitem - The extended 16-bit math primitives (\type {\omathcode} etc.) have been + The extended 16-bit math primitives (\orm {omathcode} etc.) have been removed. \stopitem \startitem The \OCP\ processing has been removed completely and as a consequence, the - following primitives have been removed: - - \start \raggedright - \type {\ocp}, \type {\externalocp}, \type {\ocplist}, \type {\pushocplist}, - \type {\popocplist}, \type {\clearocplists}, \type {\addbeforeocplist}, \type - {\addafterocplist}, \type {\removebeforeocplist}, \type {\removeafterocplist} - and \type {\ocptracelevel} - \par \stop + following primitives have been removed: \orm {ocp}, \orm {externalocp}, \orm + {ocplist}, \orm {pushocplist}, \orm {popocplist}, \orm {clearocplists}, \orm + {addbeforeocplist}, \orm {addafterocplist}, \orm {removebeforeocplist}, \orm + {removeafterocplist} and \orm {ocptracelevel}. \stopitem \startitem \LUATEX\ only understands 4~of the 16~direction specifiers of \ALEPH: \type - {TLT} (latin), \type {TRT} (arabic), \type {RTT} (cjk), \type {LTL} - (mongolian). All other direction specifiers generate an error. + {TLT} (latin), \type {TRT} (arabic), \type {RTT} (cjk), \type {LTL} (mongolian). + All other direction specifiers generate an error. In addition to a keyword + driven model we also provide an integer driven one. \stopitem \startitem The input translations from \ALEPH\ are not implemented, the related - primitives are not available: - - \start \raggedright - \type {\DefaultInputMode}, \type {\noDefaultInputMode}, \type {\noInputMode}, - \type {\InputMode}, \type {\DefaultOutputMode}, \type {\noDefaultOutputMode}, - \type {\noOutputMode}, \type {\OutputMode}, \type {\DefaultInputTranslation}, - \type {\noDefaultInputTranslation}, \type {\noInputTranslation}, \type - {\InputTranslation}, \type {\DefaultOutputTranslation}, \type - {\noDefaultOutputTranslation}, \type {\noOutputTranslation} and \type - {\OutputTranslation} - \par \stop + primitives are not available: \orm {DefaultInputMode}, \orm + {noDefaultInputMode}, \orm {noInputMode}, \orm {InputMode}, \orm + {DefaultOutputMode}, \orm {noDefaultOutputMode}, \orm {noOutputMode}, \orm + {OutputMode}, \orm {DefaultInputTranslation}, \orm + {noDefaultInputTranslation}, \orm {noInputTranslation}, \orm + {InputTranslation}, \orm {DefaultOutputTranslation}, \orm + {noDefaultOutputTranslation}, \orm {noOutputTranslation} and \orm + {OutputTranslation}. \stopitem \startitem - Several bugs have been fixed an confusing implementation details have been sorted - out. + Several bugs have been fixed and confusing implementation details have been + sorted out. \stopitem \startitem @@ -429,13 +438,13 @@ we say next applies to both these programs. \stopitem \startitem - The \type {^^} notation has been extended: after \type {^^^^} four hexadecimal - characters are expected and after \type {^^^^^^} six hexadecimal characters - have to be given. The original \TEX\ interpretation is still valid for the - \type {^^} case but the four and six variants do no backtracking, i.e.\ when - they are not followed by the right number of hexadecimal digits they issue an - error message. Because \type{^^^} is a normal \TEX\ case, we don't support the - odd number of \type {^^^^^} either. + The \type {^^} notation has been extended: after \type {^^^^} four + hexadecimal characters are expected and after \type {^^^^^^} six hexadecimal + characters have to be given. The original \TEX\ interpretation is still valid + for the \type {^^} case but the four and six variants do no backtracking, + i.e.\ when they are not followed by the right number of hexadecimal digits + they issue an error message. Because \type{^^^} is a normal \TEX\ case, we + don't support the odd number of \type {^^^^^} either. \stopitem \startitem @@ -449,9 +458,9 @@ we say next applies to both these programs. \stopitem \startitem - The page dimension related primitives \type {\pagewidth} and \type - {\pageheight} have been promoted to core primitives. The \type {\hoffset} and - \type {\voffset} primitives have been fixed. + The page dimension related primitives \lpr {pagewidth} and \lpr {pageheight} + have been promoted to core primitives. The \prm {hoffset} and \prm {voffset} + primitives have been fixed. \stopitem \startitem @@ -461,30 +470,35 @@ we say next applies to both these programs. \stopitem \startitem - The two dimension registers \type {\pagerightoffset} and \type - {\pagebottomoffset} are now core primitives. + The two dimension registers \lpr {pagerightoffset} and \lpr + {pagebottomoffset} are now core primitives. \stopitem \startitem - The direction related primitives \type {\pagedir}, \type {\bodydir}, \type - {\pardir}, \type {\textdir}, \type {\mathdir} and \type {\boxdir} are now - core primitives. + The direction related primitives \lpr {pagedir}, \lpr {bodydir}, \lpr + {pardir}, \lpr {textdir}, \lpr {mathdir} and \lpr {boxdir} are now core + primitives. \stopitem \startitem - The promotion of primitives to core primitives as well as the removed of all - others means that the initialization namespace \type {aleph} is gone. + The promotion of primitives to core primitives as well as removing of all + others means that the initialization namespace \type {aleph} that early + versions of \LUATEX\ provided is gone. \stopitem \stopitemize The above let's itself summarize as: we took the 32 bit aspects and much of the -directional mechanisms. +directional mechanisms and merged it into the \PDFTEX\ code base as starting +point for further development. Then we simplified directionality, fixed it and +opened it up. \stopsubsection \startsubsection[title=Changes from standard \WEBC] +\topicindex {\WEBC} + The compilation framework is \WEBC\ and we keep using that but without the \PASCAL\ to \CCODE\ step. This framework also provides some common features that deal with reading bytes from files and locating files in \TDS. This is what we do @@ -507,7 +521,7 @@ different: \stopitem \startitem - The \type {\openout} whatsits are not written to the log file. + The \prm {openout} whatsits are not written to the log file. \stopitem \startitem @@ -515,7 +529,8 @@ different: mode because \type {texmf.cnf} is not read: \type {shell-escape} is off (but that is not a problem because of \LUA's \type {os.execute}), and the paranoia checks on \type {openin} and \type {openout} do not happen. However, it is - easy for a \LUA\ script to do this itself by overloading \type {io.open}. + easy for a \LUA\ script to do this itself by overloading \type {io.open} and + alike. \stopitem \startitem @@ -528,17 +543,22 @@ different: \stopsection -\startsection[reference=backendprimitives,title=The backend primitives \type {\pdf*}] +\startsection[reference=backendprimitives,title=The backend primitives] + +\startsubsection[title={Less primitives}] + +\topicindex {backend} +\topicindex {\PDF+backend} In a previous section we mentioned that some \PDFTEX\ primitives were removed and others promoted to core \LUATEX\ primitives. That is only part of the story. In order to separate the backend specific primitives in de code these commands are now replaced by only a few. In traditional \TEX\ we only had the \DVI\ backend but now we have two: \DVI\ and \PDF. Additional functionality is implemented as -\quote {extensions} in \TEX speak. By separating more strickly we are able to -keep the core (fontend) clean and stable. If for some reason an extra backend -option is needed, it can be implemented without touching the core. The three -\PDF\ backend related primitives are +\quote {extensions} in \TEX\ speak. By separating more strickly we are able to +keep the core (frontend) clean and stable and isolate these extensions. If for +some reason an extra backend option is needed, it can be implemented without +touching the core. The three \PDF\ backend related primitives are: \starttyping \pdfextension command [specification] @@ -550,8 +570,12 @@ An extension triggers further parsing, depending on the command given. A variabl a (kind of) register and can be read and written, while a feedback is reporting something (as it comes from the backend it's normally a sequence of tokens). +\stopsubsection + +\startsubsection[title={\lpr{pdfextension}, \lpr {pdfvariable} and \lpr {pdffeedback}},reference=sec:pdfextensions] + In order for \LUATEX\ to be more than just \TEX\ you need to enable primitives. That -has already be the case right from the start. If you want the traditional \PDFTEX\ +has already been the case right from the start. If you want the traditional \PDFTEX\ primitives (for as far their functionality is still around) you now can do this: \starttyping @@ -604,6 +628,7 @@ The configuration related registers have become: \starttyping \edef\pdfcompresslevel {\pdfvariable compresslevel} \edef\pdfobjcompresslevel {\pdfvariable objcompresslevel} +\edef\pdfrecompress {\pdfvariable recompress} \edef\pdfdecimaldigits {\pdfvariable decimaldigits} \edef\pdfgamma {\pdfvariable gamma} \edef\pdfimageresolution {\pdfvariable imageresolution} @@ -618,6 +643,7 @@ The configuration related registers have become: \edef\pdfignoreunknownimages {\pdfvariable ignoreunknownimages} \edef\pdfgentounicode {\pdfvariable gentounicode} \edef\pdfomitcidset {\pdfvariable omitcidset} +\edef\pdfomitcharset {\pdfvariable omitcharset} \edef\pdfpagebox {\pdfvariable pagebox} \edef\pdfminorversion {\pdfvariable minorversion} \edef\pdfuniqueresname {\pdfvariable uniqueresname} @@ -657,100 +683,16 @@ macro:->[internal backend integer] macro:->[internal backend tokenlist] \stoptyping -The \type {\edef} can also be an \type {\def} but it's a bit more efficient -to expand the lookup related register beforehand. After that you can adapt -the defaults; these are: - -\starttyping -\pdfcompresslevel 9 -\pdfobjcompresslevel 1 % used: (0,9) -\pdfdecimaldigits 4 % used: (3,6) -\pdfgamma 1000 -\pdfimageresolution 71 -\pdfimageapplygamma 0 -\pdfimagegamma 2200 -\pdfimagehicolor 1 -\pdfimageaddfilename 1 -\pdfpkresolution 72 -\pdfpkfixeddpi 0 -\pdfinclusioncopyfonts 0 -\pdfinclusionerrorlevel 0 -\pdfignoreunknownimages 0 -\pdfgentounicode 0 -\pdfomitcidset 0 -\pdfpagebox 0 -\pdfminorversion 4 -\pdfuniqueresname 0 - -\pdfhorigin 1in -\pdfvorigin 1in -\pdflinkmargin 0pt -\pdfdestmargin 0pt -\pdfthreadmargin 0pt -\pdfxformmargin 0pt -\stoptyping - -If you also want some backward compatibility, you can add: - -\starttyping -\let\pdfpagewidth \pagewidth -\let\pdfpageheight \pageheight - -\let\pdfadjustspacing \adjustspacing -\let\pdfprotrudechars \protrudechars -\let\pdfnoligatures \ignoreligaturesinfont -\let\pdffontexpand \expandglyphsinfont -\let\pdfcopyfont \copyfont - -\let\pdfxform \saveboxresource -\let\pdflastxform \lastsavedboxresourceindex -\let\pdfrefxform \useboxresource - -\let\pdfximage \saveimageresource -\let\pdflastximage \lastsavedimageresourceindex -\let\pdflastximagepages\lastsavedimageresourcepages -\let\pdfrefximage \useimageresource - -\let\pdfsavepos \savepos -\let\pdflastxpos \lastxpos -\let\pdflastypos \lastypos - -\let\pdfoutput \outputmode -\let\pdfdraftmode \draftmode - -\let\pdfpxdimen \pxdimen - -\let\pdfinsertht \insertht - -\let\pdfnormaldeviate \normaldeviate -\let\pdfuniformdeviate \uniformdeviate -\let\pdfsetrandomseed \setrandomseed -\let\pdfrandomseed \randomseed - -\let\pdfprimitive \primitive -\let\ifpdfprimitive \ifprimitive - -\let\ifpdfabsnum \ifabsnum -\let\ifpdfabsdim \ifabsdim -\stoptyping - -And even: - -\starttyping -\newdimen\pdfeachlineheight -\newdimen\pdfeachlinedepth -\newdimen\pdflastlinedepth -\newdimen\pdffirstlineheight -\newdimen\pdfignoreddimen -\stoptyping +The \prm {edef} can also be a \prm {def} but it's a bit more efficient to expand +the lookup related register beforehand. The backend is derived from \PDFTEX\ so the same syntax applies. However, the \type {outline} command accepts a \type {objnum} followed by a number. No checking takes place so when this is used it had better be a valid (flushed) object. -In order to be (more or less) compatible with \PDFTEX\ we also support the -option to suppress some info: +In order to be (more or less) compatible with \PDFTEX\ we also support the option +to suppress some info but we do so via a bitset: \starttyping \pdfvariable suppressoptionalinfo \numexpr @@ -770,7 +712,7 @@ option to suppress some info: In addition you can overload the trailer id, but we don't do any checking on validity, so you have to pass a valid array. The following is like the ones -normally generated by the engine: +normally generated by the engine. You even need to include the brackets here! \starttyping \pdfvariable trailerid {[ @@ -779,8 +721,6 @@ normally generated by the engine: ]} \stoptyping -So, you even need to include the brackets! - Although we started from a merge of \PDFTEX\ and \ALEPH, by now the code base as well as functionality has diverted from those parents. Here we show the options that can be passed to the extensions. @@ -928,10 +868,113 @@ that can be passed to the extensions. {tokens} \stoptexsyntax +\stopsubsection + +\startsubsection[title={Defaults}] + +The engine sets the following defaults. + +\starttyping +\pdfcompresslevel 9 +\pdfobjcompresslevel 1 % used: (0,9) +\pdfrecompress 0 % mostly for debugging +\pdfdecimaldigits 4 % used: (3,6) +\pdfgamma 1000 +\pdfimageresolution 71 +\pdfimageapplygamma 0 +\pdfimagegamma 2200 +\pdfimagehicolor 1 +\pdfimageaddfilename 1 +\pdfpkresolution 72 +\pdfpkfixeddpi 0 +\pdfinclusioncopyfonts 0 +\pdfinclusionerrorlevel 0 +\pdfignoreunknownimages 0 +\pdfgentounicode 0 +\pdfomitcidset 0 +\pdfomitcharset 0 +\pdfpagebox 0 +\pdfminorversion 4 +\pdfuniqueresname 0 + +\pdfhorigin 1in +\pdfvorigin 1in +\pdflinkmargin 0pt +\pdfdestmargin 0pt +\pdfthreadmargin 0pt +\pdfxformmargin 0pt +\stoptyping + +\stopsubsection + +\startsubsection[title={Backward compatibility}] + +If you also want some backward compatibility, you can add: + +\starttyping +\let\pdfpagewidth \pagewidth +\let\pdfpageheight \pageheight + +\let\pdfadjustspacing \adjustspacing +\let\pdfprotrudechars \protrudechars +\let\pdfnoligatures \ignoreligaturesinfont +\let\pdffontexpand \expandglyphsinfont +\let\pdfcopyfont \copyfont + +\let\pdfxform \saveboxresource +\let\pdflastxform \lastsavedboxresourceindex +\let\pdfrefxform \useboxresource + +\let\pdfximage \saveimageresource +\let\pdflastximage \lastsavedimageresourceindex +\let\pdflastximagepages\lastsavedimageresourcepages +\let\pdfrefximage \useimageresource + +\let\pdfsavepos \savepos +\let\pdflastxpos \lastxpos +\let\pdflastypos \lastypos + +\let\pdfoutput \outputmode +\let\pdfdraftmode \draftmode + +\let\pdfpxdimen \pxdimen + +\let\pdfinsertht \insertht + +\let\pdfnormaldeviate \normaldeviate +\let\pdfuniformdeviate \uniformdeviate +\let\pdfsetrandomseed \setrandomseed +\let\pdfrandomseed \randomseed + +\let\pdfprimitive \primitive +\let\ifpdfprimitive \ifprimitive + +\let\ifpdfabsnum \ifabsnum +\let\ifpdfabsdim \ifabsdim +\stoptyping + +And even: + +\starttyping +\newdimen\pdfeachlineheight +\newdimen\pdfeachlinedepth +\newdimen\pdflastlinedepth +\newdimen\pdffirstlineheight +\newdimen\pdfignoreddimen +\stoptyping + +\stopsubsection + \stopsection \startsection[title=Directions] +\topicindex {\OMEGA} +\topicindex {\ALEPH} +\topicindex {directions} + +\startsubsection[title={Four directions}] + The directional model in \LUATEX\ is inherited from \OMEGA|/|\ALEPH\ but we tried to improve it a bit. At some point we played with recovery of modes but that was disabled later on when we found that it interfered with nested directions. That @@ -939,14 +982,23 @@ itself had as side effect that the node list was no longer balanced with respect to directional nodes which in turn can give side effects when a series of dir changes happens without grouping. -The current (0.97 onward) approach is that we again make the list balanced but -try to avoid some side effects. What happens is quite intuitive if we forget -about spaces (turned into glue) but even there what happens makes sense if you -look at it in detail. However that logic makes in|-|group switching kind of -useless when no proper nested grouping is used: switching from right to left -several times nested, results in spacing ending up after each other due to nested -mirroring. Of course a sane macro package will manage this for the user but here -we are discussing the low level dir injection. +When extending the \PDF\ backend to support directions some inconsistencies were +found and as a result we decided to support only the four models that make sense +\type {TLT} (latin), \type {TRT} (arabic), \type {RTT} (cjk) and \type {LTL} +(mongolian). + +\stopsubsection + +\startsubsection[title={How it works}] + +The approach is that we again make the list balanced but try to avoid some side +effects. What happens is quite intuitive if we forget about spaces (turned into +glue) but even there what happens makes sense if you look at it in detail. +However that logic makes in|-|group switching kind of useless when no proper +nested grouping is used: switching from right to left several times nested, +results in spacing ending up after each other due to nested mirroring. Of course +a sane macro package will manage this for the user but here we are discussing the +low level dir injection. This is what happens: @@ -1031,7 +1083,7 @@ It gets typeset as: We could define the two helpers to look back, pick up a skip, remove it and inject it after the dir node. But that way we loose the subtype information that for some applications can be handy to be kept as|-|is. This is why we now have a -variant of \type {\textdir} which injects the balanced node before the skip. +variant of \lpr {textdir} which injects the balanced node before the skip. Instead of the previous definition we can use: \startbuffer[def] @@ -1060,64 +1112,64 @@ comes out as a properly spaced: Anything more complex that this, like combination of skips and penalties, or kerns, should be handled in the input or macro package because there is no way we -can predict the expected behaviour. In fact, the \type {\linedir} is just a +can predict the expected behaviour. In fact, the \lpr {linedir} is just a convenience extra which could also have been implemented using node list parsing. +\stopsubsection + +\startsubsection[title={Controlling glue with \lpr {breakafterdirmode}}] + Glue after a dir node is ignored in the linebreak decision but you can bypass that -by setting \type {\breakafterdirmode} to~\type {1}. The following table shows the +by setting \lpr {breakafterdirmode} to~\type {1}. The following table shows the difference. Watch your spaces. \def\ShowSome#1{% - \BC - \type{#1} - \NC - \breakafterdirmode = 0 - \hsize 0pt - #1 + \BC \type{#1} + \NC \breakafterdirmode\zerocount\hsize\zeropoint#1 \NC + \NC \breakafterdirmode\plusone\hsize\zeropoint#1 \NC - \breakafterdirmode = 1 - \hsize 0pt - #1 - \NC - \NC \NR \HL + \NC \NR } -\starttabulate[|l|Tp(0pt)|w(5em)|Tp(0pt)|p|] - \HL - \BC \type{\breakafterdirmode} +\starttabulate[|l|Tp(1pt)|w(5em)|Tp(1pt)|w(5em)|] + \DB \BC \type{0} \NC \BC \type{1} \NC \NC \NR - \HL + \TB \ShowSome{pre {\textdir TLT xxx} post} \ShowSome{pre {\textdir TLT xxx }post} \ShowSome{pre{ \textdir TLT xxx} post} \ShowSome{pre{ \textdir TLT xxx }post} \ShowSome{pre { \textdir TLT xxx } post} \ShowSome{pre {\textdir TLT\relax\space xxx} post} + \LL \stoptabulate +\stopsubsection + +\startsubsection[title={Controling parshapes with \lpr {shapemode}}] Another adaptation to the \ALEPH\ directional model is control over shapes driven -by \type {\hangindent} and \type {\parshape}. This is controlled by a new parameter -\type {\shapemode}: +by \prm {hangindent} and \prm {parshape}. This is controlled by a new parameter +\lpr {shapemode}: -\starttabulate[|c|c|c|] -\BC \BC \type {\hangindent} \BC \type {\parshape} \NC \NR +\starttabulate[|c|l|l|] +\DB value \BC \prm {hangindent} \BC \prm {parshape} \NC \NR +\TB \BC \type{0} \NC normal \NC normal \NC \NR \BC \type{1} \NC mirrored \NC normal \NC \NR \BC \type{2} \NC normal \NC mirrored \NC \NR \BC \type{3} \NC mirrored \NC mirrored \NC \NR +\LL \stoptabulate -The value is reset to zero (like \type {\hangindent} and \type {\parshape}) +The value is reset to zero (like \prm {hangindent} and \prm {parshape}) after the paragraph is done with. You can use negative values to prevent -this. - -In \in {figure} [fig:shapemode] a few examples are given. +this. In \in {figure} [fig:shapemode] a few examples are given. \startplacefigure[reference=fig:shapemode,title={The effect of \type {shapemode}.}] \startcombination[2*3] @@ -1162,12 +1214,46 @@ In \in {figure} [fig:shapemode] a few examples are given. \stopcombination \stopplacefigure +\stopsubsection + +\startsubsection[title={Symbols or numbers}] + +Internally the implementation is different from \ALEPH. First of all we use no +whatsits but dedicated nodes, but also we have only 4 directions that are mapped +onto 4 numbers. A text direction node can mark the start or end of a sequence of +nodes, and therefore has two states. At the \TEX\ end we don't see these states +because \TEX\ itself will add proper end state nodes if needed. + +The symbolic names \type {TLT}, \type {TRT}, etc.\ originate in \OMEGA. In +\LUATEX\ we also have a number based model which sometimes makes more sense. + +\starttabulate[|c|l|l|] +\DB value \BC equivalent \NC \NR +\TB +\BC \type {0} \NC TLT \NC \NR +\BC \type {1} \NC TRT \NC \NR +\BC \type {2} \NC LTL \NC \NR +\BC \type {3} \NC RTT \NC \NR +\LL +\stoptabulate + +We support the \OMEGA\ primitives \orm {textdir}, \orm {pardir}, \orm {pagedir}, +\orm {pardir} and \orm {mathdir}. These accept three character keywords. The +primitives that set the direction by number are: \lpr {textdirection}, \lpr +{pardirection}, \lpr {pagedirection} and \lpr {bodydirection} and \lpr +{mathdirection}. When specifying a direction for a box you can use \type {bdir} +instead of \type {dir}. + +\stopsubsection + \stopsection \startsection[title=Implementation notes] \startsubsection[title=Memory allocation] +\topicindex {memory} + The single internal memory heap that traditional \TEX\ used for tokens and nodes is split into two separate arrays. Each of these will grow dynamically when needed. @@ -1189,14 +1275,9 @@ structures, some of the macros have been duplicated. For instance, there are now {token_info}. All access to the variable memory array is now hidden behind a macro called \type {vmem}. We mention this because using the \TEX book as reference is still quite valid but not for memory related details. Another -significate detail is that we have double linked node lists and that some nodes +significant detail is that we have double linked node lists and that most nodes carry more data. -The implementation of the growth of two arrays (via reallocation) introduces a -potential pitfall: the memory arrays should never be used as the left hand side -of a statement that can modify the array in question. Details like this are -of no concern to users. - The input line buffer and pool size are now also reallocated when needed, and the \type {texmf.cnf} settings \type {buf_size} and \type {pool_size} are silently ignored. @@ -1205,25 +1286,31 @@ ignored. \startsubsection[title=Sparse arrays] -The \type {\mathcode}, \type {\delcode}, \type {\catcode}, \type {\sfcode}, \type -{\lccode} and \type {\uccode} (and the new \type {\hjcode}) tables are now sparse -arrays that are implemented in~\CCODE. They are no longer part of the \TEX\ -\quote {equivalence table} and because each had 1.1 million entries with a few -memory words each, this makes a major difference in memory usage. +The \prm {mathcode}, \prm {delcode}, \prm {catcode}, \prm {sfcode}, \prm {lccode} +and \prm {uccode} (and the new \lpr {hjcode}) tables are now sparse arrays that +are implemented in~\CCODE. They are no longer part of the \TEX\ \quote +{equivalence table} and because each had 1.1 million entries with a few memory +words each, this makes a major difference in memory usage. Performance is not +really hurt by this. -The \type {\catcode}, \type {\sfcode}, \type {\lccode}, \type {\uccode} and \type -{\hjcode} assignments do not yet show up when using the \ETEX\ tracing routines -\type {\tracingassigns} and \type {\tracingrestores}. +The \prm {catcode}, \prm {sfcode}, \prm {lccode}, \prm {uccode} and \lpr {hjcode} +assignments don't show up when using the \ETEX\ tracing routines \prm +{tracingassigns} and \prm {tracingrestores} but we don't see that as a real +limitation. -A side|-|effect of the current implementation is that \type {\global} is now more -expensive in terms of processing than non|-|global assignments. +A side|-|effect of the current implementation is that \prm {global} is now more +expensive in terms of processing than non|-|global assignments but not many users +will notice that. The glyph ids within a font are also managed by means of a sparse array as glyph -ids can go up to index $2^{21}-1$. +ids can go up to index $2^{21}-1$ but these are never accessed directly so again +users will not notice this. \stopsubsection -\startsubsection[title=Simple single-character csnames] +\startsubsection[title=Simple single|-|character csnames] + +\topicindex {csnames} Single|-|character commands are no longer treated specially in the internals, they are stored in the hash just like the multiletter csnames. @@ -1236,16 +1323,22 @@ control sequences that uses a prefix that is otherwise impossible to obtain. \stopsubsection -\startsubsection[title=Compressed format] +\startsubsection[title=The compressed format file] + +\topicindex {format} The format is passed through \type {zlib}, allowing it to shrink to roughly half of the size it would have had in uncompressed form. This takes a bit more \CPU\ -cycles but much less disk \IO, so it should still be faster. +cycles but much less disk \IO, so it should still be faster. We use a level~3 +compression which we found to be the optimal trade|-|off between filesize and +decompression speed. \stopsubsection \startsubsection[title=Binary file reading] +\topicindex {files+binary} + All of the internal code is changed in such a way that if one of the \type {read_xxx_file} callbacks is not set, then the file is read by a \CCODE\ function using basically the same convention as the callback: a single read into a buffer @@ -1257,6 +1350,9 @@ previous code (that mostly used \type {getc} calls), it can be quite a bit faste \startsubsection[title=Tabs and spaces] +\topicindex {space} +\topicindex {newline} + We conform to the way other \TEX\ engines handle trailing tabs and spaces. For decades trailing tabs and spaces (before a newline) were removed from the input but this behaviour was changed in September 2017 to only handle spaces. We are @@ -1273,7 +1369,7 @@ to be kept. We are aware of the fact that this contradicts some of our other choices but consistency with other engines and the fact that in \KPSE\ mode a common file \IO\ layer is used can have a side effect of breaking compatibility. We still stick to our view that at the log level we can (and might be) more -incompatible. +incompatible. We already expose some more details. \stopsubsection |