2019-12-30 19:16:00

1 files changed, 440 insertions, 0 deletions

diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex b/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex
new file mode 100644
index 000000000..dca6c3781
--- /dev/null
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-modifications.tex
@@ -0,0 +1,440 @@
+% language=uk
+
+\environment luametatex-style
+
+\startcomponent luametatex-modifications
+
+\startchapter[reference=modifications,title={The original engines}]
+
+\startsection[title=The merged engines]
+
+\startsubsection[title=The rationale]
+
+\topicindex {engines}
+\topicindex {history}
+
+The first version of \LUATEX, made by Hartmut after we discussed the possibility
+of an extension language, only had a few extra primitives and it was largely the
+same as \PDFTEX. It was presented to the public in 2005. As part of the Oriental
+\TEX\ project, Taco merged substantial parts of \ALEPH\ into the code and some
+more primitives were added. Then we started more fundamental experiments. After
+many years, when the engine had become more stable, the decision was made to
+clean up the rather hybrid nature of the program. This means that some primitives
+were promoted to core primitives, often with a different name, and that others
+were removed. This also made it possible to start cleaning up the code base. In
+\in {chapter} [enhancements] we discussed some new primitives, here we will cover
+most of the adapted ones.
+
+During more than a decade stepwise new functionality was added and after 10 years
+the more of less stable version 1.0 was presented. But we continued and after
+some 15 years the \LUAMETATEX\ follow up entered its first testing stage. But
+before details about the engine are discussed in successive chapters, we first
+summarize where we started from. Keep in mind that in \LUAMETATEX\ we have a bit
+less than in \LUATEX, so this section differs from the one in the \LUATEX\
+manual.
+
+Besides the expected changes caused by new functionality, there are a number of
+not|-|so|-|expected changes. These are sometimes a side|-|effect of a new
+(conflicting) feature, or, more often than not, a change necessary to clean up
+the internal interfaces. These will also be mentioned.
+
+\stopsubsection
+
+\startsubsection[title=Changes from \TEX\ 3.1415926]
+
+\topicindex {\TEX}
+
+Of course it all starts with traditional \TEX. Even if we started with \PDFTEX,
+most still comes from original Knuthian \TEX. But we divert a bit.
+
+\startitemize
+
+\startitem
+    The current code base is written in \CCODE, not \PASCAL. The original \CWEB\
+    documentation is kept when possible and not wrapped in tagged comments. As a
+    consequence instead of one large file plus change files, we now have multiple
+    files organized in categories like \type {tex}, \type {luaf}, \type
+    {languages}, \type {fonts}, \type {libraries}, etc. There are some artifacts
+    of the conversion to \CCODE, but these got (and get) removed stepwise. The
+    documentation, which is actually comes from the mix of engines (via so called
+    change files) is kept as much as possible. Of course we want to stay as close
+    as possible to the original so that the documentation of the fundamentals
+    behind \TEX\ by Don Knuth still applies. However, because we use \CCODE, some
+    documentation is a bit off. Also, most global variables are now collected in
+    structures, but the original names were kept. There are lots of so called
+    macros too.
+\stopitem
+
+\startitem
+    See \in {chapter} [languages] for many small changes related to paragraph
+    building, language handling and hyphenation. The most important change is
+    that adding a brace group in the middle of a word (like in \type {of{}fice})
+    does not prevent ligature creation. Also, the hyphenation, ligature building
+    and kerning has been split so that we can hook in alternative or extra code
+    wherever we like. There are various options to control discretionary
+    injection and related penalties are now integrated in these nodes. Language
+    information is now bound to glyphs. The number of languages in \LUAMETATEX\
+    is smaller than in \LUATEX.
+\stopitem
+
+\startitem
+    There is no pool file, all strings are embedded during compilation. This also
+    removed some memory constraints. We kept token and node memory management
+    because it is convenient and efficient but parts were reimplemented in order
+    to remove some constraints. Token memory management is largely the same.
+\stopitem
+
+\startitem
+    The specifier \type {plus 1 fillll} does not generate an error. The extra
+    \quote {l} is simply typeset.
+\stopitem
+
+\startitem
+    The upper limit to \prm {endlinechar} and \prm {newlinechar} is 127.
+\stopitem
+
+\startitem
+    Because the backend is not built|-|in, the magnification (\prm {mag})
+    primitive is not doing nothing. A \type {shipout} just discards the content
+    of the given box. The write related primitives have to be implemented in the
+    used macro package using \LUA. None of the \PDFTEX\ derived primitives is
+    present.
+\stopitem
+
+\startitem
+    There is more control over some (formerly hard|-|coded) math properties. In fact,
+    there is a whole extra bit of math related code because we need to deal with
+    \OPENTYPE\ fonts.
+\stopitem
+
+\startitem
+    The \type {\outer} and \type {\long} prefixed are silently ignored. It is
+    permitted to use \type {\par} in math.
+\stopitem
+
+\startitem
+    Because there is no font loader, a \LUA\ variant is free to either support or
+    not the \OMEGA\ \type {ofm} file format. As there are hardly any such fonts
+    it probably makes no sense.
+\stopitem
+
+\startitem
+    The lack of a backend means that some primitives related to it are not
+    implemented. This is no big deal because it is possible to use the scanner
+    library to implement them as needed, which depends on the macro package and
+    backend.
+\stopitem
+
+\startitem
+    When detailed logging is enabled more detail is output with respect to what
+    nodes are involved. This is a side effect of the core nodes having more
+    detailed subtype information. The benefit of more detail wins from any wish
+    to be byte compatible in the logging. One can always write additional logging
+    in \LUA.
+\stopitem
+
+\stopitemize
+
+\stopsubsection
+
+\startsubsection[title=Changes from \ETEX\ 2.2]
+
+\topicindex {\ETEX}
+
+Being the de|-|facto standard extension of course we provide the \ETEX\
+features, but with a few small adaptations.
+
+\startitemize
+
+\startitem
+    The \ETEX\ functionality is always present and enabled so the prepended
+    asterisk or \type {-etex} switch for \INITEX\ is not needed.
+\stopitem
+
+\startitem
+    The \TEXXET\ extension is not present, so the primitives \type
+    {\TeXXeTstate}, \type {\beginR}, \type {\beginL}, \type {\endR} and \type
+    {\endL} are missing. Instead we used the \OMEGA|/|\ALEPH\ approach to
+    directionality as starting point, albeit it has been changed quite a bit,
+    so that we're probably not that far from \TEXXET.
+\stopitem
+
+\startitem
+    Some of the tracing information that is output by \ETEX's \prm
+    {tracingassigns} and \prm {tracingrestores} is not there. Also keep in mind
+    that tracing doesn't involve what \LUA\ does.
+\stopitem
+
+\startitem
+    Register management in \LUAMETATEX\ uses the \OMEGA|/|\ALEPH\ model, so the
+    maximum value is 65535 and the implementation uses a flat array instead of
+    the mixed flat & sparse model from \ETEX.
+\stopitem
+
+\startitem
+    Because we don't use change files on top of original \TEX, the integration of
+    \ETEX\ functionality is bit more natural, code wise.
+\stopitem
+
+\stopitemize
+
+\stopsubsection
+
+\startsubsection[title=Changes from \PDFTEX\ 1.40]
+
+\topicindex {\PDFTEX}
+
+Because we want to produce \PDF\ the most natural starting point was the popular
+\PDFTEX\ program. We inherit the stable features, dropped most of the
+experimental code and promoted some functionality to core \LUATEX\ functionality
+which in turn triggered renaming primitives. However, as the backend was dropped,
+not that much from \PDFTEX\ is present any more. Basically all we now inherit
+from \PDFTEX\ is expansion and protrusion but even that has been adapted. So
+don't expect \LUAMETATEX\ to be compatible.
+
+\startitemize
+
+\startitem
+    The experimental primitives \lpr {ifabsnum} and \lpr {ifabsdim} have been
+    promoted to core primitives.
+\stopitem
+
+\startitem
+    The primitives \lpr {ifincsname}, \lpr {expanded} and \lpr {quitvmode}
+    have become core primitives.
+\stopitem
+
+\startitem
+    As the hz (expansion) and protrusion mechanism are part of the core the
+    related primitives \lpr {lpcode}, \lpr {rpcode}, \lpr {efcode}, \lpr
+    {leftmarginkern}, \lpr {rightmarginkern} are promoted to core primitives. The
+    two commands \lpr {protrudechars} and \lpr {adjustspacing} control these
+    processes.
+\stopitem
+
+\startitem
+    In \LUAMETATEX\ three extra primitives can be used to overload the font
+    specific settings: \lpr {adjustspacingstep} (max: 100), \lpr
+    {adjustspacingstretch} (max: 1000) and \lpr {adjustspacingshrink} (max: 500).
+\stopitem
+
+\startitem
+    The hz optimization code has been partially redone so that we no longer need
+    to create extra font instances. The front- and backend have been decoupled
+    and the glyph and kern nodes carry the used values. In \LUATEX\ that made a
+    more efficient generation of \PDF\ code possible. It also resulted in much
+    cleaner code. The backend code is gone, but of course the information is
+    still carried around.
+\stopitem
+
+\startitem
+    When \lpr {adjustspacing} has value~2, hz optimization will be applied to
+    glyphs and kerns. When the value is~3, only glyphs will be treated. A value
+    smaller than~2 disables this feature. With value of~1, font expansion is
+    applied after \TEX's normal paragraph breaking routines have broken the
+    paragraph into lines. In this case, line breaks are identical to standard
+    \TEX\ behavior (as with \PDFTEX). But \unknown\ this is a left|-|over from
+    the early days of \PDFTEX\ when this feature was part of a research topic. At
+    some point level~1 might be dropped from \LUAMETATEX.
+\stopitem
+
+\startitem
+    When \lpr {protrudechars} has a value larger than zero characters at the edge
+    of a line can be made to hang out. A value of~2 will take the protrusion into
+    account when breaking a paragraph into lines. A value of~3 will try to deal
+    with right|-|to|-|left rendering; this is a still experimental feature.
+\stopitem
+
+\startitem
+    The pixel multiplier dimension \lpr {pxdimen} has be inherited as core
+    primitive.
+\stopitem
+
+\startitem
+    The primitive \lpr {tracingfonts} is now a core primitive but doesn't relate
+    to the backend.
+\stopitem
+
+\stopitemize
+
+\stopsubsection
+
+\startsubsection[title=Changes from \ALEPH\ RC4]
+
+\topicindex {\ALEPH}
+
+In \LUATEX\ we took the 32 bit aspects and much of the directional mechanisms and
+merged it into the \PDFTEX\ code base as starting point for further development.
+Then we simplified directionality, fixed it and opened it up. In \LUAMETATEX\ not
+that much of the later is left. We only have two horizontal directions. Instead
+of vertical directions we introduce an orientation model bound to boxes.
+
+The already reduced|-|to|-|four set of directions now only has two members:
+left|-|to|-|right and right|-|to|-|left. They don't do much as it is the backend
+that has to deal with them. When paragraphs are constructed a change in
+horizontal direction is irrelevant for calculating the dimensions. So, basically
+most that we do is registering state and passing that on till the backend can do
+something with it.
+
+Here is a summary of inherited functionality:
+
+\startitemize
+
+\startitem
+    The \type {^^} notation has been extended: after \type {^^^^} four
+    hexadecimal characters are expected and after \type {^^^^^^} six hexadecimal
+    characters have to be given. The original \TEX\ interpretation is still valid
+    for the \type {^^} case but the four and six variants do no backtracking,
+    i.e.\ when they are not followed by the right number of hexadecimal digits
+    they issue an error message. Because \type {^^^} is a normal \TEX\ case, we
+    don't support the odd number of \type {^^^^^} either.
+\stopitem
+
+\startitem
+    Glues {\it immediately after} direction change commands are not legal
+    breakpoints. There is a bit more sanity testing for the direction state.
+\stopitem
+
+\startitem
+    The placement of math formula numbers is direction aware and adapts
+    accordingly. Boxes carry directional information but rules don't.
+\stopitem
+
+\startitem
+    There are no direction related primitives for page and body directions. The
+    paragraph, text and math directions are specified using primitives that
+    take a number.
+\stopitem
+
+\stopitemize
+
+\stopsubsection
+
+\startsubsection[title=Changes from standard \WEBC]
+
+\topicindex {\WEBC}
+
+The \LUAMETATEX\ codebase is not dependent on the \WEBC\ framework. The
+interaction with the file system and \TDS\ is up to \LUA. There still might be
+traces but eventually the code base should be lean and mean. The \METAPOST\
+library is coded in \CWEB\ and in order to be independent from related tools,
+conversion to \CCODE\ is done with a \LUA\ script ran by, surprise, \LUAMETATEX.
+
+\stopsubsection
+
+\stopsection
+
+\startsection[title=Implementation notes]
+
+\startsubsection[title=Memory allocation]
+
+\topicindex {memory}
+
+The single internal memory heap that traditional \TEX\ used for tokens and nodes
+is split into two separate arrays. Each of these will grow dynamically when
+needed. Internally a token or node is an index into these arrays. This permits
+for an efficient implementation and is also responsible for the performance of
+the core. The original documentation in \TEX\ The Program mostly applies!
+
+\stopsubsection
+
+\startsubsection[title=Sparse arrays]
+
+The \prm {mathcode}, \prm {delcode}, \prm {catcode}, \prm {sfcode}, \prm {lccode}
+and \prm {uccode} (and the new \lpr {hjcode}) tables are now sparse arrays that
+are implemented in~\CCODE. They are no longer part of the \TEX\ \quote
+{equivalence table} and because each had 1.1 million entries with a few memory
+words each, this makes a major difference in memory usage. Performance is not
+really hurt by this.
+
+The \prm {catcode}, \prm {sfcode}, \prm {lccode}, \prm {uccode} and \lpr {hjcode}
+assignments don't show up when using the \ETEX\ tracing routines \prm
+{tracingassigns} and \prm {tracingrestores} but we don't see that as a real
+limitation. It also saves a lot of clutter.
+
+A side|-|effect of the current implementation is that \prm {global} is now more
+expensive in terms of processing than non|-|global assignments but not many users
+will notice that.
+
+The glyph ids within a font are also managed by means of a sparse array as glyph
+ids can go up to index $2^{21}-1$ but these are never accessed directly so again
+users will not notice this.
+
+\stopsubsection
+
+\startsubsection[title=Simple single|-|character csnames]
+
+\topicindex {csnames}
+
+Single|-|character commands are no longer treated specially in the internals,
+they are stored in the hash just like the multiletter csnames.
+
+The code that displays control sequences explicitly checks if the length is one
+when it has to decide whether or not to add a trailing space.
+
+Active characters are internally implemented as a special type of multi|-|letter
+control sequences that uses a prefix that is otherwise impossible to obtain.
+
+\stopsubsection
+
+\startsubsection[title=Binary file reading]
+
+\topicindex {files+binary}
+
+All of the internal code is changed in such a way that if one of the \type
+{read_xxx_file} callbacks is not set, then the file is read by a \CCODE\ function
+using basically the same convention as the callback: a single read into a buffer
+big enough to hold the entire file contents. While this uses more memory than the
+previous code (that mostly used \type {getc} calls), it can be quite a bit faster
+(depending on your \IO\ subsystem). So far we never had issues with this approach.
+
+\stopsubsection
+
+\startsubsection[title=Tabs and spaces]
+
+\topicindex {space}
+\topicindex {newline}
+
+We conform to the way other \TEX\ engines handle trailing tabs and spaces. For
+decades trailing tabs and spaces (before a newline) were removed from the input
+but this behaviour was changed in September 2017 to only handle spaces. We are
+aware that this can introduce compatibility issues in existing workflows but
+because we don't want too many differences with upstream \TEXLIVE\ we just follow
+up on that patch (which is a functional one and not really a fix). It is up to
+macro packages maintainers to deal with possible compatibility issues and in
+\LUAMETATEX\ they can do so via the callbacks that deal with reading from files.
+
+The previous behaviour was a known side effect and (as that kind of input
+normally comes from generated sources) it was normally dealt with by adding a
+comment token to the line in case the spaces and|/|or tabs were intentional and
+to be kept. We are aware of the fact that this contradicts some of our other
+choices but consistency with other engines. We still stick to our view that at
+the log level we can (and might be) more incompatible. We already expose some
+more details anyway.
+
+\stopsubsection
+
+\startsubsection[title=Logging]
+
+The information that goes into the log file can be different from \LUATEX, and
+might even differ a bit more in the future. The main reason is that inside the
+engine we have more granularity, which for instance means that we output subtype
+related information when nodes are printed. Of course we could have offered a
+compatibility mode but it serves no purpose. Over time there have been many
+subtle changes to control logs in the \TEX\ ecosystems so another one is
+bearable.
+
+In a similar fashion, there is a bit different behaviour when \TEX\ expects
+input, which in turn is a side effect of removing the interception of \type {*}
+and \type {&} which made for cleaner code (quite a bit had accumulated as side
+effect of continuous adaptations in the \TEX\ ecosystems). There was already code
+that was never executed, simply as side effect of the way \LUATEX\ initializes
+itself (one needs to enable classes of primitives for instance).
+
+\stopsubsection
+
+\stopsection
+
+\stopchapter
+
+\stopcomponent