path: root/doc/context/sources/general/manuals/luametatex/luametatex-differences.tex

% language=us runpath=texruns:manuals/luametatex

\environment luametatex-style

\startcomponent luametatex-differences

\startchapter[reference=differences,title={Differences with \LUATEX}]

As \LUAMETATEX\ is a leaner and meaner \LUATEX. This means that substantial parts and
dependencies are gone: quite some font code, all backend code with related frontend
code and of course image and font inclusion. There is also new functionality which
makes for less lean but in the end we still have less, also in terms of dependencies.
This chapter will discuss what is gone. We start with the primitives that were dropped.

\starttabulate[|l|pl|]
\BC fonts       \NC \type {\letterspacefont}
                    \type {\copyfont}
                    \type {\expandglyphsinfont}
                    \type {\ignoreligaturesinfont}
                    \type {\tagcode}
                    \type {\leftghost}
                    \type {\rightghost}
                \NC \NR
\BC backend     \NC \type {\dviextension}
                    \type {\dvivariable }
                    \type {\dvifeedback}
                    \type {\pdfextension}
                    \type {\pdfvariable }
                    \type {\pdffeedback}
                    \type {\dviextension}
                    \type {\draftmode}
                    \type {\outputmode}
                \NC \NR
\BC dimensions  \NC \type {\pageleftoffset}
                    \type {\pagerightoffset}
                    \type {\pagetopoffset}
                    \type {\pagebottomoffset}
                    \type {\pageheight}
                    \type {\pagewidth}
                \NC \NR
\BC resources   \NC \type {\saveboxresource}
                    \type {\useboxresource}
                    \type {\lastsavedboxresourceindex}
                    \type {\saveimageresource}
                    \type {\useimageresource}
                    \type {\lastsavedimageresourceindex}
                    \type {\lastsavedimageresourcepages}
                \NC \NR
\BC positioning \NC \type {\savepos}
                    \type {\lastxpos}
                    \type {\lastypos}
                \NC \NR
\BC directions  \NC \type {\textdir}
                    \type {\linedir}
                    \type {\mathdir}
                    \type {\pardir}
                    \type {\pagedir}
                    \type {\bodydir}
                    \type {\pagedirection}
                    \type {\bodydirection}
                \NC \NR
\BC randomizer  \NC \type {\randomseed}
                    \type {\setrandomseed}
                    \type {\normaldeviate}
                    \type {\uniformdeviate}
                \NC \NR
\BC utilities   \NC \type {\synctex}
                \NC \NR
\BC extensions  \NC \type {\latelua}
                    \type {\lateluafunction}
                    \type {\openout}
                    \type {\write}
                    \type {\closeout}
                    \type {\openin}
                    \type {\read}
                    \type {\readline}
                    \type {\closein}
                    \type {\ifeof}
                \NC \NR
\BC control     \NC \type {\suppressfontnotfounderror}
                    \type {\suppresslongerror}
                    \type {\suppressprimitiveerror}
                    \type {\suppressmathparerror}
                    \type {\suppressifcsnameerror}
                    \type {\suppressoutererror}
                    \type {\mathoption}
                \NC \NR
\BC system      \NC \type {\primitive}
                    \type {\ifprimitive}
                    \type {\formatname}
                \NC \NR
\BC ignored     \NC \type {\long}
                    \type {\outer}
                    \type {\mag}
                \NC \NR
\stoptabulate

The resources and positioning primitives are actually useful but can be defined
as macros that (via \LUA) inject nodes in the input that suit the macro package
and backend. The three||letter direction primitives are gone and the numeric
variants are now leading. There is no need for page and body related directions
and they don't work well in \LUATEX\ anyway. We only have two directions left.
Because we can hook in \LUA\ functions that get information about what is expected
(consumer or provider) there are plenty possibilities for adding functionality
using this scripting langauge.

The primitive related extensions were not that useful and reliable so they have
been removed. There are some new variants that will be discussed later. The \type
{\outer} and \type {\long} prefixes are gone as they don't make much sense
nowadays and them becoming dummies opened the way to something new: control
sequence properties that permit protection against as well as controlled
overloading of definitions. I don't think that (\CONTEXT) users will notice these
prefixes being gone. The definition and parsing related \type {\suppress..}
features are now default and can't be changed so related primitives are gone.

The \type {\shipout} primitive does no ship out but just erases the content of
the box unless of course that has happened already in another way. A macro
package should implement its own backend and related shipout. Talking of backend,
the extension primitives that relate to backends can be implemented as part of a
backend design using generic whatsits. There is only one type of whatsit now. In
fact we're now closer to original \TEX\ with respect to the extensions.

The \type {img} library has been removed as it's rather bound to the backend. The
\type {slunicode} library is also gone. There are some helpers in the string
library that can be used instead and one can write additional \LUA\ code if
needed. There is no longer a \type {pdf} backend library but we have an up to
date \PDF\ parsing library on board.

In the \type {node}, \type {tex} and \type {status} library we no longer have
helpers and variables that relate to the backend. The \LUAMETATEX\ engine is in
principle \DVI\ and \PDF\ unaware. There are, as mentioned, only generic whatsit
nodes that can be used for some management related tasks. For instance you can
use them to implement user nodes. More extensive status information is provided
in the overhauled status library. All libraries have additional functionality and
names of functions have been normalized (for as far as possible).

The margin kern nodes are gone and we now use regular kern nodes for them. As a
consequence there are two extra subtypes indicating the injected left or right
kern. The glyph field served no real purpose so there was no reason for a special
kind of node.

The \KPSE\ library is no longer built|-|in, but one can use an external \KPSE\
library, assuming that it is present on the system, because the engine has a so
called optional library interface to it. Because there is no backend, quite some
file related callbacks could go away. The following file related callbacks
remained (till now):

\starttyping
find_write_file find_format_file open_data_file
\stoptyping

The callbacks related to errors are changed:

\starttyping
intercept_tex_error intercept_lua_error
show_error_message show_warning_message
\stoptyping

There is a hook that gets called when one of the fundamental memory structures
gets reallocated.

\starttyping
trace_memory
\stoptyping

When you use the overload protect mechanisms, a callback can be plugged in to handle
exceptions:

\starttyping
handle_overload
\stoptyping

The (job) management hooks are kept:

\starttyping
process_jobname
start_run stop_run wrapup_run
pre_dump
start_file stop_file
\stoptyping

Because we use a more generic whatsit model, there is a new callback:

\starttyping
show_whatsit
\stoptyping

Because tracing boxes now reports a lot more information, we have a plug in for
detail:

\starttyping
get_attribute
\stoptyping

Being the core of extensibility, the typesetting callbacks of course stayed. This
is what we ended up with:

\startalign[flushleft,nothyphenated]
\tt \cldcontext{table.concat(table.sortedkeys(callbacks.list), ", ")}
\stopalign

As in \LUATEX\ font loading happens with the following callback. This time it
really needs to be set because there is no built|-|in font loader.

\starttyping
define_font
\stoptyping

There are all kinds of subtle differences in the implementation, for instance we
no longer intercept \type {*} and \type {&} as these were already replaced long
ago in \TEX\ engines by command line options. Talking of options, only a few are
left. All input goes via \LUA, even the console. One can program a terminal if
needed.

We took our time for reaching a stable state in \LUATEX. Among the reasons is the
fact that most was experimented with in \CONTEXT, which we can adapt to the
engine as we go. It took many years to decide what to keep and how to do things.
Of course there are places when things can be improved but that most likely only
happens in \LUAMETATEX. Contrary to what is sometimes suggested, the
\LUATEX|-|\CONTEXT\ \MKIV\ combination (assuming matched versions) has been quite
stable. It made no sense otherwise. Most \CONTEXT\ functionality didn't change
much at the user level. Of course there have been issues, as is natural with
everything new and beta, but we have a fast update cycle.

The same is true for \LUAMETATEX\ and \CONTEXT\ \LMTX: it can be used for
production as usual and in practice \CONTEXT\ users tend to use the beta
releases, which proves this. Of course, if you use low level features that are
experimental you're on your own. Also, as with \LUATEX\ it might take many years
before a long term stable is defined. The good news is that, when the source code
has become part of the \CONTEXT\ distribution, there is always a properly
working, more or less long term stable, snapshot.

The error reporting subsystem has been redone quite a bit but is still
fundamentally the same. We don't really assume interactive usage but if someone
uses it, it might be noticed that it is not possible to backtrack or inject
something. Of course it is no big deal to implement all that in \LUA\ if needed.
It removes a system dependency and makes for a bit cleaner code. In \CONTEXT\ we
quit on an error simply because one has to fix source anyway and runs are fast
enough. Logging provides more detail and new primitives can be used to prevent
clutter in tracing (the more complex a macro package becomes, the more extreme
tracing becomes).

There are new primitives as well as some extensions to existing primitive
functionality. These are described in following chapters but there might be
hidden treasures in the binary. If you locate them, don't automatically assume
them to stay, some might be part of experiments! There are for instance a few
csname related definers, we have integer and dimension constants, the macro
argument parser can be brought in tolerant mode, the repertoire of conditionals
has been extended, some internals can be controlled (think of normalization of
lines, hyphenation etc.), and macros can be protected against user overload. Not
all is discussed in detail in this manual but there are introductions in the
\CONTEXT\ distribution that explain them. But the \TEX\ kernel is of course
omnipresent.

\startluacode

local luametatex = tex.primitives()
local luatex     = table.load("luatex-primitives.lua")

if not luatex then
    local tex = "\\starttext \\ctxlua {table.save(tex.jobname .. '.lua',tex.primitives())} \\stoptext"

    io.savedata("luatex-primitives.tex",    tex)

    os.execute("context --luatex --once luatex-primitives")

    luatex = table.load("luatex-primitives.lua")
end


if luatex and luametatex then

    luatex     = table.tohash(luatex)
    luametatex = table.tohash(luametatex)

 -- context.page()

    context("The following primitives are available in \\LUATEX\\ but not in \\LUAMETATEX.  ")
    context("Some of these are emulated in \\CONTEXT.")

    context.blank()
    context.startcolumns { n = 2 }
        for k, v in table.sortedhash(luatex) do
            if not luametatex[k] then
                context.type(k)
                context.crlf()
            end
        end
    context.stopcolumns()
    context.blank()

 -- context.page()

    context("The following primitives are available in \\LUAMETATEX\\ only. ")
    context("At some point in time some might be added to \\LUATEX.")

    context.blank()
    context.startcolumns { n = 2 }
        for k, v in table.sortedhash(luametatex) do
            if not luatex[k] then
                context.type(k)
                context.crlf()
            end
        end
    context.stopcolumns()
    context.blank()

end

\stopluacode

As part of a bit more consistency some function names also changed. Names with an
\type {_} got that removed (as that was the minority). It's easy to provide a
back mapping if needed (just alias the functions).

{\em Todo: only mention the \LUATEX\ ones.}

\starttabulate[|l|l|l|l|]
\DB library  \BC old name          \BC new name         \BC comment \NC \NR
\TB
\NC language \NC clear_patterns    \NC clearpatterns    \NC \NR
\NC          \NC clear_hyphenation \NC clearhyphenation \NC \NR
\NC mplib    \NC italcor           \NC italic           \NC \NR
\NC          \NC pen_info          \NC peninfo          \NC \NR
\NC          \NC solve_path        \NC solvepath        \NC \NR
\NC texio    \NC write_nl          \NC writenl          \NC old name stays \NC \NR
\NC node     \NC protect_glyph     \NC protectglyph     \NC \NR
\NC          \NC protect_glyphs    \NC protectglyphs    \NC \NR
\NC          \NC unprotect_glyph   \NC unprotectglyph   \NC \NR
\NC          \NC unprotect_glyphs  \NC unprotectglyphs  \NC \NR
\NC          \NC end_of_math       \NC endofmath        \NC \NR
\NC          \NC mlist_to_hlist    \NC mlisttohlist     \NC \NR
\NC          \NC effective_glue    \NC effectiveglue    \NC \NR
\NC          \NC has_glyph         \NC hasglyph         \NC \NR
\NC          \NC first_glyph       \NC firstglyph       \NC \NR
\NC          \NC has_field         \NC hasfield         \NC \NR
\NC          \NC copy_list         \NC copylist         \NC \NR
\NC          \NC flush_node        \NC flushnode        \NC \NR
\NC          \NC flush_list        \NC flushlist        \NC \NR
\NC          \NC insert_before     \NC insertbefore     \NC \NR
\NC          \NC insert_after      \NC insertafter      \NC \NR
\NC          \NC last_node         \NC lastnode         \NC \NR
\NC          \NC is_zero_glue      \NC iszeroglue       \NC \NR
\NC          \NC make_extensible   \NC makeextensible   \NC \NR
\NC          \NC uses_font         \NC usesfont         \NC \NR
\NC          \NC is_char           \NC ischar           \NC \NR
\NC          \NC is_direct         \NC isdirect         \NC \NR
\NC          \NC is_glyph          \NC isglyph          \NC \NR
\NC          \NC is_node           \NC isnode           \NC \NR
\NC token    \NC scan_keyword      \NC scankeyword      \NC \NR
\NC          \NC scan_keywordcs    \NC scankeywordcs    \NC \NR
\NC          \NC scan_int          \NC scanint          \NC \NR
\NC          \NC scan_real         \NC scanreal         \NC \NR
\NC          \NC scan_float        \NC scanfloat        \NC \NR
\NC          \NC scan_dimen        \NC scandimen        \NC \NR
\NC          \NC scan_glue         \NC scanglue         \NC \NR
\NC          \NC scan_toks         \NC scantoks         \NC \NR
\NC          \NC scan_code         \NC scancode         \NC \NR
\NC          \NC scan_string       \NC scanstring       \NC \NR
\NC          \NC scan_argument     \NC scanargument     \NC \NR
\NC          \NC scan_word         \NC scanword         \NC \NR
\NC          \NC scan_csname       \NC scancsname       \NC \NR
\NC          \NC scan_list         \NC scanlist         \NC \NR
\NC          \NC scan_box          \NC scanbox          \NC \NR
\LL
\stoptabulate

It's all part of trying to make the code base consistent but it is sometimes a
bit annoying. However, that's why we develop this engine independent of the
\LUATEX\ code base. It's anyway a change that has been on my todo list for quite
a while because those inconsistencies annoyed me.

\stopchapter

\stopcomponent