1 files changed, 275 insertions, 2 deletions
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex b/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex
index 11e8b352d..cf8d2dca8 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex-lua.tex
@@ -189,8 +189,8 @@ cast from string to number and vise versa as this can change in future versions.
 \index {locales}
 
 In stock \LUA, many things depend on the current locale. In \LUAMETATEX, we can't
-do that, because it makes documents unportable. While \LUAMETATEX\ is running if
-forces the following locale settings:
+do that, because it makes documents non|-|portable. While \LUAMETATEX\ is running
+if forces the following locale settings:
 
 \starttyping
 LC_CTYPE=C
@@ -237,6 +237,42 @@ usage better.
 
 \stopsection
 
+\startsection[title={Files}]
+
+\startsubsection[title={File syntax}]
+
+\topicindex {files+names}
+
+\LUAMETATEX\ will accept a braced argument as a file name:
+
+\starttyping
+\input {plain}
+\openin 0 {plain}
+\stoptyping
+
+This allows for embedded spaces, without the need for double quotes. Macro
+expansion takes place inside the argument. Keep in mind that as side effect of
+delegating \IO\ to \LUA\ the \tex {openin} primitive is not provided by the
+engine and has to be implemented by the macro package. This also means that the
+limit on the number of open files is not enforced by the engine.
+
+\stopsubsection
+
+\startsubsection[title={Writing to file}]
+
+\topicindex {files+writing}
+
+Writing to a file in \TEX\ has two forms: delayed and immediate. Delayed writing
+means that the to be written text is anchored in the node list and flushed by the
+backend. As all \IO\ is delegated to \LUA, this also means that it has to deal
+with distinction. In \LUATEX\ the number of open files was already bumped to 127,
+but in \LUAMETATEX\ it depends on the macro package. The special meaning of
+channel 18 was already dropped in \LUATEX\ because we have \type {os.execute}.
+
+\stopsubsection
+
+\stopsection
+
 \startsection[title={Testing}]
 
 \topicindex {testing}
@@ -252,6 +288,243 @@ output file for instance.
 
 \stopsection
 
+\startsection[reference=internals,title={The internals}]
+
+\topicindex{nodes}
+\topicindex{boxes}
+\topicindex{\LUA}
+
+This is a reference manual and not a tutorial. This means that we discuss changes
+relative to traditional \TEX\ and also present new (or extended) functionality.
+As a consequence we will refer to concepts that we assume to be known or that
+might be explained later. Because the \LUATEX\ and \LUAMETATEX\ engines open up
+\TEX\ there's suddenly quite some more to explain, especially about the way a (to
+be) typeset stream moves through the machinery. However, discussing all that in
+detail makes not much sense, because deep knowledge is only relevant for those
+who write code not possible with regular \TEX\ and who are already familiar with
+these internals (or willing to spend time on figuring it out).
+
+So, the average user doesn't need to know much about what is in this manual. For
+instance fonts and languages are normally dealt with in the macro package that
+you use. Messing around with node lists is also often not really needed at the
+user level. If you do mess around, you'd better know what you're dealing with.
+Reading \quotation {The \TEX\ Book} by Donald Knuth is a good investment of time
+then also because it's good to know where it all started. A more summarizing
+overview is given by \quotation {\TEX\ by Topic} by Victor Eijkhout. You might
+want to peek in \quotation {The \ETEX\ manual} too.
+
+But \unknown\ if you're here because of \LUA, then all you need to know is that
+you can call it from within a run. If you want to learn the language, just read
+the well written \LUA\ book. The macro package that you use probably will provide
+a few wrapper mechanisms but the basic \prm {directlua} command that does the job
+is:
+
+\starttyping
+\directlua{tex.print("Hi there")}
+\stoptyping
+
+You can put code between curly braces but if it's a lot you can also put it in a
+file and load that file with the usual \LUA\ commands. If you don't know what
+this means, you definitely need to have a look at the \LUA\ book first.
+
+If you still decide to read on, then it's good to know what nodes are, so we do a
+quick introduction here. If you input this text:
+
+\starttyping
+Hi There ...
+\stoptyping
+
+eventually we will get a linked lists of nodes, which in \ASCII\ art looks like:
+
+\starttyping
+H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ...
+\stoptyping
+
+When we have a paragraph, we actually get something like this, where a \type
+{par} node stores some metadata and is followed by a \type {hlist} flagged as
+indent box:
+
+\starttyping
+[par] <=> [hlist] <=> H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ...
+\stoptyping
+
+Each character becomes a so called glyph node, a record with properties like the
+current font, the character code and the current language. Spaces become glue
+nodes. There are many node types and nodes can have many properties but that will
+be discussed later. Each node points back to a previous node or next node, given
+that these exist. Sometimes multiple characters are represented by one glyph
+(shape), so one can also get:
+
+\starttyping
+[par] <=> [hlist] <=> H <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
+\stoptyping
+
+And maybe some characters get positioned relative to each other, so we might
+see:
+
+\starttyping
+[par] <=> [hlist] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
+\stoptyping
+
+Actually, the above representation is one view, because in \LUAMETATEX\ we can
+choose for this:
+
+\starttyping
+[par] <=> [glue] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
+\stoptyping
+
+where glue (currently fixed) is used instead of an empty hlist (think of a \type
+{\hbox}). Options like this are available because want a certain view on these
+lists from the \LUA\ end and the result being predicable is part of that.
+
+It's also good to know beforehand that \TEX\ is basically centered around
+creating paragraphs and pages. The par builder takes a list and breaks it into
+lines. At some point horizontal blobs are wrapped into vertical ones. Lines are
+so called boxes and can be separated by glue, penalties and more. The page
+builder accumulates lines and when feasible triggers an output routine that will
+take the list so far. Constructing the actual page is not part of \TEX\ but done
+using primitives that permit manipulation of boxes. The result is handled back to
+\TEX\ and flushed to a (often \PDF) file.
+
+\starttyping
+\setbox\scratchbox\vbox\bgroup
+    line 1\par line 2
+\egroup
+
+\showbox\scratchbox
+\stoptyping
+
+The above code produces the next log lines that reveal how the engines sees a
+paragraph (wrapped in a \type {\vbox}):
+
+\starttyping[style=small]
+1:4: > \box257=
+1:4: \vbox[normal][16=1,17=1,47=1], width 483.69687, height 27.58083, depth 0.1416, direction l2r
+1:4: .\list
+1:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r
+1:4: ...\list
+1:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt
+1:4: ....\glue[left][16=1,17=1,47=1] 0.0pt
+1:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt
+1:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519
+1:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt
+1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l
+1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i
+1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n
+1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e
+1:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30
+1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000031 1
+1:4: ....\penalty[line][16=1,17=1,47=1] 10000
+1:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil
+1:4: ....\glue[right][16=1,17=1,47=1] 0.0pt
+1:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt
+1:4: ..\glue[par][16=1,17=1,47=1] 5.44995pt plus 1.81665pt minus 1.81665pt
+1:4: ..\glue[baseline][16=1,17=1,47=1] 6.79396pt
+1:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r
+1:4: ...\list
+1:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt
+1:4: ....\glue[left][16=1,17=1,47=1] 0.0pt
+1:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt
+1:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519
+1:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt
+1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l
+1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i
+1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n
+1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e
+1:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30
+1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000032 2
+1:4: ....\penalty[line][16=1,17=1,47=1] 10000
+1:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil
+1:4: ....\glue[right][16=1,17=1,47=1] 0.0pt
+1:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt
+\stoptyping
+
+The \LUATEX\ engine provides hooks for \LUA\ code at nearly every reasonable
+point in the process: collecting content, hyphenating, applying font features,
+breaking into lines, etc. This means that you can overload \TEX's natural
+behaviour, which still is the benchmark. When we refer to \quote {callbacks} we
+means these hooks. The \TEX\ engine itself is pretty well optimized but when you
+kick in much \LUA\ code, you will notices that performance drops. Don't blame and
+bother the authors with performance issues. In \CONTEXT\ over 50\% of the time
+can be spent in \LUA, but so far we didn't get many complaints about efficiency.
+Adding more callbacks makes no sense, also because at some point the performance
+hit gets too large. There are plenty of ways to achieve goals. For that reason:
+take remarks about \LUATEX, features, potential, performance etc.\ with a natural
+grain of salt.
+
+Where plain \TEX\ is basically a basic framework for writing a specific style,
+macro packages like \CONTEXT\ and \LATEX\ provide the user a whole lot of
+additional tools to make documents look good. They hide the dirty details of font
+management, language support, turning structure into typeset results, wrapping
+pages, including images, and so on. You should be aware of the fact that when you
+hook in your own code to manipulate lists, this can interfere with the macro
+package that you use. Each successive step expects a certain result and if you
+mess around to much, the engine eventually might bark and quit. It can even
+crash, because testing everywhere for what users can do wrong is no real option.
+
+When you read about nodes in the following chapters it's good to keep in mind
+what commands relate to them. Here are a few:
+
+\starttabulate[|l|l|p|]
+\DB command              \BC node          \BC explanation \NC \NR
+\TB
+\NC \prm {hbox}          \NC \nod {hlist} \NC horizontal box \NC \NR
+\NC \prm {vbox}          \NC \nod {vlist} \NC vertical box with the baseline at the bottom \NC \NR
+\NC \prm {vtop}          \NC \nod {vlist} \NC vertical box with the baseline at the top \NC \NR
+\NC \prm {hskip}         \NC \nod {glue}  \NC horizontal skip with optional stretch and shrink \NC \NR
+\NC \prm {vskip}         \NC \nod {glue}  \NC vertical skip with optional stretch and shrink \NC \NR
+\NC \prm {kern}          \NC \nod {kern}  \NC horizontal or vertical fixed skip \NC \NR
+\NC \prm {discretionary} \NC \nod {disc}  \NC hyphenation point (pre, post, replace) \NC \NR
+\NC \prm {char}          \NC \nod {glyph} \NC a character \NC \NR
+\NC \prm {hrule}         \NC \nod {rule}  \NC a horizontal rule \NC \NR
+\NC \prm {vrule}         \NC \nod {rule}  \NC a vertical rule \NC \NR
+\NC \prm {textdirection} \NC \nod {dir}   \NC a change in text direction \NC \NR
+\LL
+\stoptabulate
+
+Whatever we feed into \TEX\ at some point becomes a token which is either
+interpreted directly or stored in a linked list. A token is just a number that
+encodes a specific command (operator) and some value (operand) that further
+specifies what that command is supposed to do. In addition to an interface to
+nodes, there is an interface to tokens, as later chapters will demonstrate.
+
+Text (interspersed with macros) comes from an input medium. This can be a file,
+token list, macro body cq.\ arguments, some internal quantity (like a number),
+\LUA, etc. Macros get expanded. In the process \TEX\ can enter a group. Inside
+the group, changes to registers get saved on a stack, and restored after leaving
+the group. When conditionals are encountered, another kind of nesting happens,
+and again there is a stack involved. Tokens, expansion, stacks, input levels are
+all terms used in the next chapters. Don't worry, they loose their magic once you
+use \TEX\ a lot. You have access to most of the internals and when not, at least
+it is possible to query some state we're in or level we're at.
+
+When we talk about pack(ag)ing it can mean two things. When \TEX\ has consumed
+some tokens that represent text they are added to the current list. When the text
+is put into a so called \prm {hbox} (for instance a line in a paragraph) it
+(normally) first gets hyphenated, next ligatures are build, and finally kerns are
+added. Each of these stages can be overloaded using \LUA\ code. When these three
+stages are finished, the dimension of the content is calculated and the box gets
+its width, height and depth. What happens with the box depends on what macros do
+with it.
+
+The other thing that can happen is that the text starts a new paragraph. In that
+case some information is stored in a leading \type {par} node. Then indentation
+is appended and the paragraph ends with some glue. Again the three stages are
+applied but this time afterwards, the long line is broken into lines and the
+result is either added to the content of a box or to the main vertical list (the
+running text so to say). This is called par building. At some point \TEX\ decides
+that enough is enough and it will trigger the page builder. So, building is
+another concept we will encounter. Another example of a builder is the one that
+turns an intermediate math list into something typeset.
+
+Wrapping something in a box is called packing. Adding something to a list is
+described in terms of contributing. The more complicated processes are wrapped
+into builders. For now this should be enough to enable you to understand the next
+chapters. The text is not as enlightening and entertaining as Don Knuths books,
+sorry.
+
+\stopsection
+
 \stopchapter
 
 \stopcomponent