2018-04-13 15:02:00

1 files changed, 160 insertions, 150 deletions

diff --git a/doc/context/sources/general/manuals/luatex/luatex-lua.tex b/doc/context/sources/general/manuals/luatex/luatex-lua.tex
index 82b060440..c8df17da9 100644
--- a/doc/context/sources/general/manuals/luatex/luatex-lua.tex
+++ b/doc/context/sources/general/manuals/luatex/luatex-lua.tex
@@ -46,7 +46,9 @@ There are two situations that make \LUATEX\ behave like the \LUA\ byte compiler:
 
 In this mode, \LUATEX\ is exactly like \type {luac} from the stand alone \LUA\
 distribution, except that it does not have the \type {-l} switch, and that it
-accepts (but ignores) the \type {--luaconly} switch.
+accepts (but ignores) the \type {--luaconly} switch. The current version of \LUA\
+can dump bytecode using \type {string.dump} so we might decide to drop this
+version if \LUATEX.
 
 \subsection{Other commandline processing}
 
@@ -84,10 +86,8 @@ consequence. The following command|-|line options are understood:
 \NC \type{--version}                    \NC display version and exit \NC \NR
 \stoptabulate
 
-Some of the traditional flags are just ignored: \type {--etex}, \type
-{--translate-file}, \type {--8bit}. \type {--[no-]parse-first-line}, \type
-{--default-translate-file}. Also, we no longer support write18 because \type
-{os.execute} can do the same.
+We don't support \type {\write18} because \type {os.execute} can do the same. It
+simplifies the code and makes more write targets possible.
 
 The value to use for \type {\jobname} is decided as follows:
 
@@ -104,10 +104,10 @@ The value to use for \type {\jobname} is decided as follows:
     part following the last \type {.}) stripped off.
 \stopitem
 \startitem
-    An exception to the previous point: if the command line goes into interactive
-    mode (by starting with a command) and there are no files input via \type
-    {\everyjob} either, then the \type {\jobname} is set to \type {texput} as a
-    last resort.
+    There is an exception to the previous point: if the command line goes into
+    interactive mode (by starting with a command) and there are no files input
+    via \type {\everyjob} either, then the \type {\jobname} is set to \type
+    {texput} as a last resort.
 \stopitem
 \stopitemize
 
@@ -116,61 +116,73 @@ attaching the proper extension (\type {log}, \type {pdf}, etc.) to the found
 \type {\jobname}. These files are created in the directory pointed to by \type
 {--output-directory}, or in the current directory, if that switch is not present.
 
-\blank
-
 Without the \type {--lua} option, command line processing works like it does in
-any other web2c-based typesetting engine, except that \LUATEX\ has a few extra
-switches.
-
-If the \type {--lua} option is present, \LUATEX\ will enter an alternative mode
-of command line processing in comparison to the standard web2c programs.
-
-In this mode, a small series of actions is taken in order. First, it will parse
-the command line as usual, but it will only interpret a small subset of the
-options immediately: \type {--safer}, \type {--nosocket}, \type
-{--[no-]shell-escape}, \type {--enable-write18}, \type {--disable-write18}, \type
-{--shell-restricted}, \type {--help}, \type {--version}, and \type {--credits}.
-
-Next \LUATEX\ searches for the requested \LUA\ initialization script. If it
-cannot be found using the actual name given on the command line, a second attempt
-is made by prepending the value of the environment variable \type {LUATEXDIR}, if
-that variable is defined in the environment.
-
-Then it checks the various safety switches. You can use those to disable some
-\LUA\ commands that can easily be abused by a malicious document. At the moment,
-\type {--safer} \type {nil}s the following functions:
-
-\starttabulate[|l|l|]
-\BC library      \BC functions \NC \NR
-\NC \type {os}   \NC \type {execute} \type {exec} \type {spawn} \type {setenv} \type {rename} \type {remove} \type {tmpdir} \NC \NR
-\NC \type {io}   \NC \type {popen} \type {output} \type {tmpfile} \NC \NR
-\NC \type {lfs}  \NC \type {rmdir} \type {mkdir} \type {chdir} \type {lock} \type {touch} \NC \NR
-\stoptabulate
+any other \WEBC|-|based typesetting engine, except that \LUATEX\ has a few extra
+switches and lacks some others. Also, if the \type {--lua} option is present,
+\LUATEX\ will enter an alternative mode of command line processing in comparison
+to the standard \WEBC\ programs. In this mode, a small series of actions is taken
+in order.
 
-Furthermore, it disables loading of compiled \LUA\ libraries and it makes \type
-{io.open()} fail on files that are opened for anything besides reading.
+\startitemize[n]
 
-When \LUATEX\ starts it set the locale to a neutral value. If for some reason you
-use \type {os.locale}, you need to make sure you \type {nil} it afterwards
-because otherwise it can interfere with code that for instance generates dates.
-You can nil the locale with
+\startitem
+    First, it will parse the command line as usual, but it will only interpret a
+    small subset of the options immediately: \type {--safer}, \type {--nosocket},
+    \type {--[no-]shell-escape}, \type {--enable-write18}, \type
+    {--disable-write18}, \type {--shell-restricted}, \type {--help}, \type
+    {--version}, and \type {--credits}.
+\stopitem
 
-\starttyping
-os.setlocale(nil.nil)
-\stoptyping
+\startitem
+    Next \LUATEX\ searches for the requested \LUA\ initialization script. If it
+    cannot be found using the actual name given on the command line, a second
+    attempt is made by prepending the value of the environment variable \type
+    {LUATEXDIR}, if that variable is defined in the environment.
+\stopitem
+
+\startitem
+    Then it checks the various safety switches. You can use those to disable some
+    \LUA\ commands that can easily be abused by a malicious document. At the
+    moment, \type {--safer} \type {nil}s the following functions:
+
+    \starttabulate[|l|l|]
+        \BC library      \BC functions \NC \NR
+        \NC \type {os}   \NC \type {execute} \type {exec} \type {spawn} \type {setenv} \type {rename} \type {remove} \type {tmpdir} \NC \NR
+        \NC \type {io}   \NC \type {popen} \type {output} \type {tmpfile} \NC \NR
+        \NC \type {lfs}  \NC \type {rmdir} \type {mkdir} \type {chdir} \type {lock} \type {touch} \NC \NR
+    \stoptabulate
+
+    Furthermore, it disables loading of compiled \LUA\ libraries and it makes
+    \type {io.open()} fail on files that are opened for anything besides reading.
+\stopitem
+
+\startitem
+    When \LUATEX\ starts it set the locale to a neutral value. If for some reason
+    you use \type {os.locale}, you need to make sure you \type {nil} it
+    afterwards because otherwise it can interfere with code that for instance
+    generates dates. You can nil the locale with
+
+    \starttyping
+    os.setlocale(nil.nil)
+    \stoptyping
 
-The \type {--nosocket} option makes the socket library unavailable, so that \LUA\
-cannot use networking.
+    The \type {--nosocket} option makes the socket library unavailable, so that \LUA\
+    cannot use networking.
+
+    The switches \type {--[no-]shell-escape}, \type {--[enable|disable]-write18}, and
+    \type {--shell-restricted} have the same effects as in \PDFTEX, and additionally
+    make \type {io.popen()}, \type {os.execute}, \type {os.exec} and \type {os.spawn}
+    adhere to the requested option.
+\stopitem
 
-The switches \type {--[no-]shell-escape}, \type {--[enable|disable]-write18}, and
-\type {--shell-restricted} have the same effects as in \PDFTEX, and additionally
-make \type {io.popen()}, \type {os.execute}, \type {os.exec} and \type {os.spawn}
-adhere to the requested option.
+\startitem
+    Next the initialization script is loaded and executed. From within the
+    script, the entire command line is available in the \LUA\ table \type {arg},
+    beginning with \type {arg[0]}, containing the name of the executable. As
+    consequence warnings about unrecognized options are suppressed.
+\stopitem
 
-Next the initialization script is loaded and executed. From within the script,
-the entire command line is available in the \LUA\ table \type {arg}, beginning with
-\type {arg[0]}, containing the name of the executable. As consequence warnings
-about unrecognized options are suppressed.
+\stopitemize
 
 Command line processing happens very early on. So early, in fact, that none of
 \TEX's initializations have taken place yet. For that reason, the tables that
@@ -209,12 +221,9 @@ check \type {--progname}, or \type {--ini} and \type {--fmt}, if \type
 
 \LUA s \type {tostring} function (and \type {string.format} may return values in
 scientific notation, thereby confusing the \TEX\ end of things when it is used as
-the right|-|hand side of an assignment to a \type {\dimen} or \type {\count}.
-
-Loading dynamic \LUA\ libraries will fail if there are two \LUA\ libraries loaded
-at the same time (which will typically happen on \type {win32}, because there is
-one \LUA\ 5.3 inside \LUATEX, and another will likely be linked to the \DLL\ file
-of the module itself).
+the right|-|hand side of an assignment to a \type {\dimen} or \type {\count}. The
+output of these serializers also depend on the \LUA\ version, so in \LUA\ 5.3 you
+can get different output than from 5.2.
 
 \LUATEX\ is able to use the kpathsea library to find \type {require()}d modules.
 For this purpose, \type {package.searchers[2]} is replaced by a different loader
@@ -233,12 +242,10 @@ decides at runtime whether to use \KPATHSEA\ or the built|-|in core \LUA\
 function. It uses \KPATHSEA\ when that is already initialized at that point in
 time, otherwise it reverts to using the normal \type {package.cpath} loader.
 
-This functionality required an extension to kpathsea:
-
-\startnarrower
-There is a new kpathsea file format: \type {kpse_clua_format} that searches for
-files with extension \type {.dll} and \type {.so}. The \type {texmf.cnf} setting
-for this variable is \type {CLUAINPUTS}, and by default it has this value:
+This functionality required an extension to kpathsea. There is a new kpathsea
+file format: \type {kpse_clua_format} that searches for files with extension
+\type {.dll} and \type {.so}. The \type {texmf.cnf} setting for this variable is
+\type {CLUAINPUTS}, and by default it has this value:
 
 \starttyping
 CLUAINPUTS=.:$SELFAUTOLOC/lib/{$progname,$engine,}/lua//
@@ -248,12 +255,14 @@ This path is imperfect (it requires a \TDS\ subtree below the binaries
 directory), but the architecture has to be in the path somewhere, and the
 currently simplest way to do that is to search below the binaries directory only.
 Of course it no big deal to write an alternative loader and use that in a macro
-package.
+package. One level up (a \type {lib} directory parallel to \type {bin}) would
+have been nicer, but that is not doable because \TEXLIVE\ uses a \type
+{bin/<arch>} structure.
 
-One level up (a \type {lib} directory parallel to \type {bin}) would have been
-nicer, but that is not doable because \TEXLIVE\ uses a \type {bin/<arch>}
-structure.
-\stopnarrower
+Loading dynamic \LUA\ libraries will fail if there are two \LUA\ libraries loaded
+at the same time (which will typically happen on \type {win32}, because there is
+one \LUA\ 5.3 inside \LUATEX, and another will likely be linked to the \DLL\ file
+of the module itself).
 
 In keeping with the other \TEX|-|like programs in \TEXLIVE, the two \LUA\ functions
 \type {os.execute} and \type {io.popen}, as well as the two new functions \type
@@ -267,30 +276,28 @@ four functions above if the matching \type {texmf.cnf} variable(s) or their \typ
 requested system command. In \quote {script interpreter} runs of \LUATEX, these
 settings have no effect, and all four functions function as normal.
 
-The \type {f:read("*line")} and \type {f:lines()} functions from the io library
-have been adjusted so that they are line|-|ending neutral: any of \type {LF},
-\type {CR} or \type {CR+LF} are acceptable line endings.
-
-\type {luafilesystem} has been extended: there are two extra boolean functions
-(\type {lfs.isdir(filename)} and \type {lfs.isfile(filename)}) and one extra
-string field in its attributes table (\type {permissions}). There is an
-additional function \type {lfs.shortname()} which takes a file name and returns
-its short name on \type {win32} platforms. On other platforms, it just returns
-the given argument. The file name is not tested for existence. Finally, for
-non|-|\type {win32} platforms only, there is the new function \type
-{lfs.readlink()} hat takes an existing symbolic link as argument and returns its
-content. It returns an error on \type {win32}.
-
-The \type {string} library has an extra function: \type {string.explode(s[,m])}.
-This function returns an array containing the string argument \type {s} split
-into sub-strings based on the value of the string argument \type {m}. The second
-argument is a string that is either empty (this splits the string into
-characters), a single character (this splits on each occurrence of that
-character, possibly introducing empty strings), or a single character followed by
-the plus sign \type {+} (this special version does not create empty sub-strings).
-The default value for \type {m} is \quote {\type { +}} (multiple spaces). Note:
-\type {m} is not hidden by surrounding braces as it would be if this function was
-written in \TEX\ macros.
+Some libraries have a few more functions, either coded in \CCODE\ or in \LUA.
+For instance, when we started with \LUATEX\ we added some helpers to the \type
+{luafilesystem} namespace \type {lfs}. The two boolean functions \type
+{lfs.isdir} and \type {lfs.isfile} were speedy and better variants of what could
+be done with \type {lfs.attributes}. The additional function \type
+{lfs.shortname} takes a file name and returns its short name on \type {win32}
+platforms. Finally, for non|-|\type {win32} platforms only, we provided \type
+{lfs.readlink} that takes an existing symbolic link as argument and returns its
+name. However, the \type library evolved sop now we dropped these in favour of
+pure \LUA\ variants. The \type {shortname} helper is considered obsolete and now
+just returns the name.
+
+The \type {string} library has a few extra functions like \type
+{string.explode(s[,m])}. This function returns an array containing the string
+argument \type {s} split into sub-strings based on the value of the string
+argument \type {m}. The second argument is a string that is either empty (this
+splits the string into characters), a single character (this splits on each
+occurrence of that character, possibly introducing empty strings), or a single
+character followed by the plus sign \type {+} (this special version does not
+create empty sub-strings). The default value for \type {m} is \quote {\type { +}}
+(multiple spaces). Note: \type {m} is not hidden by surrounding braces as it
+would be if this function was written in \TEX\ macros.
 
 The \type {string} library also has six extra iterators that return strings
 piecemeal:
@@ -323,7 +330,8 @@ are useful especially in the conversion of \UTF16 encoded data into \UTF8.
 
 There is also a two|-|argument form of \type {string.dump()}. The second argument
 is a boolean which, if true, strips the symbols from the dumped data. This
-matches an extension made in \type {luajit}.
+matches an extension made in \type {luajit}. This is typically a function that
+gets adapted as \LUA\ itself progresses.
 
 The \type {string} library functions \type {len}, \type {lower}, \type {sub}
 etc.\ are not \UNICODE|-|aware. For strings in the \UTF8 encoding, i.e., strings
@@ -355,8 +363,8 @@ similar helpers too:
 \stopitemize
 
 These three functions are relative fast and don't do much checking. They can be used
-as building blocks for other helpers.
-
+as building blocks for other helpers. So, eventually we can decide to drop the
+\type {sln} library, just that you know.
 
 \blank
 
@@ -373,25 +381,37 @@ The \type {os} library has a few extra functions and variables:
     \type {os.exec(commandline)} is a variation on \type {os.execute}. Here
     \type {commandline} can be either a single string or a single table.
 
-    If the argument is a table \LUATEX\ first checks if there is a value at
-    integer index zero. If there is, this is the command to be executed.
-    Otherwise, it will use the value at integer index one. If neither are
-    present, nothing at all happens.
-
-    The set of consecutive values starting at integer~1 in the table are the
-    arguments that are passed on to the command (the value at index~1 becomes
-    \type {arg[0]}). The command is searched for in the execution path, so there
-    is normally no need to pass on a fully qualified path name.
-
-    If the argument is a string, then it is automatically converted into a table
-    by splitting on whitespace. In this case, it is impossible for the command
-    and first argument to differ from each other.
-
-    In the string argument format, whitespace can be protected by putting (part
-    of) an argument inside single or double quotes. One layer of quotes is
-    interpreted by \LUATEX, and all occurrences of \type {\"}, \type {\'} or \type
-    {\\} within the quoted text are unescaped. In the table format, there is no
-    string handling taking place.
+    \startitemize
+
+    \startitem
+        If the argument is a table \LUATEX\ first checks if there is a value at
+        integer index zero. If there is, this is the command to be executed.
+        Otherwise, it will use the value at integer index one. If neither are
+        present, nothing at all happens.
+    \stopitem
+
+    \startitem
+        The set of consecutive values starting at integer~1 in the table are the
+        arguments that are passed on to the command (the value at index~1 becomes
+        \type {arg[0]}). The command is searched for in the execution path, so
+        there is normally no need to pass on a fully qualified path name.
+    \stopitem
+
+    \startitem
+        If the argument is a string, then it is automatically converted into a
+        table by splitting on whitespace. In this case, it is impossible for the
+        command and first argument to differ from each other.
+    \stopitem
+
+    \startitem
+        In the string argument format, whitespace can be protected by putting
+        (part of) an argument inside single or double quotes. One layer of quotes
+        is interpreted by \LUATEX, and all occurrences of \type {\"}, \type {\'}
+        or \type {\\} within the quoted text are unescaped. In the table format,
+        there is no string handling taking place.
+    \stopitem
+
+    \stopitemize
 
     This function normally does not return control back to the \LUA\ script: the
     command will replace the current process. However, it will return the two
@@ -496,11 +516,27 @@ Some modules that are normally external to \LUA\ are statically linked in with
 \startitemize
 
 \startitem
+    \type {lpeg}, by Roberto Ierusalimschy, \hyphenatedurl
+    {http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html}. This library is not
+    \UNICODE|-|aware, but interprets strings on a byte|-|per|-|byte basis. This
+    mainly means that \type {lpeg.S} cannot be used with \UTF8 characters encoded
+    in more than two bytes, and thus \type {lpeg.S} will look for one of those
+    two bytes when matching, not the combination of the two. The same is true for
+    \type {lpeg.R}, although the latter will display an error message if used
+    with multibyte characters. Therefore \type {lpeg.R('aä')} results in the
+    message \type {bad argument #1 to 'R' (range must have two characters)},
+    since to \type {lpeg}, \type {ä} is two 'characters' (bytes), so \type {aä}
+    totals three. In practice this is no real issue and with some care you can
+    deal with \UNICODE\ just fine.
+\stopitem
+
+\startitem
     \type {slnunicode}, from the \type {selene} libraries, \hyphenatedurl
     {http://luaforge.net/projects/sln}. This library has been slightly extended
     so that the \type {unicode.utf8.*} functions also accept the first 256 values
     of plane~18. This is the range \LUATEX\ uses for raw binary output, as
-    explained above.
+    explained above. We have no plans to provide more like this because you can
+    basically do all that you want in \LUA.
 \stopitem
 
 \startitem
@@ -514,20 +550,6 @@ Some modules that are normally external to \LUA\ are statically linked in with
 \stopitem
 
 \startitem
-    \type {lpeg}, by Roberto Ierusalimschy, \hyphenatedurl
-    {http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html}. This library is not
-    \UNICODE|-|aware, but interprets strings on a byte|-|per|-|byte basis. This
-    mainly means that \type {lpeg.S} cannot be used with \UTF8 characters encoded
-    in more than two bytes, and thus \type {lpeg.S} will look for one of those
-    two bytes when matching, not the combination of the two. The same is true for
-    \type {lpeg.R}, although the latter will display an error message if used
-    with multibyte characters. Therefore \type {lpeg.R('aä')} results in the
-    message \type {bad argument #1 to 'R' (range must have two characters)},
-    since to \type {lpeg}, \type {ä} is two 'characters' (bytes), so \type {aä}
-    totals three. In practice this is no real issue.
-\stopitem
-
-\startitem
     \type {lzlib}, by Tiago Dionizio, \hyphenatedurl
     {http://luaforge.net/projects/lzlib/}.
 \stopitem
@@ -546,10 +568,6 @@ Some modules that are normally external to \LUA\ are statically linked in with
 
 \stopitemize
 
-At some point (this also depends on distributions) \LUATEX\ might have these
-libraries loaded on demand. For this reason you can best use \type {require} to
-make sure they are loaded.
-
 \section{Testing}
 
 For development reasons you can influence the used startup date and time. This can
@@ -579,17 +597,9 @@ When Universal Time is needed, you can pass the flag \type {utc} to the engine.
 property also works when the date and time are set by \LUATEX\ itself. It has a
 complementary entry \type {use_utc_time} in the \type {texconfig} table.
 
-\startnotabene
-    To some extend a cleaner solution would be to have a flag that disables all
-    variable data in one go (like filenames and so) but we just follow the method
-    implemented in \PDFTEX\ where primitives are used to influence other
-    properties.
-\stopnotabene
-
-\startnotabene
-    In \CONTEXT\ we provide the command line argument \type {--nodates} that does
-    bit more disabling of dates.
-\stopnotabene
+There is some control possible, for instance prevent filename to be written to
+the \PDF\ file. This is discussed elsewhere. In \CONTEXT\ we provide the command
+line argument \type {--nodates} that does bit more disabling of dates.
 
 \stopchapter