path: root/doc/context/sources/general/manuals/tools/tools-mkiv.tex

% language=uk

% author    : Hans Hagen
% copyright : PRAGMA ADE & ConTeXt Development Team
% license   : Creative Commons Attribution ShareAlike 4.0 International
% reference : pragma-ade.nl | contextgarden.net | texlive (related) distributions
% origin    : the ConTeXt distribution
%
% comment   : Because this manual is distributed with TeX distributions it comes with a rather
%             liberal license. We try to adapt these documents to upgrades in the (sub)systems
%             that they describe. Using parts of the content otherwise can therefore conflict
%             with existing functionality and we cannot be held responsible for that. Many of
%             the manuals contain characteristic graphics and personal notes or examples that
%             make no sense when used out-of-context.
%
% comment   : Some chapters might have been published in TugBoat, the NTG Maps, the ConTeXt
%             Group journal or otherwise. Thanks to the editors for corrections. Also thanks
%             to users for testing, feedback and corrections.

\usemodule[abr-02]

\setuplayout
  [width=middle,
   height=middle,
   backspace=2cm,
   topspace=1cm,
   footer=0pt,
   bottomspace=2cm]

\definecolor
  [DocumentColor]
  [r=.5]

\setuptype
  [color=DocumentColor]

\setuptyping
  [color=DocumentColor]

\setupalign
  [tolerant,stretch]

% \usetypescript
%   [iwona]
%
% \setupbodyfont
%   [iwona]

\setupbodyfont
  [ibmplex,rm]

\setuphead
  [chapter]
  [style=\bfc,
   color=DocumentColor]

\setuphead
  [section]
  [style=\bfb,
   color=DocumentColor]

\setuphead
  [sub  section]
  [style=\bfa,
   color=DocumentColor]

\setupinteraction
  [hidden]

\setupwhitespace
  [big]

\setupheadertexts
  []

\setupheadertexts
  []
  [{\DocumentColor \type {luatools mtxrun context}\quad\pagenumber}]

\usesymbols[cc]

\def\sTEXMFSTART{\type{texmfstart}}
\def\sLUATOOLS  {\type{luatools}}
\def\sMTXRUN    {\type{mtxrun}}
\def\sCONTEXT   {\type{context}}
\def\sKPSEWHICH {\type{kpsewhich}}
\def\sMKTEXLSR  {\type{mktexlsr}}
\def\sXSLTPROC  {\type{xsltproc}}

\usemodule[narrowtt]

\startdocument
  [metadata:author=Hans Hagen,
   metadata:title={Tools: luatools, mtxrun, context},
   author=Hans Hagen,
   affiliation=PRAGMA ADE,
   location=Hasselt NL,
   title=Tools,
   extra-1=luatools,
   extra-2=mtxrun,
   extra-3=context,
   support=www.contextgarden.net,
   website=www.pragma-ade.nl]

\startMPpage
    StartPage ;
        picture p ; p := image (
            for i=1 upto 21 :
                for j=1 upto 30 :
                    drawarrow (fullcircle rotated uniformdeviate 360) scaled 10 shifted (i*15,j*15) ;
                endfor ;
            endfor ;
        ) ;
        p := p ysized (bbheight(Page)-4mm) ;
        fill Page enlarged 2mm withcolor \MPcolor{DocumentColor} ;
        draw p shifted (center Page - center p) withpen pencircle scaled 2 withcolor .5white ;
        numeric dx ; dx := bbwidth(Page)/21 ;
        numeric dy ; dy := bbheight(Page)/30 ;
        p := textext("\tt\bf\white\getvariable{document}{extra-1}") xsized(14*dx) ;
        p := p shifted (-lrcorner p) shifted lrcorner Page shifted (-1dx,8dy) ;
        draw p ;
        p := textext("\tt\bf\white\getvariable{document}{extra-2}") xsized(14*dx) ;
        p := p shifted (-lrcorner p) shifted lrcorner Page shifted (-1dx,5dy) ;
        draw p ;
        p := textext("\tt\bf\white\getvariable{document}{extra-3}") xsized(14*dx) ;
        p := p shifted (-lrcorner p) shifted lrcorner Page shifted (-1dx,2dy) ;
        draw p ;
        setbounds currentpicture to Page ;
    StopPage
\stopMPpage

\startsubject[title=Contents]

\placelist[section][alternative=a]

\stopsubject

\startsection[title={Remark}]

This manual is work in progress. Feel free to submit additions or corrections.
Before you start reading, it is good to know that in order to get starting with
\CONTEXT, the easiest way to do that is to download the standalone distribution
from \type {contextgarden.net}. After that you only need to make sure that \type
{luatex} is in your path. The main command you use is then \sCONTEXT\ and
normally it does all the magic it needs itself.

\stopsection

\startsection[title={Introduction}]

Right from the start \CONTEXT\ came with programs that managed the process of
\TEX-ing. Although you can perfectly well run \TEX\ directly, it is a fact that
often multiple runs are needed as well as that registers need to be sorted.
Therefore managing a job makes sense.

First we had \TEXEXEC\ and \TEXUTIL, and both were written in \MODULA, and as
this language was not supported on all platforms the programs were rewritten in
\PERL. Following that a few more tools were shipped with \CONTEXT.

When we moved on to \RUBY\ all the \PERL\ scripts were rewritten and when
\CONTEXT\ \MKIV\ showed up, \LUA\ replaced \RUBY. As we use \LUATEX\ this means
that currently the tools and the main program share the same language. For \MKII\
scripts like \TEXEXEC\ will stay around but the idea is that there will be \LUA\
alternatives for them as well.

Because we shipped many scripts, and because the de facto standard \TEX\
directory structure expects scripts to be in certain locations we not only ship
tools but also some more generic scripts that locate and run these tools.

\stopsection

\startsection[title={The location}]

Normally you don't need to know so many details about where the scripts
are located but here they are:

\starttyping
<texroot>/scripts/context/perl
<texroot>/scripts/context/ruby
<texroot>/scripts/context/lua
<texroot>/scripts/context/stubs
\stoptyping

This hierarchy was actually introduced because \CONTEXT\ was shipped with a bunch
of tools. As mentioned, we nowadays focus on \LUA\ but we keep a few of the older
scripts around in the \PERL\ and \RUBY\ paths.Now, if you're only using \CONTEXT\
\MKIV, and this is highly recommended, you can forget about all but the \LUA\
scripts.

\stopsection

\startsection[title={The traditional finder}]

When you run scripts multiple times, and in the case of \CONTEXT\ they are even
run inside other scripts, you want to minimize the startup time. Unfortunately
the traditional way to locate a script, using \sKPSEWHICH, is not that fast,
especially in a setup with many large trees Also, because not all tasks can be
done with the traditional scripts (take format generation) we provided a runner
that could deal with this: \sTEXMFSTART. As this script was also used in more
complex workflows, it had several tasks:

\startitemize[packed]
\item locate scripts in the distribution and run them using the right
      interpreter
\item do this selectively, for instance identify the need for a run using
      checksums for potentially changed files (handy for image conversion)
\item pass information to child processes so that lookups are avoided
\item choose a distribution among several installed versions (set the root
      of the \TEX\ tree)
\item change the working directory before running the script
\item resolve paths and names on demand and launch programs with arguments
      where names are expanded controlled by prefixes (handy for
      \TEX-unware programs)
\item locate and open documentation, mostly as part the help systems in
      editors, but also handy for seeing what configuration file is used
\item act as a \KPSEWHICH\ server cq.\ client (only used in special cases,
      and using its own database)
\stopitemize

Of course there were the usual more obscure and undocumented features as
well. The idea was to use this runner as follows:

\starttyping
texmfstart texexec <further arguments>
texmfstart --tree <rootoftree> texexec <further arguments>
\stoptyping

These are just two ways of calling this program. As \sTEXMFSTART\ can initialize
the environment as well, it is basically the only script that has to be present
in the binary path. This is quite comfortable as this avoids conflicts in names
between the called scripts and other installed programs.

Of course calls like above can be wrapped in a shell script or batch file without
penalty as long as \sTEXMFSTART\ itself is not wrapped in a caller script that
applies other inefficient lookups. If you use the \CONTEXT\ minimals you can be
sure that the most efficient method is chosen, but we've seen quite inefficient
call chains elsewhere.

In the \CONTEXT\ minimals this script has been replaced by the one we will
discuss in the next section: \sMTXRUN\ but a stub is still provided.

\stopsection

\startsection[title={The current finder}]

In \MKIV\ we went a step further and completely abandoned the traditional lookup
methods and do everything in \LUA. As we want a clear separation between
functionality we have two main controlling scripts: \sMTXRUN\ and \sLUATOOLS. The
last name may look somewhat confusing but the name is just one on in a series.
\footnote {We have \type {ctxtools}, \type {exatools}, \type {mpstools}, \type
{mtxtools}, \type {pdftools}, \type {rlxtools}, \type {runtools}, \type
{textools}, \type {tmftools} and \type {xmltools}. Most if their funtionality is
already reimplemented.}

In \MKIV\ the \sLUATOOLS\ program is nowadays seldom used. It's just a drop in
for \sKPSEWHICH\ plus a bit more. In that respect it's rather dumb in that it
does not use the database, but clever at the same time because it can make one
based on the little information available when it runs. It can also be used to
generate format files either or not using \LUA\ stubs but in practice this is not
needed at all.

For \CONTEXT\ users, the main invocation of this tool is when the \TEX\ tree is
updated. For instance, after adding a font to the tree or after updating
\CONTEXT, you need to run:

\starttyping
mtxrun --generate
\stoptyping

After that all tools will know where to find stuff and how to behave well within
the tree. This is because they share the same code, mostly because they are
started using \sMTXRUN. For instance, you process a file with:

\starttyping
mtxrun --script context <somefile>
\stoptyping

Because this happens often, there's also a shortcut:

\starttyping
context <somefile>
\stoptyping

But this does use \sMTXRUN\ as well. The help information of \sMTXRUN\ is rather
minimalistic and if you have no clue what an option does, you probably never
needed it anyway. Here we discuss a few options. We already saw that we can
explicitly ask for a script:

\starttyping
mtxrun --script context <somefile>
\stoptyping

but

\starttyping
mtxrun context <somefile>
\stoptyping

also works. However, by using \type {--script} you limit te lookup to the valid
\CONTEXT\ \MKIV\ scripts. In the \TEX\ tree these have names prefixed by \type
{mtx-} and a lookup look for a plural as well. So, the next two lookups are
equivalent:

\starttyping
mtxrun --script font
mtxrun --script fonts
\stoptyping

Both will run \type {mtx-fonts.lua}. Actually, this is one of the scripts that
you might need when your font database is somehow outdated and not updated
automatically:

\starttyping
mtxrun --script fonts --reload --force
\stoptyping

Normally \sMTXRUN\ is all you need in order to run a script. However, there are a
few more options:

\ctxlua{os.execute("mtxrun > tools-mkiv-help.tmp")}

\typefile[ntyping]{tools-mkiv-help.tmp}

Don't worry,you only need those obscure features when you integrate \CONTEXT\ in
for instance a web service or when you run large projects where runs and paths
take special care.

\stopsection

\startsection[title={Updating}]

There are two ways to update \CONTEXT\ \MKIV. When you manage your
trees yourself or when you use for instance \TEXLIVE, you act as
follows:

\startitemize[packed]
\item download the file cont-tmf.zip from \type {www.pragma-ade.com} or elsewhere
\item unzip this file in a subtree, for instance \type {tex/texmf-local}
\item run \type {mtxrun --generate}
\item run \type {mtxrun --script font --reload}
\item run \type {mtxrun --script context --make}
\stopitemize

Or shorter:

\startitemize[packed]
\item run \type {mtxrun --generate}
\item run \type {mtxrun font --reload}
\item run \type {context --make}
\stopitemize

Normally these commands are not even needed, but they are a nice test if your
tree is still okay. To some extend \sCONTEXT\ is clever enough to decide if the
databases need to be regenerated and|/|or a format needs to be remade and|/|or if
a new font database is needed.

Now, if you also want to run \MKII, you need to add:

\startitemize[packed]
\item run \type {mktexlsr}
\item run \type {texexec --make}
\stopitemize

The question is, how to act when \sLUATOOLS\ and \sMTXRUN\ have been updated
themselves? In that case, after unzipping the archive, you need to do the
following:

\startitemize[packed]
\item run \type {luatools --selfupdate}
\item run \type {mtxrun --selfupdate}
\stopitemize

For quite a while we shipped so called \CONTEXT\ minimals. These zip files
contained only the resources and programs that made sense for running \CONTEXT.
Nowadays the minimals are installed and synchronized via internet. \footnote
{This project was triggered by Mojca Miklavec who is also in charge of this bit
of the \CONTEXT\ infrastructure. More information can be found at \type
{contextgarden.net}.} You can just run the installer again and no additional
commands are needed. In the console you will see several calls to \sMTXRUN\ and
\sLUATOOLS\ fly by.

\stopsection

\startsection[title={The tools}]

We only mention the tools here. The most important ones are \sCONTEXT\ and \type
{fonts}. You can ask for a list of installed scripts with:

\starttyping
mtxrun --script
\stoptyping

On my machine this gives:

\ctxlua{os.execute("mtxrun --script > tools-mkiv-help.tmp")}

\typefile[ntyping]{tools-mkiv-help.tmp}

The most important scripts are \type {mtx-fonts} and \type {mtx-context}. By
default fonts are looked up by filename (the \type {file:} prefix before font
names in \CONTEXT\ is default). But you can also lookup fonts by name (\type
{name:}) or by specification (\type {spec:}). If you want to use these two
methods, you need to generate a font database as mentioned in the previous
section. You can also use the font tool to get information about the fonts
installed on your system.

\stopsection

\startsection[title={Running \CONTEXT}]

The \sCONTEXT\ tool is what you will use most as it manages your
run.

\ctxlua{os.execute("context > tools-mkiv-help.tmp")}

\typefile[ntyping]{tools-mkiv-help.tmp}

There are few expert options too:

\ctxlua{os.execute("context --expert > tools-mkiv-help.tmp")}

\typefile[ntyping]{tools-mkiv-help.tmp}

You might as well forget about these unless you are one of the
\CONTEXT\ developers.

\stopsection

\startsection[title={Prefixes}]

A handy feature of \sMTXRUN\ (and as most features an inheritance of
\sTEXMFSTART) is that it will resolve prefixed arguments. This can be of help
when you run programs that are unaware of the \TEX\ tree but nevertheless need to
locate files in it.

\ctxlua{os.execute("mtxrun --prefixes > tools-mkiv-help.tmp")}

\typefile[ntyping]{tools-mkiv-help.tmp}

An example is:

\starttyping
mtxrun --execute xsltproc file:whatever.xsl file:whatever.xml
\stoptyping

The call to \sXSLTPROC\ will get two arguments, being the complete path to the
files (given that it can be resolved). This permits you to organize the files in
a similar was as \TEX\ files.

\stopsection

\startsection[title={Stubs}]

As the tools are written in the \LUA\ language we need a \LUA\ interpreter and of
course we use \LUATEX\ itself. On \UNIX\ we can copy \sLUATOOLS\ and \sMTXRUN\ to
files in the binary path with the same name but without suffix. Starting them in
another way is a waste of time, especially when \sKPSEWHICH\ is used to find
then, something which is useless in \MKIV\ anyway. Just use these scripts
directly as they are self contained.

For \sCONTEXT\ and other scripts that we want convenient access to, stubs are
needed, like:

\starttyping
#!/bin/sh
mtxrun --script context "$@"
\stoptyping

This is also quite efficient as the \sCONTEXT\ script \type {mtx-context} is
loaded in \sMTXRUN\ and uses the same database.

On \WINDOWS\ you can copy the scripts as|-|is and associate the suffix with
\LUATEX\ (or more precisely: \type {texlua}) but then all \LUA\ script will be
run that way which is not what you might want.

In \TEXLIVE\ stubs for starting scripts were introduced by Fabrice Popineau. Such
a stub would start for instance \sTEXMFSTART, that is: it located the script
(\PERL\ or \RUBY) in the \TEX\ tree and launched it with the right interpreter.
Later we shipped pseudo binaries of \sTEXMFSTART: a \RUBY\ interpreter plus
scripts wrapped into a self contained binary.

For \MKIV\ we don't need such methods and started with simple batch files,
similar to the \UNIX\ startup scripts. However, these have the disadvantage that
they cannot be used in other batch files without using the \type {start} command.
In \TEXLIVE\ this is taken care of by a small binary written bij T.M.\ Trzeciak
so on \TEXLIVE\ 2009 we saw a call chain from \type {exe} to \type {cmd} to \type
{lua} which is somewhat messy.

This is why we now use an adapted and stripped down version of that program that
is tuned for \sMTXRUN, \sLUATOOLS\ and \sCONTEXT. So, we moved from the original
\type {cmd} based approach to an \type {exe} one.

\starttyping
mtxrun.dll
mtxrun.exe
\stoptyping

You can copy \type {mtxrun.exe} to for instance \type {context.exe} and it will
still use \sMTXRUN\ for locating the right script. It also takes care of mapping
\sTEXMFSTART\ to \sMTXRUN. So we've removed the intermediate \type {cmd} step,
can run the script as any program, and most of all, we're as efficient as can be.
Of course this program is only meaningful for the \CONTEXT\ approach to tools.

It may all sound more complex than it is but once it works users will not notice
those details. Also, in practice not that much has changed in running the tools
between \MKII\ and \MKIV\ as we've seen no reason to change the methods.

\stopsection

\startsection[title={A detailed look at \sMTXRUN}]

This section is derived from Taco Hoekwaters presentation and article for the
2018 \CONTEXT\ meeting. You might want to read this is you want to benefit from
even the most obscure features. There is a bit of repetition with the previous
sections but so be it.

% macros specific to this section

\def\highlighted#1#2#3%
  {\testpage[4]
   \dontleavehmode
   \backgroundline[gray]{\tt\strut #1}
   \nowhitespace
   \startnarrower[left]
   \veryraggedright
   #2\par
   \doifsomething{#3}{{\it #3}}
   \stopnarrower}

\definestartstop
  [options]
  [before=\blank,
   after=\blank]

\def\option#1#2%
  {\highlighted{--#1}{#2}{}}

\definestartstop
  [mtxrunscripts]
  [before=\blank,after=\blank]

\def\mtxrunscriptv#1#2#3#4%
  {\highlighted{#1}{#3}{#4}}

\def\mtxrunscript#1#2#3%
  {\highlighted{#1}{#3}{}}

\definestartstop
  [mtxrunenvironment]
  [before=\blank,
   after=\blank]

\def\mtxrunenv#1#2%
  {\highlighted{#1}{#2}{}}

\startsubsection[title=Common flags]

Much of the code inside \MKIV\ can alter its behaviour based on either \quote
{trackers} (which add debugging information to the terminal and log output) or
\quote {directives} or \quote {experiments} (for getting code to perform some
alternate behaviour). Since this also affects the \LUA\ code within \sMTXRUN\
itself, it makes sense to list these options first.

Trackers enable more extensive status messages on the console or in \CONTEXT\
additional visual clues. Directives change behaviour that is not part of the
official interface and have no corresponding commands. Experiments are like
directives but not official (yet).

\startoptions
    \option
        {trackers}
        {show (known) trackers}
    \option
        {directives}
        {show (known) directives}
    \option
        {experiments}
        {show (known) experiments}
\stopoptions

Enabling directives, trackers and experiments:

\startoptions
    \option
        {trackers=list}
        {enable given trackers}
    \option
        {directives=list}
        {enable given directives}
    \option
        {experiments=list}
        {enable given experiments}
\stopoptions

The next tree (hidden) options are converted into \quote {directives} entries,
that are then enabled. These are just syntactic sugar for the relevant directive.

\startoptions
    \option
        {silent[=...]}
        {sets \type {logs.blocked={\%s}}}
    \option
        {errors[=...]}
        {sets \type {logs.errors={\%s}}}
    \option
        {noconsole}
        {sets \type {logs.target=file}}
\stopoptions

As you can see here, various directives (and even some trackers) have optional
arguments, which can make specifying such directives on the command line a bit of
a challenge. Explaining the details of all the directives is outside of the scope
of this article, but you can look them up in the \CONTEXT\ source by searching
for \typ {directives.register} and \typ {trackers.register}.

In verbose mode, \sMTXRUN\ itself gives more messages, and it also enables \typ
{resolvers.locating}, which is a tracker itself:

\startoptions
    \option
        {verbose}
        {give a bit more info}
\stopoptions

The \type {--timedlog} (hidden) option starts the \sMTXRUN\ output with a
timestamp line:

\startoptions
    \option
        {timedlog}
        {prepend output with a timestamp}
\stopoptions

\stopsubsection

\startsubsection[title=Setup for finding files and configurations]

The next block of options deals with the setup of \sMTXRUN\ itself. You do not
need to deal with these options unless you are messing with the \CONTEXT\
distribution yourself instead of relying on a prepackaged solution, or you need
to use kpathsea for some reason (typically in a \MKII\ environment). In
particular, \type {--progname} and \type {--tree} are often needed as well when
using the \type {kpse} options.

\startoptions
    \option
        {configurations}
        {show configuration order, alias \type {--show-configurations}}
    \option
        {resolve}
        {resolve prefixed arguments, see \type {--prefixes}, below}
\stopoptions

and:

\startoptions
    \option
        {usekpse}
        {use kpse as fallback (when no \MKIV\ and cache installed, often slower)}
    \option
        {forcekpse}
        {force using kpse (handy when no \MKIV\ and cache installed but less
         functionality)}
    \option
        {progname=str}
        {format or backend}
    \option
        {tree=pathtotree}
        {use given texmf tree (default file: \type {setuptex.tmf})}
\stopoptions

We don't provide such a \type {.tmf} file in the distribution.

\stopsubsection

\startsubsection[title=Options for finding files and reporting configurations]

Once the configuration setup is done, it makes sense to have a bunch of options
to use and|/|or query the configuration.

\startoptions
    \option
        {locate}
        {locate given filename in database (default) or system (uses the
         sub||options \type {--first}, \type {--all} and \type {--detail})}
    \option
        {autogenerate}
        {regenerate databases if needed (handy when used to run context in an
         editor)}
    \option
        {generate}
        {generate file database}
    \option
        {prefixes}
        {show supported prefixes for file searches}
    \option
        {variables}
        {show configuration variables (uses the sub||option \type {--pattern},
         and an alias is \type {--show-variables})}
    \option
        {expansions}
        {show configuration variable expansion (uses the sub||options
         \type{--pattern}, alias \type {--show-expansions})}
    \option
        {expand-braces}
        {expand complex variable}
    \option
        {resolve-path}
        {expand variable (completely resolve paths)}
    \option
        {expand-path}
        {expand variable (resolve paths)}
    \option
        {expand-var}
        {expand variable (resolves references inside variables, alias
         \type{--expand-variable})}
    \option
        {show-path}
        {show path expansion of \type {...} (alias \type {--path-value})}
    \option
        {var-value}
        {report value of variable (alias \type {--show-value})}
    \option
        {find-file}
        {report file location; it uses the sub||options \type {--all}, \type
         {--pattern}, and \type {--format}}
    \option
        {find-path}
        {report path of file}
\stopoptions

Hidden option:

\startoptions
    \option
        {format-path}
        {report format path}
\stopoptions

\stopsubsection

\startsubsection[title=Running code]

Here we come to the core functionality of \sMTXRUN: running scripts. First
there are few options that trigger how the running takes place:

\startoptions
    \option
        {path=runpath}
        {go to given path before execution}
    \option
        {ifchanged=filename}
        {only execute when given file has changed (this loads and saves an md5
         checksum from \type {filename.md5})}
    \option
        {iftouched=old,new}
        {only execute when given file has changed (time stamp)}
    \option
        {timedrun}
        {run a script or program and time its run (external)}
\stopoptions

Specifying both \type {--iftouched} and \type {--ifchanged} means both are
tested, and when either one is false, nothing will happen. These options have to
come before one of the next options:

\startoptions
    \option
        {script}
        {run an mtx script (where \LUA\ is the preferred method); it has the
         sub||options \typ {--nofiledatabase}, \typ {--autogenerate}, \type
         {--load}, and \type {--save}. The latter two are currently no|-|ops}
    \option
        {execute}
        {run a script or program externally (\type {texmfstart} method); it has
         sub||option \type {--noquotes}}
    \option
        {internal}
        {run a script using built|-|in libraries (alias is \type {--ctxlua})}
    \option
        {direct}
        {run an external program; it has the sub||option \type {--noquotes}}
\stopoptions

Since scripts potentially have their own options, any options intended for
\sMTXRUN\ itself have to come {\it before} the option that specifies the script
to run, and options for the script itself have to come {\it after} the option
that gives the script name. This is especially true when using \type {--script},
so it is important to check the order of your options.

Of the four above options, \type {--script} is the most important one, since that
is the one that finds and executes the \LUA\ \sMTXRUN\ scripts provided by the
distribution. With \typ {--nofiledatabase}, it will not attempt to resolve any
file names (which means you need either a local script or a full path name). On
the opposite side, when you also provide \typ {--autogenerate}, it will not only
attempt to resolve the file name, it will also regenerate the database if it
cannot find the script on the first try. In a future version of \CONTEXT, the
\type {--load} and \type {--save} will allow you to save|/|restore the current
command line in a file for reuse.

The shell return value of \sMTXRUN\ indicates whether the script was found. When
you specify something like \type {--script base}, \sMTXRUN\ actually searches
for \type {mtx-base.lua}, \type {mtx-bases.lua}, \type {mtx-t-base.lua}, \type
{mtx-t-bases.lua}, and \type {base.lua}, in that order. The
distribution||supplied scripts normally use \type {mtx-<name>.lua} as template.
The template names with \type {mtx-t-} prefix is reserved for third||party
scripts, and \type {<name>.lua} is just a last-ditch effort if nothing else
works. Scripts are looked for in the local path, and in whatever directories the
configuration variable \type {LUAINPUTS} points to.

The \type {--execute} option exists mostly for the non||\LUA\ {\MKII} scripts
that still exist in the distribution. It will try to find a matching interpreter
for non||\LUA\ scripts, and it is aware of a number of distribution||supplied
scripts so that if you specify \type {--execute texexec}, it knows that what you
really want to execute is \type {ruby texexec.rb}. Support is present for \RUBY\
(\type {.rb}, \LUA\ (\type {.lua}), python (\type {.py}) and \PERL\ (\type {.pl})
scripts (tested in that order). File resolving uses \type {TEXMFSCRIPTS} from the
configuration. The shell return value of \sMTXRUN\ indicates whether the script
was found and executed.

The \type {--internal} option uses the file search method of \type {--execute},
but then assumes this is a \LUA\ script and executes it internally like \type
{--script}. This is useful if you have a \LUA\ script in an odd location.

The last of the four options, \type {--direct}, directly executes an external
program. You need to give the full path for binaries not in the current shell
\type {PATH}, because no searching is done at all. The shell return value of
\sMTXRUN\ in this case is a boolean based on the return value of
\type {os.exec()}.

It is also possible to execute bare \LUA\ code directly:

\startoptions
    \option
        {evaluate}
        {run code passed on the command-line (between quotes)}
\stopoptions

\stopsubsection

\startsubsection[title=Options for maintenance of \sMTXRUN\ itself]

None of these are advertised. Normally developers should be the only ones needing
them, but if you made a change to one of the distributed libraries (maybe because
of a beta bug), you may need to run \type {--selfmerge} and \type {--selfupdate}.

\startoptions
    \option
        {selfclean}
        {remove embedded libraries}
    \option
        {selfmerge}
        {update embedded libraries in \type {mtxrun.lua}}
    \option
        {selfupdate}
        {copy \type {mtxlua.lua} to the executable directory, renamed \type
         {mtxrun}}
\stopoptions

\stopsubsection

\startsubsection[title=Creating stubs]

Stubs are little shortcuts that live in some binaries directory. For example, the
content of the \UNIX||style \sCONTEXT\ shell command is:

\starttyping
#!/bin/sh
mtxrun --script context "$@"
\stoptyping

Apart from the \sCONTEXT\ command itself (which is provided by the distribution),
use of stubs is discouraged. Still, the \sMTXRUN\ options are there because
sometimes existing workflows depend on executable tool names like \type
{ctxtools}.

\startoptions
    \option
        {makestubs}
        {create stubs for (context related) scripts}
    \option
        {removestubs}
        {remove stubs (context related) scripts}
    \option
        {stubpath=binpath}
        {paths where stubs will be written}
    \option
        {windows}
        {create windows (mswin) stubs (alias \type {--mswin})}
    \option
        {unix}
        {create unix (linux) stubs (alias \type {--linux})}
\stopoptions

\stopsubsection

\startsubsection[title=Remaining options]

The remaining options are hard to group into a subcategory. These are the
advertised options:

\startoptions
    \option
        {systeminfo}
        {show current operating system, processor, et cetera}
    \option
        {edit}
        {launch editor with found file; the editor is taken from the environment
         variable \type {MTXRUN_EDITOR}, or \type {TEXMFSTART_EDITOR}, or
         \type {EDITOR}, or as a last resort: \type {gvim}}
    \option
        {launch}
        {launch files like manuals, assumes os support (uses the sub||options
         \type {--all}, \type {--pattern} and \type {--list})}
\stopoptions

While these are sort of hidden options:

\startoptions
    \option
        {ansi}
        {colorize output to terminal using \ANSI\ escapes}
    \option
        {associate}
        {launch files like manuals, assumes os support. this function does not do
         any file searching, so you have to use either a local file or a full
         path name}
    \option
        {exporthelp}
        {output the \sMTXRUN\ \XML\ help blob (useful for creating man and \HTML\
         help pages)}
    \option
        {fmt}
        {shortcut for \type {--script base --fmt}}
    \option
        {gethelp}
        {attempt to look up remote \sCONTEXT\ command help (uses the sub||options
         \type{--command} and \type {--url})}
    \option
        {help}
        {print the \sMTXRUN\ help screen}
    \option
        {locale}
        {force setup of locale; unless you are certain you need this option, stay
         away from it, because it can interfere massively with \CONTEXT's \LUA\
         code}
    \option
        {make}
        {(re)create format files (aliases are \type {--ini} and \type {--compile})}
    \option
        {platform}
        {(alias is \type {--show-platform})}
    \option
        {run}
        {shortcut for \type {--script base --run}}
    \option
        {version}
        {print \sMTXRUN\ version}
\stopoptions

\stopsubsection

\startsubsection[title=Known scripts]

When you run \type {mtxrun --scripts}, it will output a list of \quote {known}
scripts. The definition of \quote {known} is important here: the list comprises
the scripts that are present in the same directory as \type {mtx-context.lua}
that do not have an extra hyphen in the name (like \type {mtx-t-...}scripts would
have). In a normal installation, this means it \quote {knows} almost all the
scripts that are distributed with \CONTEXT. Note: it skips over any files from
the distribution that do have an extra hyphen, like the \type {mtx-server}
support scripts.

Since this section is about \sMTXRUN, I'll just present the list of the scripts
that are \quote {known} in the current \CONTEXT\ beta as output by \sMTXRUN\
itself, and not get into detail about all of the script functionality (they all
have \type {--help} options if you want to find out more). Where we still felt the
need to explain something, there is an extra bit of text in italics.

\startmtxrunscripts
\mtxrunscript
    {babel}
    {1.20}
    {Babel Input To UTF Conversion}
\mtxrunscript
    {base}
    {1.35}
    {ConTeXt TDS Management Tool (aka luatools)}
\mtxrunscript
    {bibtex}
    {}
    {bibtex helpers (obsolete)}
\mtxrunscript
    {cache}
    {0.10}
    {ConTeXt & MetaTeX Cache Management}
\mtxrunscript
    {chars}
    {0.10}
    {MkII Character Table Generators}
\mtxrunscriptv
    {check}
    {0.10}
    {Basic ConTeXt Syntax Checking}
    {Occasionally useful on big projects, but be warned that it does not actually
     run any \TEX\ engine, so it is not 100\% reliable.}
\mtxrunscriptv
    {colors}
    {0.10}
    {ConTeXt Color Management}
    {This displays icc color tables by name}
\mtxrunscriptv
    {convert}
    {0.10}
    {ConTeXT Graphic Conversion Helpers}
    {A wrapper around ghostscript and imagemagick that offers some extra (batch
     processing) functionality.}
\mtxrunscript
    {dvi}
    {0.10}
    {ConTeXt DVI Helpers}
\mtxrunscriptv
    {epub}
    {1.10}
    {ConTeXt EPUB Helpers}
    {The EPUB manual (\type {epub-mkiv.pdf}) explains how to use this script.}
\mtxrunscriptv
    {evohome}
    {1.00}
    {Evohome Fetcher}
    {Evohome is a domotica system that controls your central heating}
\mtxrunscript
    {fcd}
    {1.00}
    {Fast Directory Change}
\mtxrunscriptv
    {flac}
    {0.10}
    {ConTeXt Flac Helpers}
    {Extracts information from \type{.flac} audio files into an \XML\ index.}
\mtxrunscript
    {fonts}
    {0.21}
    {ConTeXt Font Database Management}
\mtxrunscript
    {grep}
    {0.10}
    {Simple Grepper}
\mtxrunscript
    {interface}
    {0.13}
    {ConTeXt Interface Related Goodies}
\mtxrunscript
    {metapost}
    {0.10}
    {MetaPost to PDF processor}
\mtxrunscript
    {metatex}
    {0.10}
    {MetaTeX Process Management (obsolete)}
\mtxrunscript
    {modules}
    {1.00}
    {ConTeXt Module Documentation Generators}
\mtxrunscriptv
    {package}
    {0.10}
    {Distribution Related Goodies}
    {This script is used to create the generic \CONTEXT\ code used in
     \LUA\LATEX~c.s.}
\mtxrunscriptv
    {patterns}
    {0.20}
    {ConTeXt Pattern File Management}
    {Hyphenation patterns, that is \unknown}
\mtxrunscript
    {pdf}
    {0.10}
    {ConTeXt PDF Helpers}
\mtxrunscript
    {plain}
    {1.00}
    {Plain TeX Runner}
\mtxrunscript
    {profile}
    {1.00}
    {ConTeXt MkIV LuaTeX Profiler}
\mtxrunscript
    {rsync}
    {0.10}
    {Rsync Helpers}
\mtxrunscript
    {scite}
    {1.00}
    {Scite Helper Script}
\mtxrunscriptv
    {server}
    {0.10}
    {Simple Webserver For Helpers}
    {There are some subscripts associated with this.}
\mtxrunscript
    {synctex}
    {1.00}
    {ConTeXt SyncTeX Checker}
\mtxrunscript
    {texworks}
    {1.00}
    {TeXworks Startup Script}
\mtxrunscript
    {timing}
    {0.10}
    {ConTeXt Timing Tools}
\mtxrunscript
    {tools}
    {1.01}
    {Some File Related Goodies}
\mtxrunscript
    {unicode}
    {1.02}
    {Checker for \type{char-def.lua}}
\mtxrunscript
    {unzip}
    {0.10}
    {Simple Unzipper}
\mtxrunscript
    {update}
    {1.03}
    {ConTeXt Minimals Updater}
\mtxrunscript
    {watch}
    {1.00}
    {ConTeXt Request Watchdog}
\mtxrunscriptv
    {youless}
    {1.10}
    {YouLess Fetcher}
    {YouLess is a domotica system that tracks your home energy use.}
\stopmtxrunscripts

\stopsubsection

\startsubsection[title=Writing your own]

A well|-|written script has some required internal structure. It should start with
a module definition block. This gives some information about the module, but more
importantly, it prevents double|-|loading.

Here is an example:

\starttyping
if not modules then modules = { } end

modules ['mtx-envtest'] = {
    version   = 0.100,
    comment   = "companion to mtxrun.lua",
    author    = "Taco Hoekwater",
    copyright = "Taco Hoekwater",
    license   = "bsd"
}
\stoptyping

Next up is a variable containing the help information. The help information is actually
a bit of \XML\ stored in \LUA\ string. In the full example listing at the end of this
article, you can see what the internal structure is supposed to be like.

\starttyping
local helpinfo = [[
<?xml version="1.0"?>
<application>
  ....
</application>
]]
\stoptyping

And this help information is then used to create an instance of an \type
{application} table.

\starttyping
local application = logs.application {
    name     = "envtest",
    banner   = "Mtxrun environment demo",
    helpinfo = helpinfo,
}
\stoptyping

After this call, the \type {application} table contains (amongst some other
things) three functions that are very useful:

\startmtxrunenvironment
\mtxrunenv
    {identify()}
    {Prints out a banner identifying the current script to the user.}
\mtxrunenv
    {report(str)}
    {For printing information to the terminal with the script name as prefix.}
\mtxrunenv
    {export()}
    {Prints the \type {helpinfo} to the terminal, so it can easily be used for
     documentation purposes.}
\stopmtxrunenvironment

Next up, it is good to define your scripts' functionality in functions in a
private table. This prevents namespace pollution, and looks like this:

\starttyping
scripts          = scripts         or { }
scripts.envtest  = scripts.envtest or { }

function scripts.envtest.runtest()
    application.report("script name is " .. environment.ownname)
end
\stoptyping

And finally, identify the current script, followed by handling the provided
options (usually with an \type {if}||\type {else} statement).

\starttyping
if environment.argument("exporthelp") then
   application.export()
elseif environment.argument('test') then
   scripts.envtest.runtest()
else
    application.help()
end
\stoptyping

\stopsubsection

\startsubsection[title=Script environment]

\sMTXRUN\ includes lots of the internal \LUA\ helper libraries from \CONTEXT. We
actually maintain a version of the script without all those libraries included,
and before every (beta) \CONTEXT\ release, an amalgamated version of \sMTXRUN\ is
added to the distribution. In the merging process, all comments are stripped from
the embedded libraries, so if you want to know details, it is better to look in
the original \LUA\ source file.

Inside \sMTXRUN, the full list of embedded libraries can be queried via the array
\type {own.libs}:

\startalign[flushleft]
l-lua.lua l-macro.lua l-sandbox.lua l-package.lua l-lpeg.lua
l-function.lua l-string.lua l-table.lua l-io.lua l-number.lua l-set.lua l-os.lua
l-file.lua l-gzip.lua l-md5.lua l-url.lua l-dir.lua l-boolean.lua l-unicode.lua
l-math.lua util-str.lua util-tab.lua util-fil.lua util-sac.lua util-sto.lua
util-prs.lua util-fmt.lua trac-set.lua trac-log.lua trac-inf.lua trac-pro.lua
util-lua.lua util-deb.lua util-tpl.lua util-sbx.lua util-mrg.lua util-env.lua
luat-env.lua lxml-tab.lua lxml-lpt.lua lxml-mis.lua lxml-aux.lua lxml-xml.lua
trac-xml.lua data-ini.lua data-exp.lua data-env.lua data-tmp.lua data-met.lua
data-res.lua data-pre.lua data-inp.lua data-out.lua data-fil.lua data-con.lua
data-use.lua data-zip.lua data-tre.lua data-sch.lua data-lua.lua data-aux.lua
data-tmf.lua data -lst.lua util-lib.lua luat-sta.lua luat-fmt.lua
\stopalign

In fact, the \LUA\ table \type {own} contains some other useful stuff like the
script's actual disk name and location (\type {own.name} and \type {own.path})
and some internal variables like a list of all the locations it searches for its
embedded libraries (\type {own.list}), which is used by the \type {--selfmerge}
option and also allows the non||amalgamated version to run (since otherwise \type
{--selfmerge} could not be bootstrapped).

\sMTXRUN\ offers a programming environment that makes it easy to write \LUA\
script. The most important element of that environment is a \LUA\ table that is
conveniently called \type {environment} (\type {util-env} does the actual work of
setting that up).

The bulk of \type {environment} consists of functions and variables that deal
with the command||line given by the user as \sMTXRUN\ does quite a bit of work on
the given command||line. The goal is to safely tuck all the given options into
the \type {arguments} and \type {files} tables. This work is done by two
functions called \type {initializearguments()} and \type {splitarguments()}.
These functions are part of the \type{environment} table, but you should not need
them as they have been called already once control is passed on to your script.

\startmtxrunenvironment
\mtxrunenv
    {arguments}
    {These are the processed options to the current script. The keys are option
     names (without the leading dashes) and the value is either \type {true} or a
     string with one level of shell quotes removed.}
\mtxrunenv
    {files}
    {This array holds all the non||option arguments to the current script.
     Typically, those are supposed to be files, but they could be any text,
     really.}
\mtxrunenv
    {getargument(name,partial)}
    {Queries the \type {arguments} table using a function. Its main reason for
     existence is the \type {partial} argument, which allows scripts to accept
     shortened command||line options (alias: \type {argument()}).}
\mtxrunenv
    {setargument(name,value)}
    {Sets a value in the \type {arguments} table. This can be useful in
     complicated scripts with default options.}
\stopmtxrunenvironment

In case you need access to the full command||line, there are some possibilities:

\startmtxrunenvironment
\mtxrunenv
    {arguments_after}
    {These are the unquoted but otherwise unprocessed arguments to your script as
     an array.}
\mtxrunenv
    {arguments_before}
    {These are the unquoted but otherwise unprocessed arguments to \sMTXRUN\
     before your scripts' name (so the last entry is usually \type {--script}).}
\mtxrunenv
    {rawarguments}
    {This is the whole unprocessed command||line as an array.}
\mtxrunenv
    {originalarguments}
    {Like \type{rawarguments}, but with some top||level quotes removed.}
\mtxrunenv
    {reconstructcommandline(arg,noquote)}
    {Tries to reconstruct a command||line from its arguments. It uses \type
     {originalarguments} if no \type {arg} is given. Take care: due to the
     vagaries of shell command||line processing, this may or may not work when
     quoting is involved.}
\stopmtxrunenvironment

\type{environment} also stores various bits of information you may find useful:

\startmtxrunenvironment
\mtxrunenv
    {validengines}
    {This table contains keys for \type {luatex} and \type {luajittex}. This is
     only relevant when \sMTXRUN\ itself is called via \LUATEX's \type {luaonly}
     option.}
\mtxrunenv
    {basicengines}
    {This table maps executable names to \type{validengines} entries.}
\mtxrunenv
    {default_texmfcnf}
    {This is the \type {texmfcnf} value from \type {kpathsea}, processed for use
     with \MKIV\ in the unlikely event this is needed.}
\mtxrunenv
    {homedir}
    {The user's home directory.}
\mtxrunenv
    {ownbin}
    {The name of the binary used to call \sMTXRUN.}
\mtxrunenv
    {ownmain}
    {The mapped version of \type {ownbin}.}
\mtxrunenv
    {ownname}
    {Full name of this instance of \sMTXRUN.}
\mtxrunenv
    {ownpath}
    {The path this  instance of \sMTXRUN\ resides in.}
\mtxrunenv
    {texmfos}
    {Operating system root directory path.}
\mtxrunenv
    {texos}
    {Operating system root directory name.}
\mtxrunenv
    {texroot}
    {\CONTEXT\ root directory path.}
\stopmtxrunenvironment

As well as some functions:

\startmtxrunenvironment
\mtxrunenv
    {texfile(filename)}
    {Locates a {\TEX} file.}
\mtxrunenv
    {luafile(filename)}
    {Locates a \LUA\ file.}
\mtxrunenv
    {loadluafile(filename,version)}
    {Locates, compiles and loads a \LUA\ file, possibly in compressed \type {.luc}
     format. In the compressed case, it uses the \type {version} to make sure the
     compressed form is up||to||date.}
\mtxrunenv
    {luafilechunk(filename,silent,macros)}
    {Locates and compiles a \LUA\ file, returning its contents as data.}
\mtxrunenv
    {make_format(name,arguments)}
    {Creates a format file and stores it in the \CONTEXT\ cache, used by \type
     {mtxrun --make}.}
\mtxrunenv
    {relativepath(path,root)}
    {Returns a modified version of \type {root} based on the relative path in
     \type {path}.}
\mtxrunenv
    {run_format(name,data,more)}
    {Run a \TEX\ format file.}
\stopmtxrunenvironment

\stopsubsection

\startsubsection[title=Shell return values]

As explained earlier, the shell return value of \sMTXRUN\ normally indicates
whether the script was found. If you are running a \CONTEXT\ release newer than
September 2018 and want to modify the shell return value from within your script,
you can use \type {os.exitcode}. Whatever value you assign to that variable will
be the shell return value of your script.

\stopsubsection

\stopsection

\startsubject[title={Colofon}]

\starttabulate[|B|p|]
    \NC author    \NC \documentvariable{author},
                      \documentvariable{affiliation},
                      \documentvariable{location} \NC \NR
    \NC           \NC Taco Hoekwater, extra \sMTXRUN\ section \NC \NR
    \NC version   \NC \currentdate \NC \NR
    \NC website   \NC \documentvariable{website} \endash\
                      \documentvariable{support} \NC \NR
    \NC copyright \NC \symbol[cc][cc-by-sa-nc] \NC \NR
\stoptabulate

\stopsubject

\stopdocument