path: root/doc/context/sources/general/manuals/tools/tools-mkiv.tex

% language=uk

% author    : Hans Hagen
% copyright : PRAGMA ADE & ConTeXt Development Team
% license   : Creative Commons Attribution ShareAlike 4.0 International
% reference : pragma-ade.nl | contextgarden.net | texlive (related) distributions
% origin    : the ConTeXt distribution
%
% comment   : Because this manual is distributed with TeX distributions it comes with a rather
%             liberal license. We try to adapt these documents to upgrades in the (sub)systems
%             that they describe. Using parts of the content otherwise can therefore conflict
%             with existing functionality and we cannot be held responsible for that. Many of
%             the manuals contain characteristic graphics and personal notes or examples that
%             make no sense when used out-of-context.

\usemodule[abr-02]

\setuplayout
  [width=middle,
   height=middle,
   backspace=2cm,
   topspace=1cm,
   footer=0pt,
   bottomspace=2cm]

\definecolor
  [DocumentColor]
  [r=.5]

\setuptype
  [color=DocumentColor]

\setuptyping
  [color=DocumentColor]

\usetypescript
  [iwona]

\setupbodyfont
  [iwona]

\setuphead
  [chapter]
  [style=\bfc,
   color=DocumentColor]

\setuphead
  [section]
  [style=\bfb,
   color=DocumentColor]

\setupinteraction
  [hidden]

\setupwhitespace
  [big]

\setupheadertexts
  []

\setupheadertexts
  []
  [{\DocumentColor \type {luatools mtxrun context}\quad\pagenumber}]

\usesymbols[cc]

\def\sTEXMFSTART{\type{texmfstart}}
\def\sLUATOOLS  {\type{luatools}}
\def\sMTXRUN    {\type{mtxrun}}
\def\sCONTEXT   {\type{context}}
\def\sKPSEWHICH {\type{kpsewhich}}
\def\sMKTEXLSR  {\type{mktexlsr}}
\def\sXSLTPROC  {\type{xsltproc}}

\usemodule[narrowtt]

\startdocument
  [metadata:author=Hans Hagen,
   metadata:title={Tools: luatools, mtxrun, context},
   author=Hans Hagen,
   affiliation=PRAGMA ADE,
   location=Hasselt NL,
   title=Tools,
   extra-1=luatools,
   extra-2=mtxrun,
   extra-3=context,
   support=www.contextgarden.net,
   website=www.pragma-ade.nl]

\startMPpage
    StartPage ;
        picture p ; p := image (
            for i=1 upto 21 :
                for j=1 upto 30 :
                    drawarrow (fullcircle rotated uniformdeviate 360) scaled 10 shifted (i*15,j*15) ;
                endfor ;
            endfor ;
        ) ;
        p := p ysized (bbheight(Page)-4mm) ;
        fill Page enlarged 2mm withcolor \MPcolor{DocumentColor} ;
        draw p shifted (center Page - center p) withpen pencircle scaled 2 withcolor .5white ;
        numeric dx ; dx := bbwidth(Page)/21 ;
        numeric dy ; dy := bbheight(Page)/30 ;
        p := textext("\tt\bf\white\getvariable{document}{extra-1}") xsized(14*dx) ;
        p := p shifted (-lrcorner p) shifted lrcorner Page shifted (-1dx,8dy) ;
        draw p ;
        p := textext("\tt\bf\white\getvariable{document}{extra-2}") xsized(14*dx) ;
        p := p shifted (-lrcorner p) shifted lrcorner Page shifted (-1dx,5dy) ;
        draw p ;
        p := textext("\tt\bf\white\getvariable{document}{extra-3}") xsized(14*dx) ;
        p := p shifted (-lrcorner p) shifted lrcorner Page shifted (-1dx,2dy) ;
        draw p ;
        setbounds currentpicture to Page ;
    StopPage
\stopMPpage

\startsubject[title=Contents]

\placelist[section][alternative=a]

\stopsubject

\startsection[title={Remark}]

This manual is work in progress. Feel free to submit additions or corrections.
Before you start reading, it is good to know that in order to get starting with
\CONTEXT, the easiest way to do that is to download the standalone distribution
from \type {contextgarden.net}. After that you only need to make sure that \type
{luatex} is in your path. The main command you use is then \type {context} and
normally it does all the magic it needs itself.

\stopsection

\startsection[title={Introduction}]

Right from the start \CONTEXT\ came with programs that managed the process of
\TEX-ing. Although you can perfectly well run \TEX\ directly, it is a fact that
often multiple runs are needed as well as that registers need to be sorted.
Therefore managing a job makes sense.

First we had \TEXEXEC\ and \TEXUTIL, and both were written in \MODULA, and as
this language was not supported on all platforms the programs were rewritten in
\PERL. Following that a few more tools were shipped with \CONTEXT.

When we moved on to \RUBY\ all the \PERL\ scripts were rewritten and when
\CONTEXT\ \MKIV\ showed up, \LUA\ replaced \RUBY. As we use \LUATEX\ this means
that currently the tools and the main program share the same language. For \MKII\
scripts like \TEXEXEC\ will stay around but the idea is that there will be \LUA\
alternatives for them as well.

Because we shipped many scripts, and because the de facto standard \TEX\
directory structure expects scripts to be in certain locations we not only ship
tools but also some more generic scripts that locate and run these tools.

\stopsection

\startsection[title={The location}]

Normally you don't need to know so many details about where the scripts
are located but here they are:

\starttyping
<texroot>/scripts/context/perl
<texroot>/scripts/context/ruby
<texroot>/scripts/context/lua
<texroot>/scripts/context/stubs
\stoptyping

This hierarchy was actually introduced because \CONTEXT\ was shipped with a bunch
of tools. As mentioned, we nowadays focus on \LUA\ but we keep a few of the older
scripts around in the \PERL\ and \RUBY\ paths.Now, if you're only using \CONTEXT\
\MKIV, and this is highly recommended, you can forget about all but the \LUA\
scripts.

\stopsection

\startsection[title={The traditional finder}]

When you run scripts multiple times, and in the case of \CONTEXT\ they are even
run inside other scripts, you want to minimize the startup time. Unfortunately
the traditional way to locate a script, using \sKPSEWHICH, is not that fast,
especially in a setup with many large trees Also, because not all tasks can be
done with the traditional scripts (take format generation) we provided a runner
that could deal with this: \sTEXMFSTART. As this script was also used in more
complex workflows, it had several tasks:

\startitemize[packed]
\item locate scripts in the distribution and run them using the right
      interpreter
\item do this selectively, for instance identify the need for a run using
      checksums for potentially changed files (handy for image conversion)
\item pass information to child processes so that lookups are avoided
\item choose a distribution among several installed versions (set the root
      of the \TEX\ tree)
\item change the working directory before running the script
\item resolve paths and names on demand and launch programs with arguments
      where names are expanded controlled by prefixes (handy for
      \TEX-unware programs)
\item locate and open documentation, mostly as part the help systems in
      editors, but also handy for seeing what configuration file is used
\item act as a \KPSEWHICH\ server cq.\ client (only used in special cases,
      and using its own database)
\stopitemize

Of course there were the usual more obscure and undocumented features as
well. The idea was to use this runner as follows:

\starttyping
texmfstart texexec <further arguments>
texmfstart --tree <rootoftree> texexec <further arguments>
\stoptyping

These are just two ways of calling this program. As \sTEXMFSTART\ can initialize
the environment as well, it is basically the only script that has to be present
in the binary path. This is quite comfortable as this avoids conflicts in names
between the called scripts and other installed programs.

Of course calls like above can be wrapped in a shell script or batch file without
penalty as long as \sTEXMFSTART\ itself is not wrapped in a caller script that
applies other inefficient lookups. If you use the \CONTEXT\ minimals you can be
sure that the most efficient method is chosen, but we've seen quite inefficient
call chains elsewhere.

In the \CONTEXT\ minimals this script has been replaced by the one we will
discuss in the next section: \sMTXRUN\ but a stub is still provided.

\stopsection

\startsection[title={The current finder}]

In \MKIV\ we went a step further and completely abandoned the traditional lookup
methods and do everything in \LUA. As we want a clear separation between
functionality we have two main controlling scripts: \sMTXRUN\ and \sLUATOOLS. The
last name may look somewhat confusing but the name is just one on in a series.
\footnote {We have \type {ctxtools}, \type {exatools}, \type {mpstools}, \type
{mtxtools}, \type {pdftools}, \type {rlxtools}, \type {runtools}, \type
{textools}, \type {tmftools} and \type {xmltools}. Most if their funtionality is
already reimplemented.}

In \MKIV\ the \sLUATOOLS\ program is nowadays seldom used. It's just a drop in
for \sKPSEWHICH\ plus a bit more. In that respect it's rather dumb in that it
does not use the database, but clever at the same time because it can make one
based on the little information available when it runs. It can also be used to
generate format files either or not using \LUA\ stubs but in practice this is not
needed at all.

For \CONTEXT\ users, the main invocation of this tool is when the \TEX\ tree is
updated. For instance, after adding a font to the tree or after updating
\CONTEXT, you need to run:

\starttyping
mtxrun --generate
\stoptyping

After that all tools will know where to find stuff and how to behave well within
the tree. This is because they share the same code, mostly because they are
started using \sMTXRUN. For instance, you process a file with:

\starttyping
mtxrun --script context <somefile>
\stoptyping

Because this happens often, there's also a shortcut:

\starttyping
context <somefile>
\stoptyping

But this does use \sMTXRUN\ as well. The help information of \sMTXRUN\ is rather
minimalistic and if you have no clue what an option does, you probably never
needed it anyway. Here we discuss a few options. We already saw that we can
explicitly ask for a script:

\starttyping
mtxrun --script context <somefile>
\stoptyping

but

\starttyping
mtxrun context <somefile>
\stoptyping

also works. However, by using \type {--script} you limit te lookup to the valid
\CONTEXT\ \MKIV\ scripts. In the \TEX\ tree these have names prefixed by \type
{mtx-} and a lookup look for a plural as well. So, the next two lookups are
equivalent:

\starttyping
mtxrun --script font
mtxrun --script fonts
\stoptyping

Both will run \type {mtx-fonts.lua}. Actually, this is one of the scripts that
you might need when your font database is somehow outdated and not updated
automatically:

\starttyping
mtxrun --script fonts --reload --force
\stoptyping

Normally \sMTXRUN\ is all you need in order to run a script. However, there are a
few more options:

\ctxlua{os.execute("mtxrun > tools-mkiv-help.tmp")}

\typefile[ntyping]{tools-mkiv-help.tmp}

Don't worry,you only need those obscure features when you integrate \CONTEXT\ in
for instance a web service or when you run large projects where runs and paths
take special care.

\stopsection

\startsection[title={Updating}]

There are two ways to update \CONTEXT\ \MKIV. When you manage your
trees yourself or when you use for instance \TEXLIVE, you act as
follows:

\startitemize[packed]
\item download the file cont-tmf.zip from \type {www.pragma-ade.com} or elsewhere
\item unzip this file in a subtree, for instance \type {tex/texmf-local}
\item run \type {mtxrun --generate}
\item run \type {mtxrun --script font --reload}
\item run \type {mtxrun --script context --make}
\stopitemize

Or shorter:

\startitemize[packed]
\item run \type {mtxrun --generate}
\item run \type {mtxrun font --reload}
\item run \type {context --make}
\stopitemize

Normally these commands are not even needed, but they are a nice test if your
tree is still okay. To some extend \sCONTEXT\ is clever enough to decide if the
databases need to be regenerated and|/|or a format needs to be remade and|/|or if
a new font database is needed.

Now, if you also want to run \MKII, you need to add:

\startitemize[packed]
\item run \type {mktexlsr}
\item run \type {texexec --make}
\stopitemize

The question is, how to act when \sLUATOOLS\ and \sMTXRUN\ have been updated
themselves? In that case, after unzipping the archive, you need to do the
following:

\startitemize[packed]
\item run \type {luatools --selfupdate}
\item run \type {mtxrun --selfupdate}
\stopitemize

For quite a while we shipped so called \CONTEXT\ minimals. These zip files
contained only the resources and programs that made sense for running \CONTEXT.
Nowadays the minimals are installed and synchronized via internet. \footnote
{This project was triggered by Mojca Miklavec who is also charge of this bit of
the \CONTEXT\ infrastructure. More information can be found at \type
{contextgarden.net}.} You can just run the installer again and no additional
commands are needed. In the console you will see several calls to \sMTXRUN\ and
\sLUATOOLS\ fly by.

\stopsection

\startsection[title={The tools}]

We only mention the tools here. The most important ones are \sCONTEXT\ and \type
{fonts}. You can ask for a list of installed scripts with:

\starttyping
mtxrun --script
\stoptyping

On my machine this gives:

\ctxlua{os.execute("mtxrun --script > tools-mkiv-help.tmp")}

\typefile[ntyping]{tools-mkiv-help.tmp}

The most important scripts are \type {mtx-fonts} and \type {mtx-context}. By
default fonts are looked up by filename (the \type {file:} prefix before font
names in \CONTEXT\ is default). But you can also lookup fonts by name (\type
{name:}) or by specification (\type {spec:}). If you want to use these two
methods, you need to generate a font database as mentioned in the previous
section. You can also use the font tool to get information about the fonts
installed on your system.

\stopsection

\startsection[title={Running \CONTEXT}]

The \sCONTEXT\ tool is what you will use most as it manages your
run.

\ctxlua{os.execute("context > tools-mkiv-help.tmp")}

\typefile[ntyping]{tools-mkiv-help.tmp}

There are few exert options too:

\ctxlua{os.execute("context --expert > tools-mkiv-help.tmp")}

\typefile[ntyping]{tools-mkiv-help.tmp}

You might as well forget about these unless you are one of the
\CONTEXT\ developers.

\stopsection

\startsection[title={Prefixes}]

A handy feature of \sMTXRUN\ (and as most features an inheritance of
\sTEXMFSTART) is that it will resolve prefixed arguments. This can be of help
when you run programs that are unaware of the \TEX\ tree but nevertheless need to
locate files in it.

\ctxlua{os.execute("mtxrun --prefixes > tools-mkiv-help.tmp")}

\typefile[ntyping]{tools-mkiv-help.tmp}

An example is:

\starttyping
mtxrun --execute xsltproc file:whatever.xsl file:whatever.xml
\stoptyping

The call to \sXSLTPROC\ will get two arguments, being the complete path to the
files (given that it can be resolved). This permits you to organize the files in
a similar was as \TEX\ files.

\stopsection

\startsection[title={Stubs}]

As the tools are written in the \LUA\ language we need a \LUA\ interpreter and or
course we use \LUATEX\ itself. On \UNIX\ we can copy \sLUATOOLS\ and \sMTXRUN\ to
files in the binary path with the same name but without suffix. Starting them in
another way is a waste of time, especially when \sKPSEWHICH\ is used to find
then, something which is useless in \MKIV\ anyway. Just use these scripts
directly as they are self contained.

For \sCONTEXT\ and other scripts that we want convenient access to, stubs are
needed, like:

\starttyping
#!/bin/sh
mtxrun --script context "$@"
\stoptyping

This is also quite efficient as the \sCONTEXT\ script \type {mtx-context} is
loaded in \sMTXRUN\ and uses the same database.

On \WINDOWS\ you can copy the scripts as|-|is and associate the suffix with
\LUATEX\ (or more precisely: \type {texlua}) but then all \LUA\ script will be
run that way which is not what you might want.

In \TEXLIVE\ stubs for starting scripts were introduced by Fabrice Popineau. Such
a stub would start for instance \sTEXMFSTART, that is: it located the script
(\PERL\ or \RUBY) in the \TEX\ tree and launched it with the right interpreter.
Later we shipped pseudo binaries of \sTEXMFSTART: a \RUBY\ interpreter plus
scripts wrapped into a self contained binary.

For \MKIV\ we don't need such methods and started with simple batch files,
similar to the \UNIX\ startup scripts. However, these have the disadvantage that
they cannot be used in other batch files without using the \type {start} command.
In \TEXLIVE\ this is taken care of by a small binary written bij T.M.\ Trzeciak
so on \TEXLIVE\ 2009 we saw a call chain from \type {exe} to \type {cmd} to \type
{lua} which is somewhat messy.

This is why we now use an adapted and stripped down version of that program that
is tuned for \sMTXRUN, \sLUATOOLS\ and \sCONTEXT. So, we moved from the original
\type {cmd} based approach to an \type {exe} one.

\starttyping
mtxrun.dll
mtxrun.exe
\stoptyping

You can copy \type {mtxrun.exe} to for instance \type {context.exe} and it will
still use \sMTXRUN\ for locating the right script. It also takes care of mapping
\sTEXMFSTART\ to \sMTXRUN. So we've removed the intermediate \type {cmd} step,
can not run the script as any program, and most of all, we're as efficient as can
be.

Of course this program is only meaningful for the \CONTEXT\ approach to tools.

It may all sound more complex than it is but once it works users will not notice
those details. Als, in practice not that much has changed in running the tools
between \MKII\ and \MKIV\ as we've seen no reason to change the methods.

\stopsection

\startsubject[title={Colofon}]

\starttabulate[|B|p|]
    \NC author    \NC \documentvariable{author},
                      \documentvariable{affiliation},
                      \documentvariable{location} \NC \NR
    \NC version   \NC \currentdate \NC \NR
    \NC website   \NC \documentvariable{website} \endash\
                      \documentvariable{support} \NC \NR
    \NC copyright \NC \symbol[cc][cc-by-sa-nc] \NC \NR
\stoptabulate

\stopsubject

\stopdocument