% engine=luatex language=uk

\startcomponent hybrid-languages

\environment hybrid-environment

\startchapter[title={The language mix}]

During the third \CONTEXT\ conference that ran in parallel to Euro\TEX\ 2009 in
The Hague we had several sessions where \MKIV\ was discussed and a few upcoming
features were demonstrated. The next sections summarize some of that. It's hard
to predict the future, especially because new possibilities show up once \LUATEX\
is opened up more, so remarks about the future are not definitive.

\startsection[title={\TEX}]

From now on, if I refer to \TEX\ in the perspective of \LUATEX\ I mean \quotation
{Good Old \TEX}, the language as well as the functionality. Although \LUATEX\
provides a couple of extensions it remains pretty close to compatible to its
ancestor, certainly from the perspective of the end user.

As most \CONTEXT\ users code their documents in the \TEX\ language, this will
remain the focus of \MKIV. After all, there is no real reason to abandon it.
However, although \CONTEXT\ already stimulates users to use structure where
possible and not to use low level \TEX\ commands in the document source, we will
add a few more structural variants. For instance, we already introduced \type
{\startchapter} and \type {\startitem} in addition to \type {\chapter} and \type
{\item}.

We even go further, by using key|/|value pairs for defining section titles,
bookmarks, running headers, references, bookmarks and list entries at the start
of a chapter. And, as we carry around much more information in the (for \TEX\ so
typical) auxiliary data files, we provide extensive control over rendering the
numbers of these elements when they are recalled (like in tables of contents).
So, if you really want to use different texts for all references to a chapter
header, it can be done:

\starttyping
\startchapter
  [label=emcsquare,
   title={About $e=mc^2$},
   bookmark={einstein},
   list={About $e=mc^2$ (Einstein)},
   reference={$e=mc^2$}]

  ... content ...

\stopchapter
\stoptyping

Under the hood, the \MKIV\ code base is becoming quite a mix and once we have a
more clear picture of where we're heading, it might become even more of a hybrid.
Already for some time most of the font handling is done by \LUA, and a bit more
logic and management might move to \LUA\ as well. However, as we want to be
downward compatible we cannot go as far as we want (yet). This might change as
soon as more of the primitives have associated \LUA\ functions. Even then it will
be a trade off: calling \LUA\ takes some time and it might not pay off at all.

Some of the more tricky components, like vertical spacing, grid snapping,
balancing columns, etc.\ are already in the process of being \LUA fied and their
hybrid form might turn into complete \LUA\ driven solutions eventually. Again,
the compatibility issue forces us to follow a stepwise approach, but at the cost
of (quite some) extra development time. But whatever happens, the \TEX\ input
language as well as machinery will be there.

\stopsection

\startsection[title={\METAPOST}]

I never regret integrating \METAPOST\ support in \CONTEXT\ and a dream came true
when \MPLIB\ became part of \LUATEX. Apart from a few minor changes in the way
text integrates into \METAPOST\ graphics the user interface in \MKIV\ is the same
as in \MKII. Insofar as \LUA\ is involved, this is hidden from the user. We use
\LUA\ for managing runs and conversion of the result to \PDF. Currently
generating \METAPOST\ code by \LUA\ is limited to assisting in the typesetting of
chemical structure formulas which is now part of the core.

When defining graphics we use the \METAPOST\ language and not some \TEX|-|like
variant of it. Information can be passed to \METAPOST\ using special macros (like
\type {\MPcolor}), but most relevant status information is passed automatically
anyway.

You should not be surprised if at some point we can request information from
\TEX\ directly, because after all this information is accessible. Think of
something \type {w := texdimen(0) ;} being expanded at the \METAPOST\ end instead
of \type {w := \the\dimen0 ;} being passed to \METAPOST\ from the \TEX\ end.

\stopsection

\startsection[title={\LUA}]

What will the user see of \LUA ? First of all he or she can use this scripting
language to generate content. But when making a format or by looking at the
statistics printed at the end of a run, it will be clear that \LUA\ is used all
over the place.

So how about \LUA\ as a replacement for the \TEX\ input language? Actually, it is
already possible to make such \quotation {\CONTEXT\ \LUA\ Documents} using
\MKIV's built in functions. Each \CONTEXT\ command is also available as a \LUA\
function.

\startbuffer
\startluacode
  context.bTABLE {
      framecolor = "blue",
      align= "middle",
      style = "type",
      offset=".5ex",
    }
    for i=1,10 do
      context.bTR()
      for i=1,20 do
        local r= math.random(99)
        if r < 50 then
          context.bTD {
            background = "color",
            backgroundcolor = "blue"
          }
          context(context.white("%#2i",r))
        else
          context.bTD()
          context("%#2i",r)
        end
        context.eTD()
      end
      context.eTR()
    end
  context.eTABLE()
\stopluacode
\stopbuffer

\typebuffer

Of course it helps if you know \CONTEXT\ a bit. For instance we can as well say:

\starttyping
if r < 50 then
  context.bTD {
    background = "color",
    backgroundcolor = "blue",
    foregroundcolor = "white",
  }
else
  context.bTD()
end
context("%#2i",r)
context.eTD()
\stoptyping

And, knowing \LUA\ helps as well, since the following is more efficient:

\startbuffer
\startluacode
  local colored = {
    background = "color",
    backgroundcolor = "blue",
    foregroundcolor = "white",
  }
  local basespec = {
    framecolor = "blue",
    align= "middle",
    style = "type",
    offset=".5ex",
  }
  local bTR, eTR = context.bTR, context.eTR
  local bTD, eTD = context.bTD, context.eTD
  context.bTABLE(basespec)
    for i=1,10 do
      bTR()
      for i=1,20 do
        local r= math.random(99)
        bTD((r < 50 and colored) or nil)
        context("%#2i",r)
        eTD()
      end
      eTR()
    end
  context.eTABLE()
\stopluacode
\stopbuffer

\typebuffer

Since in practice the speedup is negligible and the memory footprint is about the
same, such optimization seldom make sense.

At some point this interface will be extended, for instance when we can use
\TEX's main (scanning, parsing and processing) loop as a so-called coroutine and
when we have opened up more of \TEX's internals. Of course, instead of putting
this in your \TEX\ source, you can as well keep the code at the \LUA\ end.

\placefigure
  {The result of the shown \LUA\ code.}
  {\getbuffer}

The script that manages a \CONTEXT\ run (also called \type {context}) will
process files with the \type {cld} suffix automatically. You can also force
processing as \LUA\ with the flag \type {--forcecld}. \footnote {Similar methods
exist for processing \XML\ files.} The \type {mtxrun} script also recognizes
\type {cld} files and delegate the call to the \type {context} script.

\starttyping
context yourfile.cld
\stoptyping

But will this replace \TEX\ as an input language? This is quite unlikely because
coding documents in \TEX\ is so convenient and there is not much to gain here. Of
course in a pure \LUA\ based workflow (for instance publishing information from
databases) it would be nice to code in \LUA, but even then it's mostly syntactic
sugar, as \TEX\ has to do the job anyway. However, eventually we will have a
quite mature \LUA\ counterpart.

\stopsection

\startsection[title={\XML}]

This is not so much a programming language but more a method of tagging your
document content (or data). As structure is rather dominant in \XML, it is quite
handy for situations where we need different output formats and multiple tools
need to process the same data. It's also a standard, although this does not mean
that all documents you see are properly structured. This in turn means that we
need some manipulative power in \CONTEXT, and that happens to be easier to do in
\MKIV\ than in \MKII.

In \CONTEXT\ we have been supporting \XML\ for a long time, and in \MKIV\ we made
the switch from stream based to tree based processing. The current implementation
is mostly driven by what has been possible so far but as \LUATEX\ becomes more
mature, bits and pieces will be reimplemented (or at least cleaned up and brought
up to date with developments in \LUATEX).

One could argue that it makes more sense to use \XSLT\ for converting \XML\ into
something \TEX, but in most of the cases that I have to deal with much effort
goes into mapping structure onto a given layout specification. Adding a bit of
\XML\ to \TEX\ mapping to that directly is quite convenient. The total amount of
code is probably smaller and it saves a processing step.

We're mostly dealing with education|-|related documents and these tend to have a
more complex structure than the final typeset result shows. Also, readability of
code is not served with such a split as most mappings look messy anyway (or
evolve that way) due to the way the content is organized or elements get abused.

There is a dedicated manual for dealing with \XML\ in \MKIV, so we only show a
simple example here. The documents to be processed are loaded in memory and
serialized using setups that are associated to elements. We keep track of
documents and nodes in a way that permits multipass data handling (rather usual
in \TEX). Say that we have a document that contains questions. The following
definitions will flush the (root element) \type {questions}:

\starttyping
\startxmlsetups xml:mysetups
    \xmlsetsetup{#1}{questions}{xml:questions}
\stopxmlsetups

\xmlregistersetup{xml:mysetups}

\startxmlsetups xml:questions
    \xmlflush{#1}
\stopxmlsetups

\xmlprocessfile{main}{somefile.xml}{}
\stoptyping

Here the \type {#1} represents the current \XML\ element. Of course we need more
associations in order to get something meaningful. If we just serialize then we
have mappings like:

\starttyping
\xmlsetsetup{#1}{question|answer}{xml:*}
\stoptyping

So, questions and answers are mapped onto their own setup which flushes them,
probably with some numbering done at the spot.

In this mechanism \LUA\ is sort of invisible but quite busy as it is responsible
for loading, filtering, accessing and serializing the tree. In this case \TEX\
and \LUA\ hand over control in rapid succession.

You can hook in your own functions, like:

\starttyping
\xmlfilter{#1}{(wording|feedback|choice)/function(cleanup)}
\stoptyping

In this case the function \type {cleanup} is applied to elements with names that
match one of the three given. \footnote {This example is inspired by one of our
projects where the cleanup involves sanitizing (highly invalid) \HTML\ data that
is embedded as a \type {CDATA} stream, a trick to prevent the \XML\ file to be
invalid.}

Of course, once you start mixing in \LUA\ in this way, you need to know how we
deal with \XML\ at the \LUA\ end. The following function show how we calculate
scores:

\starttyping
\startluacode
function xml.functions.totalscore(root)
  local n = 0
  for e in xml.collected(root,"/outcome") do
    if xml.filter(e,"action[text()='add']") then
      local m = xml.filter(e,"xml:///score/text()")
      n = n + (tonumber(m or 0) or 0)
    end
  end
  tex.write(n)
end
\stopluacode
\stoptyping

You can either use such a function in a filter or just use it as
a \TEX\ macro:

\starttyping
\startxmlsetups xml:question
  \blank
  \xmlfirst{#1}{wording}
  \startitemize
    \xmlfilter{#1}{/answer/choice/command(xml:answer:choice)}
  \stopitemize
  \endgraf
  score: \xmlfunction{#1}{totalscore}
  \blank
\stopxmlsetups

\startxmlsetups xml:answer:choice
    \startitem
        \xmlflush{#1}
    \stopitem
\stopxmlsetups
\stoptyping

The filter variant is like this:

\starttyping
\xmlfilter{#1}{./function('totalscore')}
\stoptyping

So you can take your choice and make your source look more \XML|-|ish,
\LUA|-|like or \TEX|-|wise. A careful reader might have noticed the peculiar
\type {xml://} in the function code. When used inside \MKIV, the serializer
defaults to \TEX\ so results are piped back into \TEX. This prefix forced the
regular serializer which keeps the result at the \LUA\ end.

Currently some of the \XML\ related modules, like \MATHML\ and handling of
tables, are really a mix of \TEX\ code and \LUA\ calls, but it makes sense to
move them completely to \LUA. One reason is that their input (formulas and table
content) is restricted to non|-|\TEX\ anyway. On the other hand, in order to be
able to share the implementation with \TEX\ input, it also makes sense to stick
to some hybrid approach. In any case, more of the calculations and logic will
move to \LUA, while \TEX\ will deal with the content.

A somewhat strange animal here is \XSLFO. We do support it, but the \MKII\
implementation was always somewhat limited and the code was quite complex. So,
this needs a proper rewrite in \MKIV, which will happen indeed. It's mostly a
nice exercise of hybrid technology but until now I never really needed it. Other
bits and pieces of the current \XML\ goodies might also get an upgrade.

There is already a bunch of functions and macros to filter and manipulate \XML\
content and currently the code involved is being cleaned up. What direction we go
also depends on users' demands. So, with respect to \XML\ you can expect more
support, a better integration and an upgrade of some supported \XML\ related
standards.

\startsection [title={Tools}]

Some of the tools that ship with \CONTEXT\ are also examples of hybrid usage.

Take this:

\starttyping
mtxrun --script server --auto
\stoptyping

On my machine this reports:

\starttyping
MTXrun | running at port: 31415
MTXrun | document root: c:/data/develop/context/lua
MTXrun | main index file: unknown
MTXrun | scripts subpath: c:/data/develop/context/lua
MTXrun | context services: http://localhost:31415/mtx-server-ctx-startup.lua
\stoptyping

The \type {mtxrun} script is a \LUA\ script that acts as a controller for other
scripts, in this case \type {mtx-server.lua} that is part of the regular
distribution. As we use \LUATEX\ as a \LUA\ interpreter and since \LUATEX\ has a
socket library built in, it can act as a web server, limited but quite right for
our purpose. \footnote {This application is not intentional but just a side
effect.}

The web page that pops up when you enter the given address lets you currently
choose between the \CONTEXT\ help system and a font testing tool. In \in {figure}
[fig:fonttest] you seen an example of what the font testing tool does.

\placefigure
  [here]
  [fig:fonttest]
  {An example of using the font tester.}
  {\externalfigure[mtx-server-ctx-fonttest.png][width=\textwidth]}

Here we have \LUATEX\ running a simple web server but it's not aware of having
\TEX\ on board. When you click on one of the buttons at the bottom of the screen,
the server will load and execute a script related to the request and in this case
that script will create a \TEX\ file and call \LUATEX\ with \CONTEXT\ to process
that file. The result is piped back to the browser.

You can use this tool to investigate fonts (their bad and good habits) as well as
to test the currently available \OPENTYPE\ functionality in \MKIV\ (bugs as well
as goodies).

So again we have a hybrid usage although in this case the user is not confronted
with \LUA\ and|/|or \TEX\ at all. The same is true for the other goodie, shown in
\in {figure} [fig:help]. Actually, such a goodie has always been part of the
\CONTEXT\ distribution but it has been rewritten in \LUA.

\placefigure
  [here]
  [fig:help]
  {An example of a help screen for a command.}
  {\externalfigure[mtx-server-ctx-help.png][width=\textwidth]}

The \CONTEXT\ user interface is defined in an \XML\ file, and this file is used
for several purposes: initializing the user interfaces at format generation time,
typesetting the formal command references (for all relevant interface languages),
for the wiki, and for the mentioned help goodie.

Using the mix of languages permits us to provide convenient processing of
documents that otherwise would demand more from the user than it does now. For
instance, imagine that we want to process a series of documents in the
so|-|called \EPUB\ format. Such a document is a zipped file that has a
description and resources. As the content of this archive is prescribed it's
quite easy to process it:

\starttyping
context --ctx=x-epub.ctx yourfile.epub
\stoptyping

This is equivalent to:

\starttyping
texlua mtxrun.lua --script context --ctx=x-epub.ctx yourfile.epub
\stoptyping

So, here we have \LUATEX\ running a script that itself (locates and) runs a
script \type {context}. That script loads a \CONTEXT\ job description file (with
suffix \type {ctx}). This file tells what styles to load and might have
additional directives but none of that has to bother the end user. In the
automatically loaded style we take care of reading the \XML\ files from the
zipped file and eventually map the embedded \HTML\ like files onto style elements
and produce a \PDF\ file. So, we have \LUA\ managing a run and \MKIV\ managing
with help of \LUA\ reading from zip files and converting \XML\ into something
that \TEX\ is happy with. As there is no standard with respect to the content
itself, i.e.\ the rendering is driven by whatever kind of structure is used and
whatever the \CSS\ file is able to map it onto, in practice we need an additional
style for this class of documents. But anyway it's a good example of integration.

\stopsection

\startsection [title={The future}]

Apart from these language related issues, what more is on the agenda? To mention
a few integration related thoughts:

\startitemize[packed]

\startitem
    At some point I want to explore the possibility to limit processing to just
    one run, for instance by doing trial runs without outputting anything but
    still collecting multipass information. This might save some runtime in
    demanding workflows especially when we keep extensive font loading and image
    handling in mind.
\stopitem

\startitem
    Related to this is the ability to run \MKIV\ as a service but that demands
    that we can reset the state of \LUATEX\ and actually it might not be worth
    the trouble at all given faster processors and disks. Also, it might not save
    much runtime on larger jobs.
\stopitem

\startitem
    More interesting can be to continue experimenting with isolating parts of
    \CONTEXT\ in such a way that one can construct a specialized subset of
    functionality. Of course the main body of code will always be loaded as one
    needs basic typesetting anyway.
\stopitem

\stopitemize

Of course we keep improving existing mechanisms and improve solutions using a mix
of \TEX\ and \LUA, using each language (and system) for what it can do best.

\stopsection

\stopchapter

\stopcomponent