2018-10-17 15:14:00

18 files changed, 373 insertions, 1 deletions

diff --git a/doc/context/documents/general/manuals/luatex.pdf b/doc/context/documents/general/manuals/luatex.pdf
index 82d0931a9..3af62acd7 100644
--- a/doc/context/documents/general/manuals/luatex.pdf
+++ b/doc/context/documents/general/manuals/luatex.pdf
diff --git a/doc/context/documents/general/manuals/musings.pdf b/doc/context/documents/general/manuals/musings.pdf
index 8c956b7c4..7573569b9 100644
--- a/doc/context/documents/general/manuals/musings.pdf
+++ b/doc/context/documents/general/manuals/musings.pdf
diff --git a/doc/context/documents/general/qrcs/setup-cs.pdf b/doc/context/documents/general/qrcs/setup-cs.pdf
index d1c49f8b1..db96bb54f 100644
--- a/doc/context/documents/general/qrcs/setup-cs.pdf
+++ b/doc/context/documents/general/qrcs/setup-cs.pdf
diff --git a/doc/context/documents/general/qrcs/setup-de.pdf b/doc/context/documents/general/qrcs/setup-de.pdf
index aa6cee0e1..f65a41e0d 100644
--- a/doc/context/documents/general/qrcs/setup-de.pdf
+++ b/doc/context/documents/general/qrcs/setup-de.pdf
diff --git a/doc/context/documents/general/qrcs/setup-en.pdf b/doc/context/documents/general/qrcs/setup-en.pdf
index d8dc51e14..e1e1bc1cd 100644
--- a/doc/context/documents/general/qrcs/setup-en.pdf
+++ b/doc/context/documents/general/qrcs/setup-en.pdf
diff --git a/doc/context/documents/general/qrcs/setup-fr.pdf b/doc/context/documents/general/qrcs/setup-fr.pdf
index 5b4822110..cca294dfd 100644
--- a/doc/context/documents/general/qrcs/setup-fr.pdf
+++ b/doc/context/documents/general/qrcs/setup-fr.pdf
diff --git a/doc/context/documents/general/qrcs/setup-it.pdf b/doc/context/documents/general/qrcs/setup-it.pdf
index 5e7251c8c..789088c63 100644
--- a/doc/context/documents/general/qrcs/setup-it.pdf
+++ b/doc/context/documents/general/qrcs/setup-it.pdf
diff --git a/doc/context/documents/general/qrcs/setup-mapping-cs.pdf b/doc/context/documents/general/qrcs/setup-mapping-cs.pdf
index 9e9c44e8d..7d38dea46 100644
--- a/doc/context/documents/general/qrcs/setup-mapping-cs.pdf
+++ b/doc/context/documents/general/qrcs/setup-mapping-cs.pdf
diff --git a/doc/context/documents/general/qrcs/setup-mapping-de.pdf b/doc/context/documents/general/qrcs/setup-mapping-de.pdf
index 7727c6c80..0fc61cd85 100644
--- a/doc/context/documents/general/qrcs/setup-mapping-de.pdf
+++ b/doc/context/documents/general/qrcs/setup-mapping-de.pdf
diff --git a/doc/context/documents/general/qrcs/setup-mapping-en.pdf b/doc/context/documents/general/qrcs/setup-mapping-en.pdf
index b78b69945..21537aa94 100644
--- a/doc/context/documents/general/qrcs/setup-mapping-en.pdf
+++ b/doc/context/documents/general/qrcs/setup-mapping-en.pdf
diff --git a/doc/context/documents/general/qrcs/setup-mapping-fr.pdf b/doc/context/documents/general/qrcs/setup-mapping-fr.pdf
index afd504957..8ca776be2 100644
--- a/doc/context/documents/general/qrcs/setup-mapping-fr.pdf
+++ b/doc/context/documents/general/qrcs/setup-mapping-fr.pdf
diff --git a/doc/context/documents/general/qrcs/setup-mapping-it.pdf b/doc/context/documents/general/qrcs/setup-mapping-it.pdf
index b530df9c3..a2691c1e8 100644
--- a/doc/context/documents/general/qrcs/setup-mapping-it.pdf
+++ b/doc/context/documents/general/qrcs/setup-mapping-it.pdf
diff --git a/doc/context/documents/general/qrcs/setup-mapping-nl.pdf b/doc/context/documents/general/qrcs/setup-mapping-nl.pdf
index 1d448a5af..4e1491898 100644
--- a/doc/context/documents/general/qrcs/setup-mapping-nl.pdf
+++ b/doc/context/documents/general/qrcs/setup-mapping-nl.pdf
diff --git a/doc/context/documents/general/qrcs/setup-mapping-ro.pdf b/doc/context/documents/general/qrcs/setup-mapping-ro.pdf
index f565c5543..bbd19ee43 100644
--- a/doc/context/documents/general/qrcs/setup-mapping-ro.pdf
+++ b/doc/context/documents/general/qrcs/setup-mapping-ro.pdf
diff --git a/doc/context/documents/general/qrcs/setup-nl.pdf b/doc/context/documents/general/qrcs/setup-nl.pdf
index 304a63658..a1fdf1440 100644
--- a/doc/context/documents/general/qrcs/setup-nl.pdf
+++ b/doc/context/documents/general/qrcs/setup-nl.pdf
diff --git a/doc/context/documents/general/qrcs/setup-ro.pdf b/doc/context/documents/general/qrcs/setup-ro.pdf
index 39ecb74ad..c67652ac4 100644
--- a/doc/context/documents/general/qrcs/setup-ro.pdf
+++ b/doc/context/documents/general/qrcs/setup-ro.pdf
diff --git a/doc/context/sources/general/manuals/musings/musings-roadmap.tex b/doc/context/sources/general/manuals/musings/musings-roadmap.tex
new file mode 100644
index 000000000..f8771ba42
--- /dev/null
+++ b/doc/context/sources/general/manuals/musings/musings-roadmap.tex
@@ -0,0 +1,372 @@
+% language=uk
+
+% \showfontkerns
+
+\startcomponent musings-roadmap
+
+\environment musings-style
+
+\startchapter[title={\METATEX, a roadmap}]
+
+% \startlines \setupalign[flushright]
+% Hans Hagen
+% Hasselt NL
+% September 2018
+% \stoplines
+
+\startsection[title={Introduction}]
+
+Here I will shortly wrap up the state of \LUATEX\ and \CONTEXT\ in fall 2018. I
+made the first draft of this article as preparation for the \CONTEXT\ meeting
+where we also discussed the future. I updated the text afterwards to match the
+decisions made there. It's also a personal summary of thoughts and discussions
+with team members about where to move next.
+
+\stopsection
+
+\startsection[title={The state of affairs}]
+
+After a dozen years the development of \LUATEX\ has reached a state where adding
+more functionality and|/|or opening up more of the internals makes not much
+sense. Apart from fixes and maybe some minor extensions, version 1.10 is what you
+get. Users can do enough in \LUA\ and there is not much to gain in convenience
+and performance. Of course some of the code can and will be cleaned up, as we
+still see the effects of going from \PASCAL\ to \CWEB\ to \CCODE. In the process
+consistency is on the radar so we might occasionally add a helper. But we also
+don't want to move too far away from the original code, which is for instance why
+we keep names, keys and other properties found in original \TEX, which in turn
+leads to some inconsistencies with extensions added over time. We have to accept
+that.
+
+Because \LUATEX\ development is closely related to \CONTEXT\ development,
+especially \MKIV, we've also reached the moment that we can get rid of some older
+code and assume the latest \LUATEX\ to be used. Because we do so much in \LUA\
+the question is always to what extent the benefits outweigh the drawbacks. Just
+in case you wonder why we use \LUA\ extensively, the main reason is that it is
+easier and more efficient to manage data in this language and modern typesetting
+needs much data. It also permits us to extend regular \TEX\ functionality. But,
+one should not overrate the impact: we still let \TEX\ do what \TEX\ is best at!
+
+Performance is quite important. It doesn't make sense to create a powerful
+typesetting system where processing a page takes a second. We have discussed
+performance before since one of the complaints about \LUATEX\ is that it is slow.
+A simple, basic test is this:
+
+\starttyping
+\starttext
+    \dorecurse{1000}{\input tufte \par}
+\stoptext
+\stoptyping
+
+This involves 1000 times loading a file (and reporting that on the console, which
+can influence runtime), typesetting paragraphs, splitting of a page and of course
+loading fonts and saving to the \PDF\ file. When I run this on a modest machine,
+I get these (relative) timings for the (about) 225 pages:
+
+\starttabulate[|l|c|c|c|c|]
+\BC \TEX\ engine used  \BC \PDFTEX \BC \LUATEX \BC \LUAJITTEX \BC \XETEX \NC \NR
+\BC runtime in seconds \NC 2.0     \NC  3.9    \NC 3.0        \NC 8.4    \NC \NR
+\stoptabulate
+
+Now, as expected the 8 bit \PDFTEX\ is the winner here but \LUATEX\ is not doing
+that bad. I don't know why \XETEX\ is so much slower, maybe because its 64 bit
+binary is less optimal. I once noticed that a 64 bit \PDFTEX\ performed worse on
+such a test than \LUATEX, for which I always use 64 bit binaries.
+
+If you consider that often much more is done than in this example, you can take
+my word that \LUATEX\ quickly outpaces \PDFTEX\ on more complex tasks. In that
+sense it is now our benchmark. It must be said that the \MKIV\ code is probably a
+bit more efficient than the \MKII\ code but that doesn't matter much in this
+simple test because hardly any macro magic happens here; it mostly tests basic
+font processing, paragraph building and page construction. I don't think that I
+can squeeze out more pages per second, at least not without users telling me
+where they encounter bottlenecks that don't result from their style coding. It's
+no problem to write inefficient macros (or styles) so normally a user should
+first carefully check her|/|his own work. Using a more modern \CPU\ with proper
+caching and an \SSD\ helps too.
+
+So, to summarize, we can say that with version 1.10 \LUATEX\ is sort of finished.
+Our mission is now to make \LUATEX\ robust and stable. Things can be added and
+improved, but these are small and mostly consistency related.
+
+\stopsection
+
+\startsection[title={More in \LUA}]
+
+Till now I always managed to add functionality to \CONTEXT\ without hampering
+performance too much. Of course the biggest challenge is always in handling fonts
+and common features like color because that all happens in \LUA. So, the question
+is, what if we delegate more of the core functionality to \LUA ? I will discuss a
+few options because the \CONTEXT\ developers and users need to agree on the path
+to follow. One question there is, are the possible performance hits (which can be
+an inconvenience) compensated by better and easier typesetting.
+
+Fonts, colors, special typesetting features like spaced kerning, protrusion,
+expansion, but also dropped caps, line numbering, marginal notes, tables,
+structure related things, floats and spacing are not open for much discussion.
+All the things that happen in \LUA\ combined with macros is there and will stay.
+But how about hyphenation, paragraph building and page building? And how about a
+leaner and meaner, future safe engine?
+
+Hyphenation is handled in the \TEX\ core. But in \CONTEXT\ already for years one
+can also use a \LUA\ based variant. There is room for extensions and improvements
+there. Interesting is that performance is more or less the same, so this is an
+area where we might switch to the \LUA\ method eventually. It compares to fonts,
+where node mode is more or less the standard and base mode the old way.
+
+Building the paragraphs in \LUA\ is also available in \MKIV, although it needs an
+update. Again performance is not that bad, so when we add features not possible
+(or hard to do) in regular \TEX, it might actually pay of to default to the
+par builder written in \LUA.
+
+The page builder is also doable in \LUA\ but so far I only played a bit with a
+\LUA\ based variant. I might pick up that thread. However, when we would switch to
+\LUA\ there, it might have a bit of a penalty, unless we combine it with some
+other mechanisms which is not entirely trivial, as it would mean a diversion from
+the way \TEX\ does it normally.
+
+How about math? We could at some point do math rendering in \LUA\ but because the
+core mechanism is the standard, it doesn't really makes much sense. It would also
+touch the soul of \TEX. But, I might give it a try, just for fun, so that I can
+play with it a bit. It's typically something for cold and rainy days with some
+music in the background.
+
+We already use \LUA\ in the frontend: locating and reading files in \TEX,
+\XML, \LUA\ and whatever input format. Normalization and manipulation is all
+active and available. The backend is also depending on \LUA, like support for
+special \PDF\ features and exporting to \XML . The engine still handles the page
+stream conversion, font inclusion and object management.
+
+The inclusion of images is also handled by the engine, although in \CONTEXT\ we
+can delegate \PDF\ inclusion to \LUA. Interesting is that this has no performance
+hit.
+
+With some juggling the page stream conversion can also be done in \LUA, and I
+might move that code into the \CONTEXT\ distribution. Here we do have a
+performance hit: about one second more runtime on the 14 seconds needed for the
+300 page \LUATEX\ manual and just over more than half a second on a 11 second
+\LUAJITTEX\ run. The manual has lots of tables, verbatim, indices and uses color
+as well as a more than average number of fonts and much time is spent in \LUA. So
+there is a price to pay there. I tried to speed that up but there is not much to
+gain there.
+
+So, say that we default to \LUA\ based hyphenation, which enables some new
+functionality, \LUA\ based par building, which permits some heuristics for corner
+cases, and \LUA\ based page building, which might result in more control over
+tricky cases. A total performance hit of some 5\% is probably acceptable,
+especially because by that time I might have replaced my laptop and won't notice
+the degrade. This still fits in the normal progress and doesn't really demand a
+roadmap or wider acceptance. And of course we would still use the same strategies
+as implemented in traditional \TEX\ as default anyway.
+
+\stopsection
+
+\startsection[title={A more drastic move}]
+
+More fundamental is the question whether we delegate more backend activity to
+\LUA\ code. If we decide to handle the page stream in \LUA, then the next
+question is, why not also delegate object management and font inclusion to
+\LUA. Now, keep in mind that this is all very \CONTEXT\ specific! Already for
+more than a decade we delegate a lot to \LUA, and also we have a rather tight
+control over this core functionality. This would mean that \CONTEXT\ doesn't
+really need the backend code in the engine. \footnote {For generic packages like
+TikZ we (can) provide some primitive emulators, which is rather trivial to
+implement.}
+
+That situation is actually not unique. For instance, already for a while we don't
+need the \LUATEX\ font loader either, as loading the \OPENTYPE\ files is done in
+\LUA. So, we could also get rid of the font loader code. Currently some code is
+shared with the font inclusion in the backend but that can be isolated.
+
+You can see a \TEX\ engine as being made from several parts, but the core really
+concerns only two processes: reading, storing and expanding macros on the one
+hand, and converting a stream of characters into lines, paragraphs, pages etc.
+Fonts are mostly an abstraction: they are visible in so called glyph nodes as
+font identifier (a number) and character code (also a number) properties. The
+result, nowadays being \PDF, is also an abstraction: at some point the engine
+converts the to be shipped out box in \PDF\ instructions, and in our case,
+relatively simple ones. The backend registers which characters and fonts are used
+and also includes the right resources. But, the backend is not part of the core
+as such! It has been introduced in \PDFTEX\ and is a so called extension.
+
+So, what does that all mean for a future version of \CONTEXT\ and \LUATEX ? It
+means that we can decide to follow up with a \CONTEXT\ that does more in \LUA,
+which means not hard coded in a binary, on the one hand, but that we can also
+decide to strip the engine from non|-|core code. But, given that \LUATEX\ is also
+used in other macro packages, this would mean a different engine. We cannot say
+that \LUATEX\ is stable when we also experiment with core components.
+
+We've seen folks picking up experimental versions assuming that it is a precursor
+to official code. So, in order to move on we need to avoid confusion: we need to
+use another name. Choosing a name is always tricky but as Taco already registered
+the \METATEX\ domain, and because in the \CONTEXT\ distribution you will find
+references to \METATEX, we will use that name for the future engine. Adding \LUA\
+to that name makes sense but then the name would become too long.
+
+The main difference between \METATEX\ and \LUATEX\ would be that the former has
+no file lookup library, no hardcoded font loader, and no backend generator (but
+possibly some helpers, and these need time to evolve). We're basically back where
+\TEX\ started but instead of coding these extensions in \PASCAL\ or \CCODE\ we
+use \LUA. We're also kind of back to when we first started experimenting with
+\LUATEX\ in \CONTEXT\ where test, write and rewrite were going in parallel. But,
+as said, we cannot impose that on a wide audience.
+
+If we go for such a lean and mean follow up, then we can also do a more drastic
+cleanup of obsolete code in \CONTEXT\ (dating from \ETEX, \PDFTEX, \ALEPH, etc.).
+We then are sort of back to where it all started: we go back to the basics. This
+might mean dropping some primitives (one can define them as dummy). Of course we
+could generalize some of the \CONTEXT\ code to provide the kicked out
+functionality but would that pay of? Probably not.
+
+Just for the record: replacing the handling of macros, registers, grouping, etc.\
+to \LUA\ is not really an option as the performance hit would make a large system
+like \CONTEXT\ sort of unusable: it's no option and not even considered (although
+I must admit that I have some experimental \LUA\ based \TEX\ parser code around).
+
+It is quite likely that building \METATEX\ from source for the moment will be an
+option to the build script. But we can also decide to simplify that process,
+which is possible because we only need one binary. But in general we can assume
+that one can generate \METATEX\ and \LUATEX\ from the same source. A first step
+probably is a further isolation of the backend code. The fontloader and file
+handling code already can be made optional.
+
+Given that we only need one binary (it being \LUATEX\ or \METATEX) and nowadays
+only use \OPENTYPE\ fonts, one can even start thinking of a mini distribution,
+possibly with a zipped resource tree, something we experimented with in the early
+days of \LUATEX.
+
+Another though I have been playing with is a better separation between low level
+and high level \CONTEXT\ commands, and whether the low level layer should be more
+generic in nature (so that one can run specific packages on top of it instead of
+the whole of \CONTEXT) but that might not be worth the trouble.
+
+\stopsection
+
+\startsection[title={Interlude}]
+
+If we look at the future, it's good to also look at the past. Opening up \TEX\
+the way we did has many advantages but also potential drawbacks. It works quite
+well in \CONTEXT\ because we ship an integrated package. I don't think that there
+are many users who kick in their own callbacks. It is possible but completely up
+to the user to make sure things work out well. Performance hits, interference,
+crashes: those who interfere with the internals can sort that out themselves. I'm
+not sure how well that works out in other macro packages but it is a time bomb if
+users start doing that. Of course the documented interfaces to use \LUA\ in
+\CONTEXT\ are supported. So far I think we're not yet bitten in the tail. We keep
+this aspect out of the discussion.
+
+Another important aspect is stability of the engine. Sometimes we get suggestions
+for changes or patches that works for a specific case but for sure will have side
+effects on \CONTEXT. Just as we don't test \LATEX\ side effects, \LATEX\ users
+don't check \CONTEXT. And we're not even talking of users who expect their code
+to keep working. A tight control over the source is important but cannot be we
+will not be around for ever. This means that at some point \LUATEX\ should not be
+changed any more, even when we observe side effects we want to get rid of,
+because these side effects can be in use. This is another argument for a stripped
+down engine. The less there is to mess with, the less the mess.
+
+\stopsection
+
+\startsection[title={Audience}]
+
+So how about \CONTEXT\ itself? Of course we can make it better. We can add more
+examples and more documentation. We can try to improve support. The main question
+for us (as developers) is who actually is our audience. From the mails coming to
+the \CONTEXT\ support list it looks like a rather diverse group of users.
+
+At \TEX\ meetings there are often discussions about promoting \TEX. I can agree
+on the fact that even for simple documents it makes a lot of sense to use \TEX,
+but who will take the first hurdles? How many people really produce a lot of
+documents? And how many need \TEX\ after maybe a short period of (enforced) usage
+at the university?
+
+It's not trivial to recognize the possibilities and power of the
+\LUATEX|-|\CONTEXT\ combination. We never got any serious requests for support
+from large organizations. In fact, we do use this combination in a few projects
+for educational publishers, but there it's actually the authors and editors doing
+the work. It's seldom company policy to use tools that efficiently automate
+typesetting. I dare to say that publishers are not really an audience at all:
+they normally delegate the task. They might accept \TEX\ documents but let them
+rekey or adapt far|-|far|-|away and as cheap as possible. Thinking of it, the
+main reason for Don Knuth for writing \TEX\ in the first place was the ability to
+control the look and feel and quality. It were developments at typesetters and
+publishers that triggered development of \TEX . It was user demand. And the
+success of \TEX\ was largely due to the unique personality and competence of the
+author.
+
+System integrators qualify as audience but I fear that \TEX\ is not considered
+hip and modern. It doesn't seem to matter if you can demonstrate that it can do a
+wonderful job efficiently and relatively cheap. Also the fact that an
+installation can be very stable on the long run is of no importance. Maybe that
+audience (market place) is all about \quotation {The more we have to program and
+update regularly, the merrier.}. Marketing \TEX\ is difficult.
+
+Those who render multiple products, maintain manuals, have to render many
+documents automatically qualify as audience. But often company policies,
+preferred suppliers, so called standard tools etc.\ are used as argument against
+\TEX. It's a missed opportunity.
+
+One needs a certain mindset to recognize the potential and the question is, how
+do we reach that audience. Drawing a roadmap for that is not easy but worth
+discussing. We're open for suggestions.
+
+% \footnote {It's kind of interesting that recently the \TEX\ User Group announced
+% its presence on Facebook and Twitter. Apart from wondering how that gets updated,
+% one can also wonder how many potential (or even current) users go there, given
+% that these platforms are subjected to rise and fall. I'm on neither of them and
+% don't plan to. Kids (our future users) that I know already said goodbye to them.
+% We'll see how that works out.}
+
+\stopsection
+
+\startsection[title={Conclusion}]
+
+At the \CONTEXT\ user meeting those present agreed that moving forward this way
+makes sense. This means that we will explore a lean and mean \METATEX\ alongside
+\LUATEX. There is no rush and it's all volunteer work so we will take our time
+for this. It boils down to some reshuffling of code so that we can remove the
+built|-|in font loader, file handling, and probably also \SYNCTEX\ because we can
+emulate that. Then the backend with its font inclusion code will be cleaned up a
+bit (we even discussed only supporting modern wide fonts). It's no big deal to
+adapt \CONTEXT\ to this (so it can and will support both \LUATEX\ and \METATEX).
+Eventually the backend might go away but now we're talking years ahead. By then
+we can also explore the option to make \METATEX\ start out as a \LUA\ function
+call (the main control loop) and become reentrant. There will probably not be
+many changes to the opened up \TEX\ kernel, but we might extend the \METAPOST\
+part a bit (some of that was discussed at the meeting) especially because it is a
+nice tool to visualize big data.
+
+As with \LUATEX\ development we will go in small steps so that we keep a working
+system. Of course \LUATEX\ is always there as stable fallback. The experiments
+will mostly happen in the experimental branch and binaries will be generated
+using the compile farm on the \CONTEXT\ garden, just as happens now. This also
+limits testing and exploring to the \CONTEXT\ community so that there are no side
+effects for mainstream \LUATEX\ usage.
+
+Nowadays, instead if roadmaps, we tend to use navigational gadgets that adapt
+themselves to the situation. On the road by car this can mean a detour and when
+walking around it can be going to suggested points of interest. During the
+excursion at the meeting, we noticed that after the drivers (navigators)
+synchronized their gadget with Jano, the routes that were followed differed a
+bit. We saw cars in front of going a different direction and cars behind us
+arriving from a different direction. So, even when we talk about roadmaps, our
+route can be adapted to the situation.
+
+Now here is something to think about. If you look at the \TEX\ community you will
+notice that it's an aging community. User groups seem to loose members, although
+the \CONTEXT\ group is currently still growing. Fortunately we see a new
+generation taking interest and the \CONTEXT\ users are a pleasant mix and it
+makes me stay around. I see it as an \quote {old timers} responsibility to have
+\TEX\ and its environment in a healthy state by the time I retire from it
+(although I have no plans in that direction). In parallel to the upcoming
+development I think we will also see a change in \TEX\ use and usage. This aspect
+was also discussed at the meeting and for sure will get a follow up on the
+mailing lists and future meetings. It might as well influence the decisions we
+make the upcoming years. So far \TEX\ has never failed us in it's flexibility and
+capacity to adapt, so let's end on that positive note.
+
+\stopsection
+
+\stopchapter
+
+\stopcomponent
diff --git a/doc/context/sources/general/manuals/musings/musings.tex b/doc/context/sources/general/manuals/musings/musings.tex
index a49de9265..e2787dc99 100644
--- a/doc/context/sources/general/manuals/musings/musings.tex
+++ b/doc/context/sources/general/manuals/musings/musings.tex
@@ -15,7 +15,7 @@
     \component musings-whytex
     \component musings-staygo
     \component musings-stability
-  % \component musings-roadmap
+    \component musings-roadmap
 \stopbodymatter
 
 \stopproduct