From f72c2cf29d36ae836c894bad29dfd965d1af0236 Mon Sep 17 00:00:00 2001 From: Hans Hagen Date: Sun, 18 Aug 2019 22:51:53 +0200 Subject: 2019-08-18 22:26:00 --- .../manuals/followingup/followingup-retrospect.tex | 188 +++++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100644 doc/context/sources/general/manuals/followingup/followingup-retrospect.tex (limited to 'doc/context/sources/general/manuals/followingup/followingup-retrospect.tex') diff --git a/doc/context/sources/general/manuals/followingup/followingup-retrospect.tex b/doc/context/sources/general/manuals/followingup/followingup-retrospect.tex new file mode 100644 index 000000000..b99185b77 --- /dev/null +++ b/doc/context/sources/general/manuals/followingup/followingup-retrospect.tex @@ -0,0 +1,188 @@ +% language=us + +\startcomponent followingup-retrospect + +\environment followingup-style + +\startchapter[title={Retrospect}] + +% \startsection[title={Introduction}] +% \stopsection + +At some point in a new development, and \LUAMETATEX\ feels like that, there comes +a moment when you need to make a decision. In this case the question is if we +need to make hybrid \MKIV\ and \LMTX\ files or do the same as with the transition +from \MKII\ to \MKIV: use two variants. For \TEX\ files a conditional section has +only overhead in the format generation as skipped code doesn't end up in the +format. With conditional \LUA\ code it's different: the ignored section is still +present in byte code. But even for \TEX\ code a conditional section is not +entirely invisible: encountered control sequences are still creating (bogus) hash +entries. So the question is: do we go lean and mean and do we omit historic +non|-|\LMTX\ code? + +A comparison with the transition from \MKII\ is actually relevant. For instance +right from the start \CONTEXT\ had an abstract backend layer, and support for +engines and output formats was loaded on demand. There was never any specific +code in the core. With \MKIV\ we changed the model but there is still some +abstraction. + +In \MKII\ we also had to deal with encodings and that has consequences for +font handling, language support and input encodings. In \MKIV\ all that changed: +internal all is \UTF, as is normally the input (but we can still use encodings), +and fonts are always mapped to \UNICODE. + +Anyhow, much that made sense for \MKII\ was no longer relevant for \MKIV: code +could be dropped. But some mechanisms were reimplemented using \LUA: code was +added. The user interface stayed the same but in \MKIV\ uses a conceptually +different approach deep down. Therefore the code base was split in \MKII\ and +\MKIV\ files but this transition was made stepwise. + +So should the same happen with \LMTX ? There is not that much that needs to be +added to \MKIV\ in terms of functionality. In the end, for the \TEX\ code the +differences are not that substantial, so there we can consider loading different +files. The files involved are rather stable so there is not much danger of +functionality between \MKIV\ and \LMTX\ getting out of sync. The same is true for +the \LUA\ files, although synchronization is probably more an issue there. + +Another option is to always assume that \LUAMETATEX\ is used. For testing regular +\LUATEX\ (patches) we can just use a 2019 stable \CONTEXT. But in order for users +to benefit from developments we then expect them all to move on to \LMTX. Using a +frozen 2019 version with upcoming \LUATEX\ is no big deal as we've done the same +with \MKII\ and that worked out okay. + +When we started with \CONTEXT\ development in the previous century we were doing +pretty weird things. I remember getting comments that what we did made no sense +because it was not what \TEX\ was meant for and some even suggested that it +disrupted the picture. Highly structured input, a clear separation (and +abstraction) of front and backend, inheritance and user defined styling, +integrated support for \XML, embedded \METAPOST, advanced interactive documents, +handling of fonts en encodings, the list is long. Occasionally some of the things +that came with \CONTEXT\ were ridiculed, like the fact that a script was used to +manage the (multiple) run(s), but in the end, look at how many script are around +now. Some even wondered why we used \TEX\ at all because \TEX\ was meant for +typesetting math. And who needs \XML\ let alone \MATHML ? Or interactive \PDF\ +features? Much in \CONTEXT\ and its management got smoother over time and the +\LUAMETATEX\ engine fits nicely into this evolution. It's hard to keep the +cutting edge but at least we have the instruments. + +During \BACHOTEX\ 2019 (end of April, beginning of May) this project was +presented the first time outside the \CONTEXT\ community. During that meeting +Mojca Miklavec, one of the driving forces behind \CONTEXT, upgraded the compile +farm that already was used to compile (intermediate versions of) \LUATEX\ and +\TEXLIVE\ to also compile \type {pplib} (handy for development) and \LUAMETATEX. +This permits us to fine|-|tune the \type {cmake} setup which is still work in +progress. And, also further improvements take place in the code base itself. + +One of the properties of open source is that one can build upon an existing code +base, so when at \BACHOTEX\ Arthur announced that he was going to make a merge of +\XETEX\ (which he maintains) and \LUATEX\ no one was surprised. But it could be a +strong argument for a rather strict code freeze: spin|-|offs need stability. I've +been told that there are now several projects where more libraries (like +Harfbuzz) get integrated. Those cases don't influence the parent but here +stability of the original also is expected, unless of course additional features +go in these engines, which itself creates instability, but that's another matter. +One could actually argue that the arrival of variants defeats the argument that +stability is important: if a macro package uses new features, it needs to adapt, +and naturally (temporary) issues might show up. Such are the dynamics of todays +software development. History in general shows that not that much is persistent +(or even accumulative) and programs are probably the least, so maybe the whole +stability aspect has lost its relevance. \footnote {In a similar way as that the +argument \quotation {Publishers want this or that, so we as \TEX\ community need +to provide it.} is no longer that relevant because publishing is now more a +business model than vocation.} Of course \LUAMETATEX\ is also a follow up, but +one of the ideas behind it was that I could use it as platform for (independent) +experiments that could result in code being put into \LUATEX. Also, the changes +have a limited impact: only \CONTEXT\ will be affected. \footnote {So maybe, in +the end, stability boils down to \quotation {The engine behaves the same and the +\CONTEXT\ that comes with it exploits its features as good as possible}.} + +It is not feasible to make \CONTEXT\ work with all kind of engines that in +practice are not used by its users. For instance, after \XETEX\ showed up it went +through several iterations or font rendering, so we never really spent time on +the low level features that it provided (there was no demand anyway). One cannot +simply claim that one method is better than another that replaces it and expect +constant adaptation (probably for the sake of a few potential users). There +simply is no \quote {best} engine and no \quote {perfect} solution. Another +aspect is that when we would adapt \CONTEXT\ to \LUATEX\ variants the +dependencies on specific functionality that itself depends on the outside world +is kind of unavoidable. Especially languages and fonts are fluid and for the +average user there is not that much difference in that department. Should we +really complicate matters for a few (potential) users? In \CONTEXT\ support like +that is added on demand, driven by specific needs of users who use \TEX\ for a +reason and are willing to test. + +There's enough huge and complex software around that demonstrates what happens +when programs are extended, keep growing, their code base becoming more complex. +Such a process doesn't really fit in my ideas about for \TEX. We positioned 1.10 +as long term stable, with the option to add a few handy things in the long run. +For sure there are niches to fill and it is a fact that the \TEX\ community can +deal with variants of engines: just look at the different \CJK\ engines around, +with prefixes like \type {p}, \type {up}, \type {ep}, etc. But the question is, +where does that put further \LUATEX\ development? And, more important, what +consequences does it have for the \CONTEXT\ code base? + +The reason I mention this is that I had in mind to eventually backport features +that work out well in \LUAMETATEX. I also mentioned that in order to support +stock \LUATEX\ it made no sense to split the \CONTEXT\ code base. After all, a +few conditional sections could deal with the difference between \LUATEX\ and +\LUAMETATEX: some differences could be temporary anyway. But, given recent +developments it actually made sense to split the code base: why spent time on +backporting when the engine user base is spread over different spinoffs. I can +better just assume \CONTEXT\ to exclusively use \LUAMETATEX\ and that other macro +packages use (one or more) \LUATEX\ variants. I can then keep the generic code up +to date and maybe occasionally add some proven stable features. It is also no big +deal to keep the minimum subset needed for (plain) font handling compatible, +assuming \LUATEX\ compatibility, as in the end that engine is the benchmark, +especially when I strip it a bit from features not needed outside \CONTEXT. + +Thoughts like this show how fragile plans and predictions are: within a year one +has to adapt ideas and assumptions. But it also proves that \LUAMETATEX\ was a +good choice for \CONTEXT, especially because it is bound to \CONTEXT\ +development, which keep the users independent and isolated from developments that +don't mind that much the (side) effects on \CONTEXT. + +% \footnote {I mentioned stability a few times, but this aspect is somewhat vague: +% often I see complaints about \LUATEX, or comparisons with other engines, that +% have nothing to do with the engine per se, but more with misunderstanding and|/| +% assumptions, strange usage, maybe or even likely bad user code, comparing apples +% and pears, etc. The term \type {bug} is very popular and often a preferred +% qualifications, and it sounds even more impressive when it's qualified as a bug +% one. I guess that a more tight coupling between specific engines and macro +% packages at least that aspect becomes cleaner.} + +Around the \CONTEXT\ meeting (or maybe a bit later) we hope to have the new +installation infrastructure stable too (currently it is also experimental). By +that time it will also be clear how we will proceed with the \LMTX\ project. In +the meantime I have decided so put \LUAMETATEX\ specific files alongside the +\MKIV\ files, simply because I always need to be able run stock \LUATEX. In order +to show the close relationship these files are flagged as \MKXL, so we bump from +\quote {Mark Four} to \quote {Mark Fourty}. The suffixes \type {mkiv}, \type +{mkvi} and \type {mpiv} get company from \type {mkxl}, \type {mklx} and \type +{mpxl}. Depending on backporting features, files can come and go. I'm not yet +sure about the \LUA\ files but the \type {lmt} suffix is already reserved for +future use. \footnote {This is because \LUA\ 5.4 introduces some new syntax +elements and where we can get away with the difference between 5.2 (\LUAJITTEX) +and 5.3 (\LUATEX) such a syntax change is more drastic.} All this is also driven +by (user) demand. + +Consider this (and these thoughts) a snapshot. There will be the usual reports on +experiments and developments. And in due time there will also be a manual for +\LUAMETATEX. \footnote {In fact it already lives on my machine but I'm not in +ready yet for the usual complaints about manuals, so I'm not in that much of a +hurry.} And yes, at some point I have to make up my mind with respect to +backporting features that have proven to be useful. + +% \footnote {Actually, it seems to come with the Internet: folks wining on whatever +% platform about lack of documentation (most of the \CONTEXT\ distribution actually +% is documentation and quite some articles are, have been, and will be written) or +% possible bug (always huge, even if no bug at all) without exposing much actual +% research or knowledge about these matters. Write, post and shout before thinking +% it through, increase the number hits on your profile. It's for sure a way to make +% something end up at the bottom of my to do list, if at all. A valid response +% could be: whatever did you contribute to the community that I myself (or +% \CONTEXT\ users) can benefit from. Quite likely: nothing (or little)! It looks +% like even the normally friendly \TEX\ community sometimes gets infected by this.} + +\stopchapter + +\stopcomponent -- cgit v1.2.3