From afc6f0a4de593d7164341006a7dfc5e1add977aa Mon Sep 17 00:00:00 2001 From: Hans Hagen Date: Thu, 9 Jan 2020 12:04:00 +0100 Subject: 2020-01-09 11:26:00 --- .../manuals/evenmore/evenmore-libraries.tex | 186 +++++++++++++++++++++ .../sources/general/manuals/evenmore/evenmore.tex | 1 + 2 files changed, 187 insertions(+) create mode 100644 doc/context/sources/general/manuals/evenmore/evenmore-libraries.tex (limited to 'doc/context/sources/general/manuals/evenmore') diff --git a/doc/context/sources/general/manuals/evenmore/evenmore-libraries.tex b/doc/context/sources/general/manuals/evenmore/evenmore-libraries.tex new file mode 100644 index 000000000..35c957c99 --- /dev/null +++ b/doc/context/sources/general/manuals/evenmore/evenmore-libraries.tex @@ -0,0 +1,186 @@ +% language=us + +\startcomponent evenmore-libraries + +\environment evenmore-style + +\startchapter[title={Libraries}] + +\startsection[title={Introduction}] + +The \LUAMETATEX\ binary comes with a couple of libraries built in. These normally +provide enough functionality to get a \TEX\ job done. But take the case where +need to manipulate (or convert) an image before we can include it? It would be +nice if \CONTEXT\ does that for you so having some features in the binary that +handle it make sense. However, given that such a conversion only happens once it +makes more sense to just call an external program and let that deal with it. It +is for that reason that the \CONTEXT\ code base has hardly any library related +code: most of what one wants to do can be done by calling a program. Some callers +are built in, others can be dealt with using the Adityas filter module. The most +significant runtime exception is probably accessing \SQL\ databases where it +might be more efficient to use a library call instead of calling a client. And +even then the main reason for that interface being present is the simple fact +that I (ab)use the engine to serve requests that need some kind of database +access. Another example of where we need some external program is in generating +barcodes. Here one can argue that it does make sense to do that runtime, for +instance because they change or because one doesn't like to have dozens of cached +barcode images on disk. + +In this chapter I will explain how we deal with libraries in \LUAMETATEX. Because +libraries create a dependency an approach is chosen that tries to avoid bloating +the source tree with additional header and source files. This is made easy by the +fact that we don't need full blown interfaces to libraries where all methods are +exposed. We know what we need and most of these tasks somehow relate to +typesetting which is a limited application with known demands in terms of input, +output and performance. We don't need to serve every possible scenario. + +\stopsection + +\startsection[title={Using \LUA\ libraries}] + +One approach is to use a \LUA\ library that sits between the embedded \LUA\ instance +and the external library. Say that one does this: + +\starttyping +local mylib = require("mylib") +\stoptyping + +This can locate and load the file \type {mylib.lua} which implements a bunch of +(\LUA) functions. But, it can also load a library, for instance \type +{mylib.dll}, a binary that provides functions that themselves can call external +ones. Often such a library is also responsible for some resource management which +is then done via userdata objects. Such a connector library on the one hand +refers to \LUA\ library methods (like \type {const char * str = lua_tostring (L, +1);} for fetching a \LUA\ string variable from the argument list) and on the +other hand to those in the external library (like passing that string \type {str} +to a function and passing the result back to \LUA\ with \type {lua_pushstring (L, +result);}). If we would follow that approach in \LUAMETATEX\ it means that in +addition to the main binary (on \MSWINDOWS\ that is \type {luametatex.exe}) there +is also an extra intermediate binary (on \MSWINDOWS\ that is \type {mylib.dll}) +plus the external library (on \MSWINDOWS\ that could be \type {foolib.dll}) which +itself can depend on other libraries. + +In this approach we need to compile the extra intermediate libraries alongside +the main \LUAMETATEX\ binary. Quite likely we then need access to the header +files of the external libraries too. We might even decide to put the dependencies +in our source tree. But, this is not what we like to do: it adds extra work, we +need to keep an eye on updates and operating specific patches, we complicate the +compilation, etc. This all contradicts the fact that we want \LUAMETATEX\ to be +simple. There is no need to complicate the setup just because a very few users +want to use some library. Add to this the fact that quite likely we need to +provide a version of \LUAMETATEX\ that exposes its \LUA\ related symbols which +makes for a larger binary. So, this approach is not really an option because +at the same time we like to keep the binary (and memory footprint) as small +as possible (think of running in a container or on a low energy device). + +\stopsection + +\startsection[title=A variant] + +There are a few issues when you use \LUA\ libraries from elsewhere. First of all, +you need to get hold of one that matches the version of \LUA\ that you use. There +are not that many and some only can be set up as part of a larger framework. +Also, you can find plenty of modules that seem not to be maintained (or maybe +they are just very stable and I'm wrong here). Also, not all platforms are +supported equally well. Then there is the question to what extend libraries are +to stay. What is considered to be the standard today might not be tomorrow. Even +in the rather stable \TEX\ ecosystem we see them come and go. These are all +reasons to avoid hard coded dependencies. Ideally we like users to be able to +compile \LUAMETATEX\ in the future without too must hassle. + +A couple of years after we started the \LUATEX\ project, a solution for using +libraries was implemented, called \SWIGLIB, because it uses the swig +infrastructure. It was an attempt to come up with a more or less standard +approach, a rather one|-|to|-|one mapping so that basically any library could be +interfaced. But, probably because no one really needs libraries, it never catched +on. In \MKIV\ we still support loading libraries made that way but in \LMTX\ that +code has been removed. + +As a side note: the code that deals with this in \MKIV\ also deals with version +specific loading. When we were playing with for instance \MYSQL\ libs we found +out that it made sense to be able to support different \API s, but in the end, +given the rare usage of libraries, that made no sense either. Therefore in \LMTX\ +locating libraries has version support removes and as a consequence is much +simpler (code|-|wise). + +\stopsection + +\startsection[title=Foreign function interfaces] + +Then there is a \FFI\ interface, first introduced in \LUAJITTEX\ as it is part of +\LUAJIT, and later a similar library was built|-|in \LUATEX. But \LUAJIT\ doesn't +conceptually follow \LUA\ upgrades and its future is unsure so in \LUAMETATEX\ +there is no \JIT\ variant (the \JIT\ part was never used anyway as it only slowed +down a run; we just used the \FFI\ part plus the fact that the restricted virtual +machine performs better). The \FFI\ library used in \LUATEX\ also comes from +elsewhere and it doesn't seem to be maintained any longer, so that code is to be +kept working in the perspective of \LUATEX. Both technologies hook into the +processor architecture and are somewhat complex so when their maintenance becomes +unsure we have to reconsider using them. Not all hardware platforms are supported +\footnote {As I write this only Intel works while ARM doesn't and only on +\MSWINDOWS, \LINUX\ and \OSX\ I can compile without alignment warnings} and the +functionality can differ in details per platform. To some extend we can keep +using \FFI\ in \LUATEX\ because Luigi takes care of it, but who knows when it +becomes too problematic. Does it make sense to adopt a library that needs tweaks +depending on architectures? For now we're good for \LUATEX, so for a while we're +also okay (in \MKIV). + +The nice thing about \FFI\ is that one can define the interface at runtime. Of +course this interface has to fit the current version of the library \API, but +that is doable. It is up to a user of a library to determine where it comes from. +It can be put in the \TEX\ tree but also being taken from wherever the operating +system put it in the path. Of course that can then be a bit of an issue when +there are different versions because programs can ship their own variants, but +when you use a library you probably are aware of that and know what you're doing. +A drawback of \FFI\ is that it opens up the whole machinery pretty low level, +which can be considered a risk. Some can consider that to be a security threat. +It for these reasons that \LUAMETATEX\ doesn't provide the \FFI\ feature; users +who depend on it can of course use \MKIV\ with \LUATEX. + +\stopsection + +\startsection[title=So how to proceed?] + +When a library and its \LUA\ interface are kept external the main binary has to +be compiled in a way that permits loading libraries (read: symbols need to be +known). When we use \FFI\ that is not needed. And when a library is internal we +have the disadvantage that we mentioned at the start of this chapter. + +So, how do we combine the advantages of \FFI\ (runtime binding), external +libraries (no need to have all that code in the code base) and internal libraries +(no loading issues)? At some point it stroke me that we actually can do that with +not that much effort. The solution was probably subconsciously implanted by +noticing the fact that the \LUAMETATEX\ machinery uses function pointers in some +places and the fact that when a \LUA\ library is loaded by \LUA\ itself, a +specific initialization function is called to initialize it: by combining these +concepts we can delay the binding till when a library is needed. + +In \LUAMETATEX\ we can therefore have some optional libraries that offer a +minimal interface because after all we can do a lot at the \LUA\ end. Optional +libraries register themselves in the global \type {optional} table. We're talking +of a couple of hundred lines of \CCODE\ for a simple binding. The functions in an +optional library table can be used (accessed) without loading the library and +then just do nothing useful. So, before using them you need to load the third +party library but we can safely assume that the \LUA\ wrapper code calls an +initializer when it needs some feature. That initializer, which by the way is +located at the \LUA\ end, loads the external library, and when that is successful +the needed helpers are bound by resolving function pointers. There is no +dependency when nothing is used: the main binary stays lean and mean because the +binding normally only adds a few \KB. Users can compile without dependencies and +when used performance is quite okay (no \FFI\ overhead). + +The \LUAMETATEX\ distribution only ships a few such bindings but these can serve +as example. What is shipped has a proper \LUA\ companion file and these are then +the standard one used in the \CONTEXT\ distribution. Think of \MYSQL\ and +\SQLITE\ (for databases), \ZINT\ (for barcodes), simple \CURL\ (for fetching +stuff), \GHOSTSCRIPT\ and \GRAPHICSMAGICK\ (for some conversions) bindings . When +compiled into \LUAMETATEX\ these will add some interfacing code to the main +binary but that gets compensated by the removal of the \FFI\ library. The \LUA\ +interfaces provide just enough to get us going. At some point we can consider +providing libraries as optional part of an installation because we can generate +them using the buildbot infrastructure managed by Mojca, but the core +distribution (source code) is kept clean. + +\stopsection + +\stopcomponent diff --git a/doc/context/sources/general/manuals/evenmore/evenmore.tex b/doc/context/sources/general/manuals/evenmore/evenmore.tex index 5bd751bfd..317c1548c 100644 --- a/doc/context/sources/general/manuals/evenmore/evenmore.tex +++ b/doc/context/sources/general/manuals/evenmore/evenmore.tex @@ -18,6 +18,7 @@ \component evenmore-normalization \component evenmore-expansion \component evenmore-macros + \component evenmore-libraries \stopbodymatter \stopdocument -- cgit v1.2.3