diff options
Diffstat (limited to 'doc/context/sources/general/manuals/onandon/onandon-53.tex')
-rw-r--r-- | doc/context/sources/general/manuals/onandon/onandon-53.tex | 288 |
1 files changed, 288 insertions, 0 deletions
diff --git a/doc/context/sources/general/manuals/onandon/onandon-53.tex b/doc/context/sources/general/manuals/onandon/onandon-53.tex new file mode 100644 index 000000000..0d5dc1b9c --- /dev/null +++ b/doc/context/sources/general/manuals/onandon/onandon-53.tex @@ -0,0 +1,288 @@ +% language=uk + +\startcomponent onandon-53 + +\environment onandon-environment + +\startchapter[title={From \LUA\ 5.2 to 5.3}] + +When we started with \LUATEX\ we used \LUA\ 5.1 and moved to 5.2 when that became +available. We didn't run into issues then because there were no fundamental +changes that could not be dealt with. However, when \LUA\ 5.3 was announced in +2015 we were not sure if we should make the move. The main reason was that we'd +chosen \LUA\ because of its clean design which meant that we had only one number +type: double. In 5.3 on the other hand, deep down a number can be either an +integer or a floating point quantity. + +Internally \TEX\ is mostly (up to) 32-bit integers and when we go from \LUA\ to +\TEX\ we round numbers. Nonetheless one can expect some benefits in using +integers. Performance|-|wise we didn't expect much, and memory consumption would +be the same too. So, the main question then was: can we get the same output and +not run into trouble due to possible differences in serializing numbers; after +all \TEX\ is about stability. The serialization aspect is for instance important +when we compare quantities and|/|or use numbers in hashes. + +Apart from this change in number model, which comes with a few extra helpers, +another extension in 5.3 was that bit|-|wise operations are now part of the +language. The lpeg library is still not part of stock \LUA. There is some minimal +\UTF8 support, but less than we provide in \LUATEX\ already. So, looking at these +changes, we were not in a hurry to update. Also, it made sense to wait till this +important number|-|related change was stable. + +But, a few years later, we still had it on our agenda to test, and after the +\CONTEXT\ 2017 meeting we decided to give it a try; here are some observations. A +quick test was just dropping in the new \LUA\ code and seeing if we could make a +\CONTEXT\ format. Indeed that was no big deal but a test run failed because at +some point a (for instance) \type {1} became a \type {1.0}. It turned out that +serializing has some side effects. And with some ad hoc prints for tracing (in +the \LUATEX\ source) I could figure out what went on. How numbers are seen can +(to some extent) be deduced from the \type {string.format} function, which is in +\LUA\ a combination of parsing, splitting and concatenation combined with piping +to the \CCODE\ \type {sprintf} function. \footnote {Actually, at some point I +decided to write my own formatter on top of \type {format} and I ended up with +splitting as well. It's only now that I realize why this is working out so well +(in terms of performance): simple format (single items) are passed more or less +directly to \type {sprintf} and as \LUA\ itself is fast, due to some caching, the +overhead is small compared to the built|-|in splitter method. And the \CONTEXT\ +formatter has many more options and is extensible.} + +\starttyping +local a = 2 * (1/2) print(string.format("%s", a),math.type(x)) +local b = 2 * (1/2) print(string.format("%d", b),math.type(x)) +local c = 2 print(string.format("%d", c),math.type(x)) +local d = -2 print(string.format("%d", d),math.type(x)) +local e = 2 * (1/2) print(string.format("%i", e),math.type(x)) +local f = 2.1 print(string.format("%.0f",f),math.type(x)) +local g = 2.0 print(string.format("%.0f",g),math.type(x)) +local h = 2.1 print(string.format("%G", h),math.type(x)) +local i = 2.0 print(string.format("%G", i),math.type(x)) +local j = 2 print(string.format("%.0f",j),math.type(x)) +local k = -2 print(string.format("%.0f",k),math.type(x)) +\stoptyping + +This gives the following results: + +\starttabulate[|cBT|c|T|c|cT|] +\BC a \NC 2 * (1/2)\NC s \NC 1.0 \NC float \NC \NR +\BC b \NC 2 * (1/2)\NC d \NC 1 \NC float \NC \NR +\BC c \NC 2 \NC d \NC 2 \NC integer \NC \NR +\BC d \NC -2 \NC d \NC 2 \NC integer \NC \NR +\BC e \NC 2 * (1/2)\NC i \NC 1 \NC float \NC \NR +\BC f \NC 2.1 \NC .0f \NC 2 \NC float \NC \NR +\BC g \NC 2.0 \NC .0f \NC 2 \NC float \NC \NR +\BC h \NC 2.1 \NC G \NC 2.1 \NC float \NC \NR +\BC i \NC 2.0 \NC G \NC 2 \NC float \NC \NR +\BC j \NC 2 \NC .0f \NC 2 \NC integer \NC \NR +\BC k \NC -2 \NC .0f \NC 2 \NC integer \NC \NR +\stoptabulate + +This demonstrates that we have to be careful when we need these numbers +represented as strings. In \CONTEXT\ the number of places where we had to check +for that was not that large; in fact, only some hashing related to font sizes had +to be done using explicit rounding. + +Another surprising side effect is the following. Instead of: + +\starttyping +local n = 2^6 +\stoptyping + +we now need to use: + +\starttyping +local n = 0x40 +\stoptyping + +or just: + +\starttyping +local n = 64 +\stoptyping + +because we don't want this to be serialized to \type {64.0} which is due to the +fact that a power results in a float. One can wonder if this makes sense when we +apply it to an integer. + +At any rate, once we could process a file, two documents were chosen for a +performance test. Some experiments with loops and casts had demonstrated that we +could expect a small performance hit and indeed, this was the case. Processing +the \LUATEX\ manual takes 10.7 seconds with 5.2 on my 5-year-old laptop and 11.6 +seconds with 5.3. If we consider that \CONTEXT\ spends 50\% of its time in \LUA, +then we see a 20\% performance penalty. Processing the \METAFUN\ manual (which +has lots of \METAPOST\ images) went from less than 20 seconds (\LUAJITTEX\ does +it in 16 seconds) up to more than 27 seconds. So there we lose more than 50\% on +the \LUA\ end. When we observed these kinds of differences, Luigi and I +immediately got into debugging mode, partly out of curiosity, but also because +consistent performance is important to~us. + +Because these numbers made no sense, we traced different sub-mechanisms and +eventually it became clear that the reason for the speed penalty was that the +core \typ {string.format} function was behaving quite badly in the \type {mingw} +cross-compiled binary, as seen by this test: + +\starttyping +local t = os.clock() +for i=1,1000*1000 do + -- local a = string.format("%.3f",1.23) + -- local b = string.format("%i",123) + local c = string.format("%s",123) +end +print(os.clock()-t) +\stoptyping + +\starttabulate[|c|c|c|c|c|] +\BC \BC lua 5.3 \BC lua 5.2 \BC texlua 5.3 \BC texlua 5.2 \BC \NR +\BC a \NC 0.43 \NC 0.54 \NC 3.71 (0.47) \NC 0.53 \NC \NR +\BC b \NC 0.18 \NC 0.24 \NC 3.78 (0.17) \NC 0.22 \NC \NR +\BC c \NC 0.26 \NC 0.68 \NC 3.67 (0.29) \NC 0.66 \NC \NR +\stoptabulate + +The 5.2 binaries perform the same but the 5.3 Lua binary greatly outperforms +\LUATEX, and so we had to figure out why. After all, all this integer +optimization could bring some gain! It took us a while to figure this out. The +numbers in parentheses are the results after fixing this. + +Because font internals are specified in integers one would expect a gain +in running: + +\starttyping +mtxrun --script font --reload force +\stoptyping + +and indeed that is the case. On my machine a scan results in 2561 registered +fonts from 4906 read files and with 5.2 that takes 9.1 seconds while 5.3 needs a +bit less: 8.6 seconds (with the bad format performance) and even less once that +was fixed. For a test: + +\starttyping +\setupbodyfont[modern] \tf \bf \it \bs +\setupbodyfont[pagella] \tf \bf \it \bs +\setupbodyfont[dejavu] \tf \bf \it \bs +\setupbodyfont[termes] \tf \bf \it \bs +\setupbodyfont[cambria] \tf \bf \it \bs +\starttext \stoptext +\stoptyping + +This code needs 30\% more runtime so the question is: how often do we call \type +{string.format} there? A first run (when we wipe the font cache) needs some +715,000 calls while successive runs need 115,000 calls so that slow down +definitely comes from the bad handling of \type {string.format}. When we drop in +a \LUA\ update or whatever other dependency we don't want this kind of impact. In +fact, when one uses external libraries that are or can be compiled under the +\TEX\ Live infrastructure and the impact would be such, it's bad advertising, +especially when one considers the occasional complaint about \LUATEX\ being +slower than other engines. + +The good news is that eventually Luigi was able to nail down this issue and we +got a binary that performed well. It looks like \LUA\ 5.3.4 (cross|)|compiles +badly with \GCC\ 5.3.0 and 6.3.0. + +So in the end caching the fonts takes: + +\starttabulate[||c|c|] +\BC \BC caching \BC running \NC \NR +\BC 5.2 stock \NC 8.3 \NC 1.2 \NC \NR +\BC 5.3 bugged \NC 12.6 \NC 2.1 \NC \NR +\BC 5.3 fixed \NC 6.3 \NC 1.0 \NC \NR +\stoptabulate + +So indeed it looks like 5.3 is able to speed up \LUATEX\ a bit, given that one +integrates it in the right way! Using a recent compiler is needed too, although +one can wonder when a bad case will show up again. One can also wonder why such a +slow down can mostly go unnoticed, because for sure \LUATEX\ is not the only +compiled program. + +The next examples are some edge cases that show you need to be aware +that +\startitemize[n,text,nostopper] + \startitem an integer has its limits, \stopitem + \startitem that hexadecimal numbers are integers and \stopitem + \startitem that \LUA\ and \LUAJIT\ can be different in details. \stopitem +\stopitemize + +\starttabulate[||T|T|] +\NC \NC \tx print(0xFFFFFFFFFFFFFFFF) \NC \tx print(0x7FFFFFFFFFFFFFFF) \NC \NR +\HL +\BC lua 52 \NC 1.844674407371e+019 \NC 9.2233720368548e+018 \NC \NR +\BC luajit \NC 1.844674407371e+19 \NC 9.2233720368548e+18 \NC \NR +\BC lua 53 \NC -1 \NC 9223372036854775807 \NC \NR +\stoptabulate + +So, to summarize the process. A quick test was relatively easy: move 5.3 into the +code base, adapt a little bit of internals (there were some \LUATEX\ interfacing +bits where explicit rounding was needed), run tests and eventually fix some +issues related to the Makefile (compatibility) and \CCODE\ obscurities (the slow +\type {sprintf}). Adapting \CONTEXT\ was also not much work, and the test suite +uncovered some nasty side effects. For instance, the valid 5.2 solution: + +\starttyping +local s = string.format("02X",u/1024) +local s = string.char (u/1024) +\stoptyping + +now has to become (both 5.2 and 5.3): + +\starttyping +local s = string.format("02X",math.floor(u/1024)) +local s = string.char (math.floor(u/1024)) +\stoptyping + +or (both 5.2 and (emulated or real) 5.3): + +\starttyping +local s = string.format("02X",bit32.rshift(u,10)) +local s = string.char (bit32.rshift(u,10)) +\stoptyping + +or (only 5.3): + +\starttyping +local s = string.format("02X",u >> 10)) +local s = string.char (u >> 10) +\stoptyping + +or (only 5.3): + +\starttyping +local s = string.format("02X",u//1024) +local s = string.char (u//1024) +\stoptyping + +A conditional section like: + +\starttyping +if LUAVERSION >= 5.3 then + local s = string.format("02X",u >> 10)) + local s = string.char (u >> 10) +else + local s = string.format("02X",bit32.rshift(u,10)) + local s = string.char (bit32.rshift(u,10)) +end +\stoptyping + +will fail because (of course) the 5.2 parser doesn't like that. In \CONTEXT\ we +have some experimental solutions for that but that is beyond this summary. + +In the process a few \UTF\ helpers were added to the string library so that we +have a common set for \LUAJIT\ and \LUA\ (the \type {utf8} library that was added +to 5.3 is not that important for \LUATEX). For now we keep the \type {bit32} +library on board. Of course we'll not mention all the details here. + +When we consider a gain in speed of 5-10\% with 5.3 that also means that the gain +of \LUAJITTEX\ compared to 5.2 becomes less. For instance in font processing both +engines now perform closer to the same. + +As I write this, we've just entered 2018 and after a few months of testing +\LUATEX\ with \LUA\ 5.3 we're confident that we can move the code to the +experimental branch. This means that we will use this version in the \CONTEXT\ +distribution and likely will ship this version as 1.10 in 2019, where it becomes +the default. The 2018 version of \TEX~Live will have 1.07 with \LUA\ 5.2 while +intermediate versions of the \LUA\ 5.3 binary will end up on the \CONTEXT\ +garden, probably with number 1.08 and 1.09 (who knows what else we will add or +change in the meantime). + +\stopchapter + +\stopcomponent + +% collectgarbage("count") -- two return values in 2 |