2021-04-14 22:57:00

23 files changed, 309 insertions, 130 deletions

diff --git a/doc/context/documents/general/manuals/luametatex.pdf b/doc/context/documents/general/manuals/luametatex.pdf
index 99b706c84..cc0e821fb 100644
--- a/doc/context/documents/general/manuals/luametatex.pdf
+++ b/doc/context/documents/general/manuals/luametatex.pdf
diff --git a/doc/context/sources/general/manuals/followingup/followingup-memory.tex b/doc/context/sources/general/manuals/followingup/followingup-memory.tex
new file mode 100644
index 000000000..63e3821ed
--- /dev/null
+++ b/doc/context/sources/general/manuals/followingup/followingup-memory.tex
@@ -0,0 +1,136 @@
+% language=us
+
+\startcomponent followingup-memory
+
+\environment followingup-style
+
+\startchapter[title={Memory}]
+
+\startsection[title={Introduction}]
+
+\stopsection
+
+\startsection[title={\LUA}]
+
+When you initialize \LUA\ a proper memory allocator has to be provided. The
+allocator gets an old size and new size passed. When both are zero the allocator
+can \type {free} the blob, when the new size exceeds the old size the blob has to
+be \type {realloc}'s, and otherwise an initial \type {malloc} happens. When used
+with \CONTEXT, \LUAMETATEX\ will do lots of calls to the allocator and often an
+initial allocation is followed by a reallocation, for instance because tables
+start out small but immediately grows a while after.
+
+It is for this reason that early 2021 I decided to look into alternative
+allocators. I can of course code one myself, but where a \LUATEX\ run is a one
+time event, often with growing memory usage due to all kind of accumulating
+resources, using the engine as stand alone interpreter needs a more sophisticated
+approach than just keeping a bunch of bucket pools alive: when the script engine
+runs for months or even years memory should be returned to the operating system
+occasionally. We don't want the same side effects that \HTML\ browsers have:
+during the day you need to restart them occasionally because they use up quite a
+bit of your computers memory (often for no real reason, so it probably has to do
+with keeping memory in store instead of returning it and|/|or it can be a side
+effect of a scattered pool \unknown\ who knows).
+
+Instead of reinventing that wheel I ended up with testing Daan Leijen's \type
+{mimalloc} implementation: a not bloated, not too low level, reasonable sized
+library. Some simple experiments learned that it does make a difference in
+performance. The experiment was done with the native \MICROSOFT\ compiler (msvc).
+One reason for that is that till that moment I preferred the cross compiled
+\MINGW\ versions (for cross compiling I use the \LINUX\ subsystem that comes with
+\MSWINDOWS). Although native binaries compile faster and are smaller, the cross
+compiled ones perform somewhat better (often some 5\%). Interesting is that
+making the format file is always much faster with a native binary, probably
+because the console output is supported better. When the alternative memory
+allocator is plugged into \LUA\ suddenly the native version outperforms the cross
+compiled one (also by some 5\%). The overall gain on a native binary for
+compiling the \LUAMETATEX\ manual is between~5 and~10\% which was reason enough
+to continue this experiment. As a first step the native compiled version will
+default to it, later other platforms might follow.
+
+\stopsection
+
+\startsection[title={\TEX}]
+
+Memory allocation in \TEX\ has always been done by the engine itself. At startup
+a couple of big chunks are allocated and from that smaller blobs are taken. The
+largest chunks are for nodes, tokens and the table of equivalents (including the
+hash where control sequences are mapped onto registers and macros (lists of
+tokens). Smaller chunks are used for nesting states, after group restoration
+stacks, in- and output levels, etc. In modern engines the sizes of the chunks can
+be configured, some only at format generation time. In \LUAMETATEX\ we are more
+dynamic and after an initial (minimal) chunk allocation, when needed more memory
+will be allocated on demand, in steps, until a configured size is reached. That
+size has an upper limit (which if needed can be enlarged at compilation time). A
+side effect is that we (need to) do some more checking.
+
+Node memory is special in the sense that nodes are basically offsets in a large
+array where each node has a number of slots after that offset. This is rather
+efficient in terms of performance and memory. New nodes (of any size) are taken
+from the node chunk and never returned. When freed they are appended to a list
+per size and that list serves as pool before new nodes get taken from the chunk.
+Variable size chunks are done differently, if only because we use them plenty in
+\CONTEXT\ and they can lead to (excessive and) fragmented memory usage otherwise.
+
+Tokens all have the same size so here there is only one list of free tokens.
+Because tokens and (most) nodes make it into linked lists those lists of free
+nodes and tokens are rather natural. And it's also fast. It all means that \TEX\
+itself does hardly any real memory allocation: only a few dozen large chunks. An
+exception is the string pool, where contrary to traditional \TEX\ engines, the
+\LUATEX\ (and \LUAMETATEX) engines allocate strings using \type {malloc}. Those
+strings (used for control sequences) are never freed. In other cases where
+strings are used, like in for instance \type {\csname} construction, temporary
+strings are used. The same is true for some file related operations. None of
+these are real demanding in terms of excessive allocation and freeing. Also, in
+places that matter \LUAMETATEX\ is already quite optimized so using a different
+allocator gives no gain here.
+
+Technically we could allocate nodes by using \type {malloc} but there are a few
+places in the engine that makes this hard. It can be done but then we need to
+make some conceptual changes (with regards to the way inserts are dealt with) and
+the question is if we gain much by breaking away from tradition. I guess there it
+will actually hurt performance if we change this. Another variant is where we
+allocate nodes of the same size from different pools but this doesn't bring us
+any gain either. A stringer argument is that changing the current (and historic)
+memory management of nodes will complicate the code.
+
+A bit of an exception is the flow of information between \LUA\ and \TEX. There we
+do quite some allocation but it depends on how much a macro package demands of
+that.
+
+\stopsection
+
+\startsection[title={\METAPOST}]
+
+When the \METAPOST\ library was written, Taco changed the memory allocation to be
+more dynamic. One reason for this is that the number models (scaled, double,
+decimal, binary) have their own demands. For some objects (like numbers) the
+implementation uses a pool so it sits between the way \TEX\ works and \LUA\ when
+the standard allocator is used. This means that although quite some allocation
+is demanded, often the pool can serve the requests. (We might use a few more
+pools in the future.)
+
+In \LUAMETATEX\ the memory related code has been reorganized a little so that
+(again as experiment) the \type {mimalloc} manager can be used. The performance
+gain is not as impressive as with \LUA, but we'll see how that evolves when more
+demand poses more stress.
+
+\stopsection
+
+\startsection[title={The verdict}]
+
+In \LUAMETATEX\ version 2.09.4 and later the native \MSWINDOWS\ binaries now use
+the alternative \type {mimalloc} allocator. The gain is most noticeable for \LUA\
+and a little for \TEX\ and \METAPOST. The test suite with 2550 files runs in 1200
+seconds which is quite an improvement over the \MINGW\ cross compiled binary that
+needs 1350 seconds. We do occasionally test a binary compiled with \CLANG\ but
+that one is much slower than both others (compilation also takes much more time)
+but that might improve over time. Because of these results, it is likely that
+I'll also check out the other platforms, once the \MSWINDOWS\ binaries have
+proven to be stable (those are the once I use anyway).
+
+\stopsection
+
+\stopchapter
+
+\stopcomponent
diff --git a/doc/context/sources/general/manuals/followingup/followingup.tex b/doc/context/sources/general/manuals/followingup/followingup.tex
index 7d7d17851..996673a36 100644
--- a/doc/context/sources/general/manuals/followingup/followingup.tex
+++ b/doc/context/sources/general/manuals/followingup/followingup.tex
@@ -29,6 +29,7 @@
         \component followingup-tex
         \component followingup-retrospect
         \component followingup-fonts
+        \component followingup-memory
     \stopbodymatter
 
 \stopdocument
diff --git a/doc/context/sources/general/manuals/luametatex/luametatex.tex b/doc/context/sources/general/manuals/luametatex/luametatex.tex
index a46e595ca..b7b0ab749 100644
--- a/doc/context/sources/general/manuals/luametatex/luametatex.tex
+++ b/doc/context/sources/general/manuals/luametatex/luametatex.tex
@@ -1,4 +1,4 @@
-% ------------------------   ------------------------   ------------------------
+% ------------------------   ------ ------------------   ------------------------
 % 2019-12-17  32bit  64bit   2020-01-10  32bit  64bit   2020-11-30  32bit  64bit
 % ------------------------   ------------------------   ------------------------
 % freebsd     2270k  2662k   freebsd     2186k  2558k   freebsd     2108k  2436k
diff --git a/scripts/context/lua/mtx-update.lua b/scripts/context/lua/mtx-update.lua
index b0ed9aac7..f3e22493b 100644
--- a/scripts/context/lua/mtx-update.lua
+++ b/scripts/context/lua/mtx-update.lua
@@ -266,7 +266,8 @@ update.platforms = {
     ["macosx"]         = "osx-64",
     ["osx"]            = "osx-64",
     ["osx-64"]         = "osx-64",
-    ["osx-arm64"]      = "osx-64",
+    ["osx-arm"]        = "osx-arm64",
+    ["osx-arm64"]      = "osx-arm64",
     --
  -- ["solaris-intel"]  = "solaris-intel",
     --
diff --git a/scripts/context/lua/mtxrun.lua b/scripts/context/lua/mtxrun.lua
index cc5b04b1e..19158704c 100644
--- a/scripts/context/lua/mtxrun.lua
+++ b/scripts/context/lua/mtxrun.lua
@@ -15774,7 +15774,7 @@ do -- create closure to overcome 200 locals limit
 
 package.loaded["util-zip"] = package.loaded["util-zip"] or true
 
--- original size: 19496, stripped down to: 10858
+-- original size: 19324, stripped down to: 10821
 
 if not modules then modules={} end modules ['util-zip']={
  version=1.001,
@@ -15800,19 +15800,12 @@ local getposition=files.getposition
 local band=bit32.band
 local rshift=bit32.rshift
 local lshift=bit32.lshift
-local decompress,expandsize,calculatecrc
- local zlibdecompress=zlib.decompress
- local zlibexpandsize=zlib.expandsize
- local zlibchecksum=zlib.crc32
- decompress=function(source)
-  return zlibdecompress(source,-15) 
- end
- expandsize=zlibexpandsize and function(source,targetsize)
-  return zlibexpandsize(source,targetsize,-15) 
- end or decompress
- calculatecrc=function(buffer,initial)
-  return zlibchecksum(initial or 0,buffer)
- end
+local zlibdecompress=zlib.decompress
+local zlibdecompresssize=zlib.decompresssize
+local zlibchecksum=zlib.crc32
+local decompress=function(source)   return zlibdecompress (source,-15)   end 
+local decompresssize=function(source,targetsize) return zlibdecompresssize(source,targetsize,-15) end 
+local calculatecrc=function(buffer,initial) return zlibchecksum   (initial or 0,buffer)   end
 local zipfiles={}
 utilities.zipfiles=zipfiles
 local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist  do
@@ -15937,8 +15930,8 @@ local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist  do
     setposition(handle,position)
     local result=readstring(handle,compressed)
     if data.method==8 then
-     if expandsize then
-      result=expandsize(result,data.uncompressed)
+     if decompresssize then
+      result=decompresssize(result,data.uncompressed)
      else
       result=decompress(result)
      end
@@ -25857,8 +25850,8 @@ end -- of closure
 
 -- used libraries    : l-bit32.lua l-lua.lua l-macro.lua l-sandbox.lua l-package.lua l-lpeg.lua l-function.lua l-string.lua l-table.lua l-io.lua l-number.lua l-set.lua l-os.lua l-file.lua l-gzip.lua l-md5.lua l-sha.lua l-url.lua l-dir.lua l-boolean.lua l-unicode.lua l-math.lua util-str.lua util-tab.lua util-fil.lua util-sac.lua util-sto.lua util-prs.lua util-fmt.lua util-soc-imp-reset.lua util-soc-imp-socket.lua util-soc-imp-copas.lua util-soc-imp-ltn12.lua util-soc-imp-mime.lua util-soc-imp-url.lua util-soc-imp-headers.lua util-soc-imp-tp.lua util-soc-imp-http.lua util-soc-imp-ftp.lua util-soc-imp-smtp.lua trac-set.lua trac-log.lua trac-inf.lua trac-pro.lua util-lua.lua util-deb.lua util-tpl.lua util-sbx.lua util-mrg.lua util-env.lua luat-env.lua util-zip.lua lxml-tab.lua lxml-lpt.lua lxml-mis.lua lxml-aux.lua lxml-xml.lua trac-xml.lua data-ini.lua data-exp.lua data-env.lua data-tmp.lua data-met.lua data-res.lua data-pre.lua data-inp.lua data-out.lua data-fil.lua data-con.lua data-use.lua data-zip.lua data-tre.lua data-sch.lua data-lua.lua data-aux.lua data-tmf.lua data-lst.lua libs-ini.lua luat-sta.lua luat-fmt.lua
 -- skipped libraries : -
--- original bytes    : 1026599
--- stripped bytes    : 405840
+-- original bytes    : 1026427
+-- stripped bytes    : 405705
 
 -- end library merge
 
diff --git a/scripts/context/stubs/mswin/mtxrun.lua b/scripts/context/stubs/mswin/mtxrun.lua
index cc5b04b1e..19158704c 100644
--- a/scripts/context/stubs/mswin/mtxrun.lua
+++ b/scripts/context/stubs/mswin/mtxrun.lua
@@ -15774,7 +15774,7 @@ do -- create closure to overcome 200 locals limit
 
 package.loaded["util-zip"] = package.loaded["util-zip"] or true
 
--- original size: 19496, stripped down to: 10858
+-- original size: 19324, stripped down to: 10821
 
 if not modules then modules={} end modules ['util-zip']={
  version=1.001,
@@ -15800,19 +15800,12 @@ local getposition=files.getposition
 local band=bit32.band
 local rshift=bit32.rshift
 local lshift=bit32.lshift
-local decompress,expandsize,calculatecrc
- local zlibdecompress=zlib.decompress
- local zlibexpandsize=zlib.expandsize
- local zlibchecksum=zlib.crc32
- decompress=function(source)
-  return zlibdecompress(source,-15) 
- end
- expandsize=zlibexpandsize and function(source,targetsize)
-  return zlibexpandsize(source,targetsize,-15) 
- end or decompress
- calculatecrc=function(buffer,initial)
-  return zlibchecksum(initial or 0,buffer)
- end
+local zlibdecompress=zlib.decompress
+local zlibdecompresssize=zlib.decompresssize
+local zlibchecksum=zlib.crc32
+local decompress=function(source)   return zlibdecompress (source,-15)   end 
+local decompresssize=function(source,targetsize) return zlibdecompresssize(source,targetsize,-15) end 
+local calculatecrc=function(buffer,initial) return zlibchecksum   (initial or 0,buffer)   end
 local zipfiles={}
 utilities.zipfiles=zipfiles
 local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist  do
@@ -15937,8 +15930,8 @@ local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist  do
     setposition(handle,position)
     local result=readstring(handle,compressed)
     if data.method==8 then
-     if expandsize then
-      result=expandsize(result,data.uncompressed)
+     if decompresssize then
+      result=decompresssize(result,data.uncompressed)
      else
       result=decompress(result)
      end
@@ -25857,8 +25850,8 @@ end -- of closure
 
 -- used libraries    : l-bit32.lua l-lua.lua l-macro.lua l-sandbox.lua l-package.lua l-lpeg.lua l-function.lua l-string.lua l-table.lua l-io.lua l-number.lua l-set.lua l-os.lua l-file.lua l-gzip.lua l-md5.lua l-sha.lua l-url.lua l-dir.lua l-boolean.lua l-unicode.lua l-math.lua util-str.lua util-tab.lua util-fil.lua util-sac.lua util-sto.lua util-prs.lua util-fmt.lua util-soc-imp-reset.lua util-soc-imp-socket.lua util-soc-imp-copas.lua util-soc-imp-ltn12.lua util-soc-imp-mime.lua util-soc-imp-url.lua util-soc-imp-headers.lua util-soc-imp-tp.lua util-soc-imp-http.lua util-soc-imp-ftp.lua util-soc-imp-smtp.lua trac-set.lua trac-log.lua trac-inf.lua trac-pro.lua util-lua.lua util-deb.lua util-tpl.lua util-sbx.lua util-mrg.lua util-env.lua luat-env.lua util-zip.lua lxml-tab.lua lxml-lpt.lua lxml-mis.lua lxml-aux.lua lxml-xml.lua trac-xml.lua data-ini.lua data-exp.lua data-env.lua data-tmp.lua data-met.lua data-res.lua data-pre.lua data-inp.lua data-out.lua data-fil.lua data-con.lua data-use.lua data-zip.lua data-tre.lua data-sch.lua data-lua.lua data-aux.lua data-tmf.lua data-lst.lua libs-ini.lua luat-sta.lua luat-fmt.lua
 -- skipped libraries : -
--- original bytes    : 1026599
--- stripped bytes    : 405840
+-- original bytes    : 1026427
+-- stripped bytes    : 405705
 
 -- end library merge
 
diff --git a/scripts/context/stubs/unix/mtxrun b/scripts/context/stubs/unix/mtxrun
index cc5b04b1e..19158704c 100644
--- a/scripts/context/stubs/unix/mtxrun
+++ b/scripts/context/stubs/unix/mtxrun
@@ -15774,7 +15774,7 @@ do -- create closure to overcome 200 locals limit
 
 package.loaded["util-zip"] = package.loaded["util-zip"] or true
 
--- original size: 19496, stripped down to: 10858
+-- original size: 19324, stripped down to: 10821
 
 if not modules then modules={} end modules ['util-zip']={
  version=1.001,
@@ -15800,19 +15800,12 @@ local getposition=files.getposition
 local band=bit32.band
 local rshift=bit32.rshift
 local lshift=bit32.lshift
-local decompress,expandsize,calculatecrc
- local zlibdecompress=zlib.decompress
- local zlibexpandsize=zlib.expandsize
- local zlibchecksum=zlib.crc32
- decompress=function(source)
-  return zlibdecompress(source,-15) 
- end
- expandsize=zlibexpandsize and function(source,targetsize)
-  return zlibexpandsize(source,targetsize,-15) 
- end or decompress
- calculatecrc=function(buffer,initial)
-  return zlibchecksum(initial or 0,buffer)
- end
+local zlibdecompress=zlib.decompress
+local zlibdecompresssize=zlib.decompresssize
+local zlibchecksum=zlib.crc32
+local decompress=function(source)   return zlibdecompress (source,-15)   end 
+local decompresssize=function(source,targetsize) return zlibdecompresssize(source,targetsize,-15) end 
+local calculatecrc=function(buffer,initial) return zlibchecksum   (initial or 0,buffer)   end
 local zipfiles={}
 utilities.zipfiles=zipfiles
 local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist  do
@@ -15937,8 +15930,8 @@ local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist  do
     setposition(handle,position)
     local result=readstring(handle,compressed)
     if data.method==8 then
-     if expandsize then
-      result=expandsize(result,data.uncompressed)
+     if decompresssize then
+      result=decompresssize(result,data.uncompressed)
      else
       result=decompress(result)
      end
@@ -25857,8 +25850,8 @@ end -- of closure
 
 -- used libraries    : l-bit32.lua l-lua.lua l-macro.lua l-sandbox.lua l-package.lua l-lpeg.lua l-function.lua l-string.lua l-table.lua l-io.lua l-number.lua l-set.lua l-os.lua l-file.lua l-gzip.lua l-md5.lua l-sha.lua l-url.lua l-dir.lua l-boolean.lua l-unicode.lua l-math.lua util-str.lua util-tab.lua util-fil.lua util-sac.lua util-sto.lua util-prs.lua util-fmt.lua util-soc-imp-reset.lua util-soc-imp-socket.lua util-soc-imp-copas.lua util-soc-imp-ltn12.lua util-soc-imp-mime.lua util-soc-imp-url.lua util-soc-imp-headers.lua util-soc-imp-tp.lua util-soc-imp-http.lua util-soc-imp-ftp.lua util-soc-imp-smtp.lua trac-set.lua trac-log.lua trac-inf.lua trac-pro.lua util-lua.lua util-deb.lua util-tpl.lua util-sbx.lua util-mrg.lua util-env.lua luat-env.lua util-zip.lua lxml-tab.lua lxml-lpt.lua lxml-mis.lua lxml-aux.lua lxml-xml.lua trac-xml.lua data-ini.lua data-exp.lua data-env.lua data-tmp.lua data-met.lua data-res.lua data-pre.lua data-inp.lua data-out.lua data-fil.lua data-con.lua data-use.lua data-zip.lua data-tre.lua data-sch.lua data-lua.lua data-aux.lua data-tmf.lua data-lst.lua libs-ini.lua luat-sta.lua luat-fmt.lua
 -- skipped libraries : -
--- original bytes    : 1026599
--- stripped bytes    : 405840
+-- original bytes    : 1026427
+-- stripped bytes    : 405705
 
 -- end library merge
 
diff --git a/scripts/context/stubs/win64/mtxrun.lua b/scripts/context/stubs/win64/mtxrun.lua
index cc5b04b1e..19158704c 100644
--- a/scripts/context/stubs/win64/mtxrun.lua
+++ b/scripts/context/stubs/win64/mtxrun.lua
@@ -15774,7 +15774,7 @@ do -- create closure to overcome 200 locals limit
 
 package.loaded["util-zip"] = package.loaded["util-zip"] or true
 
--- original size: 19496, stripped down to: 10858
+-- original size: 19324, stripped down to: 10821
 
 if not modules then modules={} end modules ['util-zip']={
  version=1.001,
@@ -15800,19 +15800,12 @@ local getposition=files.getposition
 local band=bit32.band
 local rshift=bit32.rshift
 local lshift=bit32.lshift
-local decompress,expandsize,calculatecrc
- local zlibdecompress=zlib.decompress
- local zlibexpandsize=zlib.expandsize
- local zlibchecksum=zlib.crc32
- decompress=function(source)
-  return zlibdecompress(source,-15) 
- end
- expandsize=zlibexpandsize and function(source,targetsize)
-  return zlibexpandsize(source,targetsize,-15) 
- end or decompress
- calculatecrc=function(buffer,initial)
-  return zlibchecksum(initial or 0,buffer)
- end
+local zlibdecompress=zlib.decompress
+local zlibdecompresssize=zlib.decompresssize
+local zlibchecksum=zlib.crc32
+local decompress=function(source)   return zlibdecompress (source,-15)   end 
+local decompresssize=function(source,targetsize) return zlibdecompresssize(source,targetsize,-15) end 
+local calculatecrc=function(buffer,initial) return zlibchecksum   (initial or 0,buffer)   end
 local zipfiles={}
 utilities.zipfiles=zipfiles
 local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist  do
@@ -15937,8 +15930,8 @@ local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist  do
     setposition(handle,position)
     local result=readstring(handle,compressed)
     if data.method==8 then
-     if expandsize then
-      result=expandsize(result,data.uncompressed)
+     if decompresssize then
+      result=decompresssize(result,data.uncompressed)
      else
       result=decompress(result)
      end
@@ -25857,8 +25850,8 @@ end -- of closure
 
 -- used libraries    : l-bit32.lua l-lua.lua l-macro.lua l-sandbox.lua l-package.lua l-lpeg.lua l-function.lua l-string.lua l-table.lua l-io.lua l-number.lua l-set.lua l-os.lua l-file.lua l-gzip.lua l-md5.lua l-sha.lua l-url.lua l-dir.lua l-boolean.lua l-unicode.lua l-math.lua util-str.lua util-tab.lua util-fil.lua util-sac.lua util-sto.lua util-prs.lua util-fmt.lua util-soc-imp-reset.lua util-soc-imp-socket.lua util-soc-imp-copas.lua util-soc-imp-ltn12.lua util-soc-imp-mime.lua util-soc-imp-url.lua util-soc-imp-headers.lua util-soc-imp-tp.lua util-soc-imp-http.lua util-soc-imp-ftp.lua util-soc-imp-smtp.lua trac-set.lua trac-log.lua trac-inf.lua trac-pro.lua util-lua.lua util-deb.lua util-tpl.lua util-sbx.lua util-mrg.lua util-env.lua luat-env.lua util-zip.lua lxml-tab.lua lxml-lpt.lua lxml-mis.lua lxml-aux.lua lxml-xml.lua trac-xml.lua data-ini.lua data-exp.lua data-env.lua data-tmp.lua data-met.lua data-res.lua data-pre.lua data-inp.lua data-out.lua data-fil.lua data-con.lua data-use.lua data-zip.lua data-tre.lua data-sch.lua data-lua.lua data-aux.lua data-tmf.lua data-lst.lua libs-ini.lua luat-sta.lua luat-fmt.lua
 -- skipped libraries : -
--- original bytes    : 1026599
--- stripped bytes    : 405840
+-- original bytes    : 1026427
+-- stripped bytes    : 405705
 
 -- end library merge
 
diff --git a/tex/context/base/mkii/cont-new.mkii b/tex/context/base/mkii/cont-new.mkii
index a0151c54d..b468ce3d0 100644
--- a/tex/context/base/mkii/cont-new.mkii
+++ b/tex/context/base/mkii/cont-new.mkii
@@ -11,7 +11,7 @@
 %C therefore copyrighted by \PRAGMA. See mreadme.pdf for
 %C details.
 
-\newcontextversion{2021.04.09 19:54}
+\newcontextversion{2021.04.14 22:55}
 
 %D This file is loaded at runtime, thereby providing an
 %D excellent place for hacks, patches, extensions and new
diff --git a/tex/context/base/mkii/context.mkii b/tex/context/base/mkii/context.mkii
index 5313df6e6..7ad92f388 100644
--- a/tex/context/base/mkii/context.mkii
+++ b/tex/context/base/mkii/context.mkii
@@ -20,7 +20,7 @@
 %D your styles an modules.
 
 \edef\contextformat {\jobname}
-\edef\contextversion{2021.04.09 19:54}
+\edef\contextversion{2021.04.14 22:55}
 
 %D For those who want to use this:
 
diff --git a/tex/context/base/mkiv/cont-new.mkiv b/tex/context/base/mkiv/cont-new.mkiv
index e81cbb7d9..7bb390194 100644
--- a/tex/context/base/mkiv/cont-new.mkiv
+++ b/tex/context/base/mkiv/cont-new.mkiv
@@ -13,7 +13,7 @@
 
 % \normalend % uncomment this to get the real base runtime
 
-\newcontextversion{2021.04.09 19:54}
+\newcontextversion{2021.04.14 22:55}
 
 %D This file is loaded at runtime, thereby providing an excellent place for hacks,
 %D patches, extensions and new features. There can be local overloads in cont-loc
diff --git a/tex/context/base/mkiv/context.mkiv b/tex/context/base/mkiv/context.mkiv
index 5062c3a10..6fa606a95 100644
--- a/tex/context/base/mkiv/context.mkiv
+++ b/tex/context/base/mkiv/context.mkiv
@@ -45,7 +45,7 @@
 %D {YYYY.MM.DD HH:MM} format.
 
 \edef\contextformat {\jobname}
-\edef\contextversion{2021.04.09 19:54}
+\edef\contextversion{2021.04.14 22:55}
 
 %D Kind of special:
 
diff --git a/tex/context/base/mkiv/status-files.pdf b/tex/context/base/mkiv/status-files.pdf
index cffa6fdb6..69556e09e 100644
--- a/tex/context/base/mkiv/status-files.pdf
+++ b/tex/context/base/mkiv/status-files.pdf
diff --git a/tex/context/base/mkiv/status-lua.pdf b/tex/context/base/mkiv/status-lua.pdf
index fea56a359..615e7d326 100644
--- a/tex/context/base/mkiv/status-lua.pdf
+++ b/tex/context/base/mkiv/status-lua.pdf
diff --git a/tex/context/base/mkiv/util-zip.lua b/tex/context/base/mkiv/util-zip.lua
index f87e391a1..e97f3a065 100644
--- a/tex/context/base/mkiv/util-zip.lua
+++ b/tex/context/base/mkiv/util-zip.lua
@@ -34,32 +34,13 @@ local band          = bit32.band
 local rshift        = bit32.rshift
 local lshift        = bit32.lshift
 
-local decompress, expandsize, calculatecrc
+local zlibdecompress     = zlib.decompress
+local zlibdecompresssize = zlib.decompresssize
+local zlibchecksum       = zlib.crc32
 
--- if flate then
---
---     decompress   = flate.flate_decompress
---     calculatecrc = flate.update_crc32
---
--- else
-
-    local zlibdecompress = zlib.decompress
-    local zlibexpandsize = zlib.expandsize
-    local zlibchecksum   = zlib.crc32
-
-    decompress = function(source)
-        return zlibdecompress(source,-15) -- auto
-    end
-
-    expandsize = zlibexpandsize and function(source,targetsize)
-        return zlibexpandsize(source,targetsize,-15) -- auto
-    end or decompress
-
-    calculatecrc = function(buffer,initial)
-        return zlibchecksum(initial or 0,buffer)
-    end
-
--- end
+local decompress     = function(source)            return zlibdecompress    (source,-15)            end -- auto
+local decompresssize = function(source,targetsize) return zlibdecompresssize(source,targetsize,-15) end -- auto
+local calculatecrc   = function(buffer,initial)    return zlibchecksum      (initial or 0,buffer)   end
 
 local zipfiles      = { }
 utilities.zipfiles  = zipfiles
@@ -201,8 +182,8 @@ local openzipfile, closezipfile, unzipfile, foundzipfile, getziphash, getziplist
                 setposition(handle,position)
                 local result = readstring(handle,compressed)
                 if data.method == 8 then
-                    if expandsize then
-                        result = expandsize(result,data.uncompressed)
+                    if decompresssize then
+                        result = decompresssize(result,data.uncompressed)
                     else
                         result = decompress(result)
                     end
diff --git a/tex/context/base/mkxl/cldf-lmt.lmt b/tex/context/base/mkxl/cldf-lmt.lmt
index f3121eadc..1c1b6a353 100644
--- a/tex/context/base/mkxl/cldf-lmt.lmt
+++ b/tex/context/base/mkxl/cldf-lmt.lmt
@@ -6,7 +6,7 @@ if not modules then modules = { } end modules ['cldf-lmt'] = {
     license   = "see context related readme files"
 }
 
-local next, load = next, load
+local next, load, tonumber = next, load, tonumber
 local gmatch, gsub, byte = string.gmatch, string.gsub, string.byte
 
 local setmetatableindex    = table.setmetatableindex
@@ -627,7 +627,7 @@ context.luatables = {
 local tables = { }
 local stack  = setmetatableindex("table")
 
-interfaces.implement {
+implement {
     name      = "droptablegroup",
     public    = true,
     actions   = function()
@@ -699,7 +699,7 @@ local function newtable(array)
     if not tables[name] then
         local t = { }
         tables[name] = t
-        interfaces.implement {
+        implement {
             name    = name,
             public  = true,
             usage   = "value",
@@ -886,7 +886,7 @@ implement {
     end
 }
 
-interfaces.implement {
+implement {
     name      = "bitwise",
     public    = true,
     usage     = "value",
@@ -933,7 +933,7 @@ interfaces.implement {
 
 local escape = function(s) return "\\" .. byte(s) end
 
-interfaces.implement {
+implement {
     name    = "ctxluamatch",
     public  = true,
     usage   = "value",
@@ -947,3 +947,65 @@ interfaces.implement {
         return none_code
     end,
 }
+
+-- yes or no ...
+
+do
+
+    local codes         = { }
+    tex.codes           = codes
+
+    local global_code   = tex.flagcodes.global
+
+    local savelua       = token.save_lua
+    local isdefined     = token.is_defined
+
+    local newsparse     = sparse.new
+    local setsparse     = sparse.set
+    local wipesparse    = sparse.wipe
+    local restoresparse = sparse.restore
+
+    -- local function isglobal(n) -- maybe a general helper
+    --     return n and (tonumber(n) & global_code)
+    -- end
+
+    local registerfunction = context.functions.register
+
+    implement {
+        name      = "codedef",
+        public    = true,
+        protected = true,
+        actions   = function(what)
+            local name = scancsname(true)
+         -- if isdefined(name) then
+         --     wipesparse(codes[name]) -- better make a wipe helper if ever needed
+         -- else
+                local code    = newsparse()
+                local restore = registerfunction(function() restoresparse(code) end)
+                implement {
+                    name      = name,
+                    public    = true,
+                    protected = true,
+                    usage     = "value",
+                    actions   = function(what)
+                        local n = scaninteger()
+                        if what == "value" then
+                            return integer_code, code[n]
+                        else
+                            local v = scaninteger(true)
+                         -- if isglobal(what) then
+                            if what and (tonumber(what) & global_code) then
+                                setsparse(code,"global",n,v)
+                            else
+                                savelua(restore,true) -- only once
+                                setsparse(code,n,v)
+                            end
+                        end
+                    end,
+                }
+                codes[name] = code
+         -- end
+        end,
+    }
+
+end
diff --git a/tex/context/base/mkxl/cont-new.mkxl b/tex/context/base/mkxl/cont-new.mkxl
index 87175b22d..f99a9ce23 100644
--- a/tex/context/base/mkxl/cont-new.mkxl
+++ b/tex/context/base/mkxl/cont-new.mkxl
@@ -13,7 +13,7 @@
 
 % \normalend % uncomment this to get the real base runtime
 
-\newcontextversion{2021.04.09 19:54}
+\newcontextversion{2021.04.14 22:55}
 
 %D This file is loaded at runtime, thereby providing an excellent place for hacks,
 %D patches, extensions and new features. There can be local overloads in cont-loc
diff --git a/tex/context/base/mkxl/context.mkxl b/tex/context/base/mkxl/context.mkxl
index 84bdd0277..103481265 100644
--- a/tex/context/base/mkxl/context.mkxl
+++ b/tex/context/base/mkxl/context.mkxl
@@ -29,7 +29,7 @@
 %D {YYYY.MM.DD HH:MM} format.
 
 \immutable\edef\contextformat {\jobname}
-\immutable\edef\contextversion{2021.04.09 19:54}
+\immutable\edef\contextversion{2021.04.14 22:55}
 
 %overloadmode 1 % check frozen / warning
 %overloadmode 2 % check frozen / error
diff --git a/tex/context/base/mkxl/lpdf-lmt.lmt b/tex/context/base/mkxl/lpdf-lmt.lmt
index 8d8220230..800f56ca3 100644
--- a/tex/context/base/mkxl/lpdf-lmt.lmt
+++ b/tex/context/base/mkxl/lpdf-lmt.lmt
@@ -42,9 +42,9 @@ local loaddata = io.loaddata
 
 local bpfactor             = number.dimenfactors.bp
 
--- local md5HEX               = md5.HEX
 local osuuid               = os.uuid
-local zlibcompress         = (xzip or zlib).compress
+local zlibcompress         = xzip.compress
+local zlibcompresssize     = xzip.compresssize
 
 local nuts                 = nodes.nuts
 local tonut                = nodes.tonut
@@ -101,6 +101,19 @@ local report_objects = logs.reporter("backend","objects")
 local trace_objects  = false  trackers.register("backend.objects", function(v) trace_objects = v end)
 local trace_details  = false  trackers.register("backend.details", function(v) trace_details = v end)
 
+-- experiment:
+
+local function compressdata(data,size)
+    local guess = ((size // 4096) + 1) * 2048
+    local comp  = zlibcompresssize(data,3,guess)
+ -- if comp then
+ --     report()
+ --     report("size %i, guess %i, result %i => %s / %s",size,guess,#comp,guess>=#comp and "hit" or "miss")
+ --     report()
+ -- end
+    return comp
+end
+
 -- we collect them:
 
 local flushers = { }
@@ -2021,25 +2034,28 @@ local addtocache, flushcache, cache do
             list = concat(list," ")
             data[0] = list
             data = concat(data,"\010",0,d)
+            local size = #data
             local strobj = pdfdictionary {
                 Type  = p_ObjStm,
                 N     = d,
                 First = #list + 1,
             }
             objects[cache] = offset
-            local b = nil
-            local e = s_stream_e
+            local fb
             if compress then
-                local comp = zlibcompress(data,3)
-                if comp and #comp < #data then
+--                 local comp = zlibcompress(data,3)
+                local comp = compressdata(data,size)
+                if comp and #comp < size then
                     data = comp
-                    b = f_stream_b_d_c(cache,strobj(),#data)
+                    fb = f_stream_b_d_c
                 else
-                    b = f_stream_b_d_u(cache,strobj(),#data)
+                    fb = f_stream_b_d_u
                 end
             else
-                b = f_stream_b_d_u(cache,strobj(),#data)
+                fb = f_stream_b_d_u
             end
+            local b = fb(cache,strobj(),size)
+            local e = s_stream_e
             flush(f,b)
             flush(f,data)
             flush(f,e)
@@ -2274,7 +2290,8 @@ local function flushstreamobj(data,n,dict,comp,nolength)
         if nolength then
             b = f_stream_b_d_r(n,dict)
         elseif comp ~= false and compress and size > threshold then
-            local compdata = zlibcompress(data,3)
+--             local compdata = zlibcompress(data,3)
+            local compdata = compressdata(data,size)
             if compdata then
                 local compsize = #compdata
                 if compsize > size - threshold then
@@ -2298,7 +2315,8 @@ local function flushstreamobj(data,n,dict,comp,nolength)
         if nolength then
             data = f_stream_d_r(n,dict,data)
         elseif comp ~= false and compress and size > threshold then
-            local compdata = zlibcompress(data,3)
+--             local compdata = zlibcompress(data,3)
+            local compdata = compressdata(data,size)
             if compdata then
                 local compsize = #compdata
                 if compsize > size - threshold then
@@ -2635,6 +2653,7 @@ local openfile, closefile  do
                 end
                 objects[0] = rep("\0",1+nofbytes+1)
                 local data = concat(objects,"",0,nofobjects)
+                local size = #data
                 local xref = pdfdictionary {
                     Type  = pdfconstant("XRef"),
                     Size  = nofobjects + 1,
@@ -2643,17 +2662,20 @@ local openfile, closefile  do
                     Info  = info,
                     ID    = trailerid and pdfarray { pdfliteral(trailerid,true), pdfliteral(trailerid,true) } or nil,
                 }
+                local fb
                 if compress then
-                    local comp = zlibcompress(data,3)
+--                     local comp = zlibcompress(data,3)
+                    local comp = compressdata(data,size)
                     if comp then
                         data = comp
-                        flush(f,f_stream_b_d_c(nofobjects,xref(),#data))
+                        fb = f_stream_b_d_c
                     else
-                        flush(f,f_stream_b_d_u(nofobjects,xref(),#data))
+                        fb = f_stream_b_d_u
                     end
                 else
-                    flush(f,f_stream_b_d_u(nofobjects,xref(),#data))
+                    fb = f_stream_b_d_u
                 end
+                flush(f,fb(nofobjects,xref(),size))
                 flush(f,data)
                 flush(f,s_stream_e)
                 flush(f,f_startxref(xrefoffset))
diff --git a/tex/context/base/mkxl/mult-ini.mkxl b/tex/context/base/mkxl/mult-ini.mkxl
index 17f63ef93..b357c3e12 100644
--- a/tex/context/base/mkxl/mult-ini.mkxl
+++ b/tex/context/base/mkxl/mult-ini.mkxl
@@ -19,8 +19,10 @@
 
 \unprotect
 
-\registerctxluafile{mult-ini}{autosuffix}
-\registerctxluafile{mult-fmt}{initexonly}
+\pushoverloadmode
+    \registerctxluafile{mult-ini}{autosuffix}
+    \registerctxluafile{mult-fmt}{initexonly}
+\popoverloadmode
 
 %D \macros
 %D   [constanten,variabelen,commands]
diff --git a/tex/context/fonts/mkiv/pagella-math.lfg b/tex/context/fonts/mkiv/pagella-math.lfg
index d4ae14162..0e8ac1f70 100644
--- a/tex/context/fonts/mkiv/pagella-math.lfg
+++ b/tex/context/fonts/mkiv/pagella-math.lfg
@@ -1,5 +1,6 @@
 local kern_V   = { bottomright = { { kern = -200 } } }
 local kern_W   = { bottomright = { { kern = -100 } } }
+-- local kern_f   = { bottomright = { { kern = -100 } } }
 local offset_f = { xoffset = "llx" }
 
 -- Beware of updates in ssty slots!
@@ -27,6 +28,7 @@ return {
          -- [0x1D44A]     = kern_W, -- 𝑊
          -- ["1:0x1D44A"] = kern_W, -- needed for compact
          -- ["2:0x1D44A"] = kern_W, -- needed for compact
+--             ["*:0x1D453"] = kern_f,
             ["*:0x1D449"] = kern_V, -- 𝑉
             ["*:0x1D44A"] = kern_W, -- 𝑊
         },
diff --git a/tex/generic/context/luatex/luatex-fonts-merged.lua b/tex/generic/context/luatex/luatex-fonts-merged.lua
index 868d224f3..1b594098e 100644
--- a/tex/generic/context/luatex/luatex-fonts-merged.lua
+++ b/tex/generic/context/luatex/luatex-fonts-merged.lua
@@ -1,6 +1,6 @@
 -- merged file : c:/data/develop/context/sources/luatex-fonts-merged.lua
 -- parent file : c:/data/develop/context/sources/luatex-fonts.lua
--- merge date  : 2021-04-09 19:54
+-- merge date  : 2021-04-14 22:55
 
 do -- begin closure to overcome local limits and interference