diff options
23 files changed, 309 insertions, 130 deletions
diff --git a/doc/context/documents/general/manuals/luametatex.pdf b/doc/context/documents/general/manuals/luametatex.pdf Binary files differindex 99b706c84..cc0e821fb 100644 --- a/doc/context/documents/general/manuals/luametatex.pdf +++ b/doc/context/documents/general/manuals/luametatex.pdf diff --git a/doc/context/sources/general/manuals/followingup/followingup-memory.tex b/doc/context/sources/general/manuals/followingup/followingup-memory.tex new file mode 100644 index 000000000..63e3821ed --- /dev/null +++ b/doc/context/sources/general/manuals/followingup/followingup-memory.tex @@ -0,0 +1,136 @@ +% language=us + +\startcomponent followingup-memory + +\environment followingup-style + +\startchapter[title={Memory}] + +\startsection[title={Introduction}] + +\stopsection + +\startsection[title={\LUA}] + +When you initialize \LUA\ a proper memory allocator has to be provided. The +allocator gets an old size and new size passed. When both are zero the allocator +can \type {free} the blob, when the new size exceeds the old size the blob has to +be \type {realloc}'s, and otherwise an initial \type {malloc} happens. When used +with \CONTEXT, \LUAMETATEX\ will do lots of calls to the allocator and often an +initial allocation is followed by a reallocation, for instance because tables +start out small but immediately grows a while after. + +It is for this reason that early 2021 I decided to look into alternative +allocators. I can of course code one myself, but where a \LUATEX\ run is a one +time event, often with growing memory usage due to all kind of accumulating +resources, using the engine as stand alone interpreter needs a more sophisticated +approach than just keeping a bunch of bucket pools alive: when the script engine +runs for months or even years memory should be returned to the operating system +occasionally. We don't want the same side effects that \HTML\ browsers have: +during the day you need to restart them occasionally because they use up quite a +bit of your computers memory (often for no real reason, so it probably has to do +with keeping memory in store instead of returning it and|/|or it can be a side +effect of a scattered pool \unknown\ who knows). + +Instead of reinventing that wheel I ended up with testing Daan Leijen's \type +{mimalloc} implementation: a not bloated, not too low level, reasonable sized +library. Some simple experiments learned that it does make a difference in +performance. The experiment was done with the native \MICROSOFT\ compiler (msvc). +One reason for that is that till that moment I preferred the cross compiled +\MINGW\ versions (for cross compiling I use the \LINUX\ subsystem that comes with +\MSWINDOWS). Although native binaries compile faster and are smaller, the cross +compiled ones perform somewhat better (often some 5\%). Interesting is that +making the format file is always much faster with a native binary, probably +because the console output is supported better. When the alternative memory +allocator is plugged into \LUA\ suddenly the native version outperforms the cross +compiled one (also by some 5\%). The overall gain on a native binary for +compiling the \LUAMETATEX\ manual is between~5 and~10\% which was reason enough +to continue this experiment. As a first step the native compiled version will +default to it, later other platforms might follow. + +\stopsection + +\startsection[title={\TEX}] + +Memory allocation in \TEX\ has always been done by the engine itself. At startup +a couple of big chunks are allocated and from that smaller blobs are taken. The +largest chunks are for nodes, tokens and the table of equivalents (including the +hash where control sequences are mapped onto registers and macros (lists of +tokens). Smaller chunks are used for nesting states, after group restoration +stacks, in- and output levels, etc. In modern engines the sizes of the chunks can +be configured, some only at format generation time. In \LUAMETATEX\ we are more +dynamic and after an initial (minimal) chunk allocation, when needed more memory +will be allocated on demand, in steps, until a configured size is reached. That +size has an upper limit (which if needed can be enlarged at compilation time). A +side effect is that we (need to) do some more checking. + +Node memory is special in the sense that nodes are basically offsets in a large +array where each node has a number of slots after that offset. This is rather +efficient in terms of performance and memory. New nodes (of any size) are taken +from the node chunk and never returned. When freed they are appended to a list +per size and that list serves as pool before new nodes get taken from the chunk. +Variable size chunks are done differently, if only because we use them plenty in +\CONTEXT\ and they can lead to (excessive and) fragmented memory usage otherwise. + +Tokens all have the same size so here there is only one list of free tokens. +Because tokens and (most) nodes make it into linked lists those lists of free +nodes and tokens are rather natural. And it's also fast. It all means that \TEX\ +itself does hardly any real memory allocation: only a few dozen large chunks. An +exception is the string pool, where contrary to traditional \TEX\ engines, the +\LUATEX\ (and \LUAMETATEX) engines allocate strings using \type {malloc}. Those +strings (used for control sequences) are never freed. In other cases where +strings are used, like in for instance \type {\csname} construction, temporary +strings are used. The same is true for some file related operations. None of +these are real demanding in terms of excessive allocation and freeing. Also, in +places that matter \LUAMETATEX\ is already quite optimized so using a different +allocator gives no gain here. + +Technically we could allocate nodes by using \type {malloc} but there are a few +places in the engine that makes this hard. It can be done but then we need to +make some conceptual changes (with regards to the way inserts are dealt with) and +the question is if we gain much by breaking away from tradition. I guess there it +will actually hurt performance if we change this. Another variant is where we +allocate nodes of the same size from different pools but this doesn't bring us +any gain either. A stringer argument is that changing the current (and historic) +memory management of nodes will complicate the code. + +A bit of an exception is the flow of information between \LUA\ and \TEX. There we +do quite some allocation but it depends on how much a macro package demands of +that. + +\stopsection + +\startsection[title={\METAPOST}] + +When the \METAPOST\ library was written, Taco changed the memory allocation to be +more dynamic. One reason for this is that the number models (scaled, double, +decimal, binary) have their own demands. For some objects (like numbers) the +implementation uses a pool so it sits between the way \TEX\ works and \LUA\ when +the standard allocator is used. This means that although quite some allocation +is demanded, often the pool can serve the requests. (We might use a few more +pools in the future.) + +In \LUAMETATEX\ the memory related code has been reorganized a little so that +(again as experiment) the \type {mimalloc} manager can be used. The performance +gain is not as impressive as with \LUA, but we'll see how that evolves when more +demand poses more stress. + +\stopsection + +\startsection[title={The verdict}] + +In \LUAMETATEX\ version 2.09.4 and later the native \MSWINDOWS\ binaries now use +the alternative \type {mimalloc} allocator. The gain is most noticeable for \LUA\ +and a little for \TEX\ and \METAPOST. The test suite with 2550 files runs in 1200 +seconds which is quite an improvement over the \MINGW\ cross compiled binary that +needs 1350 seconds. We do occasionally test a binary compiled with \CLANG\ but +that one is much slower than both others (compilation also takes much more time) +but that might improve over time. Because of these results, it is likely that +I'll also check out the other platforms, once the \MSWINDOWS\ binaries have +proven to be stable (those are the once I use anyway). + +\stopsection + +\stopchapter + +\stopcomponent diff --git a/doc/context/sources/general/manuals/followingup/followingup.tex b/doc/context/sources/general/manuals/followingup/followingup.tex index 7d7d17851..996673a36 100644 --- a/doc/context/sources/general/manuals/followingup/followingup.tex +++ b/doc/context/sources/general/manuals/followingup/followingup.tex @@ -29,6 +29,7 @@ \component followingup-tex \component followingup-retrospect \component followingup-fonts + \component followingup-memory \stopbodymatter \stopdocument diff --git a/doc/context/sources/general/manuals/luametatex/luametatex.tex b/doc/context/sources/general/manuals/luametatex/luametatex.tex index a46e595ca..b7b0ab749 100644 --- a/doc/context/sources/general/manuals/luametatex/luametatex.tex +++ b/doc/context/sources/general/manuals/luametatex/luametatex.tex @@ -1,4 +1,4 @@ -% ------------------------ ------------------------ ------------------------ +% ------------------------ ------ ------------------ ------------------------ % 2019-12-17 32bit 64bit 2020-01-10 32bit 64bit 2020-11-30 32bit 64bit % ------------------------ ------------------------ ------------------------ % freebsd 2270k 2662k freebsd 2186k 2558k freebsd 2108k 2436k diff --git a/scripts/context/lua/mtx-update.lua b/scripts/context/lua/mtx-update.lua index b0ed9aac7..f3e22493b 100644 --- a/scripts/context/lua/mtx-update.lua +++ b/scripts/context/lua/mtx-update.lua @@ -266,7 +266,8 @@ update.platforms = { ["macosx"] = "osx-64", ["osx"] = "osx-64", ["osx-64"] = "osx-64", - ["osx-arm64"] = "osx-64", + ["osx-arm"] = "osx-arm64", + ["osx-arm64"] = "osx-arm64", -- -- ["solaris-intel"] = "solaris-intel", -- diff --git a/scripts/context/lua/mtxrun.lua b/scripts/context/lua/mtxrun.lua index cc5b04b1e..19158704c 100644 --- a/scripts/context/lua/mtxrun.lua +++ b/scripts/context/lua/mtxrun.lua @@ -15774,7 +15774,7 @@ do -- create closure to overcome 200 locals limit package.loaded["util-zip"] = package.loaded["util-zip"] or true --- original size: 19496, stripped down to: 10858 +-- original size: 19324, stripped down to: 10821 if not modules then modules={} end modules ['util-zip']={ version=1.001, @@ -15800,19 +15800,12 @@ local getposition=files.getposition local band=bit32.band local rshift=bit32.rshift local lshift=bit32.lshift -local decompress,expandsize,calculatecrc - local zlibdecompress=zlib.decompress - local zlibexpandsize=zlib.expandsize - local zlibchecksum=zlib.crc32 - decompress=function(source) - return zlibdecompress(source,-15) - end - expandsize=zlibexpandsize and function(source,targetsize) - return zlibexpandsize(source,targetsize,-15) - end or decompress - calculatecrc=function(buffer,initial) - return zlibchecksum(initial or 0,buffer) - end +local zlibdecompress=zlib.decompress +local zlibdecompresssize=zlib.decompresssize +local zlibchecksum=zlib.crc32 +local decompress=function(source) return zlibdecompress (source,-15) end +local decompresssize=function(source,targetsize) return zlibdecompresssize(source,targetsize,-15) end +local calculatecrc=function(buffer,initial) return zlibchecksum (initial or 0,buffer) end local zipfiles={} utilities.zipfiles=zipfiles local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist do @@ -15937,8 +15930,8 @@ local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist do setposition(handle,position) local result=readstring(handle,compressed) if data.method==8 then - if expandsize then - result=expandsize(result,data.uncompressed) + if decompresssize then + result=decompresssize(result,data.uncompressed) else result=decompress(result) end @@ -25857,8 +25850,8 @@ end -- of closure -- used libraries : l-bit32.lua l-lua.lua l-macro.lua l-sandbox.lua l-package.lua l-lpeg.lua l-function.lua l-string.lua l-table.lua l-io.lua l-number.lua l-set.lua l-os.lua l-file.lua l-gzip.lua l-md5.lua l-sha.lua l-url.lua l-dir.lua l-boolean.lua l-unicode.lua l-math.lua util-str.lua util-tab.lua util-fil.lua util-sac.lua util-sto.lua util-prs.lua util-fmt.lua util-soc-imp-reset.lua util-soc-imp-socket.lua util-soc-imp-copas.lua util-soc-imp-ltn12.lua util-soc-imp-mime.lua util-soc-imp-url.lua util-soc-imp-headers.lua util-soc-imp-tp.lua util-soc-imp-http.lua util-soc-imp-ftp.lua util-soc-imp-smtp.lua trac-set.lua trac-log.lua trac-inf.lua trac-pro.lua util-lua.lua util-deb.lua util-tpl.lua util-sbx.lua util-mrg.lua util-env.lua luat-env.lua util-zip.lua lxml-tab.lua lxml-lpt.lua lxml-mis.lua lxml-aux.lua lxml-xml.lua trac-xml.lua data-ini.lua data-exp.lua data-env.lua data-tmp.lua data-met.lua data-res.lua data-pre.lua data-inp.lua data-out.lua data-fil.lua data-con.lua data-use.lua data-zip.lua data-tre.lua data-sch.lua data-lua.lua data-aux.lua data-tmf.lua data-lst.lua libs-ini.lua luat-sta.lua luat-fmt.lua -- skipped libraries : - --- original bytes : 1026599 --- stripped bytes : 405840 +-- original bytes : 1026427 +-- stripped bytes : 405705 -- end library merge diff --git a/scripts/context/stubs/mswin/mtxrun.lua b/scripts/context/stubs/mswin/mtxrun.lua index cc5b04b1e..19158704c 100644 --- a/scripts/context/stubs/mswin/mtxrun.lua +++ b/scripts/context/stubs/mswin/mtxrun.lua @@ -15774,7 +15774,7 @@ do -- create closure to overcome 200 locals limit package.loaded["util-zip"] = package.loaded["util-zip"] or true --- original size: 19496, stripped down to: 10858 +-- original size: 19324, stripped down to: 10821 if not modules then modules={} end modules ['util-zip']={ version=1.001, @@ -15800,19 +15800,12 @@ local getposition=files.getposition local band=bit32.band local rshift=bit32.rshift local lshift=bit32.lshift -local decompress,expandsize,calculatecrc - local zlibdecompress=zlib.decompress - local zlibexpandsize=zlib.expandsize - local zlibchecksum=zlib.crc32 - decompress=function(source) - return zlibdecompress(source,-15) - end - expandsize=zlibexpandsize and function(source,targetsize) - return zlibexpandsize(source,targetsize,-15) - end or decompress - calculatecrc=function(buffer,initial) - return zlibchecksum(initial or 0,buffer) - end +local zlibdecompress=zlib.decompress +local zlibdecompresssize=zlib.decompresssize +local zlibchecksum=zlib.crc32 +local decompress=function(source) return zlibdecompress (source,-15) end +local decompresssize=function(source,targetsize) return zlibdecompresssize(source,targetsize,-15) end +local calculatecrc=function(buffer,initial) return zlibchecksum (initial or 0,buffer) end local zipfiles={} utilities.zipfiles=zipfiles local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist do @@ -15937,8 +15930,8 @@ local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist do setposition(handle,position) local result=readstring(handle,compressed) if data.method==8 then - if expandsize then - result=expandsize(result,data.uncompressed) + if decompresssize then + result=decompresssize(result,data.uncompressed) else result=decompress(result) end @@ -25857,8 +25850,8 @@ end -- of closure -- used libraries : l-bit32.lua l-lua.lua l-macro.lua l-sandbox.lua l-package.lua l-lpeg.lua l-function.lua l-string.lua l-table.lua l-io.lua l-number.lua l-set.lua l-os.lua l-file.lua l-gzip.lua l-md5.lua l-sha.lua l-url.lua l-dir.lua l-boolean.lua l-unicode.lua l-math.lua util-str.lua util-tab.lua util-fil.lua util-sac.lua util-sto.lua util-prs.lua util-fmt.lua util-soc-imp-reset.lua util-soc-imp-socket.lua util-soc-imp-copas.lua util-soc-imp-ltn12.lua util-soc-imp-mime.lua util-soc-imp-url.lua util-soc-imp-headers.lua util-soc-imp-tp.lua util-soc-imp-http.lua util-soc-imp-ftp.lua util-soc-imp-smtp.lua trac-set.lua trac-log.lua trac-inf.lua trac-pro.lua util-lua.lua util-deb.lua util-tpl.lua util-sbx.lua util-mrg.lua util-env.lua luat-env.lua util-zip.lua lxml-tab.lua lxml-lpt.lua lxml-mis.lua lxml-aux.lua lxml-xml.lua trac-xml.lua data-ini.lua data-exp.lua data-env.lua data-tmp.lua data-met.lua data-res.lua data-pre.lua data-inp.lua data-out.lua data-fil.lua data-con.lua data-use.lua data-zip.lua data-tre.lua data-sch.lua data-lua.lua data-aux.lua data-tmf.lua data-lst.lua libs-ini.lua luat-sta.lua luat-fmt.lua -- skipped libraries : - --- original bytes : 1026599 --- stripped bytes : 405840 +-- original bytes : 1026427 +-- stripped bytes : 405705 -- end library merge diff --git a/scripts/context/stubs/unix/mtxrun b/scripts/context/stubs/unix/mtxrun index cc5b04b1e..19158704c 100644 --- a/scripts/context/stubs/unix/mtxrun +++ b/scripts/context/stubs/unix/mtxrun @@ -15774,7 +15774,7 @@ do -- create closure to overcome 200 locals limit package.loaded["util-zip"] = package.loaded["util-zip"] or true --- original size: 19496, stripped down to: 10858 +-- original size: 19324, stripped down to: 10821 if not modules then modules={} end modules ['util-zip']={ version=1.001, @@ -15800,19 +15800,12 @@ local getposition=files.getposition local band=bit32.band local rshift=bit32.rshift local lshift=bit32.lshift -local decompress,expandsize,calculatecrc - local zlibdecompress=zlib.decompress - local zlibexpandsize=zlib.expandsize - local zlibchecksum=zlib.crc32 - decompress=function(source) - return zlibdecompress(source,-15) - end - expandsize=zlibexpandsize and function(source,targetsize) - return zlibexpandsize(source,targetsize,-15) - end or decompress - calculatecrc=function(buffer,initial) - return zlibchecksum(initial or 0,buffer) - end +local zlibdecompress=zlib.decompress +local zlibdecompresssize=zlib.decompresssize +local zlibchecksum=zlib.crc32 +local decompress=function(source) return zlibdecompress (source,-15) end +local decompresssize=function(source,targetsize) return zlibdecompresssize(source,targetsize,-15) end +local calculatecrc=function(buffer,initial) return zlibchecksum (initial or 0,buffer) end local zipfiles={} utilities.zipfiles=zipfiles local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist do @@ -15937,8 +15930,8 @@ local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist do setposition(handle,position) local result=readstring(handle,compressed) if data.method==8 then - if expandsize then - result=expandsize(result,data.uncompressed) + if decompresssize then + result=decompresssize(result,data.uncompressed) else result=decompress(result) end @@ -25857,8 +25850,8 @@ end -- of closure -- used libraries : l-bit32.lua l-lua.lua l-macro.lua l-sandbox.lua l-package.lua l-lpeg.lua l-function.lua l-string.lua l-table.lua l-io.lua l-number.lua l-set.lua l-os.lua l-file.lua l-gzip.lua l-md5.lua l-sha.lua l-url.lua l-dir.lua l-boolean.lua l-unicode.lua l-math.lua util-str.lua util-tab.lua util-fil.lua util-sac.lua util-sto.lua util-prs.lua util-fmt.lua util-soc-imp-reset.lua util-soc-imp-socket.lua util-soc-imp-copas.lua util-soc-imp-ltn12.lua util-soc-imp-mime.lua util-soc-imp-url.lua util-soc-imp-headers.lua util-soc-imp-tp.lua util-soc-imp-http.lua util-soc-imp-ftp.lua util-soc-imp-smtp.lua trac-set.lua trac-log.lua trac-inf.lua trac-pro.lua util-lua.lua util-deb.lua util-tpl.lua util-sbx.lua util-mrg.lua util-env.lua luat-env.lua util-zip.lua lxml-tab.lua lxml-lpt.lua lxml-mis.lua lxml-aux.lua lxml-xml.lua trac-xml.lua data-ini.lua data-exp.lua data-env.lua data-tmp.lua data-met.lua data-res.lua data-pre.lua data-inp.lua data-out.lua data-fil.lua data-con.lua data-use.lua data-zip.lua data-tre.lua data-sch.lua data-lua.lua data-aux.lua data-tmf.lua data-lst.lua libs-ini.lua luat-sta.lua luat-fmt.lua -- skipped libraries : - --- original bytes : 1026599 --- stripped bytes : 405840 +-- original bytes : 1026427 +-- stripped bytes : 405705 -- end library merge diff --git a/scripts/context/stubs/win64/mtxrun.lua b/scripts/context/stubs/win64/mtxrun.lua index cc5b04b1e..19158704c 100644 --- a/scripts/context/stubs/win64/mtxrun.lua +++ b/scripts/context/stubs/win64/mtxrun.lua @@ -15774,7 +15774,7 @@ do -- create closure to overcome 200 locals limit package.loaded["util-zip"] = package.loaded["util-zip"] or true --- original size: 19496, stripped down to: 10858 +-- original size: 19324, stripped down to: 10821 if not modules then modules={} end modules ['util-zip']={ version=1.001, @@ -15800,19 +15800,12 @@ local getposition=files.getposition local band=bit32.band local rshift=bit32.rshift local lshift=bit32.lshift -local decompress,expandsize,calculatecrc - local zlibdecompress=zlib.decompress - local zlibexpandsize=zlib.expandsize - local zlibchecksum=zlib.crc32 - decompress=function(source) - return zlibdecompress(source,-15) - end - expandsize=zlibexpandsize and function(source,targetsize) - return zlibexpandsize(source,targetsize,-15) - end or decompress - calculatecrc=function(buffer,initial) - return zlibchecksum(initial or 0,buffer) - end +local zlibdecompress=zlib.decompress +local zlibdecompresssize=zlib.decompresssize +local zlibchecksum=zlib.crc32 +local decompress=function(source) return zlibdecompress (source,-15) end +local decompresssize=function(source,targetsize) return zlibdecompresssize(source,targetsize,-15) end +local calculatecrc=function(buffer,initial) return zlibchecksum (initial or 0,buffer) end local zipfiles={} utilities.zipfiles=zipfiles local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist do @@ -15937,8 +15930,8 @@ local openzipfile,closezipfile,unzipfile,foundzipfile,getziphash,getziplist do setposition(handle,position) local result=readstring(handle,compressed) if data.method==8 then - if expandsize then - result=expandsize(result,data.uncompressed) + if decompresssize then + result=decompresssize(result,data.uncompressed) else result=decompress(result) end @@ -25857,8 +25850,8 @@ end -- of closure -- used libraries : l-bit32.lua l-lua.lua l-macro.lua l-sandbox.lua l-package.lua l-lpeg.lua l-function.lua l-string.lua l-table.lua l-io.lua l-number.lua l-set.lua l-os.lua l-file.lua l-gzip.lua l-md5.lua l-sha.lua l-url.lua l-dir.lua l-boolean.lua l-unicode.lua l-math.lua util-str.lua util-tab.lua util-fil.lua util-sac.lua util-sto.lua util-prs.lua util-fmt.lua util-soc-imp-reset.lua util-soc-imp-socket.lua util-soc-imp-copas.lua util-soc-imp-ltn12.lua util-soc-imp-mime.lua util-soc-imp-url.lua util-soc-imp-headers.lua util-soc-imp-tp.lua util-soc-imp-http.lua util-soc-imp-ftp.lua util-soc-imp-smtp.lua trac-set.lua trac-log.lua trac-inf.lua trac-pro.lua util-lua.lua util-deb.lua util-tpl.lua util-sbx.lua util-mrg.lua util-env.lua luat-env.lua util-zip.lua lxml-tab.lua lxml-lpt.lua lxml-mis.lua lxml-aux.lua lxml-xml.lua trac-xml.lua data-ini.lua data-exp.lua data-env.lua data-tmp.lua data-met.lua data-res.lua data-pre.lua data-inp.lua data-out.lua data-fil.lua data-con.lua data-use.lua data-zip.lua data-tre.lua data-sch.lua data-lua.lua data-aux.lua data-tmf.lua data-lst.lua libs-ini.lua luat-sta.lua luat-fmt.lua -- skipped libraries : - --- original bytes : 1026599 --- stripped bytes : 405840 +-- original bytes : 1026427 +-- stripped bytes : 405705 -- end library merge diff --git a/tex/context/base/mkii/cont-new.mkii b/tex/context/base/mkii/cont-new.mkii index a0151c54d..b468ce3d0 100644 --- a/tex/context/base/mkii/cont-new.mkii +++ b/tex/context/base/mkii/cont-new.mkii @@ -11,7 +11,7 @@ %C therefore copyrighted by \PRAGMA. See mreadme.pdf for %C details. -\newcontextversion{2021.04.09 19:54} +\newcontextversion{2021.04.14 22:55} %D This file is loaded at runtime, thereby providing an %D excellent place for hacks, patches, extensions and new diff --git a/tex/context/base/mkii/context.mkii b/tex/context/base/mkii/context.mkii index 5313df6e6..7ad92f388 100644 --- a/tex/context/base/mkii/context.mkii +++ b/tex/context/base/mkii/context.mkii @@ -20,7 +20,7 @@ %D your styles an modules. \edef\contextformat {\jobname} -\edef\contextversion{2021.04.09 19:54} +\edef\contextversion{2021.04.14 22:55} %D For those who want to use this: diff --git a/tex/context/base/mkiv/cont-new.mkiv b/tex/context/base/mkiv/cont-new.mkiv index e81cbb7d9..7bb390194 100644 --- a/tex/context/base/mkiv/cont-new.mkiv +++ b/tex/context/base/mkiv/cont-new.mkiv @@ -13,7 +13,7 @@ % \normalend % uncomment this to get the real base runtime -\newcontextversion{2021.04.09 19:54} +\newcontextversion{2021.04.14 22:55} %D This file is loaded at runtime, thereby providing an excellent place for hacks, %D patches, extensions and new features. There can be local overloads in cont-loc diff --git a/tex/context/base/mkiv/context.mkiv b/tex/context/base/mkiv/context.mkiv index 5062c3a10..6fa606a95 100644 --- a/tex/context/base/mkiv/context.mkiv +++ b/tex/context/base/mkiv/context.mkiv @@ -45,7 +45,7 @@ %D {YYYY.MM.DD HH:MM} format. \edef\contextformat {\jobname} -\edef\contextversion{2021.04.09 19:54} +\edef\contextversion{2021.04.14 22:55} %D Kind of special: diff --git a/tex/context/base/mkiv/status-files.pdf b/tex/context/base/mkiv/status-files.pdf Binary files differindex cffa6fdb6..69556e09e 100644 --- a/tex/context/base/mkiv/status-files.pdf +++ b/tex/context/base/mkiv/status-files.pdf diff --git a/tex/context/base/mkiv/status-lua.pdf b/tex/context/base/mkiv/status-lua.pdf Binary files differindex fea56a359..615e7d326 100644 --- a/tex/context/base/mkiv/status-lua.pdf +++ b/tex/context/base/mkiv/status-lua.pdf diff --git a/tex/context/base/mkiv/util-zip.lua b/tex/context/base/mkiv/util-zip.lua index f87e391a1..e97f3a065 100644 --- a/tex/context/base/mkiv/util-zip.lua +++ b/tex/context/base/mkiv/util-zip.lua @@ -34,32 +34,13 @@ local band = bit32.band local rshift = bit32.rshift local lshift = bit32.lshift -local decompress, expandsize, calculatecrc +local zlibdecompress = zlib.decompress +local zlibdecompresssize = zlib.decompresssize +local zlibchecksum = zlib.crc32 --- if flate then --- --- decompress = flate.flate_decompress --- calculatecrc = flate.update_crc32 --- --- else - - local zlibdecompress = zlib.decompress - local zlibexpandsize = zlib.expandsize - local zlibchecksum = zlib.crc32 - - decompress = function(source) - return zlibdecompress(source,-15) -- auto - end - - expandsize = zlibexpandsize and function(source,targetsize) - return zlibexpandsize(source,targetsize,-15) -- auto - end or decompress - - calculatecrc = function(buffer,initial) - return zlibchecksum(initial or 0,buffer) - end - --- end +local decompress = function(source) return zlibdecompress (source,-15) end -- auto +local decompresssize = function(source,targetsize) return zlibdecompresssize(source,targetsize,-15) end -- auto +local calculatecrc = function(buffer,initial) return zlibchecksum (initial or 0,buffer) end local zipfiles = { } utilities.zipfiles = zipfiles @@ -201,8 +182,8 @@ local openzipfile, closezipfile, unzipfile, foundzipfile, getziphash, getziplist setposition(handle,position) local result = readstring(handle,compressed) if data.method == 8 then - if expandsize then - result = expandsize(result,data.uncompressed) + if decompresssize then + result = decompresssize(result,data.uncompressed) else result = decompress(result) end diff --git a/tex/context/base/mkxl/cldf-lmt.lmt b/tex/context/base/mkxl/cldf-lmt.lmt index f3121eadc..1c1b6a353 100644 --- a/tex/context/base/mkxl/cldf-lmt.lmt +++ b/tex/context/base/mkxl/cldf-lmt.lmt @@ -6,7 +6,7 @@ if not modules then modules = { } end modules ['cldf-lmt'] = { license = "see context related readme files" } -local next, load = next, load +local next, load, tonumber = next, load, tonumber local gmatch, gsub, byte = string.gmatch, string.gsub, string.byte local setmetatableindex = table.setmetatableindex @@ -627,7 +627,7 @@ context.luatables = { local tables = { } local stack = setmetatableindex("table") -interfaces.implement { +implement { name = "droptablegroup", public = true, actions = function() @@ -699,7 +699,7 @@ local function newtable(array) if not tables[name] then local t = { } tables[name] = t - interfaces.implement { + implement { name = name, public = true, usage = "value", @@ -886,7 +886,7 @@ implement { end } -interfaces.implement { +implement { name = "bitwise", public = true, usage = "value", @@ -933,7 +933,7 @@ interfaces.implement { local escape = function(s) return "\\" .. byte(s) end -interfaces.implement { +implement { name = "ctxluamatch", public = true, usage = "value", @@ -947,3 +947,65 @@ interfaces.implement { return none_code end, } + +-- yes or no ... + +do + + local codes = { } + tex.codes = codes + + local global_code = tex.flagcodes.global + + local savelua = token.save_lua + local isdefined = token.is_defined + + local newsparse = sparse.new + local setsparse = sparse.set + local wipesparse = sparse.wipe + local restoresparse = sparse.restore + + -- local function isglobal(n) -- maybe a general helper + -- return n and (tonumber(n) & global_code) + -- end + + local registerfunction = context.functions.register + + implement { + name = "codedef", + public = true, + protected = true, + actions = function(what) + local name = scancsname(true) + -- if isdefined(name) then + -- wipesparse(codes[name]) -- better make a wipe helper if ever needed + -- else + local code = newsparse() + local restore = registerfunction(function() restoresparse(code) end) + implement { + name = name, + public = true, + protected = true, + usage = "value", + actions = function(what) + local n = scaninteger() + if what == "value" then + return integer_code, code[n] + else + local v = scaninteger(true) + -- if isglobal(what) then + if what and (tonumber(what) & global_code) then + setsparse(code,"global",n,v) + else + savelua(restore,true) -- only once + setsparse(code,n,v) + end + end + end, + } + codes[name] = code + -- end + end, + } + +end diff --git a/tex/context/base/mkxl/cont-new.mkxl b/tex/context/base/mkxl/cont-new.mkxl index 87175b22d..f99a9ce23 100644 --- a/tex/context/base/mkxl/cont-new.mkxl +++ b/tex/context/base/mkxl/cont-new.mkxl @@ -13,7 +13,7 @@ % \normalend % uncomment this to get the real base runtime -\newcontextversion{2021.04.09 19:54} +\newcontextversion{2021.04.14 22:55} %D This file is loaded at runtime, thereby providing an excellent place for hacks, %D patches, extensions and new features. There can be local overloads in cont-loc diff --git a/tex/context/base/mkxl/context.mkxl b/tex/context/base/mkxl/context.mkxl index 84bdd0277..103481265 100644 --- a/tex/context/base/mkxl/context.mkxl +++ b/tex/context/base/mkxl/context.mkxl @@ -29,7 +29,7 @@ %D {YYYY.MM.DD HH:MM} format. \immutable\edef\contextformat {\jobname} -\immutable\edef\contextversion{2021.04.09 19:54} +\immutable\edef\contextversion{2021.04.14 22:55} %overloadmode 1 % check frozen / warning %overloadmode 2 % check frozen / error diff --git a/tex/context/base/mkxl/lpdf-lmt.lmt b/tex/context/base/mkxl/lpdf-lmt.lmt index 8d8220230..800f56ca3 100644 --- a/tex/context/base/mkxl/lpdf-lmt.lmt +++ b/tex/context/base/mkxl/lpdf-lmt.lmt @@ -42,9 +42,9 @@ local loaddata = io.loaddata local bpfactor = number.dimenfactors.bp --- local md5HEX = md5.HEX local osuuid = os.uuid -local zlibcompress = (xzip or zlib).compress +local zlibcompress = xzip.compress +local zlibcompresssize = xzip.compresssize local nuts = nodes.nuts local tonut = nodes.tonut @@ -101,6 +101,19 @@ local report_objects = logs.reporter("backend","objects") local trace_objects = false trackers.register("backend.objects", function(v) trace_objects = v end) local trace_details = false trackers.register("backend.details", function(v) trace_details = v end) +-- experiment: + +local function compressdata(data,size) + local guess = ((size // 4096) + 1) * 2048 + local comp = zlibcompresssize(data,3,guess) + -- if comp then + -- report() + -- report("size %i, guess %i, result %i => %s / %s",size,guess,#comp,guess>=#comp and "hit" or "miss") + -- report() + -- end + return comp +end + -- we collect them: local flushers = { } @@ -2021,25 +2034,28 @@ local addtocache, flushcache, cache do list = concat(list," ") data[0] = list data = concat(data,"\010",0,d) + local size = #data local strobj = pdfdictionary { Type = p_ObjStm, N = d, First = #list + 1, } objects[cache] = offset - local b = nil - local e = s_stream_e + local fb if compress then - local comp = zlibcompress(data,3) - if comp and #comp < #data then +-- local comp = zlibcompress(data,3) + local comp = compressdata(data,size) + if comp and #comp < size then data = comp - b = f_stream_b_d_c(cache,strobj(),#data) + fb = f_stream_b_d_c else - b = f_stream_b_d_u(cache,strobj(),#data) + fb = f_stream_b_d_u end else - b = f_stream_b_d_u(cache,strobj(),#data) + fb = f_stream_b_d_u end + local b = fb(cache,strobj(),size) + local e = s_stream_e flush(f,b) flush(f,data) flush(f,e) @@ -2274,7 +2290,8 @@ local function flushstreamobj(data,n,dict,comp,nolength) if nolength then b = f_stream_b_d_r(n,dict) elseif comp ~= false and compress and size > threshold then - local compdata = zlibcompress(data,3) +-- local compdata = zlibcompress(data,3) + local compdata = compressdata(data,size) if compdata then local compsize = #compdata if compsize > size - threshold then @@ -2298,7 +2315,8 @@ local function flushstreamobj(data,n,dict,comp,nolength) if nolength then data = f_stream_d_r(n,dict,data) elseif comp ~= false and compress and size > threshold then - local compdata = zlibcompress(data,3) +-- local compdata = zlibcompress(data,3) + local compdata = compressdata(data,size) if compdata then local compsize = #compdata if compsize > size - threshold then @@ -2635,6 +2653,7 @@ local openfile, closefile do end objects[0] = rep("\0",1+nofbytes+1) local data = concat(objects,"",0,nofobjects) + local size = #data local xref = pdfdictionary { Type = pdfconstant("XRef"), Size = nofobjects + 1, @@ -2643,17 +2662,20 @@ local openfile, closefile do Info = info, ID = trailerid and pdfarray { pdfliteral(trailerid,true), pdfliteral(trailerid,true) } or nil, } + local fb if compress then - local comp = zlibcompress(data,3) +-- local comp = zlibcompress(data,3) + local comp = compressdata(data,size) if comp then data = comp - flush(f,f_stream_b_d_c(nofobjects,xref(),#data)) + fb = f_stream_b_d_c else - flush(f,f_stream_b_d_u(nofobjects,xref(),#data)) + fb = f_stream_b_d_u end else - flush(f,f_stream_b_d_u(nofobjects,xref(),#data)) + fb = f_stream_b_d_u end + flush(f,fb(nofobjects,xref(),size)) flush(f,data) flush(f,s_stream_e) flush(f,f_startxref(xrefoffset)) diff --git a/tex/context/base/mkxl/mult-ini.mkxl b/tex/context/base/mkxl/mult-ini.mkxl index 17f63ef93..b357c3e12 100644 --- a/tex/context/base/mkxl/mult-ini.mkxl +++ b/tex/context/base/mkxl/mult-ini.mkxl @@ -19,8 +19,10 @@ \unprotect -\registerctxluafile{mult-ini}{autosuffix} -\registerctxluafile{mult-fmt}{initexonly} +\pushoverloadmode + \registerctxluafile{mult-ini}{autosuffix} + \registerctxluafile{mult-fmt}{initexonly} +\popoverloadmode %D \macros %D [constanten,variabelen,commands] diff --git a/tex/context/fonts/mkiv/pagella-math.lfg b/tex/context/fonts/mkiv/pagella-math.lfg index d4ae14162..0e8ac1f70 100644 --- a/tex/context/fonts/mkiv/pagella-math.lfg +++ b/tex/context/fonts/mkiv/pagella-math.lfg @@ -1,5 +1,6 @@ local kern_V = { bottomright = { { kern = -200 } } } local kern_W = { bottomright = { { kern = -100 } } } +-- local kern_f = { bottomright = { { kern = -100 } } } local offset_f = { xoffset = "llx" } -- Beware of updates in ssty slots! @@ -27,6 +28,7 @@ return { -- [0x1D44A] = kern_W, -- 𝑊 -- ["1:0x1D44A"] = kern_W, -- needed for compact -- ["2:0x1D44A"] = kern_W, -- needed for compact +-- ["*:0x1D453"] = kern_f, ["*:0x1D449"] = kern_V, -- 𝑉 ["*:0x1D44A"] = kern_W, -- 𝑊 }, diff --git a/tex/generic/context/luatex/luatex-fonts-merged.lua b/tex/generic/context/luatex/luatex-fonts-merged.lua index 868d224f3..1b594098e 100644 --- a/tex/generic/context/luatex/luatex-fonts-merged.lua +++ b/tex/generic/context/luatex/luatex-fonts-merged.lua @@ -1,6 +1,6 @@ -- merged file : c:/data/develop/context/sources/luatex-fonts-merged.lua -- parent file : c:/data/develop/context/sources/luatex-fonts.lua --- merge date : 2021-04-09 19:54 +-- merge date : 2021-04-14 22:55 do -- begin closure to overcome local limits and interference |