summaryrefslogtreecommitdiff
path: root/src/luaotfload-database.lua
diff options
context:
space:
mode:
Diffstat (limited to 'src/luaotfload-database.lua')
-rw-r--r--src/luaotfload-database.lua124
1 files changed, 99 insertions, 25 deletions
diff --git a/src/luaotfload-database.lua b/src/luaotfload-database.lua
index 24c2745..0557c1a 100644
--- a/src/luaotfload-database.lua
+++ b/src/luaotfload-database.lua
@@ -8,31 +8,105 @@ if not modules then modules = { } end modules ['luaotfload-database'] = {
--[[doc--
- Some statistics:
-
- a) TL 2012, mkluatexfontdb --force
- b) v2.4, luaotfload-tool --update --force
- c) v2.4, luaotfload-tool --update --force --formats=+afm,pfa,pfb
- d) Context, mtxrun --script fonts --reload --force
-
- (Keep in mind that Context does index fewer fonts since it
- considers only the contents of the minimals tree, not the
- tex live one!)
-
- time (m:s) peak VmSize (kB)
- a 1:19 386 018
- b 0:37 715 797
- c 2:27 1 017 674
- d 0:44 1 082 313
-
- Most of the increase in memory consumption from version 1.x to 2.2+
- can be attributed to the move from single-pass to a multi-pass
- approach to building the index: Information is first gathered from
- all reachable fonts and only afterwards processed, classified and
- discarded. Also, there is a good deal of additional stuff kept in
- the database now: two extra tables for file names and font families
- have been added, making font lookups more efficient while improving
- maintainability of the code.
+ With version 2.7 we killed of the Fontforge libraries in favor of
+ the Lua implementation of the OT format reader. There were many
+ reasons to do this on top of the fact that FF won’t be around at
+ version 1.0 anymore: In addition to maintainability, memory safety
+ and general code hygiene, the new reader shows an amazing
+ performance: Scanning of the 3200 font files on my system takes
+ around 23 s now, as opposed to 74 s with the Fontforge libs. Memory
+ usage has improved drastically as well, as illustrated by these
+ profiles:
+
+ GB
+ 1.324^ #
+ | #
+ | ::#
+ | : #::
+ | @: #:
+ | @: #:
+ | @@@: #:
+ | @@@ @: #:
+ | @@ @ @ @: #:
+ | @ @@:@ @ @: #: :
+ | @@ : @ :@ @ @: #: :
+ | @@: ::@@ :@@ ::@ :@ @ @: #: : :::
+ | @@ : :: @@ :@ : @ :@ @ @: #: : :: :
+ | @@ : ::: @@ :@ ::: @ :@ @ @: #: :: :: :
+ | @@@@ :: :::: @@ :@ : : @ :@ @ @: #: :: :::: ::
+ | :@ @@ :: :::: @@ :@ : :: : @ :@ @ @: #: :: :: :: ::
+ | @:@ @@ :: ::::: @@ :@ : :::: : @ :@ @ @: #: :::: :::: :: ::
+ | @@:@ @@ :: ::::::: @@ :@ : : :: : @ :@ @ @: #: ::: @: :: :: ::@
+ | @@@:@ @@ :: :: ::::: @@ :@ ::: :: : @ :@ @ @: #: ::: @: :: :: ::@
+ | @@@@@:@ @@ ::::::: ::::: @@ :@ ::: :: : @ :@ @ @: #: ::: ::@: :: :: ::@
+ 0 +----------------------------------------------------------------------->GB
+ 0 16.29
+
+ This is the memory usage during a complete database rebuild with
+ the Fontforge libraries. The same action using the new
+ ``getinfo()`` method gives a different picture:
+
+ MB
+ 43.37^ #
+ | @ @ @#
+ | @@ @ @ @# :
+ | @@@ : @: @ @ @# :
+ | @ @@@ : : @: @ @: :@# :
+ | @ @ @@@ : : @: @ @: :@# :
+ | @ : : :@ @@@:::: @::@ @: :@#:: :
+ | :: : @ : @ : :::@ @ :@@@:::::@::@ @:::@#::::
+ | : @ : :: : :@:: :@: :::::@ @ :@@@:::::@::@:@:::@#::::
+ | :: :@ : @ ::@:@:::@:: :@: :::::@: :@ :@@@:::::@::@:@:::@#::::
+ | :: :@::: :@ ::@:@: :@::::@::::::::@:::@::@@@:::::@::@:@:::@#::::
+ | :@::::@::: :@:::@:@: :@::::@::::::::@:::@::@@@:::::@::@:@:::@#::::
+ | :::::@::::@::: :@:::@:@: :@::::@::::::::@:::@::@@@:::::@::@:@:::@#::::
+ | ::: :@::::@::: :@:::@:@: :@::::@::::::::@:::@::@@@:::::@::@:@:::@#::::
+ | :::: :@::::@::: :@:::@:@: :@::::@::::::::@:::@::@@@:::::@::@:@:::@#::::
+ | :::: :@::::@::: :@:::@:@: :@::::@::::::::@:::@::@@@:::::@::@:@:::@#::::
+ | :::: :@::::@::: :@:::@:@: :@::::@::::::::@:::@::@@@:::::@::@:@:::@#::::
+ | :::: :@::::@::: :@:::@:@: :@::::@::::::::@:::@::@@@:::::@::@:@:::@#::::
+ | :::: :@::::@::: :@:::@:@: :@::::@::::::::@:::@::@@@:::::@::@:@:::@#::::
+ | :::: :@::::@::: :@:::@:@: :@::::@::::::::@:::@::@@@:::::@::@:@:::@#::::
+ 0 +----------------------------------------------------------------------->GB
+ 0 3.231
+
+ FF peaks at around 1.4 GB after 12.5 GB worth of allocations,
+ whereas the Lua implementation arrives at around 45 MB after 3.2 GB
+ total:
+
+ impl time(B) total(B) useful-heap(B) extra-heap(B)
+ fontforge 12,496,407,184 1,421,150,144 1,327,888,638 93,261,506
+ lua 3,263,698,960 45,478,640 37,231,892 8,246,748
+
+ Much of the inefficiency of Fontforge is a direct consequence of
+ having to parse the entire font to extract what essentially boils
+ down to a couple hundred bytes of metadata per font. Since some
+ information like design sizes (oh, Adobe!) is stuffed away in
+ Opentype tables, the vastly more efficient approach of
+ fontloader.info() proves insufficient for indexing. Thus, we ended
+ up using fontloader.open() which causes even the character tables
+ to be parsed, which incidentally are responsible for most of the
+ allocations during that peak of 1.4 GB measured above, along with
+ the encodings:
+
+ 20.67% (293,781,048B) 0x6A8F72: SplineCharCreate (splineutil.c:3878)
+ 09.82% (139,570,318B) 0x618ACD: _FontViewBaseCreate (fontviewbase.c:84)
+ 08.77% (124,634,384B) 0x6A8FB3: SplineCharCreate (splineutil.c:3885)
+ 04.53% (64,436,904B) in 80 places, all below massif's threshold (1.00%)
+ 02.68% (38,071,520B) 0x64E14E: addKernPair (parsettfatt.c:493)
+ 01.04% (14,735,320B) 0x64DE7D: addPairPos (parsettfatt.c:452)
+ 39.26% (557,942,484B) 0x64A4E0: PsuedoEncodeUnencoded (parsettf.c:5706)
+
+ What gives? For 2.7 we expect a rougher transition than a year back
+ due to the complete revamp of the OT loading code. Breakage of
+ fragile aspects like font and style names has been anticipated and
+ addressed prior to the 2016 pretest release. In contrast to the
+ earlier approach of letting FF do a complete dump and then harvest
+ identifiers from the output we now have to coordinate with upstream
+ as to which fields are actually needed in order to provide a
+ similarly acceptable name → file lookup. On the bright side, these
+ things are a lot simpler to fix than the rather tedious work of
+ having users update their Luatex binary =)
--doc]]--