diff options
Diffstat (limited to 'doc/context')
21 files changed, 2402 insertions, 20 deletions
diff --git a/doc/context/documents/general/manuals/lowlevel-alignments.pdf b/doc/context/documents/general/manuals/lowlevel-alignments.pdf Binary files differindex abeebfee9..86b9a60b1 100644 --- a/doc/context/documents/general/manuals/lowlevel-alignments.pdf +++ b/doc/context/documents/general/manuals/lowlevel-alignments.pdf diff --git a/doc/context/presentations/context/2021/context-2021-compactfonts.pdf b/doc/context/presentations/context/2021/context-2021-compactfonts.pdf Binary files differnew file mode 100644 index 000000000..fb3f448ba --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-compactfonts.pdf diff --git a/doc/context/presentations/context/2021/context-2021-compactfonts.tex b/doc/context/presentations/context/2021/context-2021-compactfonts.tex new file mode 100644 index 000000000..7d2d3e482 --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-compactfonts.tex @@ -0,0 +1,609 @@ +% language=us + +\usemodule[present-boring,abbreviations-logos] + + +\definehighlight[nb][style=bold,color=middlegray,define=no] + +\definecolor[maincolor][g=.3] + +\startdocument + [title={COMPACT FONTS}, + banner={a different approach to scaling}, + location={context\enspace {\bf 2021}\enspace meeting}] + +\starttitle[title=How \TEX\ uses fonts] + +\startitemize + \startitem + In a \TEX\ engine there is a 1-to-1 mapping from characters in the input + to a slot in a font. + \stopitem + \startitem + Combinations of characters can be mapped onto one shape: ligatures. + \stopitem + \startitem + Hyphenation is (also) controlled by the input encoding of a font which + imposes some limitations. + \stopitem + \startitem + In the typeset result characters (glyphs) can be (re)positioned relatively: kerns. + \stopitem + \startitem + This all happens in a very interwoven way (e.g. inside the par builder). + \stopitem +\stopitemize + +\stoptitle + +\starttitle[title=How traditional \TEX\ sees fonts] + +\startitemize + \startitem + A font is just a (binary) blob of data with information where characters + have a width, height, depth and italic correction. + \stopitem + \startitem + The engine only needs the dimensions and when the same font is used with a + different scale a copy with scaled dimensions is used. + \stopitem + \startitem + There can be recipes for ligatures and (more complex) so called math extensible + shapes (like arrows, wide accents, fences and radicals) + \stopitem + \startitem + There are pairwise specification for kerning. + \stopitem + \startitem + A handful of font parameters is provided in order to control for instance spacing. + \stopitem + \startitem + Virtual fonts are just normal fonts but they have recipes for the backend to + construct shapes: \TEX\ uses the normal metric file, the backend also the virtual + specification file. + \stopitem +\stopitemize + +\blank[2*big] + +An example of a definition and usage: + +\starttyping +\font\cs somename [somescale] test {\cs test} test +\stoptyping + +\stoptitle + +\starttitle[title=\UNICODE\ engines] + +\startitemize + \startitem + There are no fundamental differences between a traditional engine and an + \UNICODE\ aware one with respect to what \TEX\ needs from fonts: + dimensions. + \stopitem + \startitem + There can be more than 256 characters in a font. + \stopitem + \startitem + Some mechanisms have to be present for applying font features (like ligaturing and + kerning). + \stopitem + \startitem + In \LUATEX\ hyphenation, ligaturing and kerning are clearly separate steps. + \stopitem + \startitem + In \LUATEX\ callbacks can do lots of things with fonts and characters. + \stopitem + \startitem + Math fonts are handled differently and there is more info that comes from the font. + \stopitem + \startitem + In \LUATEX\ features like expansion are implemented differently, i.e.\ no + copies of fonts are made and applied expansion travels with the glyphs. + \stopitem +\stopitemize + +\stoptitle + +\starttitle[title=Overhead] + +\startitemize + \startitem + In all engines scales copies are used. For traditional \TEX\ a copy is cheap, but a + font that supports more of \UNICODE\ cna be quite large. + \stopitem + \startitem + Different feature sets in principle make for a different font (this can be different + per macro package). + \stopitem + \startitem + Every math scale needs three font instances: text, script and scriptscript. + \stopitem + \startitem + In \CONTEXT\ lots of sharing takes place and a lot of low level + optimizations happens (memory, performance). We always made sure (already + in \MKII) that the font system was not the bottleneck. + \stopitem + \startitem + In most cases font definitions are dynamic, and we delay initialization + as much as possible. + \stopitem + \startitem + In \MKIV\ font features can be dynamic which saves a lot of memory at the + cost of some extra processing overhead. + \stopitem +\stopitemize + +\stoptitle + +\starttitle[title=Design sizes] + +\startitemize + \startitem + A macro package's font handling is complicated by design sizes. The low level + code can look complex too. + \stopitem + \startitem + There are hardly any fonts that come in design sizes (basically only Computer + Modern) so we need to support that. + \stopitem + \startitem + Design sizes make sense for math fonts where very small characters and + symbols are used in scripts. + \stopitem + \startitem + In \OPENTYPE\ fonts smaller math design sizes are part of the font (that + then gets loaded three times with different \type {ssty} settings). + \stopitem +\stopitemize + +\startlinecorrection +\dontleavehmode\scale[frame=on,height=5ex]{$\textstyle 2$}\quad +\dontleavehmode\scale[frame=on,height=5ex]{$\scriptstyle 2$}\quad +\dontleavehmode\scale[frame=on,height=5ex]{$\scriptscriptstyle 2$}\quad\quad +\dontleavehmode\scale[frame=on,height=2ex]{$\textstyle 2$}\quad +\dontleavehmode\scale[frame=on,height=2ex]{$\scriptstyle 2$}\quad +\dontleavehmode\scale[frame=on,height=2ex]{$\scriptscriptstyle 2$}\quad\quad +\dontleavehmode\scale[frame=on,width=4em]{\colored[r=.6,a=1,t=.5]{$\textstyle 2$}}\hskip-4em +\dontleavehmode\scale[frame=on,width=4em]{\colored[g=.6,a=1,t=.5]{$\scriptstyle 2$}}\hskip-4em +\dontleavehmode\scale[frame=on,width=4em]{\colored[b=.6,a=1,t=.5]{$\scriptscriptstyle 2$}}\quad\quad +\dontleavehmode{$\textstyle 2$}\quad +\dontleavehmode{$\scriptstyle 2$}\quad +\dontleavehmode{$\scriptscriptstyle 2$}\quad +\stoplinecorrection + +\stoptitle + +\starttitle[title=The system] + +\startitemize + \startitem + Each bodyfont size (often a few are defined) has regular, bold, italic, + bold italic, slanted, bold slanted, etc. There can be unscaled instances + (design size) or scaled ones. + \stopitem + \startitem + Within a bodyfont size we have size variants: smaller {\tx x} and {\tx xx}, + larger {\tfa a}, {\tfb b}, {\tfc c} and {\tfd d}. + \stopitem + \startitem + In \MKII\ we inherit smallcap and oldstyle setups but in \MKIV\ one can either + do that dynamic or define an additional bodyfont instance. + \stopitem + \startitem + A document can use a mix of fonts mixed setup, so there is a namespace + used per family. + \stopitem + \startitem + In \OPENTYPE\ symbols are more naturally part of a font, as are emoji and + colored shapes. + \stopitem + \startitem + Lack of variants can result in artificially slanted of boldened fonts (more + advanced in \MKIV). Variable fonts are not special, but demand some work + when loading and embedding. + \stopitem + \startitem + Runtime virtual fonts are part of \MKIV\ as are fallback shapes. + \stopitem + \startitem + Languages and scripts can introduce an extra dimension to operate on. + \stopitem + \startitem + Math is greatly simplified by the fact that families are part of a font. + \stopitem + \startitem + Bold math fonts are provided by the boldening feature. + \stopitem +\stopitemize + +\stoptitle + +\starttitle[title={Practice}] + +\startitemize + \startitem + Due to optimizations the actual number of loaded instances is normally + small but there are exceptions. + \stopitem + \startitem + The \LUAMETATEX\ manual has 158 instances most of which are math (six per + used bodyfont for math plus a regular: Lucida, Pagella, Latin Modern, + Dejavu, Cambria). + \stopitem + \startitem + The final \PDF\ file has 21 embedded fonts (subsets). + \stopitem + \startitem + The fact that different sizes each have an instance and that for math + we need three per size plus one for r2l, while the only difference + is scale, makes one think of other solutions. + \stopitem + \startitem + Using duplicates of huge \CJK\ fonts makes it even more noticeable. + \stopitem +\stopitemize + +But there is a way out! + +\stoptitle + +\starttitle[title=Glyph scaling] + +We can scale glyphs in three ways: + +\startbuffer +\definedfont[lmroman10-regular*default]% +test {\glyphxscale 2500 test} test +test {\glyphyscale 2500 test} test +test {\glyphscale 2500 test} test +\stopbuffer + +\typebuffer {\getbuffer} + +We do need to honor grouping: + +\startbuffer +\definedfont[lmroman10-regular*default]% +e{\glyphxscale 2500 ff}icient +ef{\glyphxscale 2500 f}icient +ef{\glyphxscale 2500 fi}cient +e{\glyphxscale 2500 ffi}cient +\stopbuffer + +\typebuffer {\getbuffer} + +These scales can also be part of a font definition so we can do with less +instances. + +\stoptitle + +\starttitle[title=Implications] + +\startitemize + \startitem + The text processing part of the engine has to scale on the fly (e.g.\ + when dimensions are needed in paragraph building and (box) packing. + \stopitem + \startitem + Font processing (features) already distinguishes per instance but now also + has to differentiate by (upto three) scales (we use the same font id + for scaled instances). + \stopitem + \startitem + The math machinery has to check for smaller sizes and also scale + dimensions on the fly. + \stopitem + \startitem + For math that means two times scaling: the main scale (like glyph scale) plus + the relative scaling of script and scriptscript. + \stopitem + \startitem + When they are used font dimensions and math parameters are to be scaled as are + rules in some math extensibles. + \stopitem + \startitem + This all has to be done efficiently so that there is no significant drop in + performance. + \stopitem + \startitem + But quite a bit less memory is used and loading time has become less too. + \stopitem +\stopitemize + +Some day this will become default (which means a sentimental process of dropping +ancient code): + +\starttyping +\enableexperiments[fonts.compact] +\stoptyping + +\stoptitle + +\starttitle[title=Details] + +The tricks shown on this page and following pages have been in place and tested +for a while now. Because \TEX\ uses integers a scale value of 1000 means 1.000, +as in other mechanisms: + +\startbuffer +\definedfont[lmroman10-regular*default]% +test {\glyphscale 2500 test} test +\stopbuffer + +\typebuffer {\getbuffer} + +% \glyphtextscale +% \glyphscriptscale +% \glyphscriptscriptscale + +\page + +The font definition mechanism supports horizontal and vertical scaling: + +\startbuffer +\definefont[FooA][Serif*default @ 12pt 1800 500] +\definefont[FooB][Serif*default @ 12pt 0.85 0.4] +\definefont[FooC][Serif*default @ 12pt] + +There is also a new command: + +\definetweakedfont[runwider] [xscale=1.5] +\definetweakedfont[runtaller][yscale=2.5,xscale=.8,yoffset=-.2ex] + +{\FooA test test \runwider test test \runtaller test test}\par +{\FooB test test \runwider test test \runtaller test test}\par +{\FooC test test \runwider test test \runtaller test test}\par +\stopbuffer + +\typebuffer + +\getbuffer + +\page + +Tweaking also works in math: + +\startbuffer +\definetweakedfont[squeezed][xscale=0.9] +\definetweakedfont[squoozed][xscale=0.7] + +\startlines +$a = b^2 + \sqrt{c}$ +{\squeezed $a = b^2 + \sqrt{c}$} +{\squoozed $a = b^2 + \sqrt{c}$} +{$\squoozed a = b^2 + \sqrt{c}$} +{$\scriptstyle a = b^2 + \sqrt{c}$} +\stoplines +\stopbuffer + +\typebuffer + +\getbuffer + +\page + +\startbuffer +\startcombination[3*1] + {\bTABLE + \bTR \bTD foo \eTD \bTD[style=\squeezed] $x = 1$ \eTD \eTR + \bTR \bTD oof \eTD \bTD[style=\squeezed] $x = 2$ \eTD \eTR + \eTABLE} {local} + {\bTABLE[style=\squeezed] + \bTR \bTD $x = 1$ \eTD \bTD $x = 3$ \eTD \eTR + \bTR \bTD $x = 2$ \eTD \bTD $x = 4$ \eTD \eTR + \eTABLE} {global} + {\bTABLE[style=\squeezed\squeezed\squeezed\squeezed] + \bTR \bTD $x = 1$ \eTD \bTD $x = 3$ \eTD \eTR + \bTR \bTD $x = 2$ \eTD \bTD $x = 4$ \eTD \eTR + \eTABLE} {multiple} +\stopcombination +\stopbuffer + +\typebuffer + +\startlinecorrection +\getbuffer +\stoplinecorrection + +\page + +Tweaking can be combined with styles: + +\startbuffer +\definetweakedfont[MyLargerFontA][scale=2000,style=bold] +test {\MyLargerFontA test} test +\stopbuffer + +\typebuffer + +\getbuffer + +\page + +We can use negative scale values: + +\startbuffer +\bTABLE[align=middle] + \bTR + \bTD a{\glyphxscale 1000 \glyphyscale 1000 bc}d \eTD + \bTD a{\glyphxscale 1000 \glyphyscale -1000 bc}d \eTD + \bTD a{\glyphxscale -1000 \glyphyscale -1000 bc}d \eTD + \bTD a{\glyphxscale -1000 \glyphyscale 1000 bc}d \eTD + \eTR + \bTR + \bTD \tttf +1000 +1000 \eTD + \bTD \tttf +1000 -1000 \eTD + \bTD \tttf -1000 -1000 \eTD + \bTD \tttf -1000 +1000 \eTD + \eTR +\eTABLE +\stopbuffer + +\typebuffer + +\startlinecorrection +\getbuffer +\stoplinecorrection + +\page + +There are more glyph properties: + +\startbuffer +\ruledhbox{ + \ruledhbox{\glyph yoffset 1ex options 0 123} % left curly brace + \ruledhbox{\glyph xoffset .5em yoffset 1ex options "C0 123} + \ruledhbox{oeps{\glyphyoffset 1ex \glyphxscale 800 + \glyphyscale\glyphxscale oeps}oeps} +} +\stopbuffer + +\typebuffer \scale[width=\textwidth]{\getbuffer} + +\page + +Glyph scales are internal counter values, like any other: + +\startbuffer +\samplefile{jojomayer} +{\glyphyoffset .8ex + \glyphxscale 700 \glyphyscale\glyphxscale + \samplefile{jojomayer}} +{\glyphyscale\numexpr3*\glyphxscale/2\relax + \samplefile{jojomayer}} +{\glyphyoffset -.2ex + \glyphxscale 500 \glyphyscale\glyphxscale + \samplefile{jojomayer}} +\samplefile{jojomayer} +\stopbuffer + +\typebuffer + +\getbuffer + +\page + +There is a helper for real (float) scales: + +\startbuffer +1200 : \the\numericscale1200 +1.20 : \the\numericscale1.200 +\stopbuffer + +\typebuffer + +These lines produce the same integer: + +\startlines\tttf +\getbuffer +\stoplines + +\page + +You can do this but it's not always predictable (depends on outer scales too): + +\startbuffer[definition] +\def\UnKernedTeX + {T% + {\glyph xoffset -.2ex yoffset -.4ex `E}% + {\glyph xoffset -.4ex options "60 `X}} +\stopbuffer + +\startbuffer[example] +We use \UnKernedTeX\ and {\bf \UnKernedTeX} and {\bs \UnKernedTeX}: +the slanted version could use some more left shifting of the E. +\stopbuffer + +\typebuffer[definition,example] + +\startnarrower + \getbuffer[definition,example] +\stopnarrower + +Marks and cursive features can interfere, so this is an alternative: + +\startbuffer[definition] +\def\UnKernedTeX + {T\glyph left .2ex raise -.4ex `E\glyph left .2ex `X\relax} +\stopbuffer + +\typebuffer[definition] + +\startnarrower + \getbuffer[definition,example] +\stopnarrower + +\page + +Yet another glyph option: + +\starttyping +\multiply\glyphscale by 2 <{\darkgray \glyph `M}> +\multiply\glyphscale by 2 <{\darkgray \glyph raise 3pt `M}> +\multiply\glyphscale by 2 <{\darkgray \glyph raise -3pt `M}> +\multiply\glyphscale by 2 <{\darkgray \glyph left 3pt `M}> +\multiply\glyphscale by 2 <{\darkgray \glyph right 2pt `M}> +\multiply\glyphscale by 2 <{\darkgray \glyph left 3pt right 2pt `M}> +\multiply\glyphscale by 2 <{\darkgray \glyph left -3pt `M}> +\multiply\glyphscale by 2 <{\darkgray \glyph right -2pt `M}> +\multiply\glyphscale by 2 <{\darkgray \glyph left -3pt right -2pt `M}> +\stoptyping + +\startlinecorrection +\bTABLE[align=middle,width=.33\textwidth] + \bTR + \bTD \showglyphs \multiply\glyphscale by 2 <{\darkgray \glyph `M}>\eTD + \bTD \showglyphs \multiply\glyphscale by 2 <{\darkgray \glyph raise 3pt `M}>\eTD + \bTD \showglyphs \multiply\glyphscale by 2 <{\darkgray \glyph raise -3pt `M}>\eTD + \eTR + \bTR[frame=off] + \bTD \tttf \eTD + \bTD \tttf raise 3pt \eTD + \bTD \tttf raise -3pt \eTD + \eTR + \bTR + \bTD \showglyphs \multiply\glyphscale by 2 <{\darkgray \glyph left 3pt `M}>\eTD + \bTD \showglyphs \multiply\glyphscale by 2 <{\darkgray \glyph right 2pt `M}>\eTD + \bTD \showglyphs \multiply\glyphscale by 2 <{\darkgray \glyph left 3pt right 2pt `M}>\eTD + \eTR + \bTR[frame=off] + \bTD \tttf left 3pt \eTD + \bTD \tttf right 2pt\eTD + \bTD \tttf left 3pt right 2pt\eTD + \eTR + \bTR + \bTD \showglyphs \multiply\glyphscale by 2 <{\darkgray \glyph left -3pt `M}>\eTD + \bTD \showglyphs \multiply\glyphscale by 2 <{\darkgray \glyph right -2pt `M}>\eTD + \bTD \showglyphs \multiply\glyphscale by 2 <{\darkgray \glyph left -3pt right -2pt `M}>\eTD + \eTR + \bTR[frame=off] + \bTD \tttf left -3pt \eTD + \bTD \tttf right -2pt \eTD + \bTD \tttf left -3pt right -2pt \eTD + \eTR +\eTABLE +\stoplinecorrection + +\page + +A larger example: + +\startbuffer +\glyphscale 4000 +\vl\glyph `M\vl\quad +\vl\glyph raise .2em `M\vl\quad +\vl\glyph left .3em `M\vl\quad +\vl\glyph right .2em`M\vl\quad +\vl\glyph left -.2em right -.2em`M\vl\quad +\vl\glyph raise -.2em right .4em`M\vl +\stopbuffer + +\typebuffer + +{\getbuffer} + +\stoptitle + +\stopdocument diff --git a/doc/context/presentations/context/2021/context-2021-localcontrol.pdf b/doc/context/presentations/context/2021/context-2021-localcontrol.pdf Binary files differnew file mode 100644 index 000000000..f9f96aa41 --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-localcontrol.pdf diff --git a/doc/context/presentations/context/2021/context-2021-localcontrol.tex b/doc/context/presentations/context/2021/context-2021-localcontrol.tex new file mode 100644 index 000000000..72c1114be --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-localcontrol.tex @@ -0,0 +1,369 @@ +% language=us + +% TAKE FROM LOWLEVEL-EXPANSION.TEX + +\usemodule[system-tokens] + +\usemodule[present-boring,abbreviations-logos] + +\definehighlight[nb][style=bold,color=middlegray,define=no] + +\startdocument + [title={PROGRAMMING}, + banner={local control}, + location={context\enspace {\bf 2021}\enspace meeting}] + +\starttitle[title=Expansion] + +\TEX\ can be in several so called input reading modes: + +Users mostly see it reading from the source file(s). Characters are picked up and +interpreted. Depending on what token it becomes some action takes place. + +\starttyping +1 \setbox0\hbox to 10pt{2} \count0=3 \the\count0 \multiply\count0 by 4 +\stoptyping + +\startitemize + +\startitem + The \type {1} gets typeset because characters like that are seen as text. +\stopitem + +\startitem + The \type {\setbox} primitive triggers picking up a register number, then + goes on scanning for a box specification and that itself will typeset a + sequence of whatever until the group ends. +\stopitem + +\startitem + The \type {count} primitive triggers scanning for a register number (or + reference) and then scans for a number; the equal sign is optional. +\stopitem + +\startitem + The \type {the} primitive injects some value into the current input stream + (it does so by entering a new input level). +\stopitem + +\startitem + The \type {multiply} primitive picks up a register specification and + multiplies that by the next scanned number. The \type {by} is optional. +\stopitem + +\startitem + Printing from \LUA\ and scanning tokens with e.g.\ \type {\scantokens} is like + reading (pseudo) files. +\stopitem + +\stopitemize + +\stoptitle + +\starttitle[title=Expansion] + +\startbuffer[def] +\def\TestA {1 \setbox0\hbox{2} \count0=3 \the\count0} +\stopbuffer + +\startbuffer[edef] +\edef\TestB{1 \setbox0\hbox{2} \count0=3 \the\count0} +\stopbuffer + +\typebuffer[def] +\typebuffer[edef] + +\getbuffer[def] +\getbuffer[edef] + +\blank[2*big] + +\startcolumns[n=2] \switchtobodyfont[8pt] +\luatokentable\TestA +\column +\luatokentable\TestB +\stopcolumns + +\stoptitle + +\starttitle[title=Local control] + +\startbuffer[edef] +\edef\TestB{1 \setbox0\hbox{2} \count0=3 \the\count0} +\stopbuffer + +\startbuffer[ldef] +\edef\TestC{1 \setbox0\hbox{2} \localcontrolled{\count0=3} \the\count0} +\stopbuffer + +\typebuffer[edef] +\typebuffer[ldef] + +\getbuffer[edef] +\getbuffer[ldef] + +\blank[2*big] + +\startcolumns[n=2] \switchtobodyfont[8pt] +\luatokentable\TestB +\column +\luatokentable\TestC +\stopcolumns + +\stoptitle + +\starttitle[title=Side effects] + +\startbuffer[edef] +\edef\TestB{1 \setbox0\hbox{2} \count0=3 \the\count0} +\stopbuffer + +\startbuffer[ldef] +\edef\TestD{\localcontrolled{1 \setbox0\hbox{2} \count0=3 \the\count0}} +\stopbuffer + +\typebuffer[edef] +\typebuffer[ldef] + +\getbuffer[edef]\getbuffer[ldef]\quad{\darkgray\leftarrow\space Watch how the results end up here!} + +\blank[2*big] + +\startcolumns[n=2] \switchtobodyfont[8pt] +\luatokentable\TestB +\column +\luatokentable\TestD +\stopcolumns + +\stoptitle + +\starttitle[title=Usage] + +\startbuffer[def] +\def\WidthOf#1% + {\beginlocalcontrol + \setbox0\hbox{#1}% + \endlocalcontrol + \wd0 } +\stopbuffer + +\startbuffer[use] +\scratchdimen\WidthOf{The Rite Of Spring} + +\the\scratchdimen +\stopbuffer + +\typebuffer[def] +\typebuffer[use] + +\getbuffer[def]\getbuffer[use] + +\stoptitle + +\starttitle[title=Not always pretty] + +\startbuffer[def] +\def\WidthOf#1% + {\dimexpr + \beginlocalcontrol + \begingroup + \setbox0\hbox{#1}% + \expandafter + \endgroup + \expandafter + \endlocalcontrol + \the\wd0 + \relax} +\stopbuffer + +\startbuffer[use] +\scratchdimen\WidthOf{The Rite Of Spring} + +\the\scratchdimen +\stopbuffer + +\typebuffer[def] +\typebuffer[use] + +\getbuffer[def]\getbuffer[use] + +\stoptitle + +\starttitle[title=The \LUA\ end] + +Right from the start the way to get something into \TEX\ from \LUA\ has been the +print functions. But we can also go local (immediate). There are several methods: + +\startitemize +\startitem via a set token register \stopitem +\startitem via a defined macro \stopitem +\startitem via a string \stopitem +\stopitemize + +Among the things to keep in mind are catcodes, scope and expansion (especially in +when the result itself ends up in macros). + +\stoptitle + +\starttitle[title=Via a token register] + +% runlocal : macro|toks expand group + +\startbuffer[set] +\toks0={\setbox0\hbox{The Rite Of Spring (Igor Stravinsky)}} +\toks2={\setbox0\hbox{The Rite Of Spring (Joe Parrish)}} +\stopbuffer + +\typebuffer[set] + +\startbuffer[run] +\startluacode +tex.runlocal(0) context("[1: %p]",tex.box[0].width) +tex.runlocal(2) context("[2: %p]",tex.box[0].width) +\stopluacode +\stopbuffer + +\typebuffer[run] + +\start \getbuffer[set,run] \stop + +\stoptitle + +\starttitle[title=Via a token macro] + +\startbuffer[set] +\def\TestA{\setbox0\hbox{The Rite Of Spring (Igor Stravinsky)}} +\def\TestB{\setbox0\hbox{The Rite Of Spring (Joe Parrish)}} +\stopbuffer + +\typebuffer[set] + +\startbuffer[run] +\startluacode +tex.runlocal("TestA") context("[3: %p]",tex.box[0].width) +tex.runlocal("TestB") context("[4: %p]",tex.box[0].width) +\stopluacode +\stopbuffer + +\typebuffer[run] + +\start \getbuffer[set,run] \stop + +\stoptitle + +\starttitle[title=Via a string] + +\startbuffer[run] +\startluacode +tex.runstring([[\setbox0\hbox{The Rite Of Spring (Igor Stravinsky)}]]) + +context("[5: %p]",tex.box[0].width) + +tex.runstring([[\setbox0\hbox{The Rite Of Spring (Joe Parrish)}]]) + +context("[6: %p]",tex.box[0].width) +\stopluacode +\stopbuffer + +\typebuffer[run] + +\start \getbuffer[run] \stop + +\blank[2*big] + +A bit more high level: + +\starttyping +context.runstring([[\setbox0\hbox{(Here \bf 1.2345)}]]) +context.runstring([[\setbox0\hbox{(Here \bf %.3f)}]],1.2345) +\stoptyping + +\stoptitle + +\starttitle[title=Locked in \LUA] + +\startbuffer[run] +\startluacode +token.setmacro("TestX",[[\setbox0\hbox{The Rite Of Spring (Igor)}]]) +tex.runlocal("TestX") +context("[7: %p]",tex.box[0].width) +\stopluacode +\stopbuffer + +\typebuffer[run] + +\start \getbuffer[run] \stop + +\startbuffer[run] +\startluacode +tex.scantoks(0,tex.ctxcatcodes,[[\setbox0\hbox{The Rite Of Spring (Joe)}]]) +tex.runlocal(0) +context("[8: %p]",tex.box[0].width) +\stopluacode +\stopbuffer + +\typebuffer[run] + +\start \getbuffer[run] \stop + +\stoptitle + +\starttitle[title=Order matters] + +A lot this relates to pushing stuff into the input which is stacked. Compare: + +\startbuffer[run] +\startluacode +context("[HERE 1]") +context("[HERE 2]") +\stopluacode +\stopbuffer + +\typebuffer[run] + +\start \getbuffer[run] \stop + +with this: + +\startbuffer[run] +\startluacode +tex.pushlocal() context("[HERE 1]") tex.poplocal() +tex.pushlocal() context("[HERE 2]") tex.poplocal() +\stopluacode +\stopbuffer + +\typebuffer[run] + +\start \getbuffer[run] \stop + +% \startluacode +% tex.runlocal(function() context("[Here 1]") end) +% context("[Here 12]") +% tex.runlocal(function() context("[Here 2]") end) +% \stopluacode + +% tex.pushlocal % swaps order +% tex.poplocal +% tex.quitlocal + +% token.expandmacro + +% str : serialized +% true : wrap next {} +% table : {serialzed} {serialzed} {serialzed} +% token : inject +% number: catcode table + +% tex.expandasvalue: +% (kind, like expand_macro) + +% tex.runstring: +% [catcode] string expand grouped + +% tex.runlocal +% function|number(register)|string(macro)|userdata(token) / expand / grouped + +% mplib.expandtex +% mpx, kind, macro, arguments + +\stopdocument diff --git a/doc/context/presentations/context/2021/context-2021-luametafun.pdf b/doc/context/presentations/context/2021/context-2021-luametafun.pdf Binary files differnew file mode 100644 index 000000000..25c7b95a4 --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-luametafun.pdf diff --git a/doc/context/presentations/context/2021/context-2021-luametafun.tex b/doc/context/presentations/context/2021/context-2021-luametafun.tex new file mode 100644 index 000000000..333c68682 --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-luametafun.tex @@ -0,0 +1,362 @@ +% language=us + +\usemodule[present-boring,abbreviations-logos] + +\setuptolerance[verytolerant,stretch] + +\startdocument + [title={LUAMETAFUN}, + banner={the new interfaces}, + location={context\enspace {\bf 2021}\enspace meeting}] + +\starttitle[title=Three subsystems] + +\startitemize + +\startitem + The core of the engine in still \TEX. It all starts there. +\stopitem + +\startitem + The \TEX\ internals are opened up via \LUA. We can call out to \LUA\ from + \TEX\ and to some extend from \LUA\ to \TEX. Quite often we push back + something via the input channels. +\stopitem + +\startitem + The \METAPOST\ library is accessed from \LUA. So from within \TEX\ there is + always \LUA\ in between. Results go back via \LUA. The library can also call + out to \LUA\ and from there to \TEX. +\stopitem + +\startitem + This means that all three major components of \LUAMETATEX\ can talk to each + other and use each others capabilities. +\stopitem + +\startitem + With \LUA\ in the center, we also have access to other mechanism, for instance + fonts, graphics and libraries. +\stopitem + +\startitem + In \CONTEXT\ the \LUA\ language also permits using \XML, \JSON, \CSV, \SQL\ + databases and other input that can be dealt with programmatically. +\stopitem + +\startitem + All has been reasonably optimized for efficiency and performance. +\stopitem + +\stopitemize + +\stoptitle + +\starttitle[title=The (\LUATEX) library] + +\startitemize + +\startitem + Turning \METAPOST\ into a library has been a subproject of the \LUATEX\ project. The + semi|-|official team (Taco, Jacko, Hans, Luigi) got John Hobbies blessing. +\stopitem + +\startitem + This was a rather massive (and impressive) operation by Taco because multiple + number models were to be supported and the internals had to be made such that + different backends were possible. All with remaining perfect (DEK) compatibility. +\stopitem + +\startitem + The \METAPOST\ library serves both the stand alone program and \LUATEX. +\stopitem + +\startitem + That means the \POSTSCRIPT\ backend is built in plus some basic (\TYPEONE) + font handling. We support \PDF\ output via the \METAPOST\ \LUA\ backend (in + \MKII\ that is done by parsing the \POSTSCRIPT\ and specials). +\stopitem + +\startitem + In addition there is \PNG\ and \SVG\ output. It helps that \METAPOST\ output is + rather simple. +\stopitem + +\startitem + The \LUATEX\ engine uses the \LUA\ backend which represents the result in \LUA\ tables + resembling the \METAPOST\ internal representation. +\stopitem + +\startitem + The library supports scaled and double (internal) but also binary and decimal + number models that use (linked in) libraries. +\stopitem + +% this would force a page +% +% \startitem +% The library is quite stable and Taco transferred maintenance to Luigi. +% \stopitem + +\stopitemize + +\stoptitle + +\starttitle[title=The (\LUAMETATEX) library] + +\startitemize + +\startitem + We don't need the \POSTSCRIPT\ backend (which only does \TYPEONE\ anyway). +\stopitem + +\startitem + We also have no use for \SVG\ and \PNG\ output. +\stopitem + +\startitem + The binary number model has no advantages over the decimal one but brings + quite some dependency with it (library code). +\stopitem + +\startitem + The \TYPEONE\ font support is not used in \CONTEXT\ because we handle text + differently. +\stopitem + +\startitem + All this means that we can do with a smaller (simplified) \METAPOST\ library. +\stopitem + +\startitem + The codebase has been overhauled. We still have \type {.w} files (\CWEB) but + use a \LUA\ script to convert that to \CCODE\ which means that we have better + control over how it comes out. +\stopitem + +\startitem + As with \LUATEX\ the file \IO, message handling etc.\ now largely goes via \LUA; + it is more integrated. +\stopitem + +\startitem + The same is true for scanning interfaces and return values (injectors). That also + made for more symbolic coding. +\stopitem + +\startitem + Memory management (allocation) is under engine control (as with \TEX\ and \LUA); + we use a common high performance allocator library. +\stopitem + +\stopitemize + +\stoptitle + +\starttitle[title=The (\LUAMETATEX) library] + +\startitemize + +\startitem + Some already present mechanism have been extended, for instance clips have + pre- and postscripts. +\stopitem + +\startitem + A grouping wrapper has been added (handy for some graphic trickery supported in the + backend.) +\stopitem + +\startitem + The \type {runscript} primitive supports symbolic references to functions (of course to + be provided at the \LUA\ end). +\stopitem + +\startitem + The \type {runscript} return values can be more native, in addition to the already + present (default) \type {scantokens} support. +\stopitem + +\startitem + Internals are extended with booleans and strings. +\stopitem + +\startitem + Output (paths, clips etc) can be stacked in a different order. +\stopitem + +\startitem + There are additional statistics available. +\stopitem + +\startitem + In some places performance could be improved. +\stopitem + +\startitem + In the meantime it can be considered a major upgrade and (for various + reasons) backporting to \LUATEX\ makes no sense. And yes, all errors are + mine. +\stopitem + +\stopitemize + +\starttitle[title=The \LUA fication] + + {\em See Taco's presentation where he gives some examples.} + +\stoptitle + +\starttitle[title=Callbacks] + +We need to hook in some functions: \blank[2*big] + +\starttabulate[|T|T|T|p|] + \HL + \NC \NC find_file \NC (name,mode,kind) \NC locate a file (usually within the \TDS\ setup) \NC \NR + \NC f \NC open_file \NC (name,mode,kind) \NC open given file \NC \NR + \NC \NC close_file \NC (handle) \NC close opened file \NC \NR + \NC s \NC read_file \NC (handle,size) \NC read from file \NC \NR + \NC \NC write_file \NC (handle,str) \NC write to file \NC \NR + \HL + \NC s \NC run_script \NC (code,size,index) \NC run the given string as \LUA\ script \NC \NR + \NC s \NC make_text \NC (str,size,mode) \NC process btex/etex \NC \NR + \HL + \NC \NC run_internal \NC (action,index,kind,name) \NC act on internal definition \NC \NR + \HL + \NC n \NC run_overload \NC (property,name,mode) \NC process overload check \NC \NR + \HL + \NC \NC run_logger \NC (target,str,size) \NC process log message \NC \NR + \NC \NC run_error \NC (message,help,interaction) \NC handle error (message) \NC \NR + \NC \NC run_warning \NC (message) \NC handle warning \NC \NR + \HL +\stoptabulate + +% check_overload shipout_backend + +\stoptitle + +\starttitle[title=Two calling methods] + +The runner can be called as: + +\starttyping +runscript("mp.MyFunction()") +\stoptyping + +which implies at the \LUA\ end: + +\starttyping +function mp.MyFunction() + ... +end +\stoptyping + +Here the callback function is responsible for loading the string and executing it. + +Alternatively one can say: + +\starttyping +runscript <number> +\stoptyping + +The number can be intercepted at the \LUA\ end to do some associated action. + +\stoptitle + +\starttitle[title=Variables] + +We can do: + +\starttyping +lua.mp.MyFunction("foo",123,true) +\stoptyping + +which in the end is equivalent to: + +\starttyping +runscript("mp.MyFunction('foo',123,true)") +\stoptyping + +Alternatively one can pick up values by scanning: like \type {scannext}, \type +{scanexpression}, \type {scantoken}, \type {scansymbol}, \type {scannumeric}, +\type {scaninteger}, \type {scanboolean}, \type {scanstring}, \type {scanpair}, +\type {scancolor}, \type {scancmykcolor}, \type {scantransform}, \type +{scanpath}, \type {scanpen}, etc. + +\stoptitle + +\starttitle[title=Return values] + +The runner can return: + +\startitemize + +\startitem + a string that gets fed into the \type {scantokens} primitive +\stopitem + +\startitem + a numeric or boolean that gets injected as native \METAPOST\ object +\stopitem + +\startitem + a table that gets concatenated and fed into the \type {scantokens} primitive +\stopitem + +\startitem + \type {true} and a second argument that gets converted into a native \METAPOST\ object +\stopitem + +\startitem + in the last case the number of table elements determines the object +\stopitem + +\stopitemize + +Instead of returning a value one can inject: \type {injectnumeric}, \type +{injectinteger}, \type {injectboolean}, \type {injectstring}, \type {injectpair}, +\type {injectcolor}, \type {injectcmykcolor}, \type {injecttransform}, \type +{injectpath}, \type {injectwhatever}, etc.\ and these accept one or more values +and|/|or tables. + +These mechanisms might evolve a bit over time. Lots of examples can be found in +the \type {mlib-*.lmt} files. + +\stoptitle + +\starttitle[title=Parameters] + +\startitemize + \startitem + The new interfaces permit us to program quite robust parameter driven + interfaces that (sort of) match the way we do things at the \TEX\ end. + \stopitem + \startitem + The distribution has several examples of usage and more will be added. + \stopitem + \startitem + Macros that use the new mechanisms can be recognized by the \type {lmt_} + prefix. + \stopitem +\stopitemize + +\starttyping +lmt_mytrick [ + somestring = "test", + somenumeric = 123, + someboolean = true, + somecolor = (1, 0, 1), + somepath = fullsquare scaled 10cm, + somelist = { (0, 0), (1, 3), (8, 9) }, + sometable = [ + somenumeric = 321, + ], +] ; +\stoptyping + +{\em Show the pattern of defining these at the \LUA\ end and in \METAPOST\ files.} + +\stoptitle + +\stopdocument diff --git a/doc/context/presentations/context/2021/context-2021-math.pdf b/doc/context/presentations/context/2021/context-2021-math.pdf Binary files differnew file mode 100644 index 000000000..dd343899b --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-math.pdf diff --git a/doc/context/presentations/context/2021/context-2021-math.tex b/doc/context/presentations/context/2021/context-2021-math.tex new file mode 100644 index 000000000..fce4ca4d5 --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-math.tex @@ -0,0 +1,236 @@ +% language=us + +\usemodule[present-boring,abbreviations-logos] + +\definecolor[maincolor] [r=.4] +\definecolor[extracolor][s=.1] + +\startdocument + [title={MATH}, + banner={complete control}, + location={context\enspace {\bf 2021}\enspace meeting}] + +\starttitle[title=The benchmark] + +\startitemize + +\startitem + Traditional \TEX\ has set the benchmark for typesetting math. +\stopitem + +\startitem + It uses three categories of fonts: alphabet, symbol, extensible. +\stopitem + +\startitem + Limitations in fonts have lead to some curious handling of dimensions. +\stopitem + +\startitem + The fact that there is \type {\over} makes for multipass scanning and + processing. +\stopitem + +\startitem + The last family counts (when entering the second pass) so one has to use very + controlled font switching. +\stopitem + +\startitem + Some features (limits, integrals, primes) rely on special macros and parsing. +\stopitem + +\startitem + One can end up in tricky font switching. Also, quite some fonts are loaded in + order to set up the machinery. +\stopitem + +\stopitemize + +\stoptitle + +\starttitle[title=Todays reality] + +\startitemize + +\startitem + We now use \OPENTYPE\ math fonts. In fact, in \MKIV\ we always used only one + font instance (unless we mix font families). +\stopitem + +\startitem + We operate in the \UNICODE\ domain, so in the end there was no need to bump the + number of families in \LUATEX. +\stopitem + +\startitem + In \CONTEXT\ we use only a few font families: regular, regular r2l, bold, bold r2l. +\stopitem + +\startitem + All the tricky stuff is done with the help of \LUA. This already started very + early in the \LUATEX\ project and has not changed. +\stopitem + +\startitem + But we still run into issues because the available fonts are inconsistent, + incompatible, have issues and that is unlikely to change. +\stopitem + +\startitem + We have a virtual font system in place that was used during the transition (when no + fonts were available). +\stopitem + +\startitem + In \CONTEXT\ there are various ways to deal with shortcomings in or extensions to fonts. +\stopitem + +\startitem + At some point we need to make up our minds, accept the issues with fonts, and + just fix things runtime. +\stopitem + +\stopitemize + +\stoptitle + +\starttitle[title=The engine] + +\startitemize + +\startitem + In \LUATEX\ we started with a hybrid approach but eventually ended up with + more split code paths. +\stopitem + +\startitem + In \LUAMETATEX\ nearly all aspects of the engine had made configurable and are under + user control. +\stopitem + +\startitem + This permits experiments where we can apply old methods onto new fonts. +\stopitem + +\stopitemize + +\stoptitle + +\starttitle[title=The tricky things] + +\startitemize + +\startitem + Spacing in traditional math is a combination of widths and italic correction: + we need to add them and sometimes afterwards subtract the italic correction. + In \OPENTYPE\ math we don't lie about dimensions and apply italic correction + selectively (as we have staircase kerns). +\stopitem + +\startitem + Special symbols like primes are quite inconsistent wrt dimensions and positioning + and we need to catch that. We also need robust ways to collapse them to the proper + \UNICODE\ symbol. +\stopitem + +\startitem + Larger variants (these \type {\bigg} things) needs some attention too. +\stopitem + +\startitem + We need to map characters onto the right shaped (alphabets) because hardly + anyone will enter the \UNICODE\ math characters directly. +\stopitem + +\startitem + We might want to fix scripts, italics, kern pairs etc. +\stopitem + +\stopitemize + +\stoptitle + +\starttitle[title=Implementation] + +Among the new features (introduced over a period of time) are: + +\startitemize + +\startitem + All kind of inter|-|class spacing parameters (in addition to what \LUATEX\ + already provides). +\stopitem + +\startitem + Dozens of \OPENTYPE\ font related spacing parameters (more that traditional + \TEX). +\stopitem + +\startitem + Opened up additional font (and taste) related parameters (hard coded in + traditional \TEX). +\stopitem + +\startitem + Opened up style related parameters (hard coded in traditional \TEX). +\stopitem + +\startitem + Control codes that make the engine follow a different (the traditional) code + path (which can come in handy when testing or writing manuals). +\stopitem + +\startitem + The ability to simplify the result a bit wrt characters (the engine loves to + box a lot). +\stopitem + +\startitem + Efficient font, style and parameter scaling by reusing fonts and thereby + limiting the number of instances. +\stopitem + +\startitem + More options to special (constructed) symbols. +\stopitem + +\startitem + Let math related nodes carry around more control properties and states. +\stopitem + +\startitem + Support of local changes to math parameters. +\stopitem + +\stopitemize + +\stoptitle + +\stopdocument + +% mml +% vf chars for extensibles + +% noads.handlers.showtree +% noads.handlers.unscript +% noads.handlers.unstack +% noads.handlers.variants +% noads.handlers.relocate +% noads.handlers.families +% noads.handlers.render +% noads.handlers.collapse +% noads.handlers.fixscripts +% noads.handlers.domains +% noads.handlers.autofences +% noads.handlers.resize +% noads.handlers.respace +% noads.handlers.alternates +% noads.handlers.tags +% noads.handlers.italics +% noads.handlers.kernpairs +% noads.handlers.classes + +% builders.kernel.mlisttohlist +% noads.handlers.makeup +% noads.handlers.align diff --git a/doc/context/presentations/context/2021/context-2021-overloadprotection.pdf b/doc/context/presentations/context/2021/context-2021-overloadprotection.pdf Binary files differnew file mode 100644 index 000000000..c6335f32d --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-overloadprotection.pdf diff --git a/doc/context/presentations/context/2021/context-2021-overloadprotection.tex b/doc/context/presentations/context/2021/context-2021-overloadprotection.tex new file mode 100644 index 000000000..5143e855a --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-overloadprotection.tex @@ -0,0 +1,298 @@ +% language=us + +\usemodule[present-boring,abbreviations-logos] + + +\definehighlight[nb][style=bold,color=middlegray,define=no] + +\setuptolerance[verytolerant,stretch] + +\definecolor[maincolor][r=.3,b=.3] + +\startdocument + [title={OVERLOAD PROTECTION}, + banner={the downside of macros}, + location={context\enspace {\bf 2021}\enspace meeting}] + +% TEX MP + +\starttitle[title=Primitives] + +A \TEX\ engine comes with a whole set of primitive operations for: + +\startitemize[packed] + \startitem + accessing internal variables + \stopitem + \startitem + defining macros + \stopitem + \startitem + controlling expansion + \stopitem + \startitem + constructing boxes + \stopitem + \startitem + finalizing pages + \stopitem + \startitem + defining characters (text and math) + \stopitem + \startitem + inserting kerns, glue and penalties + \stopitem + \startitem + defining fonts + \stopitem + \startitem + dealing with languages (hyphenation) + \stopitem + \startitem + testing properties + \stopitem + \startitem + manipulating tokens + \stopitem + \startitem + managing inserts + \stopitem + \startitem + handling marks + \stopitem + \startitem + grouping + \stopitem + \startitem + mathematics + \stopitem + \startitem + tracing + \stopitem +\stopitemize + +\stoptitle + +\starttitle[title=Macros] + +\startitemize + \startitem + Macros are commands defined by the user and/or a macro package. + \stopitem + \startitem + They can overload a primitive which then can confuse the whole machinery. + \stopitem + \startitem + A macro package can alias primitives for instance \type {\relax} can be + replaced by \type {\foo_relax} after \typ {\let \foo_relax \relax}. + \stopitem + \startitem + That only when (at definition time) the \type {_} is a letter. By using such + a character some protection against overload is provided. + \stopitem + \startitem + In \CONTEXT\ we often use(d) aliases like \type {\normalrelax} but of course + these can also be overloaded. + \stopitem + \startitem + We only overload a very few primitives, for instance \type {\language}. + \stopitem + \startitem + Users who overload primitives are \quote {on their own} and \quote {without + support}. + \stopitem + \startitem + An easy way to protect yourself is using \type {\CamelCase} names. + \stopitem +\stopitemize + +\stoptitle + +\starttitle[title=Overload protection] + +\startitemize + \startitem + The \LUAMETATEX\ engine has overload protection built in for the \TEX\ engine + as well as provides means to do that for \METAPOST. + \stopitem + \startitem + In \LMTX\ all commands have been tagged accordingly (which was quite some work). + \stopitem + \startitem + Processing \type {s-system-macros.mkxl} gives an overview. + \stopitem + \startitem + Overload protection is off by default but can be turned on: + +\starttyping +\enabledirectives[overloadmode=warning] +\enabledirectives[overloadmode=error] +\stoptyping + + \stopitem + \startitem + I myself always run with the error variant and make sure that the manuals obey + the rules. + \stopitem + \startitem + Modules and/or styles (and in a few cases the core code) can cheat and use: + +\starttyping +\pushoverloadmode + ........................ + ........................ +\popoverloadmode +\stoptyping + + \stopitem +\stopitemize + +\stoptitle + +\starttitle[title=Details] + +\startitemize + \startitem + Traditional \TEX\ has a few so called prefixes: \type {\global}, \type {\outer}, + \type {\long}, and type \type {\immediate}. + \stopitem + \startitem + The \ETEX\ engine adds \type {\protected} (because we already had that in + \CONTEXT\ we use what we (also already) had: \type {\unexpanded}). + \stopitem + \startitem + In \LUATEX\ we can force macros to be always long, something that we do + in \MKIV\ (as in \MKII). + \stopitem + \startitem + In \LUAMETATEX\ the \type {\outer} and \type {\long} prefixes have been + dropped (they are ignored). + \stopitem + \startitem + In \LUAMETATEX\ the \type {\protected} prefix acts like in other engines + but protection is implemented more naturally. + \stopitem + \startitem + In addition \LUAMETATEX\ has new prefixes: \type {\frozen}, \type + {\permanent}, \type {\immutable}, \type {\mutable}, \type {\noaligned}, + \type {\instance}, \type {\untraced}, \type {\tolerant}, \type + {\overloaded}, \type {\aliased}, \type {\immediate} and an experimental + \type {\semiprotected}, + \stopitem + \startitem + Some prefixes end up as properties of macros, some influence scanning (for instance + in alignments and when calling \LUA\ functions). There is no noticeable runtime + overhead. + \stopitem + \startitem + The \type {\meaningfull} primitive is like \type {\meaning} but also reports + properties set by prefixes; there is also \type {\meaningless}. + \stopitem +\stopitemize + +\stoptitle + +\starttitle[title=Prefixes] + +Regular definitions: + +\startitemize + \startitem + \type {\global}: defines a macro or sets a register value out of scope. + \stopitem + \startitem + \type {\outer}: is used to issue a warning when a macro defined as such was + used nested (just ignored in \LUAMETATEX). + \stopitem + \startitem + \type {\long}: triggers a warning when an argument of a macro contains a + \type {\par} equivalent (just ignored in \LUAMETATEX). + \stopitem + \startitem + \type {\protected}: makes a macro unexpandable (being itself) in an \type + {\edef} equivalent situation (where it can get out of hands). + \stopitem + \startitem + \type {\semiprotected}: is like \type {\protected} but the property is + ignored when \type {\semiexpanded} is applied. + \stopitem +\stopitemize + +Special case: + +\startitemize + \startitem + \type {\immediate}: tells a backend primitive to come into action + immediately instead of creating a delayed action (via a whatsit node). In + \LUAMETATEX\ we have no built|-|in backend so there is signals a \LUA\ + interface function this property. + \stopitem +\stopitemize + +\page + +Scanning related: + +\startitemize + \startitem + \type {\noaligned}: tags a macro as valid peek ahead macro when scanning + alignments; normally that only makes sense for \type {\protected} macros. + This permits cleaner macros in for instance table mechanisms (no + unexpected expansion side effects). + \stopitem + \startitem + \type {\untraced}: this flag makes a macro report itself as primitive in + traces which is sometimes nicer that presenting a user with some + confusing meaning. + \stopitem + \startitem + \type {\tolerant}: a prefix that makes the macro argument parser accept + all kind of new argument parsing options and continue when delimited + arguments fail. This makes macros with optional arguments produce less + noise when they are traced but more important, it makes for cleaner low + level interfaces. + \stopitem +\stopitemize + + +\page + +Overload protection (primitives are protected against overloads by default): + +\startitemize + \startitem + \type {\aliased}: create a reference (using \type {\let}) that also inherits the + properties. + \stopitem + \startitem + \type {\permanent}: sets a flag that makes a macro (or definition) immune for + redefinition. + \stopitem + \startitem + \type {\frozen}: prevents overloading but one can bypass that with some + more effort. + \stopitem + \startitem + \type {\immutable}: makes a (normally variable definition) fixed, for instance + constants. + \stopitem + \startitem + \type {\mutable}: a flag showing that this macro or definition can be used for + anything (so the macro package also has to assume that). + \stopitem + \startitem + \type {\instance}: just a flag that can be used to signal that a macro (or definition) + is an instance of a more general concept. + \stopitem + \startitem + \type {\overloaded}: bypass a frozen flag (property). + \stopitem +\stopitemize + +\blank[2*big] + +{\em Show some examples in the source code and editor.} + +\stoptitle + +\stopdocument diff --git a/doc/context/presentations/context/2021/context-2021-paragraphs.pdf b/doc/context/presentations/context/2021/context-2021-paragraphs.pdf Binary files differnew file mode 100644 index 000000000..9b32a6576 --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-paragraphs.pdf diff --git a/doc/context/presentations/context/2021/context-2021-paragraphs.tex b/doc/context/presentations/context/2021/context-2021-paragraphs.tex new file mode 100644 index 000000000..7a4e566d8 --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-paragraphs.tex @@ -0,0 +1,162 @@ +% language=us + +\usemodule[present-boring,abbreviations-logos] + +\definecolor[maincolor] [r=.4,g=.4] +\definecolor[extracolor][s=.1] + +\startdocument + [title={PARAGRAPHS}, + banner={a bit of an upgrade}, + location={context\enspace {\bf 2021}\enspace meeting}] + +\starttitle[title=Note] + +\startitemize + +\startitem + Some of the following already is present for a while and has been discussed + at previous meetings. +\stopitem + +\startitem + But \unknown\ occasionally some minor tweak gets added so consider this to be + an update. +\stopitem + +\stopitemize + +\starttitle[title=Spacing] + +\startitemize + +\startitem + Spaces in \TEX\ become glue nodes (with optional stretch and shrink). +\stopitem + +\startitem + In traditional \TEX\ these glue nodes are ref counted copies of the current + spacing related variables. +\stopitem + +\startitem + In \LUATEX\ we make real copies so that when we mess with the node list + changes to glue don't affect other instances. +\stopitem + +\stopitemize + +\starttitle[title=Parameters] + +\startitemize + +\startitem + In traditional \TEX\ the paragraphs bound properties that are in effect + when \type {\par} happens are used when breaking into lines. +\stopitem + +\startitem + In \LUAMETATEX\ the paragraphs bound properties are stored with the + paragraph and can be frozen when they are set. +\stopitem + +\startitem + This gives a more predictable (and robust) way of manipulating a + paragraph. +\stopitem + +\startitem + We can for instance get rid of grouping side effects that interfere with + \type {\everypar}. +\stopitem + +\startitem + Currently three dozen parameters are tracked but they are grouped in + categories. +\stopitem + +\blank[2*big] {\em (show code and examples)} + +\stopitemize + +\stoptitle + +\starttitle[title=Wrapping] + +\startitemize + +\startitem + Doing something in front of a paragraph is taken care of by good old + \type {\everypar}. +\stopitem + +\startitem + In \LUAMETATEX\ we also have \type {\everybeforepar} but so far in \CONTEXT\ + we haven't used that. +\stopitem + +\startitem + Adding something to the end of a paragraph can be tricky so we have + a wrapper mechanism: \type {\wrapuppar}. +\stopitem + +\startitem + The \type {\wrapuppar} primitive is similar to \type {\atendofgroup} in the + sense that it accumulates tokens (so no \type {\endofpar}). +\stopitem + +\startitem + Normally these primitives are not used directly but managed by a more general + system of handling paragraphs. +\stopitem + +\stopitemize + +\blank[2*big] {\em (show code and examples)} + +\stoptitle + +\starttitle[title=Normalizing] + +\startitemize + +\startitem + In order to see consistent paragraphs at the \LUA\ end in \LUAMETATEX\ + we can normalize the lines that come from the paragraph builder. +\stopitem + +\startitem + Normalization results in: + + \startitemize + \startitem + the first line having: indent skip + \stopitem + \startitem + each line having: left hang, left skip, right skip, right hang + \stopitem + \startitem + the last line having: left parfill skip, right parfill skip + \stopitem + \stopitemize +\stopitem + +\startitem + It is controlled by \type {\normalizelinemode} which has additional flags for + swapping hanging indentation and par shapes, breaking after dir nodes, + removing margin kerns and clipping the line width. +\stopitem + +\startitem + The clipping options avoids the side effects of \TEX\ using shifts which has + the side effect of unreal dimensions. This is one of the tricks|/|properties + of the traditional engine that is perfectly fine until we open up things. +\stopitem + +\stopitemize + +\blank[2*big] {\em (show code and examples)} + +\stoptitle + +\stopdocument diff --git a/doc/context/presentations/context/2021/context-2021-programming.pdf b/doc/context/presentations/context/2021/context-2021-programming.pdf Binary files differnew file mode 100644 index 000000000..86f61bed9 --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-programming.pdf diff --git a/doc/context/presentations/context/2021/context-2021-programming.tex b/doc/context/presentations/context/2021/context-2021-programming.tex new file mode 100644 index 000000000..907a9ebfa --- /dev/null +++ b/doc/context/presentations/context/2021/context-2021-programming.tex @@ -0,0 +1,325 @@ +% language=us + +\usemodule[present-boring,abbreviations-logos] + +\definehighlight[nb][style=bold,color=middlegray,define=no] + +\startdocument + [title={PROGRAMMING}, + banner={the way \CONTEXT\ is set up}, + location={context\enspace {\bf 2021}\enspace meeting}] + +\starttitle[title=Levels] + +When you look at \CONTEXT\ bottom|--|up (engine|--|interface) you will notice: \blank[2*big] + +\startnarrower + +\startitemize[n] + +\startitem + \highlight[nb]{primitives:} this is what the engine comes with +\stopitem +\startitem + \highlight[nb]{infrastructure:} basic management of data structures +\stopitem +\startitem + \highlight[nb]{helpers:} macros that hide complexity +\stopitem +\startitem + \highlight[nb]{subsystems:} collections of macros that implement functionality +\stopitem +\startitem + \highlight[nb]{mechanisms:} these combine various subsystems +\stopitem +\startitem + \highlight[nb]{modules:} extra functionality (uses 1--5) +\stopitem +\startitem + \highlight[nb]{styles:} handling sources and layout (uses 4--6) +\stopitem + +\stopitemize + +\stopnarrower + +\blank[2*big] Users normally see \CONTEXT\ top|--|down (usage|--|hacking). + +\stoptitle + +% % % + +\starttitle[title=Styles] + +\startitemize + +\startitem + These are prebuilt solutions for common as well as rare situations. +\stopitem + +\startitem + The system comes with some styles: the \type {s-*} files. +\stopitem + +\startitem + Right from the start the idea was that you get some reasonable default. +\stopitem + +\startitem + And if you want more you stepwise define your style as you go. +\stopitem + +\startitem + It really is part of the game: exploration. +\stopitem + +\startitem + Solving your problem is a nice challenge. +\stopitem + +\startitem + If you want a completely predefined setup, shop somewhere else. +\stopitem + +\stopitemize + +\stoptitle + +% % % + +\starttitle[title=Modules] + +\startitemize + +\startitem + A lot of functionality is built in. +\stopitem + +\startitem + This helps to keep the system consistent. +\stopitem + +\startitem + We could cheat and ship thousands of few|--|liner styles but don't do that. +\stopitem + +\startitem + There are a few mechanisms that don't really fit into the core, so these are + implemented as modules that do fit into the interface: the \type {m-*} and + \type {x-*} files. +\stopitem + +\startitem + Users can build and share their solutions which has resulted in some + third party modules: the \type {t-*} files. +\stopitem + +\startitem + For (a few, often old) private files I use \type {p-*} name scheme. +\stopitem + +\stopitemize + +\stoptitle + +% % % + +\starttitle[title=Mechanisms] + +\startitemize + +\startitem + These are combinations of subsystems but often they cannot really be distinguished. +\stopitem + +\startitem + Examples are notes, that combine notations, lists, references, descriptions etc. +\stopitem + +\stopitemize + +\stoptitle + +% % % + +\starttitle[title=Subsystems] + +\startitemize + +\startitem + This is what users see and can configure +\stopitem + +\startitem + Most are (conceptually) rather old but evolved over time. There are no + fundamental differences between \MKIV\ and \LMTX, but the later is hopefully + a bit cleaner. +\stopitem + +\startitem + Examples are fonts, languages, color, structure (sectioning, lists, + constructions, itemgroups, references), spacing, graphics, bibliographies, + positioning, numbering and layout. +\stopitem + +\startitem + More hidden are the backend, export and \XML\ interfaces. +\stopitem + +\startitem + Some have subsystems themselves, like widgets that relate to a specific + backend. +\stopitem + +\startitem + There are often dependencies between subsystems which makes that it's not really a + hierarchy. A more strict separation would demand much more overhead. +\stopitem + +\stopitemize + +\stoptitle + +% % % + +\starttitle[title=Helpers] + +\startitemize + +\startitem + These provide basic programming help. +\stopitem + +\startitem + Examples are macros for comparing things, loops, list processing, + argument handling. +\stopitem + +\startitem + But more abstract box manipulations also fits in here. +\stopitem + +\startitem + Some subsystems, like \XML\ and bibliographies provide more specific low + level helpers. +\stopitem + +\stopitemize + +\stoptitle + +% % % + +\starttitle[title=Infrastructure] + +\startitemize + +\startitem + The engine provides counters, dimension and other registers that need + to be managed in order to avoid clashes in usage. +\stopitem + +\startitem + Many of the helpers, subsystems and mechanisms fall back on common rather + low level functions (\LUA) and macros (using primitives). +\stopitem + +\stopitemize + +\stoptitle + +% % % + +\starttitle[title=Primitives] + +\startitemize + +\startitem + This is what the engine provides: the built|-|in commands and features. +\stopitem + +\startitem + In addition to the visible primitives there are \LUA\ interfaces and these + permit adding extra primitives. +\stopitem + +\startitem + In \LUAMETATEX\ we have the core \TEX\ set but a few were dropped because we + don't have a backend and a different \IO\ subsystem (so they have to be + emulated). +\stopitem + +\startitem + We also have some of the \ETEX\ primitives and very few of the \PDFTEX\ ones + but I now consider for instance expansion and protrusion extensions to be + kind of \ETEX. +\stopitem + +\startitem + There are additional \LUATEX\ primitives but some were dropped, again because + of the backend, so we emulate some, and also because some were experimental. +\stopitem + +\startitem + There are quite some new primitives and existing mechanisms have been extended, + cleaned up and (hopefully) improved. +\stopitem + +\stopitemize + +% % % + +\starttitle[title=The shift] + +\startitemize + +\startitem + There have always been complaints about \TEX\ as a language (what makes me + wonder why those who complain use it.) +\stopitem + +\startitem + Although there are some extensions to the language in \ETEX, follow|-|ups have + not really succeeded in this area. +\stopitem + +\startitem + At some point I decided that code in the categories 1|--|4 cold benefit from + extensions. +\stopitem + +\startitem + That also meant that we use less of the low helpers. It makes the code look a + bit more \TEX. +\stopitem + +\startitem + It also means less clutter, in code as well in tracing. Often the code + becomes simpler too. +\stopitem + +\startitem + The idea is that \TEX\ becomes a bit more a programming language. +\stopitem + +\startitem + Of course it takes away the \quotation {Watch me, I can do real dirty \TEX\ + hacking!} brawling. +\stopitem + +\startitem + It also can take away some of the complaints. +\stopitem + +\startitem + And it definitely adds some fun. +\stopitem + +\stopitemize + +{\em During the week we show some of the implementation (in Visual Studio) and +examples of applications. We also write a small extension (the {\tttf dk} unit)} + +\stoptitle + +\stopdocument diff --git a/doc/context/scripts/mkiv/mtx-pdf.html b/doc/context/scripts/mkiv/mtx-pdf.html index e724148ca..2932b942c 100644 --- a/doc/context/scripts/mkiv/mtx-pdf.html +++ b/doc/context/scripts/mkiv/mtx-pdf.html @@ -43,6 +43,7 @@ <tr><th>--metadata</th><td></td><td>show metadata xml blob</td></tr> <tr><th>--pretty</th><td></td><td>replace newlines in metadata</td></tr> <tr><th>--fonts</th><td></td><td>show used fonts (--detail)</td></tr> + <tr><th>--object</th><td></td><td>show object"/></td></tr> <tr><th/><td/><td/></tr> </table> <br/> diff --git a/doc/context/scripts/mkiv/mtx-pdf.man b/doc/context/scripts/mkiv/mtx-pdf.man index 1535c8af8..ea74f2aa1 100644 --- a/doc/context/scripts/mkiv/mtx-pdf.man +++ b/doc/context/scripts/mkiv/mtx-pdf.man @@ -22,6 +22,9 @@ replace newlines in metadata .TP .B --fonts show used fonts (--detail) +.TP +.B --object +show object"/> .SH AUTHOR More information about ConTeXt and the tools that come with it can be found at: diff --git a/doc/context/scripts/mkiv/mtx-pdf.xml b/doc/context/scripts/mkiv/mtx-pdf.xml index 9b3d8f8fe..08b04d85e 100644 --- a/doc/context/scripts/mkiv/mtx-pdf.xml +++ b/doc/context/scripts/mkiv/mtx-pdf.xml @@ -12,11 +12,13 @@ <flag name="metadata"><short>show metadata xml blob</short></flag> <flag name="pretty"><short>replace newlines in metadata</short></flag> <flag name="fonts"><short>show used fonts (<ref name="detail)"/></short></flag> + <flag name="object"><short>show object"/></short></flag> </subcategory> <subcategory> <example><command>mtxrun --script pdf --info foo.pdf</command></example> <example><command>mtxrun --script pdf --metadata foo.pdf</command></example> <example><command>mtxrun --script pdf --metadata --pretty foo.pdf</command></example> + <example><command>mtxrun --script pdf --stream=4 foo.pdf</command></example> </subcategory> </category> </flags> diff --git a/doc/context/sources/general/manuals/lowlevel/lowlevel-alignments.tex b/doc/context/sources/general/manuals/lowlevel/lowlevel-alignments.tex index 79fc86aae..1b8bc6552 100644 --- a/doc/context/sources/general/manuals/lowlevel/lowlevel-alignments.tex +++ b/doc/context/sources/general/manuals/lowlevel/lowlevel-alignments.tex @@ -205,7 +205,8 @@ So let's summarize what happens: pending cell spans \stopitem \startitem - flush the result to the current list + flush the result to the current list, unless packages in a box a \type + {\halign} is seen as paragraph and rows as lines (such a table can split) \stopitem \stopitemize @@ -214,28 +215,31 @@ look ahead for a \type {\noalign}, \type {\cr}, \type {\omit} or \type {\span} and when doing that it has to expand what comes. This can give side effects and often results in obscure error messages. When for instance an \type {\if} is seen and expanded, the wrong branch can be entered. And when you use protected macros -embedded alignment commands are not seen at all. Also, nesting \type {\noalign} -is not permitted. +embedded alignment commands are not seen at all; of course they still need to +produce valid operations in the current context. All these side effects are to be handled in a macro package when it wraps alignments in a high level interface and \CONTEXT\ does that for you. But because the code doesn't always look pretty then, in \LUAMETATEX\ the alignment mechanism has been extended a bit over time. -The first extension was to permit nested usage of \type {\noalign}. This has -resulted of a little reorganization of the code. A next extension showed up when -overload protection was introduced and extra prefixes were added. We can signal -the scanner that a macro is actually a \type {\noalign} variant: \footnote {A -better prefix would have been \type {\peekaligned} because in the meantime other -alignment primitives also can use this property.} +Nesting \type {\noalign} is normally not permitted (but one can redefine this +primitive such that a macro package nevertheless handles it). The first extension +permits nested usage of \type {\noalign}. This has resulted of a little +reorganization of the code. A next extension showed up when overload protection +was introduced and extra prefixes were added. We can signal the scanner that a +macro is actually a \type {\noalign} variant: \footnote {One can argue for using +the name \type {\peekaligned} because in the meantime other alignment primitives +also can use this property.} \starttyping[option=TEX] \noaligned\protected\def\InBetween{\noalign{...}} \stoptyping -This extension resulted in a second bit of reorganization (think of internal -command codes and such) but still the original processing of alignments was -there. +Here the \type {\InBetween} macro will get the same treatment as \type {\noalign} +and it will not trigger an error. This extension resulted in a second bit of +reorganization (think of internal command codes and such) but still the original +processing of alignments was there. A third overhaul of the code actually did lead to some adaptations in the way alignments are constructed so let's move on to that. @@ -307,10 +311,13 @@ a test document where you want to show font properties in a table and that font supports a script with some ten thousand glyphs, you can imagine that it accumulates and in \LUATEX\ (and \LUAMETATEX) nodes are larger so it is one of these cases where in \CONTEXT\ we get messages on the console that node memory is -bumped. +bumped. \footnote {I suppose it was a coincidence that a few weeks after these +features came available a user consulted the mailing list about a few thousand +page table that made the engine run out of memory, something that could be cured +by enabling these new features.} After playing a bit with stripping zero tab skips I found that the code would not -really benefit from such a feature: lots of extra tests made if quite ugly. As a +really benefit from such a feature: lots of extra tests made it quite ugly. As a result a first alternative was to just strip zero skips before an alignment got flushed. At least we're then a bit leaner in the processes that come after it. This feature is now available as one of the normalizer bits. @@ -403,7 +410,7 @@ which renders as: A reason to have boxes here is that it enforces a cell width but that is done at the cost of an extra wrapper. In \LUAMETATEX\ the \type {hlist} nodes are rather large because we have more options than in original \TEX, for instance offsets -and orientation. So, in a table with 10K rows of 4 cells yet get 40K extra \type +and orientation. In a table with 10K rows of 4 cells yet get 40K extra \type {hlist} nodes allocated. Now, one can argue that we have plenty of memory but being lazy is not really a sign of proper programming. @@ -501,7 +508,8 @@ In this example we specify the characters in the cells. We still need to add a specifier in the preamble definition because that will trigger the plugin. \startbuffer -\starttabulate[|lG{}|lG{}|] +\starttabulate[|lG{}|rG{}|] +\NC left \NC right \NC\NR \NC \showglyphs \setalignmentcharacter{.}1.1 \NC \setalignmentcharacter{.}1.1 \NC\NR \NC \showglyphs \setalignmentcharacter{,}11,11 \NC \setalignmentcharacter{,}11,11 \NC\NR \NC \showglyphs \setalignmentcharacter{=}111=111 \NC \setalignmentcharacter{=}111=111 \NC\NR @@ -513,7 +521,8 @@ specifier in the preamble definition because that will trigger the plugin. You can mix these approaches: \startbuffer -\starttabulate[|lG{.}|lG{}|] +\starttabulate[|lG{.}|rG{}|] +\NC left \NC right \NC\NR \NC 1.1 \NC \setalignmentcharacter{.}1.1 \NC\NR \NC 11.11 \NC \setalignmentcharacter{.}11.11 \NC\NR \NC 111.111 \NC \setalignmentcharacter{.}111.111 \NC\NR diff --git a/doc/context/sources/general/manuals/lowlevel/lowlevel-inserts.tex b/doc/context/sources/general/manuals/lowlevel/lowlevel-inserts.tex index caf3198b9..541249066 100644 --- a/doc/context/sources/general/manuals/lowlevel/lowlevel-inserts.tex +++ b/doc/context/sources/general/manuals/lowlevel/lowlevel-inserts.tex @@ -99,8 +99,8 @@ At the \TEX\ the builder is triggered automatically in the mentioned cases but a the \LUA\ end you can use \type {tex.triggerbuildpage()} to flush the pending contributions. -The properties that relate to the page look like counter and dimension registers ut -they are not. These variables are global and managed differently. +The properties that relate to the page look like counter and dimension registers +but they are not. These variables are global and managed differently. \starttabulate \NC \type {\pagegoal} \NC the available space \NC \NR @@ -201,6 +201,12 @@ behind. (This is experimental.) \stopsectionlevel +\startsectionlevel[title=Migration] + +{\em Todo, nothing new there, so no hurry.} + +\stopsectionlevel + \startsectionlevel[title=Callbacks] {\em Todo, nothing new there, so no hurry.} diff --git a/doc/context/sources/general/manuals/metafun/metafun-examples.tex b/doc/context/sources/general/manuals/metafun/metafun-examples.tex index 3c16f96dd..d9da5fea4 100644 --- a/doc/context/sources/general/manuals/metafun/metafun-examples.tex +++ b/doc/context/sources/general/manuals/metafun/metafun-examples.tex @@ -3431,7 +3431,7 @@ We demonstrate the differences with an example. The result can be seen in \startbuffer \startcombination {\startMPcode - picture first, second ; + picture first, second ; first := outlinetext.p("N") ; first := first scaled 10 ; second := outlinetext.p("T") ; second := second scaled 10 ; second := second rotatedaround(center second, 5) shifted (1,-1) ; |