1 files changed, 397 insertions, 0 deletions
diff --git a/doc/context/sources/general/manuals/evenmore/evenmore-paragraphs.tex b/doc/context/sources/general/manuals/evenmore/evenmore-paragraphs.tex
new file mode 100644
index 000000000..c8958a487
--- /dev/null
+++ b/doc/context/sources/general/manuals/evenmore/evenmore-paragraphs.tex
@@ -0,0 +1,397 @@
+% language=us runpath=texruns:manuals/evenmore
+
+% End of July 2020 I decided to look into some of the backlog items and paragraphs
+% are on them, as are inserts and so. The usual musical time stamp is buying the
+% excellent Prince Live at the Aladdin Las Vegas DVD which I ran between the
+% moments that I had enough of coding. Some tracks, like Family Name, fit perfectly
+% in mid 2020.
+
+% https://www.youtube.com/watch?v=f8FAJXPBdOg
+
+\environment evenmore-style
+
+\startcomponent evenmore-paragraphs
+
+\enableexperiments[paragraphs.freeze]
+
+\startchapter[title=Paragraphs]
+
+{\em This is mostly a wrapup of some developments, and definitely not a tutorial.
+What is described here is experimental and successive version of \CONTEXT\ \LMTX\
+will explore their potential. As long a users stay away from the low level
+primitives we can try to guarantee consistent behavior (and catch side effect or
+deal with known issues).}
+
+\startsection[title=Freezing]
+
+A well known property of paragraphs is that when the moment is there to split
+into lines, the current state of variables drives it. There are a lot of quite
+some variables involved. The most significant one is the \type {\hsize}. Take
+this:
+
+\startbuffer
+\bgroup
+    {\bf Ward:}
+    \hsize .25\textwidth
+    {\bf Ward:} \samplefile{ward}
+    \hsize .75\textwidth % last value
+\egroup
+\stopbuffer
+
+\typebuffer
+
+This gives:
+
+{\forgetparagraphfreezing \getbuffer}
+
+But wait, why do we get the full text width here? The reason is that by the time
+the paragraph ends, after the \type {\egroup}, the \type {\hsize} is set back to
+what it was before the group started.
+
+\startbuffer
+\bgroup
+    \hsize .25\textwidth
+    {\bf Ward:} \samplefile{ward}
+    \hsize .75\textwidth % last value
+    \par
+\egroup
+\stopbuffer
+
+\typebuffer
+
+This gives:
+
+{\forgetparagraphfreezing \getbuffer}
+
+The last \type {\hsize} specified is used. That is not really a problem, but the
+fact that we need to explicitly end a paragraph before the group ends actually
+is, and in a moment we will see an example where that matters a lot. First a
+general solution to this problem is discussed. In the next example, we do group,
+but inside that group we take a snapshot of the \type {\hsize}:
+
+\startbuffer
+\bgroup
+    \hsize .80\textwidth
+    \dontleavehmode
+    \snapshotpar "000001
+    {\bf Ward:} \samplefile{ward}
+\egroup
+\stopbuffer
+
+\typebuffer
+
+This time we get:
+
+{\forgetparagraphfreezing \getbuffer}
+
+The magic number used in the snapshot relates to the \type {\hsize}.
+
+\startcolumns[n=2]
+\starttabulate[|T||]
+\NC 0x\uchexnumbers{\hsizefrozenparcode        } \NC hsize \NC \NR
+\NC 0x\uchexnumbers{\skipfrozenparcode         } \NC leftskip rightskip \NC \NR
+\NC 0x\uchexnumbers{\hangfrozenparcode         } \NC hangindent hangafter \NC \NR
+\NC 0x\uchexnumbers{\indentfrozenparcode       } \NC parindent \NC \NR
+\NC 0x\uchexnumbers{\parfillfrozenparcode      } \NC parfillskip parfillleftskip \NC \NR
+\NC 0x\uchexnumbers{\adjustfrozenparcode       } \NC adjustspacing adjustspacingstep adjustspacingshrink adjustspacingstretch \NC \NR
+\NC 0x\uchexnumbers{\protrudefrozenparcode     } \NC protrudechars \NC \NR
+\NC 0x\uchexnumbers{\tolerancefrozenparcode    } \NC pretolerance tolerance \NC \NR
+\NC 0x\uchexnumbers{\stretchfrozenparcode      } \NC emergencystretch \NC \NR
+\NC 0x\uchexnumbers{\loosenessfrozenparcode    } \NC looseness \NC \NR
+\NC 0x\uchexnumbers{\lastlinefrozenparcode     } \NC lastlinefit \NC \NR
+\NC 0x\uchexnumbers{\linepenaltyfrozenparcode  } \NC linepenalty interlinepenalty interlinepenalties \NC \NR
+\NC 0x\uchexnumbers{\clubpenaltyfrozenparcode  } \NC clubpenalty clubpenalties \NC \NR
+\NC 0x\uchexnumbers{\widowpenaltyfrozenparcode } \NC widowpenalty widowpenalties displaywidowpenalty displaywidowpenalties \NC \NR
+\NC 0x\uchexnumbers{\brokenpenaltyfrozenparcode} \NC brokenpenalty \NC \NR
+\NC 0x\uchexnumbers{\demeritsfrozenparcode     } \NC adjdemerits doublehyphendemerits finalhyphendemerits \NC \NR
+\NC 0x\uchexnumbers{\shapefrozenparcode        } \NC parshape \NC \NR
+\NC 0x\uchexnumbers{\linefrozenparcode         } \NC baselineskip lineskip lineskiplimit \NC \NR
+\NC                                              \NC \NC \NR
+\NC 0xFFFFFFF                                    \BC all of them \NC \NR
+\stoptabulate
+\stopcolumns
+
+In practice you will set them all on one go, so:
+
+\starttyping
+\snapshotpar "FFFFFFF
+\stoptyping
+
+How often do we need such a feature? Actually more often than one thinks,
+especially when we have an unpredictable situation. For instance, when you
+typeset from an \XML\ source you often don't know what you get, and you can have
+cases that end up like this:
+
+\startbuffer
+\placefigure[left,none]{}{} {Ward: \bf  \dorecurse{3}{\samplefile{ward}} } \par
+\placefigure[left,none]{}{} {\bf Ward:} \dorecurse{3}{\samplefile{ward}}   \par
+\stopbuffer
+
+\typebuffer
+
+This might render as:
+
+{\forgetparagraphfreezing \getbuffer} \forgetsidefloats % needed due to interference
+
+The placement of such a figure is hooked into \type {\everypar} and uses hanging
+indentation. Like \type {\hsize}, \type {\hangafter} and \type {\hangindent} can
+be forgotten before the paragraph ends. In \MKII\ and \MKIV\ the recommended
+solution is to always start a paragraph explicitly, with a strut, forced
+indentation of preferably:
+
+\startbuffer
+\dontleavehmode {Ward: \bf  \dorecurse{3}{\samplefile{ward} } } \par
+\dontleavehmode {\bf Ward:} \dorecurse{3}{\samplefile{ward} }   \par
+\stopbuffer
+
+\typebuffer
+
+In an \XML\ mapping we can hide it but in a regular \TEX\ source this is not
+pretty. With little effort we can do the snapping automatically, so that we get:
+
+\placefigure[left,none]{}{} {Ward: \bf  \dorecurse{3}{\samplefile{ward} } } \par
+\placefigure[left,none]{}{} {\bf Ward:} \dorecurse{3}{\samplefile{ward} }   \par
+
+and this is what \CONTEXT\ \LMTX\ will do once we're sure that the snapshot
+feature behaves well and has no side effects. There is of course some overhead
+involved in taking snapshots, keeping track of the values and accessing them
+later, but it is rewarding.
+
+In addition to the numeric \type {\snapshotpar} primitive there is also another
+way to take s snapshot. As with the numeric variant, it only takes a snapshot when
+in horizontal mode: there has to be a so called local par node at the head of the
+current list. The next code shows some how to play with some of what \CONTEXT\
+offers:
+
+\starttyping
+\setuplayout[alternative=doublesided]
+
+\starttext
+
+\startbuffer
+    \dorecurse{8}{
+        \interlinepenalties 1 \maxcard
+        CASE 1: \samplefile{tufte} \par
+    } \page
+
+    \dorecurse{8}{
+        CASE 2: \samplefile{tufte}
+        \interlinepenalties 1 \maxcard \par
+    } \page
+
+    \dorecurse{8}{
+        CASE 3: \samplefile{tufte}
+        \interlinepenalties 1 \maxcard \freezeparagraphproperties \par
+    } \page
+
+    \dorecurse{8}{
+        CASE 4: \samplefile{tufte}
+        \frozen \interlinepenalties 1 \maxcard \par
+    } \page
+
+    \dorecurse{8}{
+        CASE 5: \samplefile{tufte}
+        \frozen \interlinepenalty \maxcard \par
+    } \page
+\stopbuffer
+
+\typebuffer \page \getbuffer
+
+\stoptext
+\stoptyping
+
+When you process this you will notice that the \type {\frozen} prefix also
+snapshots the parameter that gets set. Now, there is a pitfall here: some for
+these settings are persistent, i.e. they are not reset after a paragraph has been
+typeset. For instance, \type {\tolerance} is a general setting, but \type
+{\hangindent} is a one shot setting: it's value gets reset after the paragraph
+has been dealt with.
+
+Here is another test one can run to see what happens:
+
+\starttyping
+\dontleavehmode
+\defrostparagraphproperties
+\writestatus{state}{after  start            \uchexnumbers{\the\snapshotpar}}%
+\writestatus{state}{before set   hangindent \uchexnumbers{\the\snapshotpar}}%
+\frozen\hangindent10pt
+\writestatus{state}{after  set   hangindent \uchexnumbers{\the\snapshotpar}}%
+\writestatus{state}{before set   looseness  \uchexnumbers{\the\snapshotpar}}%
+\frozen\looseness 1
+\writestatus{state}{after  set   looseness  \uchexnumbers{\the\snapshotpar}}%
+\writestatus{state}{before set   hangafter  \uchexnumbers{\the\snapshotpar}}%
+\frozen\hangafter 2
+\writestatus{state}{after  set   hangafter  \uchexnumbers{\the\snapshotpar}}%
+\begingroup
+\writestatus{state}{before set   rightskip  \uchexnumbers{\the\snapshotpar}}%
+\frozen\rightskip2cm
+\writestatus{state}{after  set   rightskip  \uchexnumbers{\the\snapshotpar}}%
+\endgroup
+\writestatus{state}{before reset hangindent \uchexnumbers{\the\snapshotpar}}%
+\snapshotpar-\frozenhangindentcode
+\writestatus{state}{after  reset hangindent \uchexnumbers{\the\snapshotpar}}%
+\writestatus{state}{before reset hangafter  \uchexnumbers{\the\snapshotpar}}%
+\snapshotpar-\frozenhangaftercode
+\writestatus{state}{after  reset hangafter  \uchexnumbers{\the\snapshotpar}}%
+... content ...
+\stoptyping
+
+You can group an assignment and then take a snapshot. That way the change doesn't
+affect following paragraphs, unless of course to did a global assignment. In
+\CONTEXT\ we have a bunch of constants that can be used instead of the hard to
+remember bit positions. The \type {\frozen} prefix can also be used with for
+instance a \type {\advance} operation. Of course it only has effect for those
+(internal) parameters that relate to a paragraph.
+
+Keep in mind that what is show here will evolve: in \CONTEXT\ \LMTX\ we will
+snapshot by default and the core macros are aware of this fact. Although the way
+\CONTEXT\ is set up makes it relatively easy to make this paradigm shift users
+should anyway be aware of this change when they do their own low level tweaking,
+but in that case they probably already are aware of possible interferences.
+
+\stopsection
+
+\startsection[title=Wrapping up]
+
+Another new (low level) feature is wrapping up a paragraph. Traditional \TEX\
+comes with the powerful \type {\everypar} and in \LUAMETATEX\ we now have \type
+{\wrapuppar}. This primitive collects tokens that will be expanded just before
+the paragraph ends. Here is an example:
+
+\startbuffer
+\dontleavehmode
+\wrapuppar{\hfill {\bf ONE}}%
+\wrapuppar{\crlf\strut\hfill {\bf TWO}\hfill\strut}%
+\wrapuppar{\crlf\strut {\bf THREE}\hfill\strut {\bf FOUR}}%
+\samplefile{ward}
+
+\samplefile{ward}
+\stopbuffer
+
+\typebuffer
+
+We can only wrapup when we are in a paragraph although one can of course use the
+\type {\wrapuppar} command inside an \type {\everypar} if needed.
+
+\getbuffer
+
+An more useful example is the following. We leave it to the reader to check it
+out:
+
+\starttyping
+\dorecurse{10}{
+    \bgroup
+        \advance\hsize by -#1cm\relax
+        \dontleavehmode
+        \wrapuppar{\strut\nobreak\hfill\nobreak QED}%
+        \samplefile{ward}
+    \egroup
+    \par
+}
+\stoptyping
+
+\stopsection
+
+\startsection[title=Insertions]
+
+The concept of inserts is kind of complicated. They are nodes in a list that make
+separate streams. An application of inserts are footnotes. In the text flow a
+symbol is typeset (like a raised number) and the note itself becomes an insert.
+When a paragraph is broken into lines, these inserts end up in to be boxed line,
+but when the line is actually wrapped in a box, these inserts are collected and
+injected after the line. The page builder will then take their dimensions into
+account when it comes to breaking pages. Depending on how strict the rules are
+the inserts will end up on the same page, move, of be broken into lines.
+
+This all works well as long as the inserts are not burried into boxes: they then
+are invisible to the mechanism described before. Take the following example:
+
+\starttyping
+\dontleavehmode
+l\hbox{h\footnote{h1} test}
+l\hbox{h\footnote{h2}} test
+l\hbox{h\footnote{h3}} test
+l\footnote{l4} test
+l\footnote{l5} test
+l\hbox{h\footnote{h6} test}
+l\hbox{\hbox{\hbox{h\footnote{h7} test}}}
+l\footnote{l8} test
+\par
+\stoptyping
+
+% \starttabulate
+% \NC test \NC test \footnote{before} \samplefile{tufte} \footnote{after}\NC \NR
+% ...
+% \NC test \NC test \footnote{before} \samplefile{tufte} \footnote{after}\NC \NR
+% \stoptabulate
+
+In the engines used with \MKII\ only a few footnotes actually show up. In \MKIV\
+the situation is slightly better because there we use some trickery to migrate
+these notes to the outer level. But even there it's not perfect because the order
+changes. We can actually fix that (and do so) but it comes at a performance penalty
+so this is why in \MKIV\ dealing with this is optional.
+
+\start
+    \def\Hi#1{\high{\tx#1}}
+    \dontleavehmode
+    \hbox{lh\H1 test lh\H2 test lh\H3 test l\H4 test l\H5 test lh\H6 test lh\H7 test l\H8 test}
+
+    \starttabulate[|||||||||||]
+    \NC \MKII \NC \PDFTEX\ & \XETEX         \NC \Hi4 l4 \NC \Hi5 l5 \NC \Hi8 l8 \NC         \NC         \NC         \NC         \NC         \NC \NR
+    \NC \MKIV \footnote{In older versions.} \NC \LUATEX             \NC \Hi1 h1 \NC \Hi2 h2 \NC \Hi3 h3 \NC \Hi6 h6 \NC \Hi7 h7 \NC \Hi4 l4 \NC \Hi5 l5 \NC \Hi8 l8 \NC \NR
+    \NC \LMTX \NC \LUAMETATEX               \NC \Hi1 h1 \NC \Hi2 h2 \NC \Hi3 h3 \NC \Hi4 l4 \NC \Hi5 l5 \NC \Hi6 h6 \NC \Hi7 h7 \NC \Hi8 l8 \NC \NR
+    \stoptabulate
+\stop
+
+However, when you look at the results of \LMTX\ you will notice that the
+situation is better. This is because we have some code to \LUAMETATEX\ that can
+better deal with some cases. Combined with some \LUA\ magic (as in \MKIV) we get
+the right order at hardly any runtime overhead. It must be noted that the
+original \TEX\ engine for good reason works as it does because there the actual
+typesetting (read: resolving glyphs, hyphenation, applying ligatures and kerns,
+etc.) is interwoven with the main scanning|/|expanding loops in order to be
+efficient on the machines of those times. In \LUATEX\ we have these stages
+separated but the code dealing with inserts is the same, if only because we have
+to be compatible. In \LUAMETATEX\ we have again a bit simpler code because we use
+the fact that lists are double linked, which also makes it possible to add some
+magic code dealing with nested inserts without obscuring the code. It comes of
+course at a bit performance hit and the nodes related to lists also because
+larger but they are already much larger than in other engines, so we don't care
+too much about that. It is anyway a mechanism that need to be enabled explicitly.
+
+\stopsection
+
+\stopchapter
+
+\stopcomponent
+
+% \starttext
+
+%     \def\TestA  {\registerparwrapper       {A}  {[\ignorespaces}{\removeunwantedspaces]\showparwrapperstate{A}}}
+%     \def\TestB#1{\registerparwrapper       {B#1}{(\ignorespaces}{\removeunwantedspaces)\showparwrapperstate{B#1}}}
+%     \def\TestC  {\registerparwrapper       {C}  {<\ignorespaces}{\removeunwantedspaces>\showparwrapperstate{C}\forgetparwrapper}}
+%     \def\TestR  {\registerparwrapperreverse{R}  {<\ignorespaces}{\removeunwantedspaces>\showparwrapperstate{R}}}
+
+%     \start
+%         \TestA
+%         \dorecurse{3}{1.#1 before \ruledvbox{\hsize2em\raggedcenter\TestB1 !\par} after\par} \blank
+%         \dorecurse{3}{2.#1 before \ruledvbox{\hsize3em\raggedcenter        !\par} after\par} \blank
+%         \dorecurse{3}{3.#1 before \ruledvbox{\hsize4em\raggedcenter\TestB2 !}     after\par} \blank
+%         \forgetparwrapper
+%         \dorecurse{3}{4.#1 before \ruledvbox{\hsize5em\raggedcenter\TestB3 !}     after\par} \blank
+%         \TestC
+%         \dorecurse{3}{5.#1 before \ruledvbox{\hsize2em\raggedcenter\TestA   !}     after\par} \blank
+%     \stop
+
+
+%     \start
+%         \TestA
+%         \dorecurse{3}{6.#1  before after\par} \blank
+%         \TestB4
+%         \dorecurse{3}{7.#1 before after\par} \blank
+%         \TestB5
+%         \TestR
+%         \dorecurse{3}{8.#1 before after\par} \blank
+%     \stop
+
+% \stoptext