path: root/doc/context/sources/general/manuals/evenmore/evenmore-paragraphs.tex

% language=us runpath=texruns:manuals/evenmore

% End of July 2020 I decided to look into some of the backlog items and paragraphs
% are on them, as are inserts and so. The usual musical time stamp is buying the
% excellent Prince Live at the Aladdin Las Vegas DVD which I ran between the
% moments that I had enough of coding. Some tracks, like Family Name, fit perfectly
% in mid 2020.

% https://www.youtube.com/watch?v=f8FAJXPBdOg

\environment evenmore-style

\startcomponent evenmore-paragraphs

\enableexperiments[paragraphs.freeze]

\startchapter[title=Paragraphs]

{\em This is mostly a wrapup of some developments, and definitely not a tutorial.
What is described here is experimental and successive version of \CONTEXT\ \LMTX\
will explore their potential. As long a users stay away from the low level
primitives we can try to guarantee consistent behavior (and catch side effect or
deal with known issues).}

\startsection[title=Freezing]

A well known property of paragraphs is that when the moment is there to split
into lines, the current state of variables drives it. There are a lot of quite
some variables involved. The most significant one is the \type {\hsize}. Take
this:

\startbuffer
\bgroup
    {\bf Ward:}
    \hsize .25\textwidth
    {\bf Ward:} \samplefile{ward}
    \hsize .75\textwidth % last value
\egroup
\stopbuffer

\typebuffer

This gives:

{\forgetparagraphfreezing \getbuffer}

But wait, why do we get the full text width here? The reason is that by the time
the paragraph ends, after the \type {\egroup}, the \type {\hsize} is set back to
what it was before the group started.

\startbuffer
\bgroup
    \hsize .25\textwidth
    {\bf Ward:} \samplefile{ward}
    \hsize .75\textwidth % last value
    \par
\egroup
\stopbuffer

\typebuffer

This gives:

{\forgetparagraphfreezing \getbuffer}

The last \type {\hsize} specified is used. That is not really a problem, but the
fact that we need to explicitly end a paragraph before the group ends actually
is, and in a moment we will see an example where that matters a lot. First a
general solution to this problem is discussed. In the next example, we do group,
but inside that group we take a snapshot of the \type {\hsize}:

\startbuffer
\bgroup
    \hsize .80\textwidth
    \dontleavehmode
    \snapshotpar "000001
    {\bf Ward:} \samplefile{ward}
\egroup
\stopbuffer

\typebuffer

This time we get:

{\forgetparagraphfreezing \getbuffer}

The magic number used in the snapshot relates to the \type {\hsize}.

\startcolumns[n=2]
\starttabulate[|T||]
\NC 0x\uchexnumbers{\hsizefrozenparcode        } \NC hsize \NC \NR
\NC 0x\uchexnumbers{\skipfrozenparcode         } \NC leftskip rightskip \NC \NR
\NC 0x\uchexnumbers{\hangfrozenparcode         } \NC hangindent hangafter \NC \NR
\NC 0x\uchexnumbers{\indentfrozenparcode       } \NC parindent \NC \NR
\NC 0x\uchexnumbers{\parfillfrozenparcode      } \NC parfillskip parfillleftskip \NC \NR
\NC 0x\uchexnumbers{\adjustfrozenparcode       } \NC adjustspacing adjustspacingstep adjustspacingshrink adjustspacingstretch \NC \NR
\NC 0x\uchexnumbers{\protrudefrozenparcode     } \NC protrudechars \NC \NR
\NC 0x\uchexnumbers{\tolerancefrozenparcode    } \NC pretolerance tolerance \NC \NR
\NC 0x\uchexnumbers{\stretchfrozenparcode      } \NC emergencystretch \NC \NR
\NC 0x\uchexnumbers{\loosenessfrozenparcode    } \NC looseness \NC \NR
\NC 0x\uchexnumbers{\lastlinefrozenparcode     } \NC lastlinefit \NC \NR
\NC 0x\uchexnumbers{\linepenaltyfrozenparcode  } \NC linepenalty interlinepenalty interlinepenalties \NC \NR
\NC 0x\uchexnumbers{\clubpenaltyfrozenparcode  } \NC clubpenalty clubpenalties \NC \NR
\NC 0x\uchexnumbers{\widowpenaltyfrozenparcode } \NC widowpenalty widowpenalties displaywidowpenalty displaywidowpenalties \NC \NR
\NC 0x\uchexnumbers{\brokenpenaltyfrozenparcode} \NC brokenpenalty \NC \NR
\NC 0x\uchexnumbers{\demeritsfrozenparcode     } \NC adjdemerits doublehyphendemerits finalhyphendemerits \NC \NR
\NC 0x\uchexnumbers{\shapefrozenparcode        } \NC parshape \NC \NR
\NC 0x\uchexnumbers{\linefrozenparcode         } \NC baselineskip lineskip lineskiplimit \NC \NR
\NC                                              \NC \NC \NR
\NC 0xFFFFFFF                                    \BC all of them \NC \NR
\stoptabulate
\stopcolumns

In practice you will set them all on one go, so:

\starttyping
\snapshotpar "FFFFFFF
\stoptyping

How often do we need such a feature? Actually more often than one thinks,
especially when we have an unpredictable situation. For instance, when you
typeset from an \XML\ source you often don't know what you get, and you can have
cases that end up like this:

\startbuffer
\placefigure[left,none]{}{} {Ward: \bf  \dorecurse{3}{\samplefile{ward}} } \par
\placefigure[left,none]{}{} {\bf Ward:} \dorecurse{3}{\samplefile{ward}}   \par
\stopbuffer

\typebuffer

This might render as:

{\forgetparagraphfreezing \getbuffer} \forgetsidefloats % needed due to interference

The placement of such a figure is hooked into \type {\everypar} and uses hanging
indentation. Like \type {\hsize}, \type {\hangafter} and \type {\hangindent} can
be forgotten before the paragraph ends. In \MKII\ and \MKIV\ the recommended
solution is to always start a paragraph explicitly, with a strut, forced
indentation of preferably:

\startbuffer
\dontleavehmode {Ward: \bf  \dorecurse{3}{\samplefile{ward} } } \par
\dontleavehmode {\bf Ward:} \dorecurse{3}{\samplefile{ward} }   \par
\stopbuffer

\typebuffer

In an \XML\ mapping we can hide it but in a regular \TEX\ source this is not
pretty. With little effort we can do the snapping automatically, so that we get:

\placefigure[left,none]{}{} {Ward: \bf  \dorecurse{3}{\samplefile{ward} } } \par
\placefigure[left,none]{}{} {\bf Ward:} \dorecurse{3}{\samplefile{ward} }   \par

and this is what \CONTEXT\ \LMTX\ will do once we're sure that the snapshot
feature behaves well and has no side effects. There is of course some overhead
involved in taking snapshots, keeping track of the values and accessing them
later, but it is rewarding.

In addition to the numeric \type {\snapshotpar} primitive there is also another
way to take s snapshot. As with the numeric variant, it only takes a snapshot when
in horizontal mode: there has to be a so called local par node at the head of the
current list. The next code shows some how to play with some of what \CONTEXT\
offers:

\starttyping
\setuplayout[alternative=doublesided]

\starttext

\startbuffer
    \dorecurse{8}{
        \interlinepenalties 1 \maxcard
        CASE 1: \samplefile{tufte} \par
    } \page

    \dorecurse{8}{
        CASE 2: \samplefile{tufte}
        \interlinepenalties 1 \maxcard \par
    } \page

    \dorecurse{8}{
        CASE 3: \samplefile{tufte}
        \interlinepenalties 1 \maxcard \freezeparagraphproperties \par
    } \page

    \dorecurse{8}{
        CASE 4: \samplefile{tufte}
        \frozen \interlinepenalties 1 \maxcard \par
    } \page

    \dorecurse{8}{
        CASE 5: \samplefile{tufte}
        \frozen \interlinepenalty \maxcard \par
    } \page
\stopbuffer

\typebuffer \page \getbuffer

\stoptext
\stoptyping

When you process this you will notice that the \type {\frozen} prefix also
snapshots the parameter that gets set. Now, there is a pitfall here: some for
these settings are persistent, i.e. they are not reset after a paragraph has been
typeset. For instance, \type {\tolerance} is a general setting, but \type
{\hangindent} is a one shot setting: it's value gets reset after the paragraph
has been dealt with.

Here is another test one can run to see what happens:

\starttyping
\dontleavehmode
\defrostparagraphproperties
\writestatus{state}{after  start            \uchexnumbers{\the\snapshotpar}}%
\writestatus{state}{before set   hangindent \uchexnumbers{\the\snapshotpar}}%
\frozen\hangindent10pt
\writestatus{state}{after  set   hangindent \uchexnumbers{\the\snapshotpar}}%
\writestatus{state}{before set   looseness  \uchexnumbers{\the\snapshotpar}}%
\frozen\looseness 1
\writestatus{state}{after  set   looseness  \uchexnumbers{\the\snapshotpar}}%
\writestatus{state}{before set   hangafter  \uchexnumbers{\the\snapshotpar}}%
\frozen\hangafter 2
\writestatus{state}{after  set   hangafter  \uchexnumbers{\the\snapshotpar}}%
\begingroup
\writestatus{state}{before set   rightskip  \uchexnumbers{\the\snapshotpar}}%
\frozen\rightskip2cm
\writestatus{state}{after  set   rightskip  \uchexnumbers{\the\snapshotpar}}%
\endgroup
\writestatus{state}{before reset hangindent \uchexnumbers{\the\snapshotpar}}%
\snapshotpar-\frozenhangindentcode
\writestatus{state}{after  reset hangindent \uchexnumbers{\the\snapshotpar}}%
\writestatus{state}{before reset hangafter  \uchexnumbers{\the\snapshotpar}}%
\snapshotpar-\frozenhangaftercode
\writestatus{state}{after  reset hangafter  \uchexnumbers{\the\snapshotpar}}%
... content ...
\stoptyping

You can group an assignment and then take a snapshot. That way the change doesn't
affect following paragraphs, unless of course to did a global assignment. In
\CONTEXT\ we have a bunch of constants that can be used instead of the hard to
remember bit positions. The \type {\frozen} prefix can also be used with for
instance a \type {\advance} operation. Of course it only has effect for those
(internal) parameters that relate to a paragraph.

Keep in mind that what is show here will evolve: in \CONTEXT\ \LMTX\ we will
snapshot by default and the core macros are aware of this fact. Although the way
\CONTEXT\ is set up makes it relatively easy to make this paradigm shift users
should anyway be aware of this change when they do their own low level tweaking,
but in that case they probably already are aware of possible interferences.

\stopsection

\startsection[title=Wrapping up]

Another new (low level) feature is wrapping up a paragraph. Traditional \TEX\
comes with the powerful \type {\everypar} and in \LUAMETATEX\ we now have \type
{\wrapuppar}. This primitive collects tokens that will be expanded just before
the paragraph ends. Here is an example:

\startbuffer
\dontleavehmode
\wrapuppar{\hfill {\bf ONE}}%
\wrapuppar{\crlf\strut\hfill {\bf TWO}\hfill\strut}%
\wrapuppar{\crlf\strut {\bf THREE}\hfill\strut {\bf FOUR}}%
\samplefile{ward}

\samplefile{ward}
\stopbuffer

\typebuffer

We can only wrapup when we are in a paragraph although one can of course use the
\type {\wrapuppar} command inside an \type {\everypar} if needed.

\getbuffer

An more useful example is the following. We leave it to the reader to check it
out:

\starttyping
\dorecurse{10}{
    \bgroup
        \advance\hsize by -#1cm\relax
        \dontleavehmode
        \wrapuppar{\strut\nobreak\hfill\nobreak QED}%
        \samplefile{ward}
    \egroup
    \par
}
\stoptyping

\stopsection

\startsection[title=Insertions]

The concept of inserts is kind of complicated. They are nodes in a list that make
separate streams. An application of inserts are footnotes. In the text flow a
symbol is typeset (like a raised number) and the note itself becomes an insert.
When a paragraph is broken into lines, these inserts end up in to be boxed line,
but when the line is actually wrapped in a box, these inserts are collected and
injected after the line. The page builder will then take their dimensions into
account when it comes to breaking pages. Depending on how strict the rules are
the inserts will end up on the same page, move, of be broken into lines.

This all works well as long as the inserts are not burried into boxes: they then
are invisible to the mechanism described before. Take the following example:

\starttyping
\dontleavehmode
l\hbox{h\footnote{h1} test}
l\hbox{h\footnote{h2}} test
l\hbox{h\footnote{h3}} test
l\footnote{l4} test
l\footnote{l5} test
l\hbox{h\footnote{h6} test}
l\hbox{\hbox{\hbox{h\footnote{h7} test}}}
l\footnote{l8} test
\par
\stoptyping

% \starttabulate
% \NC test \NC test \footnote{before} \samplefile{tufte} \footnote{after}\NC \NR
% ...
% \NC test \NC test \footnote{before} \samplefile{tufte} \footnote{after}\NC \NR
% \stoptabulate

In the engines used with \MKII\ only a few footnotes actually show up. In \MKIV\
the situation is slightly better because there we use some trickery to migrate
these notes to the outer level. But even there it's not perfect because the order
changes. We can actually fix that (and do so) but it comes at a performance penalty
so this is why in \MKIV\ dealing with this is optional.

\start
    \def\Hi#1{\high{\tx#1}}
    \dontleavehmode
    \hbox{lh\H1 test lh\H2 test lh\H3 test l\H4 test l\H5 test lh\H6 test lh\H7 test l\H8 test}

    \starttabulate[|||||||||||]
    \NC \MKII \NC \PDFTEX\ & \XETEX         \NC \Hi4 l4 \NC \Hi5 l5 \NC \Hi8 l8 \NC         \NC         \NC         \NC         \NC         \NC \NR
    \NC \MKIV \footnote{In older versions.} \NC \LUATEX             \NC \Hi1 h1 \NC \Hi2 h2 \NC \Hi3 h3 \NC \Hi6 h6 \NC \Hi7 h7 \NC \Hi4 l4 \NC \Hi5 l5 \NC \Hi8 l8 \NC \NR
    \NC \LMTX \NC \LUAMETATEX               \NC \Hi1 h1 \NC \Hi2 h2 \NC \Hi3 h3 \NC \Hi4 l4 \NC \Hi5 l5 \NC \Hi6 h6 \NC \Hi7 h7 \NC \Hi8 l8 \NC \NR
    \stoptabulate
\stop

However, when you look at the results of \LMTX\ you will notice that the
situation is better. This is because we have some code to \LUAMETATEX\ that can
better deal with some cases. Combined with some \LUA\ magic (as in \MKIV) we get
the right order at hardly any runtime overhead. It must be noted that the
original \TEX\ engine for good reason works as it does because there the actual
typesetting (read: resolving glyphs, hyphenation, applying ligatures and kerns,
etc.) is interwoven with the main scanning|/|expanding loops in order to be
efficient on the machines of those times. In \LUATEX\ we have these stages
separated but the code dealing with inserts is the same, if only because we have
to be compatible. In \LUAMETATEX\ we have again a bit simpler code because we use
the fact that lists are double linked, which also makes it possible to add some
magic code dealing with nested inserts without obscuring the code. It comes of
course at a bit performance hit and the nodes related to lists also because
larger but they are already much larger than in other engines, so we don't care
too much about that. It is anyway a mechanism that need to be enabled explicitly.

\stopsection

\stopchapter

\stopcomponent

% \starttext

%     \def\TestA  {\registerparwrapper       {A}  {[\ignorespaces}{\removeunwantedspaces]\showparwrapperstate{A}}}
%     \def\TestB#1{\registerparwrapper       {B#1}{(\ignorespaces}{\removeunwantedspaces)\showparwrapperstate{B#1}}}
%     \def\TestC  {\registerparwrapper       {C}  {<\ignorespaces}{\removeunwantedspaces>\showparwrapperstate{C}\forgetparwrapper}}
%     \def\TestR  {\registerparwrapperreverse{R}  {<\ignorespaces}{\removeunwantedspaces>\showparwrapperstate{R}}}

%     \start
%         \TestA
%         \dorecurse{3}{1.#1 before \ruledvbox{\hsize2em\raggedcenter\TestB1 !\par} after\par} \blank
%         \dorecurse{3}{2.#1 before \ruledvbox{\hsize3em\raggedcenter        !\par} after\par} \blank
%         \dorecurse{3}{3.#1 before \ruledvbox{\hsize4em\raggedcenter\TestB2 !}     after\par} \blank
%         \forgetparwrapper
%         \dorecurse{3}{4.#1 before \ruledvbox{\hsize5em\raggedcenter\TestB3 !}     after\par} \blank
%         \TestC
%         \dorecurse{3}{5.#1 before \ruledvbox{\hsize2em\raggedcenter\TestA   !}     after\par} \blank
%     \stop


%     \start
%         \TestA
%         \dorecurse{3}{6.#1  before after\par} \blank
%         \TestB4
%         \dorecurse{3}{7.#1 before after\par} \blank
%         \TestB5
%         \TestR
%         \dorecurse{3}{8.#1 before after\par} \blank
%     \stop

% \stoptext