path: root/doc/context/sources/general/manuals/hybrid/hybrid-grouping.tex

\startcomponent hybrid-grouping

\startbuffer[MyAbstract]
\StartAbstract
    In this article I will discuss a few things that are hard to do in
    traditional \TEX, but reasonable well in \LUATEX.
\StopAbstract
\stopbuffer

\doifmodeelse {tugboat} {
    \usemodule[tug-01,abr-01]
    \setvariables
      [tugboat]
      [columns=yes]
    \setvariables
      [tugboat]
      [year=2010,
       volume=99,
       number=9,
       page=99]
    \setvariables
      [tugboat]
      [title=Grouping,
       subtitle=A few things you can do with LUATEX,
       keywords=,
       author=Hans Hagen,
       address=PRAGMA ADE\\Ridderstraat 27\\8061GH Hasselt NL,
       email=pragma@wxs.nl]
    %
    % we use a buffer as abstract themselves are buffers and
    % inside macros we loose line endings and such
    \getbuffer[MyAbstract]
    %
    \StartArticle
} {
    \environment hybrid-environment
    \startchapter[title={Grouping}]
}

\setupbars[rulethickness=.15] % nicer

\startsection [title={Variants}]

After using \TEX\ for a while you get accustomed to one of its interesting
concepts: grouping. Programming languages like \PASCAL\ and \MODULA\ have
keywords \type {begin} and \type {end}. So, one can say:

\starttyping
if test then begin
    print_bold("test 1")
    print_bold("test 2")
end
\stoptyping

Other languages provide a syntax like:

\starttyping
if test {
    print_bold("test 1")
    print_bold("test 2")
}
\stoptyping

So, in those languages the \type {begin} and \type {end} and|/|or the curly
braces define a \quote {group} of statements. In \TEX\ on the other hand we have:

\starttyping
test \begingroup \bf test \endgroup test
\stoptyping

Here the second \type {test} comes out in a bold font and the switch to bold
(basically a different font is selected) is reverted after the group is closed.
So, in \TEX\ grouping deals with scope and not with grouping things together.

In other languages it depends on the language of locally defined variables are
visible afterwards but in \TEX\ they're really local unless a \type {\global}
prefix (or one of the shortcuts) is used.

In languages like \LUA\ we have constructs like:

\starttyping
for i=1,100 do
    local j = i + 20
    ...
end
\stoptyping

Here \type {j} is visible after the loop ends unless prefixed by \type {local}.
Yet another example is \METAPOST:

\starttyping
begingroup ;
    save n ; numeric n ; n := 10 ;
    ...
endgroup ;
\stoptyping

Here all variables are global unless they are explicitly saved inside a group.
This makes perfect sense as the resulting graphic also has a global (accumulated)
property. In practice one rarely needs grouping, contrary to \TEX\ where one
really wants to keep changes local, if only because document content is so
unpredictable that one never knows when some change in state happens.

In principle it is possible to carry over information across a group boundary.
Consider this somewhat unrealistic example:

\starttyping
\begingroup
    \leftskip 10pt
    \begingroup
        ....
        \advance\leftskip 10pt
        ....
    \endgroup
\endgroup
\stoptyping

How do we carry the advanced leftskip over the group boundary without using a
global assignment which could have more drastic side effects? Here is the trick:

\starttyping
\begingroup
    \leftskip 10pt
    \begingroup
        ....
        \advance\leftskip 10pt
        ....
        \expandafter
    \endgroup
    \expandafter \leftskip \the\leftskip
\endgroup
\stoptyping

This is typical the kind of code that gives new users the creeps but normally
they never have to do that kind of coding. Also, that kind of tricks assumes that
one knows how many groups are involved.

\stopsection

\startsection [title={Implication}]

What does this all have to do with \LUATEX\ and \MKIV ? The user interface of
\CONTEXT\ provide lots of commands like:

\starttyping
\setupthis[style=bold]
\setupthat[color=green]
\stoptyping

Most of them obey grouping. However, consider a situation where we use \LUA\ code
to deal with some aspect of typesetting, for instance numbering lines or adding
ornamental elements to the text. In \CONTEXT\ we flag such actions with
attributes and often the real action takes place a bit later, for instance when a
paragraph or page becomes available.

A comparable pure \TEX\ example is the following:

\starttyping
{test test \bf test \leftskip10pt test}
\stoptyping

Here the switch to bold happens as expected but no leftskip of 10pt is applied.
This is because the set value is already forgotten when the paragraph is actually
typeset. So in fact we'd need:

\starttyping
{test test \bf test \leftskip10pt test \par}
\stoptyping

Now, say that we have:

\starttyping
{test test test \setupflag[option=1] \flagnexttext test}
\stoptyping

We flag some text (using an attribute) and expect it to get a treatment where
option~1 is used. However, the real action might take place when \TEX\ deals with
the paragraph or page and by that time the specific option is already forgotten
or it might have gotten another value. So, the rather natural \TEX\ grouping does
not work out that well in a hybrid situation.

As the user interface assumes a consistent behaviour we cannot simply make these
settings global even if this makes much sense in practice. One solution is to
carry the information with the flagged text i.e.\ associate it somehow in the
attribute's value. Of course, as we never know in advance when this information
is used, this might result in quite some states being stored persistently.

A side effect of this \quote {problem} is that new commands might get suboptimal
user interfaces (especially inheritance or cloning of constructs) that are
somewhat driven by these \quote {limitations}. Of course we may wonder if the end
user will notice this.

To summarize this far, we have three sorts of grouping to deal with:

\startitemize[item]

\startitem
    \TEX's normal grouping model limits its scope to the local situation and
    normally has only direct and local consequences. We cannot carry information
    over groups.
\stopitem

\startitem
    Some of \TEX's properties are applied later, for instance when a paragraph or
    page is typeset and in order to make \quote {local} changes effective, the
    user needs to add explicit paragraph ending commands (like \type {\par} or
    \type {\page}).
\stopitem

\startitem
    Features dealt with asynchronously by \LUA\ are at that time unaware of
    grouping and variables set that were active at the time the feature was
    triggered so there we need to make sure that our settings travel with the
    feature. There is not much that a user can do about it as this kind of
    management has to be done by the feature itself.
\stopitem

\stopitemize

\stopsection

It is the third case that we will give an example of in the next section. We
leave it up to the user if it gets noticed on the user interface.

\startsection [title={An example}]

A group of commands that has been reimplemented using a hybrid solution is
underlining or more generic: bars. Just take a look at the following examples and
try to get an idea on how to deal with grouping. Keep in mind that:

\startitemize[packed]
\startitem
    Colors are attributes and are resolved in the backend, so way after the
    paragraph has been typeset.
\stopitem
\startitem
    Overstrike is also handled by an attribute and gets applied in the backend as
    well, before colors are applied.
\stopitem
\startitem
    Nested overstrikes might have different settings.
\stopitem
\startitem
    An overstrike rule either inherits from the text or has its own color
    setting.
\stopitem
\stopitemize

First an example where we inherit color from the text:

\startbuffer
\definecolor[myblue][b=.75]
\definebar[myoverstrike][overstrike][color=]

Test \myoverstrike{%
    Test \myoverstrike{\myblue
        Test \myoverstrike{Test}
        Test}
    Test}
Test
\stopbuffer

\typebuffer \getbuffer

Because color is also implemented using attributes and processed later on we can
access that information when we deal with the bar.

The following example has its own color setting:

\startbuffer
\definecolor[myblue][b=.75]
\definecolor[myred] [r=.75]
\definebar[myoverstrike][overstrike][color=myred]

Test \myoverstrike{%
    Test \myoverstrike{\myblue
        Test \myoverstrike{Test}
        Test}
    Test}
Test
\stopbuffer

\typebuffer \getbuffer

See how we can color the levels differently:

\startbuffer
\definecolor[myblue] [b=.75]
\definecolor[myred]  [r=.75]
\definecolor[mygreen][g=.75]

\definebar[myoverstrike:1][overstrike][color=myblue]
\definebar[myoverstrike:2][overstrike][color=myred]
\definebar[myoverstrike:3][overstrike][color=mygreen]

Test \myoverstrike{%
    Test \myoverstrike{%
        Test \myoverstrike{Test}
        Test}
    Test}
Test
\stopbuffer

\typebuffer \getbuffer

Watch this:

\startbuffer
\definecolor[myblue] [b=.75]
\definecolor[myred]  [r=.75]
\definecolor[mygreen][g=.75]

\definebar[myoverstrike][overstrike][max=1,dy=0,offset=.5]
\definebar[myoverstrike:1][myoverstrike][color=myblue]
\definebar[myoverstrike:2][myoverstrike][color=myred]
\definebar[myoverstrike:3][myoverstrike][color=mygreen]

Test \myoverstrike{%
    Test \myoverstrike{%
        Test \myoverstrike{Test}
        Test}
    Test}
Test
\stopbuffer

\typebuffer \getbuffer

It this the perfect user interface? Probably not, but at least it keeps the
implementation quite simple.

The behaviour of the \MKIV\ implementation is roughly the same as in \MKII,
although now we specify the dimensions and placement in terms of the ratio of the
x-height of the current font.

\startbuffer
Test \overstrike{Test \overstrike{Test \overstrike{Test} Test} Test} Test \blank
Test \underbar  {Test \underbar  {Test \underbar  {Test} Test} Test} Test \blank
Test \overbar   {Test \overbar   {Test \overbar   {Test} Test} Test} Test \blank
Test \underbar  {Test \overbar   {Test \overstrike{Test} Test} Test} Test \blank
\stopbuffer

\typebuffer \getbuffer

As an extra this mechanism can also provide simple backgrounds. The normal
background mechanism uses \METAPOST\ and the advantage is that we can use
arbitrary shapes but it also carries some limitations. When the development of
\LUATEX\ is a bit further along the road I will add the possibility to use
\METAPOST\ shapes in this mechanism.

Before we come to backgrounds, first take a look at these examples:

\startbuffer
\startbar[underbar]  \input zapf \stopbar \blank
\startbar[underbars] \input zapf \stopbar \blank
\stopbuffer

\typebuffer \getbuffer

First notice that it is no problem to span multiple lines and that hyphenation is
not influenced at all. Second you can see that continuous rules are also
possible. From such a continuous rule to a background is a small step:

\startbuffer
\definebar
  [backbar]
  [offset=1.5,rulethickness=2.8,color=blue,
   continue=yes,order=background]

\definebar
  [forebar]
  [offset=1.5,rulethickness=2.8,color=blue,
   continue=yes,order=foreground]
\stopbuffer

\typebuffer \getbuffer

The following example code looks messy but this has to do with the fact that we
want properly spaced sample injection.

\startbuffer
from here
    \startcolor[white]%
        \startbar[backbar]%
            \input zapf
            \removeunwantedspaces
        \stopbar
    \stopcolor
\space till here
\blank
from here
    \startbar[forebar]%
        \input zapf
        \removeunwantedspaces
    \stopbar
\space till here
\stopbuffer

\typebuffer \getbuffer

Watch how we can use the order to hide content. By default rules are drawn on top
of the text.

Nice effects can be accomplished with transparencies:

\startbuffer
\definecolor [tblue] [b=.5,t=.25,a=1]
\setupbars [backbar] [color=tblue]
\setupbars [forebar] [color=tblue]
\stopbuffer

\typebuffer \getbuffer

We use as example:

\startbuffer[sample]
from here {\white \backbar{test test}
    \backbar {nested nested} \backbar{also also}} till here
from here {\white \backbar{test test
    \backbar {nested nested}          also also}} till here
from here {\white \backbar{test test
    \backbar {nested nested}          also also}} till here
\stopbuffer

\typebuffer[sample] \getbuffer[sample]

The darker nested variant is just the result of two transparent bars on top of
each other. We can limit stacking, for instance:

\startbuffer
\setupbars[backbar][max=1]
\setupbars[forebar][max=1]
\stopbuffer

\typebuffer \getbuffer

This gives

\getbuffer[sample]

There are currently some limitations that are mostly due to the fact that we use
only one attribute for this feature and a change in value triggers another
handling. So, we have no real nesting here.

The default commands are defined as follows:

\starttyping
\definebar[overstrike]  [method=0,dy= 0.4,offset= 0.5]
\definebar[underbar]    [method=1,dy=-0.4,offset=-0.3]
\definebar[overbar]     [method=1,dy= 0.4,offset= 1.8]

\definebar[overstrikes] [overstrike] [continue=yes]
\definebar[underbars]   [underbar]   [continue=yes]
\definebar[overbars]    [overbar]    [continue=yes]
\stoptyping

As the implementation is rather non|-|intrusive you can use bars
almost everywhere. You can underbar a whole document but equally
well you can stick to fooling around with for instance formulas.

\startbuffer
\definecolor [tred]   [r=.5,t=.25,a=1]
\definecolor [tgreen] [g=.5,t=.25,a=1]
\definecolor [tblue]  [b=.5,t=.25,a=1]

\definebar [mathred]   [backbar] [color=tred]
\definebar [mathgreen] [backbar] [color=tgreen]
\definebar [mathblue]  [backbar] [color=tblue]

\startformula
    \mathred{e} = \mathgreen{\white mc} ^ {\mathblue{\white e}}
\stopformula
\stopbuffer

\typebuffer

We get:

\getbuffer

We started this chapter with some words on grouping. In the examples you see no
difference between adding bars and for instance applying color. However you need
to keep in mind that this is only because behind the screens we keep the current
settings along with the attribute. In practice this is only noticeable when you
do lots of (local) changes to the settings. Take:

\starttyping
{test test test \setupbars[color=red] \underbar{test} test}
\stoptyping

This results in a local change in settings, which in turn will associate a new
attribute to \type {\underbar}. So, in fact the following underbar becomes a
different one than previous underbars. When the page is prepared, the unique
attribute value will relate to those settings. Of course there are more
mechanisms where such associations take place.

\stopsection

\startsection [title={More to come}]

Is this all there is? No, as usual the underlying mechanisms can be used for
other purposes as well. Take for instance inline notes:

\startbuffer
According to the wikipedia this is the longest English word:
pneumonoultramicroscopicsilicovolcanoconiosis~\shiftup {other long
words are pseudopseudohypoparathyroidism and
flocci­nauci­nihili­pili­fication}. Of course in languages like Dutch and
German we can make arbitrary long words by pasting words together.
\stopbuffer

\typebuffer

This will produce:

\getbuffer

I wonder when users really start using such features.

\stopsection

\startsection [title={Summary}]

Although under the hood the \MKIV\ bar commands are quite different from their
\MKII\ counterparts users probably won't notice much difference at first sight.
However, the new implementation does not interfere with the par builder and other
mechanisms. Plus, it is configurable and it offers more functionality. However,
as it is processed rather delayed, side effects might occur that are not
foreseen.

So, if you ever notice such unexpected side effects, you know where it might
result from: what you asked for is processed much later and by then the
circumstances might have changed. If you suspect that it relates to grouping
there is a simple remedy: define a new bar command in the document preamble
instead of changing properties mid|-|document. After all, you are supposed to
separate rendering and content in the first place.

\stopsection

\doifmodeelse {tugboat} {
    \StopArticle
} {
    \stopchapter
}

\stopcomponent