path: root/doc/context/sources/general/manuals/bidi/bidi.tex

% language=uk

% \showglyphs

\usemodule[abr-04]

\dontcomplain

\startbuffer[preamble-fonts]
\definefontfallback
  [Serif] [scheherazaderegular*arabic]
  [arabic] [force=yes,rscale=1.5]
\definefontfallback
  [SerifBold] [scheherazadebold*arabic]
  [arabic] [force=yes,rscale=1.5]
\definefontfallback
  [SerifItalic] [scheherazaderegular*arabic]
  [arabic] [force=yes,rscale=1.5]
\definefontfallback
  [SerifBoldItalic] [scheherazadebold*arabic]
  [arabic] [force=yes,rscale=1.5]

\definefontfallback
  [Serif] [sileot*hebrew]
  [hebrew] [force=yes]
\definefontfallback
  [SerifBold] [sileot*hebrew]
  [hebrew] [force=yes]
\definefontfallback
  [SerifItalic] [sileot*hebrew]
  [hebrew] [force=yes]
\definefontfallback
  [SerifBoldItalic] [sileot*hebrew]
  [hebrew] [force=yes]

\definefontfallback
  [Mono] [almfixed*none]
  [arabic] [force=yes]

\definefontfallback
  [Mono] [sileot*none]
  [hebrew] [force=yes,factor=1] % factor forces a monospace

\setupbodyfont
  [dejavu,10pt]
\stopbuffer

\startbuffer[preamble-languages]
\setuplanguage[ar][font=arabic,bidi=right]
\setuplanguage[he][font=hebrew,bidi=right]
\stopbuffer

\getbuffer[preamble-fonts]
\getbuffer[preamble-languages]

\setuplayout
  [backspace=15mm,
   topspace=15mm,
   footer=0pt,
   width=middle,
   height=middle]

\setuptyping
  [color=middleblue]

\setuptype
  [color=middleblue]

\definecolor
  [maincolor]
  [middleblue]

\setupwhitespace
  [big]

\setuphead
  [color=darkyellow]

\setuphead
  [chapter]
  [style=\bfc]

\setuphead
  [section]
  [style=\bfb]

\setuphead
  [subsection]
  [style=\bfa]

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\startluacode
    local report = logs.reporter("directions","check")
    local line   = 0
    function nodes.tracers.checkdirections(head)
        line = line + 1
        report("line: %i",line)
        for n in nodes.traverse_id(nodes.nodecodes.dir,head) do
            report("  %s (%i,%i)",n.dir,n.subtype,n.direction)
        end
        return head, false
    end

    nodes.tasks.appendaction("contributers","after","nodes.tracers.checkdirections")
    nodes.tasks.disableaction("contributers","nodes.tracers.checkdirections")
\stopluacode

\installtextracker
   {directions.check}
   {\ctxlua{nodes.tasks.enableaction("contributers","nodes.tracers.checkdirections")}}
   {\ctxlua{nodes.tasks.disableaction("contributers","nodes.tracers.checkdirections")}}

% \enabletrackers[directions.check]
% \disabletrackers[directions.check]

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\starttext

\startMPpage

    picture p, q, r, s ;

    p := textext("l2r") xsized .9PaperWidth ;
    q := textext("r2l") xsized .9PaperWidth ;
    r := textext("a few tips") xsized .9PaperWidth ;
    s := textext("\bf\ss hans\quad\space\quad hagen") xsized .5bbheight(p);

    p := p shifted - llcorner p ;
    q := q shifted - llcorner q ;
    r := r shifted - llcorner r ;
    s := s shifted - llcorner s ;

    fill Page withcolor "darkyellow" ;

    p := p shifted (.05PaperWidth,ypart .5[ulcorner Page, urcorner Page]-1.1bbheight(p)) ;
    q := q shifted (.05PaperWidth,ypart .5[ulcorner Page, urcorner Page]-1.1bbheight(p)-1.15bbheight(q)) ;
    r := r shifted (.05PaperWidth,ypart .5[llcorner Page, lrcorner Page]+0.3bbheight(r)) ;
    s := s shifted (.66PaperWidth,ypart .5[llcorner Page, lrcorner Page]+1.5bbheight(s)) ;

    draw p withcolor "lightgray" ;
    draw q withcolor "lightgray" ;
    draw r withcolor "middleblue" ;
    draw s withcolor "lightgray" ;

\stopMPpage

\startchapter[title=Introduction]

With \CONTEXT\ you can typeset in two directions: from left to right and from
right to left. In fact you can also combine these two directions, like this:

\startbuffer
There are many {\righttoleft \maincolor \it scripts in use} and some run into the
other direction. However, there is {\righttoleft \maincolor \it no fixed relation
{\lefttoright \black \it between the} direction of the script} and cars being
driven left or right of the road.
\stopbuffer

\typebuffer

\getbuffer

Even someone not familiar with right to left typesetting can see what happens
here, or not? In fact Luigi Scarso pointed out that the \type {fixed} reversed
into {\righttoleft \type {fixed}} but not in the example where {\bf fixed}
becomes {\righttoleft \bf fixed}. This signals an important property of the way
the text gets processed: you input something, at some points font features get
applied (like ligatures) and in the end the resulting glyph stream is reversed.
By that time the combination of {\bf f}+{\bf i} has become {\bf fi}! So, be
prepared for surprises.

This manual is written by a left to right user so don't expect a manual on
semitic typesetting. Also don't expect a (yet) complete manual. I'll add whatever
comes to mind. This is not a manual about Hebrew or Arabic, if only because I
can't read any of those scripts (languages). I leave that to others to cover.

{\em This is work in progress! So expect errors and typos.}

\startlines
Hans Hagen
Hasselt, NL
\stoplines

\stopchapter

\startchapter[title={Setting up fonts}]

So let's see how Arabic and Hebrew come out:

\startbuffer
The sentence \quotation {I have no clue what this means.} is translated (by
Google Translate) into \quotation {\ar \righttoleft ليس لدي أي فكرة عما يعنيه هذا.}
which is then translated back to \quotation {I have no idea what this means.} so
maybe arabic has no clue what a clue is. The suggested Arabic pronunciation is
\quotation {lays laday 'ayu fikrat eamaa yaenih hadha}. Hebrew also likes ideas
more: \quotation {\he \righttoleft אין לי מושג מה זה אומר}.
\stopbuffer

\typebuffer

\getbuffer

According to Idris Hamid the Arabic should actually be this: \quotation {\ar
لَيْسَ لَدَيَّ أَيُّ فِكْرَةٍ عَمَّا يَعْنِيهِ هٰذَا} and the transliteration \quotation {Laysa
ladayya ayyu fikratin ʿammā yaʿnihi hādhā}.

The \CONTEXT\ (or any \TEX) ecosystem deals with languages and fonts. Languages
(that relate to scripts) have specific characteristics, like running from right
to left, and fonts provide a repertoire of glyphs and features. There is no real
(standard) relationship between these. In for instance browsers, there are
automatic fallback systems for missing characters in a font: another font is
taken. These fallbacks are often not easy to tweak.

In this document we use Dejavu and although that font has Arabic shapes in its
monospace variant, the serifs come without them (at least when I write this
down). Before we actually define the bodyfont we hook in some fallbacks. The
typescript for Dejavu has lines like this:

\starttyping
\definefontsynonym
  [SerifBoldItalic]
  [name:dejavuserifbolditalic]
  [features=default,
   fallbacks=SerifBoldItalic]
\stoptyping

This permits us to do this:

\typebuffer[preamble-fonts]

In addition we set up the languages:

\typebuffer[preamble-languages]

The following example demonstrates what the effects of these commands are:

\startbuffer
{لَيْسَ لَدَيَّ أَيُّ فِكْرَةٍ عَمَّا يَعْنِيهِ هٰذَا.}
{אין לי מושג מה זה אומר.}
{\righttoleft لَيْسَ لَدَيَّ أَيُّ فِكْرَةٍ عَمَّا يَعْنِيهِ هٰذَا.}
{\righttoleft אין לי מושג מה זה אומר.}
{\ar \righttoleft لَيْسَ لَدَيَّ أَيُّ فِكْرَةٍ عَمَّا يَعْنِيهِ هٰذَا.}
{\he \righttoleft אין לי מושג מה זה אומר.}
{\ar لَيْسَ لَدَيَّ أَيُّ فِكْرَةٍ عَمَّا يَعْنِيهِ هٰذَا.}
{\he אין לי מושג מה זה אומר.}
\stopbuffer

\typebuffer

\startlines
\getbuffer
\stoplines

In principle you can also rely on automatic direction changes, for instance
by using the following command:

\starttyping
\setupdirections
  [bidi=global,
   method=three]
\stoptyping

But that doesn't do a font switch for you, nor does it do any of the other
language related settings. It really helps if you properly tag your
document content, as in:

\starttyping
{\ar لَيْسَ لَدَيَّ أَيُّ فِكْرَةٍ عَمَّا يَعْنِيهِ هٰذَا.}
{\he אין לי מושג מה זה אומר.}
\stoptyping

One reason to set the \type {font} parameter for a language is that it will
activate the right features in a font. Instead of falling back on some default,
we can be very specific in what we want to enable.

\stopchapter

\startchapter[title=A mixed layout]

The typesetting engine normally works from left to right and top to bottom. Going
from right to left actually involved two decisions:

\startitemize[packed]
\startitem the direction of the display elements, the paragraphs \stopitem
\startitem the direction of the inline text, the lines \stopitem
\stopitemize

The first one is kept track of in a state variable. Every paragraph starts with
a node that carries, among other information, that state. This node is added
automatically and does not interfere with the typesetting. The inline direction
is more intrusive as it is marked by nodes that indicate the beginning and end
of a reversed strip. This mechanism is rather reliable and normally works out
well. Take this:

\startbuffer
left {\righttoleft right} left
left{ \righttoleft right} left
left {\righttoleft right }left
left{ \righttoleft right }left
\stopbuffer

\typebuffer

You can see that we need to be careful with spaces as they can end up inside or
outside a substream and by swapping next to each other:

\startlines
\getbuffer
\stoplines

We can wrap the lines in boxes as in:

\startbuffer
\hbox{left\space{\bf\righttoleft right}\space left}
\hbox{left{\bf\space \righttoleft right}\space left}
\hbox{left\space{\bf\righttoleft right\space}left}
\hbox{left{\bf\space\righttoleft right\space}left}
\stopbuffer

\typebuffer

\definecolor[ShineThrough][s=0,a=1,t=.2]

When visualize the spaces we get this:

\startlines\ShineThrough
\showmakeup[space,hbox]\getbuffer
\stoplines

The space of a normal and bold font in the same family normally is the same but
let's mix with a larger size:

\startbuffer
\hbox{left {\bfa\righttoleft right} left}
\hbox{left{\bfa\space \righttoleft right} left}
\hbox{left {\bfa\righttoleft right }left}
\hbox{left{\bfa\space\righttoleft right }left}
\stopbuffer

\typebuffer

Now we get the following. As you can see, it really matters where we put the
braces.

\startlines
\ShineThrough\showmakeup[space,hbox]\getbuffer
\stoplines

Once you are accustomed to tagging and \TEX\ you will probably not fall into
these traps. In \in {figure} [fig:spaces] we show a large version.

\startplacefigure[location=top,title={Watch your spaces!},reference=fig:spaces]
    \scale
      [width=\hsize]
      {\vbox{\ShineThrough\showmakeup[space,hbox]\getbuffer}}
\stopplacefigure


The \type {\righttoleft} command actually has two meanings. This can best be seen
from an example.

\startbuffer
\righttoleft \bf How will this come out?
\stopbuffer

\typebuffer

\start \getbuffer \par \stop

\startbuffer
And \righttoleft \bf how will this come out?
\stopbuffer

\typebuffer

\start \getbuffer \par \stop

When we start a paragraph (or in \TEX\ speak: when we are still in vertical mode)
the paragraph direction as well as the inline direction is set. Otherwise only
the inline direction is set. There are low level \TEX\ commands (primitives) to
set the direction but you can best {\em not} use these because we need to do a
bit more than that.

There are quite some low level commands related to changing directions. Some deal
with the layout, some with boxes. We might provide more in the future.

\starttabulate[|l|p|]
\FL
\NC \type {\lefttoright}                  \NC l2r dir node or paragraph property \NC \NR
\NC \type {\righttoleft}                  \NC r2l dir node or paragraph property \NC \NR
\NC \type {\checkedlefttoright}           \NC l2r dir node or paragraph property (unless already set) \NC \NR
\NC \type {\checkedrighttoleft}           \NC r2l dir node or paragraph property (unless already set) \NC \NR
\ML
\NC \type {\synchronizeinlinedirection}   \NC pickup a (possibly) reset state \NC \NR
\NC \type {\synchronizelayoutdirection}   \NC pickup a (possibly) reset state \NC \NR
\NC \type {\synchronizedisplaydirection}  \NC pickup a (possibly) reset state \NC \NR
\ML
\NC \type {\righttolefthbox}              \NC r2l \type {\hbox} \NC \NR
\NC \type {\lefttorighthbox}              \NC l2r \type {\hbox} \NC \NR
\NC \type {\righttoleftvbox}              \NC r2l \type {\vbox} \NC \NR
\NC \type {\lefttorightvbox}              \NC l2r \type {\vbox} \NC \NR
\NC \type {\righttoleftvtop}              \NC r2l \type {\vtop} \NC \NR
\NC \type {\lefttorightvtop}              \NC l2r \type {\vtop} \NC \NR
\ML
\NC \type {\leftorrighthbox}              \NC l2r or r2l \type {\hbox} \NC \NR
\NC \type {\leftorrightvbox}              \NC l2r or r2l \type {\vbox} \NC \NR
\NC \type {\leftorrightvtop}              \NC l2r or r2l \type {\vtop} \NC \NR
\ML
\NC \type {\autodirhbox}                  \NC l2r or r2l \type {\hbox} (a bit more clever) \NC \NR
\NC \type {\autodirvbox}                  \NC l2r or r2l \type {\vbox} (a bit more clever) \NC \NR
\NC \type {\autodirvtop}                  \NC l2r or r2l \type {\vtop} (a bit more clever) \NC \NR
\ML
\NC \type {\bidilre}                      \NC character \type {U+202A}, enforce l2r state    \NC \NR
\NC \type {\bidirle}                      \NC character \type {U+202B}, enforce r2l state    \NC \NR
\NC \type {\bidipop}                      \NC character \type {U+202C}, return to last state \NC \NR
\NC \type {\bidilro}                      \NC character \type {U+202D}, override l2r state   \NC \NR
\NC \type {\bidirlo}                      \NC character \type {U+202E}, override r2l state   \NC \NR
\NC \type {\lefttorightmark} \type {\lrm} \NC character \type {U+200E}, l2r indicator        \NC \NR
\NC \type {\righttoleftmark} \type {\rlm} \NC character \type {U+200F}, r2l indicator        \NC \NR
\ML
\NC \type {\dirlre}                       \NC switch to l2r mode using \type {\bidilre} or \lefttoright \NC \NR
\NC \type {\dirrle}                       \NC switch to r2l mode using \type {\bidirle} or \righttoleft \NC \NR
\NC \type {\dirlro}                       \NC enforce l2r mode   using \type {\bidilro} or \lefttoright \NC \NR
\NC \type {\dirrlo}                       \NC enforce r2l mode   using \type {\bidirlo} or \righttoleft \NC \NR
\ML
\NC \type {\naturalhbox}                  \NC a normal l2r \type {hbox}  \NC \NR
\NC \type {\naturalvbox}                  \NC a normal l2r \type {vbox}  \NC \NR
\NC \type {\naturalvtop}                  \NC a normal l2r \type {vtop}  \NC \NR
\NC \type {\naturalhpack}                 \NC a normal l2r \type {hpack} \NC \NR
\NC \type {\naturalvpack}                 \NC a normal l2r \type {vpack} \NC \NR
\LL
\stoptabulate

When we talk about layout, we mean the overall layout, concerning the document as
a whole. We can have a dominantly l2r, dominantly r2l or mixed setup. In a next
chapter we will give more details on the dominant setup. Here we stick to
mentioning that the document flow direction is set with

\starttyping
\setupalign[r2l] % or r2l
\stoptyping

When a command to setup an environment has a \type {align} parameter, the same
keywords can be uses as part of the specification. \footnote {We haven't tested
all situations and possible interferences. Just report anomalies to the mailing
list.}

\stopchapter

\startchapter[title={Numbering and positioning}]

todo: columns (direction key), numbers (conversionsets), margins (begin/end), etc

\stopchapter

\startchapter[title={The \LUA\ interface}]

We assume that you run \CONTEXT\ \MKIV\ in combination with \LUATEX. Direction
support in this engine has been improved over time. Originally the \OMEGA\
(\ALEPH) direction model was used but in the meantime it has been stripped to the
basics, and what used to be so called whatsits (extension nodes) are now first
class nodes. In the \LUA\ interface we still support the:

\startitemize[packed]
\startitem
    \type {TLT} and \type {TRT} keywords for local par nodes
\stopitem
\startitem
    \type {+TLT}, \type {-TLT}, \type {+TRT} and \type {-TRT} keywords for
    direction nodes
\stopitem
\stopitemize

However, because we use dedicated nodes, and because nodes actually store numbers
and not strings we also expose the numeric model:

\starttabulate[|l|l|l|]
\NC 0 \NC \type {TLT} \NC left to right \NC \NR
\NC 1 \NC \type {TRT} \NC right to left \NC \NR
\NC 2 \NC \type {LTL} \NC not used in context \NC \NR
\NC 3 \NC \type {RTT} \NC not used in context \NC \NR
\stoptabulate

These values are used for local par nodes as well as direction nodes. In addition
a direction node has a subtype:

\starttabulate[|l|l|l|]
\NC 0 \NC \type {normal} \NC comparable to \type {+} \NC \NR
\NC 1 \NC \type {cancel} \NC comparable to \type {-} \NC \NR
\stoptabulate

The \type {dir} field uses the strings, the \type {direction} field the number
(both of course share the same internal node variable).

\stopchapter

\stoptext

% \defineconversionset [\s!default]             [] [numbers]
% \defineconversionset [\v!number]              [] [numbers]
% \defineconversionset [\v!pagenumber]          [] [numbers]
% \defineconversionset [\v!appendix:\s!default] [Romannumerals,Characters] [numbers]
% \defineconversionset [\v!formula]             [numbers,characters] % no \v! ?