diff options
Diffstat (limited to 'doc/context/sources/general/manuals/hybrid/hybrid-goodies.tex')
| -rw-r--r-- | doc/context/sources/general/manuals/hybrid/hybrid-goodies.tex | 648 |
1 files changed, 648 insertions, 0 deletions
diff --git a/doc/context/sources/general/manuals/hybrid/hybrid-goodies.tex b/doc/context/sources/general/manuals/hybrid/hybrid-goodies.tex new file mode 100644 index 000000000..f6a2317f0 --- /dev/null +++ b/doc/context/sources/general/manuals/hybrid/hybrid-goodies.tex @@ -0,0 +1,648 @@ +% language=uk + +\usetypescriptfile[type-husayni] + +\startcomponent hybrid-goodies + +\environment hybrid-environment + +% this will change + +\definefontfeature + [husayni-none] + [analyze=yes,mode=node, + language=dflt,script=arab, + ccmp=yes] + +\definefontfeature + [husayni-default] + [analyze=yes,mode=node, + language=dflt,script=arab, + ccmp=yes,init=yes,medi=yes,fina=yes, + rlig=yes,calt=yes,salt=yes,anum=yes, + kern=yes,curs=yes,mark=yes,mkmk=yes, + ss01=yes,ss03=yes,ss10=yes,ss12=yes,ss15=yes,ss16=yes, + ss19=yes,ss24=yes,ss25=yes,ss26=yes,ss27=yes,ss31=yes, + ss34=yes,ss35=yes,ss36=yes,ss37=yes,ss38=yes,ss41=yes, + ss43=yes] + +\definefontfeature + [husayni-first-order] + [script=arab,ss01=yes,ss03=yes,ss05=yes, + ss10=yes,ss12=yes,ss15=yes,ss16=yes,ss19=yes,ss24=yes, + ss25=yes,ss26=yes,ss27=yes,ss31=yes,ss34=yes,ss35=yes, + ss36=yes,ss37=yes,ss38=yes,ss41=yes,ss42=yes,ss43=yes] + +\definefontfeature + [husayni-stack-jiim-multi-level] + [script=arab,ss05=yes] + +\definefontfeature + [husayni-minimal-stretching] + [script=arab, + ss05=yes,ss09=yes,ss06=yes,ss13=yes,ss17=yes,ss40=yes, + js11=yes,js14=yes,js16=yes] + +\definefontfeature + [husayni-maximal-stretching] + [script=arab, + ss05=yes,ss09=yes,ss06=yes,ss13=yes,ss17=yes,ss40=yes, + js13=yes,js14=yes,js16=yes] + +\definefontfeature + [husayni-chop-haa] + [script=arab, + ss05=yes,ss09=yes,ss06=yes,ss13=yes,ss17=yes,ss54=yes] + +\definefontfeature + [husayni-normal] + [goodies=husayni, + featureset=default] + +\definefont[ArabicFontNone][husayni*husayni-none at 40pt] +\definefont[ArabicFontFull][husayni*husayni-default at 40pt] % husayni-normal + +\startchapter[title={Font Goodies}] + +\startsection[title={Introduction}] + +The Oriental \TEX\ project is one of the first and more ambitious users of +\LUATEX. A major undertaking in this project is the making of a rather full +features and complex font for typesetting Arabic. As the following text will show +some Arabic, you might get the impression that I'm an expert but be warned that +I'm far from that. But as Idris compensates this quite well the team has a lot of +fun in figuring out how to achieve our goals using \OPENTYPE\ technology in +combination with \LUATEX\ and \MKIV. A nice side effect of this is that we end up +with some neat tricks in the \CONTEXT\ core. + +Before we come to some of these goodies, an example of Arabic is given that +relates quite well to the project. It was first used at the euro\TEX\ 2009 +meeting. Take the following 6 shapes: + +\starttabulate[|c|c|c|c|c|c|] +\NC \ArabicFontFull ل \NC \ArabicFontFull و \NC \ArabicFontFull ا \NC \ArabicFontFull ت \NC \ArabicFontFull ي \NC \ArabicFontFull خ \NC \NR +\NC \type{l} \NC \type{w} \NC \type{ā} \NC \type{t} \NC \type{ī} \NC \type{kh} \NC \NR +\stoptabulate + +With these we can make the name \LUATEX\ and as we use a nice script we can +forget about the lowered~E. Putting these characters in sequence is not enough as +Arabic typesetting has to mimick the subtle aspects of scribes. + +In Latin scripts we have mostly one|-|to|-|one and many|-|to|-|one substitutions. +These can happen in sequence which in in practice boils down to multiple passes +over the stream of characters. In this process sometimes surrounding characters +(or shapes) play a role, for instance ligatures are not always wanted and their +coming into existence might depend on neighbouring characters. In some cases +glyphs have to be (re)positioned relative to each other. While in Latin scripts +the number of substitutions and positioning is not that large but in advanced +Arabic fonts it can be pretty extensive. + +With \OPENTYPE\ we have some machinery available, so we try to put as much logic +in the font as possible. However, in addition we have some dedicated optimizing +routines. The whole process is split into a couple if stages. + +The so called First|-|Order Analysis puts a given character into isolated, +initial, middle, or final state. Next, the Second|-|Order Analysis looks at the +characters and relates this state to what characters precede or succeed it. Based +on that state we do character substitutions. There can be multiple analysis and +replacements in sequence. We can do some simple aesthetic stretching and +additional related replacements. We need to attach identity marks and vowels in +proper but nice looking places. In most cases we're then done. Contrary to other +fonts we don't use many ligatures but compose characters. + +The previous steps already give reasonable results and implementing it also +nicely went along with the development of \LUATEX\ and \CONTEXT\ \MKIV. Currently +we're working on extending and perfecting the font to support what we call +Third|-|Order Contextual Analysis. This boils down to an interplay between the +paragraph builder and additional font features. In order to get pleasing spacing +we apply further substitutions, this time with wider or narrower shapes. When +this is done we need to reattach identity marks and vowels. Optionally we can +apply \HZ\ like stretching as a finishing touch but so far we didn't follow that +route yet. + +So, let's see how we can typeset the word \LUATEX\ in Arabic using some of these +techniques. + +\startlines +no order (kh ī t ā w [u] l)\hfilll {\righttoleft\ArabicFontNone لُواتيخ} +first order \hfilll {\subff{husayni-first-order}\righttoleft\ArabicFontFull لُواتيخ} +second order \hfilll {\righttoleft\ArabicFontFull لُواتيخ} +second order (Jiim-stacking) \hfilll {\addff{husayni-stack-jiim-multi-level}\righttoleft\ArabicFontFull لُواتيخ} +minimal stretching \hfilll {\addff{husayni-minimal-stretching}\righttoleft\ArabicFontFull لُواتيخ} +maximal stretching (level 3) \hfilll {\addff{husayni-maximal-stretching}\righttoleft\ArabicFontFull لُواتيخ} +chopped letter khaa (for e.g.\ underlining) \hfilll {\addff{husayni-chop-haa}\righttoleft\ArabicFontFull لُواتيخ} +\stoplines + +As said, this font is quite complex in the sense that it has many features and +associated lookups. In addition to the usual features we have stylistic and +justification variants. As these are not standardized (after all, each font can +have its own look and feel and associated treatments) we store some information +in the goodies files that ship with this font. + +\startbuffer[stylistics] +\startluacode + local goodies = fonts.goodies.load("husayni") + local stylistics = goodies and goodies.stylistics + if stylistics then + local col, row, type = context.NC, context.NR, context.type + context.starttabulate { "|l|pl|" } + col() context("feature") col() context("meaning") col() row() + for feature, meaning in table.sortedpairs(stylistics) do + col() type(feature) col() type(meaning) col() row() + end + context.stoptabulate() + end +\stopluacode +\stopbuffer + +\getbuffer[stylistics] + +It is highly unlikely that a user will remember all these features, which is why +there will be a bunch of predefined combinations. These are internalized as +follows: + +\startbuffer[featuresets] +\startluacode + local goodies = fonts.goodies.load("husayni") + local featuresets = goodies and goodies.featuresets + if featuresets then + local col, row, type = context.NC, context.NR, context.type + context.starttabulate { "|l|pl|" } + col() context("featureset") col() context("definitions") col() row() + for featureset, definitions in table.sortedpairs(featuresets) do + col() type(featureset) col() + for k, v in table.sortedpairs(definitions) do + type(string.format("%s=%s",k,tostring(v))) + context.quad() + end + col() row() + end + context.stoptabulate() + end +\stopluacode +\stopbuffer + +\getbuffer[featuresets] + +\stopsection + +\startsection[title={Color}] + +One of the objectives of the oriental \TEX\ project is to bring color to typeset +Arabic. When Idris started making samples with much manual intervention it was +about time to figure out if it could be supported by a bit of \LUA\ code. + +As the colorization concerns classes of glyphs (like vowels) this is something +that can best be done after all esthetics have been sorted out. Because things +like coloring are not part of font technology and because we don't want to misuse +the \OPENTYPE\ feature mechanisms for that, the solution lays in an extra file +that describes these goodies. + +\startbuffer[goodies-1] +\definefontfeature + [husayni-colored] + [goodies=husayni, + colorscheme=default, + featureset=default] +\stopbuffer + +\startbuffer[goodies-2] +\start + \definedfont[husayni*husayni-colored at 72pt] + \righttoleft + \resetfontcolorscheme لُواتيخ ألف ليلة وليلة \par + \setfontcolorscheme [1]لُواتيخ ألف ليلة وليلة \crlf + \setfontcolorscheme [2]لُواتيخ ألف ليلة وليلة \crlf +\stop +\stopbuffer + +\getbuffer[goodies-1,goodies-2] + +The second and third of these three lines have colored vowels and identity marks. +So how did we get the colors? There are actually two mechanisms involved in this: + +\startitemize[packed] +\startitem we need to associate colorschemes with classed of glyphs \stopitem +\startitem we need to be able to turn on and off coloring \stopitem +\stopitemize + +The first is done by loading goodies and selecting a colorscheme: + +\typebuffer[goodies-1] + +Turning on and off coloring is done with two commands (we might provide a proper +environment for this) as shown in: + +\typebuffer[goodies-2] + +If you look closely at the feature definition you'll notice that we also choose a +default featureset. For most (latin) fonts the regular feature definitions are +convenient, but for fonts that are used for Arabic there are preferred +combinations of features as there can be many. + +Currently the font we use here has the following colorschemes: + +\startbuffer[colorschemes] +\startluacode + local goodies = fonts.goodies.load("husayni") + local colorschemes = goodies and goodies.colorschemes + if colorschemes then + local col, row, type = context.NC, context.NR, context.type + context.starttabulate { "|l|pl|" } + col() context("colorscheme") col() context("numbers") col() row() + for colorscheme, numbers in table.sortedpairs(colorschemes) do + col() type(colorscheme) col() + for i=1,#numbers do + type(i) + context.quad() + end + col() row() + end + context.stoptabulate() + end +\stopluacode +\stopbuffer + +\getbuffer[colorschemes] + +\stopsection + +\startsection[title={The goodies file}] + +In principle a goodies files can contain anuy data that makes sense but in order +to be useable some entries have a prescribed structure. A goodies file looks as +follows: + +\starttyping +return { + name = "husayni", + version = "1.00", + comment = "Goodies that complement the Husayni font by Idris Samawi Hamid.", + author = "Idris Samawi Hamid and Hans Hagen", + featuresets = { + default = { + key = value, <table>, ... + }, + ... + }, + stylistics = { + key = value, ... + }, + colorschemes = { + default = { + [1] = { + "glyph_a.one", "glyph_b.one", ... + }, + ... + } + } +} +\stoptyping + +We already saw the list of special features and these are defined in the \type +{stylistics} stable. In this document, that list was typeset using the following +(hybrid) code: + +\typebuffer[stylistics] + +The table with colorscheme that we showed is generated with: + +\getbuffer[colorschemes] + +In a similar fashion we typeset the featuresets: + +\typebuffer[featuresets] + +The unprocessed \type {featuresets} table can contain one or more +named sets and each set can be a mixture of tables and key value +pairs. Say that we have: + +\starttyping + default = { + kern = "yes", { ss01 = "yes" }, { ss02 = "yes" }, "mark" + } +\stoptyping + +Given the previous definition, the order of processing is as follows. + +\startitemize[packed,n] +\startitem \type {{ ss01 = "yes" }} \stopitem +\startitem \type {{ ss02 = "yes" }} \stopitem +\startitem \type {mark} (set to \type {"yes"}) \stopitem +\startitem \type {kern = "yes"} \stopitem +\stopitemize + +So, first we process the indexed part if the list, and next the hash. Already set +values are not set again. The advantage of using a \LUA\ table is that you can +simplify definitions. Before we return the table we can define local variables, +like: + +\starttyping +local one = { ss01 = "yes" } +local two = { ss02 = "yes" } +local pos = { kern = "yes", mark = "yes" } +\stoptyping + +and use them in: + +\starttyping +default = { + one, two, pos +} +\stoptyping + +That way we we can conveniently define all kind of interesting combinations +without the need for many repetitive entries. + +The \type {colorsets} table has named subtables that are (currently) indexed by +number. Each number is associated with a color (at the \TEX\ end) and is coupled +to a list of glyphs. As you can see here, we use the name of the glyph. We prefer +this over an index (that can change during development of the font). We cannot +use \UNICODE\ points as many such glyphs are just variants and have no unique +code. + +\stopsection + +\startsection[title={Optimizing Arabic}] + +\usemodule[abr-01,narrowtt] + +\enabletrackers[fonts.goodies,nodes.optimizer] + +The ultimate goal of the Oriental \TEX\ project is to improve the look and feel +of a paragraph. Because \TEX\ does a pretty good job on breaking the paragraph +into lines, and because complicating the paragraph builder is not a good idea, we +finally settled on improving the lines that result from the par builder. This +approach is rather close to what scribes do and the advanced Husayni font +provides features that support this. + +In principle the current optimizer can replace character expansion but that would +slow down considerably. Also, for that we first have to clean up the experimental +\LUA\ based par builder. + +After several iterations the following approach was chosen. + +\startitemize + +\startitem + We typeset the paragraph with an optimal feature set. In our case this is + \type {husayni-default}. +\stopitem + +\startitem + Next we define two sets of additional features: one that we can apply to + shrink words, and one that does the opposite. +\stopitem + +\startitem + When the line has a badness we don't like, we either stepwise shrink words or + stretch them, depending on how bad things are. +\stopitem + +\stopitemize + +The set that takes care of shrinking is defined as: + +\starttyping +\definefontfeature + [shrink] + [husayni-default] + [flts=yes,js17=yes,ss05=yes,ss11=yes,ss06=yes,ss09=yes] +\stoptyping + +Stretch has a few more variants: + +\starttyping +\definefontfeature + [minimal_stretching] + [husayni-default] + [js11=yes,js03=yes] +\definefontfeature + [medium_stretching] + [husayni-default] + [js12=yes,js05=yes] +\definefontfeature + [maximal_stretching] + [husayni-default] + [js13=yes,js05=yes,js09=yes] +\definefontfeature + [wide_all] + [husayni-default] + [js11=yes,js12=yes,js13=yes,js05=yes,js09=yes] +\stoptyping + +Next we define a font solution: + +\starttyping +\definefontsolution + [FancyHusayni] + [goodies=husayni, + less=shrink, + more={minimal_stretching,medium_stretching,maximal_stretching,wide_all}] +\stoptyping + +Because these featuresets relate quite closely to the font design we don't use +this way if defining but put the definitions in the goodies file: + +\startntyping + ..... + featuresets = { -- here we don't have references to featuresets + default = { + default, + }, + minimal_stretching = { + default, js11 = yes, js03 = yes, + }, + medium_stretching = { + default, js12=yes, js05=yes, + }, + maximal_stretching= { + default, js13 = yes, js05 = yes, js09 = yes, + }, + wide_all = { + default, js11 = yes, js12 = yes, js13 = yes, js05 = yes, js09 = yes, + }, + shrink = { + default, flts = yes, js17 = yes, ss05 = yes, ss11 = yes, ss06 = yes, ss09 = yes, + }, + }, + solutions = { -- here we have references to featuresets, so we use strings! + experimental = { + less = { "shrink" }, + more = { "minimal_stretching", "medium_stretching", "maximal_stretching", "wide_all" }, + }, + }, + ..... +\stopntyping + +Now the definition looks much simpler: + +\startbuffer +\definefontsolution + [FancyHusayni] + [goodies=husayni, + solution=experimental] +\stopbuffer + +% unhbox to show stretch - shrink + +\typebuffer \getbuffer + +{\em I want some funny text (complete with translation). Actually I want all +examples translated.} + +\startbuffer[sample] +قد صعدنا +ذرى الحقائق بأقدام النبوة و الولاية و نورنا +سبع طبقات أعلام الفتوى بالهداية فنحن ليوث +الوغى و غيوث الندى و طعان العدى و فينا السيف و +القلم في العاجل و لواء الحمد +و الحوض في الآجل و أسباطنا حلفاء +الدين و خلفاء النبيين و مصابيح الأمم و مفاتيح +الكرم فالكليم ألبس حلة الاصطفاء لما عهدنا +منه الوفاء و روح القدس في جنان الصاقورة ذاق من +حدائقنا الباكورة و شيعتنا الفئة الناجية و +الفرقة الزاكية و صاروا لنا ردءا و صونا و على +الظلمة ألبا و عونا و سينفجر لهم ينابيع +الحيوان بعد لظى النيران لتمام آل حم و طه و +الطواسين من السنين و هذا الكتاب درة من درر +الرحمة و قطرة من بحر الحكمة و كتب الحسن بن +علي العسكري في سنة أربع و خمسين و مائتين +\stopbuffer + +\startbuffer +\definedfont[husayni*husayni-default at 24pt] +% todo: factor ivm grid, so the next line looks hackery: +\expanded{\setuplocalinterlinespace[line=\the\dimexpr2\lineheight]} +\setfontsolution[FancyHusayni]% command will change +\enabletrackers[builders.paragraphs.solutions.splitters.colors] +\righttoleft \getbuffer[sample] \par +\disabletrackers[builders.paragraphs.solutions.splitters.colors] +\resetfontsolution +\stopbuffer + +In the following example the yellow words are stretched and the green ones are +shrunken.\footnote {Make sure that the paragraph is finished (for instance using +\type {\par} before resetting it.)} + +\typebuffer + +\start \getbuffer \stop + +% \setfontsolution[FancyHusayni]x\par\resetfontsolution +% \setfontsolution[FancyHusayni]x\par\resetfontsolution +% \setfontsolution[FancyHusayni]x\par\resetfontsolution +% \setfontsolution[FancyHusayni]x\par\resetfontsolution +% \setfontsolution[FancyHusayni]x\par\resetfontsolution +% \setfontsolution[FancyHusayni]x\par\resetfontsolution +% \setfontsolution[FancyHusayni]x\par\resetfontsolution +% \setfontsolution[FancyHusayni]x\par\resetfontsolution + +% \startbuffer[sample] +% \dorecurse{50}{الحمد \recurselevel\space} +% \stopbuffer + +This mechanism is somewhat experimental as is the (user) interface. It is also +rather slow compared to normal processing. There is room for improvement but I +will do that when other components are more stable so that simple variants (that +we can use here) can be derived. + +When criterium~0 used above is changed into for instance~5 processing is faster. +When you enable a preroll processing is more time consuming. Examples of settings +are: + +\starttyping +\setupfontsolutions[method={preroll,normal},criterium=2] +\setupfontsolutions[method={preroll,random},criterium=5] +\setupfontsolutions[method=reverse,criterium=8] +\setupfontsolutions[method=random,criterium=2] +\stoptyping + +Using a preroll is slower because it first tries all variants and then settles +for the best; otherwise we process the first till the last solution till the +criterium is satisfied. + +% {\em Todo: show normal, reverse and random.} +% {\em Todo: bind setting to paragraph.} + +\stopsection + +\startsection[title={Protrusion and expansion}] + +There are two entries in the goodies file that relate to advanced parbuilding: +\type {protrusions} and \type {expansions}. + +\starttyping +protrusions = { + vectors = { + pure = { + [0x002C] = { 0, 1 }, -- comma + [0x002E] = { 0, 1 }, -- period + ..... + } + } +} +\stoptyping + +These vectors are similar to the ones defined globally but the vectors defined in +a goodie file are taken instead when present. + +\stopsection + +\startsection[title={Filenames and properties}] + +As filenames and properties of fonts are somewhat of an inconsistent mess, we can +use the goodies to provide more information: + +\starttyping +files = { + name = "antykwapoltawskiego", -- shared + list = { + ["AntPoltLtCond-Regular.otf"] = { + -- name = "antykwapoltawskiego", + style = "regular", + weight = "light", + width = "condensed", + }, + ..... + } + } +} +\stoptyping + +Internally this will become a lookup tree so that we can have a predictable +specifier: + +\starttyping +\definefont[MyFontA][antykwapoltawskiego-bold-italic] +\definefont[MyFontB][antykwapoltawskiego-normal-italic-condensed] +\definefont[MyFontC][antykwapoltawskiego-light-regular-semicondensed] +\stoptyping + +Of course one needs to load the goodies. One way to force that is: + +\starttyping +\loadfontgoodies[antykwapoltawskiego] +\stoptyping + +The Antykwa Poltawskiego family is rather large and provides all kind of +combinations. + +\startbuffer +\usemodule[fonts-goodies] +\showfontgoodiesfiles[name=antykwapoltawskiego] +\stopbuffer + +\startpacked +\getbuffer +\stoppacked + +This list is generated with: + +\typebuffer + +\stopsection + +\stopchapter + +\stopcomponent |