2016-08-25 14:18:00

20 files changed, 258 insertions, 18 deletions

diff --git a/doc/context/sources/general/manuals/still/still-acrobat-math.png b/doc/context/sources/general/manuals/still/still-acrobat-math.png
new file mode 100644
index 000000000..e2f842a36
--- /dev/null
+++ b/doc/context/sources/general/manuals/still/still-acrobat-math.png
diff --git a/doc/context/sources/general/manuals/still/still-acrobat-text.png b/doc/context/sources/general/manuals/still/still-acrobat-text.png
new file mode 100644
index 000000000..1c0d97e91
--- /dev/null
+++ b/doc/context/sources/general/manuals/still/still-acrobat-text.png
diff --git a/doc/context/sources/general/manuals/still/still-math-stripe.png b/doc/context/sources/general/manuals/still/still-math-stripe.png
new file mode 100644
index 000000000..606ea7961
--- /dev/null
+++ b/doc/context/sources/general/manuals/still/still-math-stripe.png
diff --git a/doc/context/sources/general/manuals/still/still-math.tex b/doc/context/sources/general/manuals/still/still-math.tex
index 473d664b7..9918b5c79 100644
--- a/doc/context/sources/general/manuals/still/still-math.tex
+++ b/doc/context/sources/general/manuals/still/still-math.tex
@@ -416,7 +416,7 @@ appearance depends on the font used but operating systems tend to consult
 multiple fonts when the core font of the editor doesn't provide a glyph.
 
 \startlinecorrection
-    \externalfigure[math-stripe.png][width=\textwidth]
+    \externalfigure[still-math-stripe.png][width=\textwidth]
 \stoplinecorrection
 
 Here I show a small portion of the stripe with buttons that inject the shown
diff --git a/doc/context/sources/general/manuals/still/still-one.tex b/doc/context/sources/general/manuals/still/still-one.tex
new file mode 100644
index 000000000..7d0c392c7
--- /dev/null
+++ b/doc/context/sources/general/manuals/still/still-one.tex
@@ -0,0 +1,234 @@
+% language=uk
+
+\usemodule[art-01,abr-02] \setupbodyfont[11pt]
+
+\starttext
+
+\startchapter[title=\LUATEX\ going stable]
+
+\startsection[title=Introduction]
+
+We're closing in on version 1.0 of \LUATEX\ and at the time of this writing (mid
+April 2016) we're at version 0.95. The last decade we've reported on a regular
+basis about progress in user group journals, \CONTEXT\ related documents and the
+\LUATEX\ manual and it makes no sense to repeat ourselves.
+
+So where do we stand now? I will not go into details about what is available in
+\LUATEX, for that you consult the manual but will stick to the larger picture
+instead.
+
+\stopsection
+
+\startsection[title=What is it]
+
+First of all, as the name suggests, \LUATEX\ has the \LUA\ scripting engine on
+board. Currently we're still at version 5.2 and the reason for not going 5.3 is
+mainly because it has a different implementation of numbers and we cannot foresee
+side effects. We will test this when we move on to \LUATEX\ version 2.0.
+
+The second part of the name indicates that we have some kind of \TEX\ and we
+think we managed to remain largely compatible with the traditional engine. We
+took most of \ETEX, much of \PDFTEX\ and some from \ALEPH\ (\OMEGA). On top of
+that we added a few new primitives and extended others.
+
+If you look at the building blocks of \TEX, you can roughly recognize these:
+
+\startitemize
+\startitem
+    an input parser (tokenizer) that includes macro expansion; its working is
+    well described, of course in the \TEX\ book, but more than three decades of
+    availability has made \TEX's behaviour rather well documented
+\stopitem
+\startitem
+    a list builder that links basic elements like characters (tagged with font
+    information), rules, boxes, glue and kerns together in a double linked
+    list of so called nodes (and noads in intermediate math lists)
+\stopitem
+\startitem
+    a language subsystem that is responsible for hyphenating words using so called
+    patterns and exceptions
+\stopitem
+\startitem
+    a font subsystem that provides information about glyphs properties, and that
+    also makes it possible to construct math symbols from snippets; it also makes
+    sure that the backend knows what to embed
+\stopitem
+\startitem
+    a paragraph builder that breaks a long list into lines and a page builder
+    that splits of chunks that can be wrapped into pages; this is all done within
+    given constraints using a model of rewards and penalties
+\stopitem
+\startitem
+    a first class math renderer that set the standard and has inspired modern
+    math font technology
+\stopitem
+\startitem
+    mechanisms for dealing with floating data, marking page related info, wrapping
+    stuff in boxes, adding glue, penalties and special information
+\stopitem
+\startitem
+    a backend that is responsible for wrapping everything typeset in a format that
+    can be printed and viewed
+\stopitem
+\stopitemize
+
+So far we're still talking of a rather generic variant of \TEX\ with \LUA\ as
+extension language. Next we zoom in on some details.
+
+\stopsection
+
+\startsection[title=Where it differs]
+
+Given experiences with discussing extensions to the engine and given the fact
+that there is never really an agreement about what makes sense or not, the
+decission was made to not extend the engine any more than really needed but to
+provide hooks to do that in \LUA. And, time has proven that this is a feasible
+approach. On the one hand we are as good as possible faithful to the original,
+and at the same time we can deal with todays and near future demands.
+
+Tokenization still happens as before but we can also write input parsers
+ourselves. You can intercept the raw input when it gets read from file, but you
+can also create scanners that you can sort of plug into the parser. Both are a
+compromise between convenience and speed but powerful enough. At the input end we
+now can group catcode changes (catcodes are properties of characters that control
+how they are interpreted) into tables so that switching between regimes is fast.
+
+You can in great detail influence how data gets read from files because the \IO\
+subsystem is opened up. In fact, you have the full power of \LUA\ available when
+doing so. At the same time you can print back from \LUA\ into the input stream.
+
+The input that makes in into \TEX, either or not intercepted and manipulated
+beforehand, is to be in \UTF8. What comes out to the terminal and log is also
+\UTF8, and internally all codepaths work with wide characters. Some memory
+constraints have been lifted, and character related commands accept large
+numbers. This comes at a price, which means that in practice the \LUATEX\ engine
+can be several times slower than the 8|-|bit \PDFTEX, but of course in practice
+performance is mostly determined by the efficiency of macro package, so it might
+actually be faster in situations that would stress its ancestors.
+
+Node lists travel through \TEX\ and can be intercepted at many points. That way
+you can add additional manipulations. You can for instance rely on \TEX\ for
+hyphenation, ligature building and kerning but you can also plug in alternatives.
+For this purpose these stages are clearly separated and less integrated (deep
+down) than in traditional \TEX. There are helpers for accessing lists of nodes,
+individual nodes and you can box those lists too (this is called packing). You
+can adapt, create and destroy node lists at will, as long as you make sure you
+feed back into \TEX\ something that makes sense.
+
+In order to control (or communicate with) nodes from the \TEX\ end, an attribute
+mechanism was added that makes it possible to bind properties to nodes when they
+get added to lists. At the \TEX\ end you can set an attribute that then gets
+assigned to the currently injected nodes, while at the \LUA\ end you can query
+the node for these attributes and their values.
+
+The language subsystem is re|-|implemented and behaves mostly the same as in the
+original \TEX\ program. It has a few extensions and permits runtime loading of
+patterns. In addition to language support we also have basic script support, that
+is: directional information is now part of the stream and contrary to \ALEPH\
+that wraps this into extension whatsits, in \LUATEX\ we have directional nodes as
+core nodes.
+
+The font subsystem is opened up in such a way that you can pass your own fonts to
+the core. You can even construct virtual fonts. This open approach makes it
+possible to support \OPENTYPE\ fonts and whatever format will show up in the
+future. Of course the backend needs to embed the right data in the result file
+but by then the hard work is already done. This approach fits into the always
+present wish of users (and package writers) to be able to implement whatever
+crazy thought one comes up with.
+
+The paragraph builder is a somewhat cleaned up variant of the \PDFTEX\ one,
+combined with directional and boundary support from \ALEPH. The protrusion and
+expansion mechanism have been redone in such a way that the front- and backend
+code is better separated and is somewhat more efficient now. As one can intercept
+the paragraph builder, additional functionality can be injected before, after or
+at some stages in the process.
+
+Of course we have kept the math engine but, because we now need to support
+\OPENTYPE\ math, alternative code paths have been added to deal with the kind of
+information that such fonts provide. We also took the opportunity to open up the
+math machinery a bit so that one can control rendering of some more complex
+elements and set the spacing between elements. Because \TEX\ users are quite
+traditional we had to stop somewhere, simply because legacy code has to be dealt
+with.
+
+Most mentioned auxiliary mechanisms can be accessed via the node lists, for
+instance you can locate inserts and marks in them. The backend related whatsit
+nodes can be recognized as well. At any time one can query and set \TEX\
+registers and intercept boxed material. Of course some knowledge of the inner
+working of \TEX\ helps here.
+
+The backend code is as much as possible separated from the frontend code (but
+there is still some work to do there). As in \PDFTEX\ you can of course inject
+arbitrary \PDF\ code and make feature rich documents. This flexibility keeps
+\TEX\ current.
+
+\stopsection
+
+\startsection[title=Extras]
+
+Is that all? No, apart from some minor extensions that might help to make
+programming somewhat easier \TEX, there are a few more fundamental additions.
+
+Images and reusable content (boxes) are now part of the core instead of them
+being wrapped into backend specific whatsits, although of course the backend has
+to provide support for it. This is more natural in the frontend (and user
+interface) and also more consistent in the engine itself. All backend
+functionality is now collected in three primitives that take arguments. This
+permits a cleaner separation between front- and backend.
+
+Then there is the \METAPOST\ library, a feature already present for many years
+now. It provides \TEX\ with some graphic capabilities that, given the origin,
+fits nicely into the whole. The \LUATEX\ and \MPLIB\ project started about the
+same time and right from the start it was our plan to combine both.
+
+One of the extras is of course \LUA. It not only permits us to interface to the
+internals of \TEX, but it also provides the user with a way to manipulate data.
+Even if you never use \LUA\ to access internals, it might still be found useful
+for occasionally doing things that are hard to accomplish using the macro
+langage.
+
+In addition to stock \LUA\ we include the \LPEG\ library, an image reading
+library (related to the backend) including read access to \PDF\ files via the
+used poppler library, parsing of \PDF\ content streams, zip compression, access
+to the file system, the ability to run commands and socket support. Some of this
+might become external libraries at some point, as we want to keep the expected
+core functionality lean and mean. A nice extra is that we provide \LUAJITTEX, a
+compatible variant that has a faster \LUA\ virtual machine on board.
+
+\stopsection
+
+\startsection[title=Follow up]
+
+The interfaces that we have now have to a large extent evolved to what we had in
+mind. We started with simple experiments: just \LUA\ plus a bit of access to
+registers. Then the Oriental \TEX\ project (with Idris Samawi Hamid) made it
+possible to speed up development and conversion to \CCODE\ and opening up took
+off. After that we gradually moved forward.
+
+That doesn't mean that we're done yet. The \LUATEX\ 1.0 engine will not change
+much. We might add a few things, and for sure we will keep working on the code
+base. The move from \PASCAL\ to \CCODE\ \WEB\ (an impressive job by itself), as
+well as merging functionality of engines (kind of a challenge when you want to
+remain compatible), opening up via \LUA\ (which possibilities even surprised us),
+and experimenting (\CONTEXT\ users paid the price for that) took quite some time,
+also because we played with proofs of concept. It helped that we used the engine
+exclusively for real typesetting related work ourselves.
+
+We will continue to clean up and document the source and stepwise improve the
+manual. If you followed the development of \CONTEXT, you will have noticed that
+\MKIV\ is heavily relying on the \LUA\ interface so stability is important
+(although we can relatively easy adapt to future developments as we did in the
+past). However, the fact that other packages support \LUATEX\ means that we also
+need to keep the 1.0 engine stable. Our challenge is to provide stability on the
+one hand, but not limit ourselves to much on the other. We'll keep you posted on
+what comes next.
+
+\blank
+
+Hans, Hartmut, Luigi, Taco
+
+\stopsection
+
+\stopchapter
+
+\stoptext
diff --git a/doc/context/sources/general/manuals/still/still-sumatra-math.png b/doc/context/sources/general/manuals/still/still-sumatra-math.png
new file mode 100644
index 000000000..0eb1417dc
--- /dev/null
+++ b/doc/context/sources/general/manuals/still/still-sumatra-math.png
diff --git a/doc/context/sources/general/manuals/still/still-sumatra-text.png b/doc/context/sources/general/manuals/still/still-sumatra-text.png
new file mode 100644
index 000000000..191c1a0ef
--- /dev/null
+++ b/doc/context/sources/general/manuals/still/still-sumatra-text.png
diff --git a/doc/context/sources/general/manuals/still/still.tex b/doc/context/sources/general/manuals/still/still.tex
index 28bc60c79..5fe279672 100644
--- a/doc/context/sources/general/manuals/still/still.tex
+++ b/doc/context/sources/general/manuals/still/still.tex
@@ -34,6 +34,7 @@
     \component still-profiling
     %component still-fonts.tex     % unfinished
     %component still-newlines.tex  % unfinished
+    \component still-viewing
     %component still-plain
     \component still-opentypemath
     \component still-expanding
diff --git a/tex/context/base/context-version.pdf b/tex/context/base/context-version.pdf
index 69116a012..ffa980a47 100644
--- a/tex/context/base/context-version.pdf
+++ b/tex/context/base/context-version.pdf
diff --git a/tex/context/base/mkiv/cont-new.mkiv b/tex/context/base/mkiv/cont-new.mkiv
index 06fa9865c..0886f6e81 100644
--- a/tex/context/base/mkiv/cont-new.mkiv
+++ b/tex/context/base/mkiv/cont-new.mkiv
@@ -11,7 +11,7 @@
 %C therefore copyrighted by \PRAGMA. See mreadme.pdf for
 %C details.
 
-\newcontextversion{2016.08.24 21:16}
+\newcontextversion{2016.08.25 14:14}
 
 %D This file is loaded at runtime, thereby providing an excellent place for
 %D hacks, patches, extensions and new features.
diff --git a/tex/context/base/mkiv/context.mkiv b/tex/context/base/mkiv/context.mkiv
index 83a0d0162..1ca5dbb5e 100644
--- a/tex/context/base/mkiv/context.mkiv
+++ b/tex/context/base/mkiv/context.mkiv
@@ -39,7 +39,7 @@
 %D up and the dependencies are more consistent.
 
 \edef\contextformat {\jobname}
-\edef\contextversion{2016.08.24 21:16}
+\edef\contextversion{2016.08.25 14:14}
 \edef\contextkind   {beta}
 
 %D For those who want to use this:
diff --git a/tex/context/base/mkiv/file-lib.lua b/tex/context/base/mkiv/file-lib.lua
index b8ba69427..62cf938ba 100644
--- a/tex/context/base/mkiv/file-lib.lua
+++ b/tex/context/base/mkiv/file-lib.lua
@@ -48,12 +48,7 @@ function resolvers.uselibrary(specification) -- todo: reporter
         end
         for i=1,#files do
             local filename = files[i]
-            if loaded[filename] then
-                -- next one
-            else
-                if onlyonce then
-                    loaded[filename] = true -- todo: base this on return value
-                end
+            if not loaded[filename] then
                 local foundname = nil
                 local barename  = removesuffix(filename)
                 -- direct search (we have an explicit suffix)
@@ -76,10 +71,18 @@ function resolvers.uselibrary(specification) -- todo: reporter
                         end
                     end
                 end
-                if foundname then
-                    action(name,foundname)
-                elseif failure then
-                    failure(name)
+                if not loaded[foundname] then
+                    if foundname then
+                        action(name,foundname)
+                        if onlyonce then
+                            loaded[foundname] = true -- todo: base this on return value
+                        end
+                    elseif failure then
+                        failure(name)
+                    end
+                    if onlyonce then
+                        loaded[filename]  = true -- todo: base this on return value
+                    end
                 end
             end
         end
diff --git a/tex/context/base/mkiv/font-mps.lua b/tex/context/base/mkiv/font-mps.lua
index d56c25731..39c17a1cc 100644
--- a/tex/context/base/mkiv/font-mps.lua
+++ b/tex/context/base/mkiv/font-mps.lua
@@ -289,7 +289,7 @@ function metapost.output(kind,font,char,advance,shift,ex)
             if glyphs then
                 local glyf = glyphs[index]
                 if glyf then
-                    local units     = shapedata.units or 1000
+                    local units     = 1000 -- factor already takes shapedata.units into account
                     local yfactor   = (sc/units) * parameters[font].factor / 655.36
                     local xfactor   = yfactor
                     local shift     = shift or 0
diff --git a/tex/context/base/mkiv/meta-imp-outlines.mkiv b/tex/context/base/mkiv/meta-imp-outlines.mkiv
index 199dadb32..7629c0c92 100644
--- a/tex/context/base/mkiv/meta-imp-outlines.mkiv
+++ b/tex/context/base/mkiv/meta-imp-outlines.mkiv
@@ -67,7 +67,7 @@ function metapost.showglyph(specification)
         end
         local glyph = shapeglyphs[index]
         if glyph and (glyph.segments or glyph.sequence) then
-            local units  = data.fontheader and data.fontheader.emsize or 1000
+            local units  = shapedata.units or 1000
             local factor = 100/units
             local paths  = metapost.paths(glyph,factor)
             if #paths > 0 and glyph.boundingbox and glyph.width then
diff --git a/tex/context/base/mkiv/publ-imp-apa.mkvi b/tex/context/base/mkiv/publ-imp-apa.mkvi
index 5d11ef62f..9492b44e8 100644
--- a/tex/context/base/mkiv/publ-imp-apa.mkvi
+++ b/tex/context/base/mkiv/publ-imp-apa.mkvi
@@ -318,7 +318,9 @@
   [apa:\s!cite:author]
   [\c!left={(},
    \c!right={)},
-   \c!inbetween={\btxcomma}]
+   \c!inbetween={\btxcomma},
+   \c!separator:names:3={\btxcomma\texampersand\space},
+   \c!separator:names:4={\btxspace\texampersand\space}]
 
 \definebtx
   [apa:\s!cite:default]
@@ -326,7 +328,7 @@
 
 \definebtx
   [apa:\s!cite:authoryears]
-  [apa:\s!cite:authoryear]
+  [apa:\s!cite:author]
   [\c!left=,
    \c!right=,
    \c!inbetween={\btxspace}]
diff --git a/tex/context/base/mkiv/status-files.pdf b/tex/context/base/mkiv/status-files.pdf
index b1184e1eb..50eaedba1 100644
--- a/tex/context/base/mkiv/status-files.pdf
+++ b/tex/context/base/mkiv/status-files.pdf
diff --git a/tex/context/base/mkiv/status-lua.pdf b/tex/context/base/mkiv/status-lua.pdf
index 829a8576a..61861f2bd 100644
--- a/tex/context/base/mkiv/status-lua.pdf
+++ b/tex/context/base/mkiv/status-lua.pdf
diff --git a/tex/context/interface/mkiv/i-context.pdf b/tex/context/interface/mkiv/i-context.pdf
index 86a044447..d2bf8cfbd 100644
--- a/tex/context/interface/mkiv/i-context.pdf
+++ b/tex/context/interface/mkiv/i-context.pdf
diff --git a/tex/context/interface/mkiv/i-readme.pdf b/tex/context/interface/mkiv/i-readme.pdf
index 0f70e266c..dc1a443e2 100644
--- a/tex/context/interface/mkiv/i-readme.pdf
+++ b/tex/context/interface/mkiv/i-readme.pdf
diff --git a/tex/generic/context/luatex/luatex-fonts-merged.lua b/tex/generic/context/luatex/luatex-fonts-merged.lua
index 2bac536cd..d86763204 100644
--- a/tex/generic/context/luatex/luatex-fonts-merged.lua
+++ b/tex/generic/context/luatex/luatex-fonts-merged.lua
@@ -1,6 +1,6 @@
 -- merged file : c:/data/develop/context/sources/luatex-fonts-merged.lua
 -- parent file : c:/data/develop/context/sources/luatex-fonts.lua
--- merge date  : 08/24/16 21:16:06
+-- merge date  : 08/25/16 14:14:14
 
 do -- begin closure to overcome local limits and interference