2020-06-25 10:58:00

9 files changed, 1469 insertions, 18 deletions

diff --git a/doc/context/documents/general/manuals/cld-mkiv.pdf b/doc/context/documents/general/manuals/cld-mkiv.pdf
index bac02979f..63f1c36dc 100644
--- a/doc/context/documents/general/manuals/cld-mkiv.pdf
+++ b/doc/context/documents/general/manuals/cld-mkiv.pdf
diff --git a/doc/context/documents/general/manuals/luametatex.pdf b/doc/context/documents/general/manuals/luametatex.pdf
index bd3333189..9c9f1c5bd 100644
--- a/doc/context/documents/general/manuals/luametatex.pdf
+++ b/doc/context/documents/general/manuals/luametatex.pdf
diff --git a/doc/context/sources/general/manuals/cld/cld-introduction.tex b/doc/context/sources/general/manuals/cld/cld-introduction.tex
index 82fcb8007..a1a74cde4 100644
--- a/doc/context/sources/general/manuals/cld/cld-introduction.tex
+++ b/doc/context/sources/general/manuals/cld/cld-introduction.tex
@@ -26,6 +26,67 @@ that is available at the \TEX\ end, or you can use \LUA\ code to do the work, or
 you can use a combination. So, from now on, in \CONTEXT\ you can code your style
 and document source in (a mixture of) \TEX, \XML, \METAPOST\ and in \LUA.
 
+% \CONTEXT\ being a \TEX\ macro package quite some action happens at the \TEX\ end;
+% although we do a lot in \LUA, there is no need to abandon the macro language. If
+% I wanted a typesetting system written in procedural language I'd already come up
+% with one but I have no need for it and \TEX\ is more fun anyway. But the fact
+% that we mix these conceptual different languages means that we need ways to
+% communicate between them. The possibilities to communicate and switch between
+% \TEX, \LUA, and of course \METAPOST\ have evolved over time. It started with just
+% \type {\directlua} but we now can use tokens scanners that permit nice
+% interfaces. So, as a consequence the interfaces in \CONTEXT\ also evolved,
+% although mostly deep down out of sight of users: \CONTEXT\ always tries to offer
+% abstract interfaces that as designed in a way that permits upward compatible
+% upgrades.
+
+% Going from \LUA\ to \TEX\ is mostly done with the \type {context} command. The
+% most low level approach would be to use the \type {tex.print} functions but using
+% the \type {context} interface is often better. Going from \TEX\ to \LUA\ is done
+% with \type {\ctxlua} and friends, btu again, you can decide to go low level with
+% \type {\directlua}. Indeed, there is no need to use these interfaces and quite
+% often we see users use low level calls in combination with macros that pass
+% arguments and there's nothing wrong with that. We see no reason to block or
+% overload them.
+
+% In these internet times it is no problem to find comments on code and coding
+% practices (of course not seldom by people who themselves produce code that
+% deserves a lot of comment but get away with it). The same is true for complaints
+% about bugs that not always are put in the perspective of the total amount of well
+% working code or experiments; on the average \TEX\ systems behave quite well and
+% have not that many bugs, but some low level features they can be confusing. We
+% don't impose a coding style in user files but we do so in the core, but that only
+% hampers ourselves.
+
+% Another issue is complaints about manuals or lack of documentation, again, often
+% by people who themselves never produced something that those who they complain
+% about can use, but, that said, lack of documentation has the benefit that one can
+% learn by playing around. There are plenty of examples to learn from, also in the
+% over ten thousand pages of \CONTEXT\ documentation and articles. And of course
+% there is the test suite which has examples. Some users even like to look into the
+% source code to learn some tricks. Now, the internet is not always the friendly
+% and tolerant environment that its marketing suggests and one can run into
+% patronizing comments on how to do thing, but: we don't (and cannot) enforce best
+% practices.
+
+% However, when you define \LUA\ functions you should {\em not} overload existing
+% functions without knowing what you deal with, simply because the whole \CONTEXT\
+% ecosystem could be affected. Don't expect support when you do so. The same is
+% true for \TEX\ macros and even more so for primitives. Although there is some
+% protection going on we think the system should be as open as possible, so little
+% is hidden for the user. \footnote {In the core code and modules that come with
+% \CONTEXT\ we are rather rigourous. There we have patterns that need to be
+% followed. Hacking around and patching at will is not an option there. In the
+% rather open system that a macro package provides it is no problem to quickly mess
+% up the lot with bad code.}
+
+% The reason for mentioning this is that we do see users come up with (often
+% surprising) solutions and we invite them to keep playing around, even with low
+% level code. When writing a document, playing around can be a nice distraction.
+% But, using a high level interface has some benefits too. It gives a bit of
+% protection against bad interactions with other code. It also often involves less
+% coding. Performance might be better although that is not always a real issue as
+% there are plenty of ways to make a \TEX\ system slow anyway.
+
 In the following chapters I will introduce typesetting in \LUA, but as we rely on
 \CONTEXT\ it is unavoidable that some regular \CONTEXT\ code shows up. The fact
 that you can ignore backslashes does not mean that you can do without knowledge
diff --git a/doc/context/sources/general/manuals/cld/cld-luafunctions.tex b/doc/context/sources/general/manuals/cld/cld-luafunctions.tex
index b2760e05b..c48b852fb 100644
--- a/doc/context/sources/general/manuals/cld/cld-luafunctions.tex
+++ b/doc/context/sources/general/manuals/cld/cld-luafunctions.tex
@@ -2306,20 +2306,22 @@ one provided by \LUATEX\ as it is more precise:
 os.gettimeofday()
 \stoptyping
 
-There is also a more extensive variant:
-
-\starttyping
-os.times()
-\stoptyping
-
-This one is platform dependent and returns a table with \type {utime} (use time),
-\type {stime} (system time), \type {cutime} (children user time), and \type
-{cstime} (children system time).
+% % This one is gone in luametatex:
+%
+% There is also a more extensive variant:
+%
+% \starttyping
+% os.times()
+% \stoptyping
+%
+% This one is platform dependent and returns a table with \type {utime} (use time),
+% \type {stime} (system time), \type {cutime} (children user time), and \type
+% {cstime} (children system time).
 
 \stopsummary
 
 \ShowLuaExampleThree {os} {gettimeofday} {}
-\ShowLuaExampleTwo   {os} {times}        {}
+%ShowLuaExampleTwo   {os} {times}        {}
 
 \startsummary[title={runtime}]
 
diff --git a/doc/context/sources/general/manuals/cld/cld-macros.tex b/doc/context/sources/general/manuals/cld/cld-macros.tex
index a177db9f8..7b6f578e7 100644
--- a/doc/context/sources/general/manuals/cld/cld-macros.tex
+++ b/doc/context/sources/general/manuals/cld/cld-macros.tex
@@ -128,9 +128,9 @@ local function startmore(opt_1)
     context.startnarrower()
 end
 
-local function stopmore(opt_1)
+local function stopmore()
     context.stopnarrower()
-    context("stop more, options: %s",interfaces.tolist(opt_1))
+    context("stop more")
     context.stopnarrower()
 end
 
@@ -175,7 +175,7 @@ If needed you can access the body of a macro. Take for instance:
 
 \typebuffer \getbuffer
 
-The following example demonsttaies how we can look inside these macros. You need
+The following example demonstrates how we can look inside these macros. You need
 to be aware of the fact that the whole blob of \LUA\ codes is finished before we
 return to \TEX, so when we pipe the meaning of \type {TestB} back to \TEX\ it
 only gets expanded afterwards. We can use a function to get back to \LUA. It's
diff --git a/doc/context/sources/general/manuals/cld/cld-mkiv.tex b/doc/context/sources/general/manuals/cld/cld-mkiv.tex
index 5c35fa4e7..9267732fa 100644
--- a/doc/context/sources/general/manuals/cld/cld-mkiv.tex
+++ b/doc/context/sources/general/manuals/cld/cld-mkiv.tex
@@ -74,6 +74,8 @@
     \component cld-logging
     \component cld-luafunctions
     \component cld-ctxfunctions
+    \component cld-scanners
+    \component cld-variables
     \component cld-callbacks
     \component cld-backendcode
     \component cld-goodies
@@ -84,8 +86,8 @@
     \component cld-files
 \stopbodymatter
 
-% \startbackmatter
-%     \component cld-index
-% \stopbackmatter
+\startbackmatter
+    \component cld-index
+\stopbackmatter
 
 \stopproduct
diff --git a/doc/context/sources/general/manuals/cld/cld-scanners.tex b/doc/context/sources/general/manuals/cld/cld-scanners.tex
new file mode 100644
index 000000000..e0dfc30b0
--- /dev/null
+++ b/doc/context/sources/general/manuals/cld/cld-scanners.tex
@@ -0,0 +1,1046 @@
+% language=uk
+
+\startcomponent cld-ctxscanners
+
+\environment cld-environment
+
+\startchapter[title={Scanners}]
+
+\startsection[title={Introduction}]
+
+\index {implementors}
+\index {scanners}
+
+Here we discuss methods to define macros that directly interface with the \LUA\
+side. It involves all kind of scanners. There are actually more than we discuss
+here but some are meant for low level usage. What is describe here has been used
+for ages and works quite well.
+
+{\em We don't discuss some of the more obscure options here. Some are there just
+because we need them as part of bootstrapping or initializing code and are of no
+real use to users.}
+
+\stopsection
+
+\startsection[title={A teaser first}]
+
+Most of this chapter is examples and you learn \TEX\ (and \LUA) best by just
+playing around. The nice thing about \TEX\ is that it's all about visual output,
+so that's why in the next examples we just typeset some of what we just scanned.
+Of course in practice the \type {actions} will be more complex.
+
+\unexpanded\def\showmeaning#1%
+  {\begingroup
+   \dontleavehmode
+   \ttbf\string #1\space
+   \tttf\meaning#1%
+   \endgroup}
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroA",
+        public    = true,
+        arguments = "string",
+        actions   = function(s)
+            context("(%s)",s)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\startbuffer[usage]
+\MyMacroA{123}
+\MyMacroA{abc}
+\edef\temp{\MyMacroA{abc}}
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+By default a macro gets defined in the \type {\clf_} namespace but the \type
+{public} option makes it visible. This default indicates that it is actually a
+low level mechanism in origin. More often than not these interfaces are used like
+this:
+
+\starttyping
+\def\MyMacro#1{... \clf_MyMacroA{#1} ...}
+\stoptyping
+
+When we look at the meaning of \type {\MyMacroA} we get:
+
+\blank \showmeaning\MyMacroA \blank
+
+And when we apply this macro as:
+
+\typebuffer[usage]
+
+We get
+
+\blank \getbuffer[usage] \blank
+
+The meaning of \type {\temp} is:
+
+\blank \showmeaning\temp \blank
+
+We can also define the macro to be protected (\type {\unexpanded}) in \CONTEXT\
+speak). We can overload existing scanners but unless we specify the \type
+{overload} option, we get a warning on the console.
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroA",
+        public    = true,
+--      overload  = true,
+        protected = true,
+        arguments = "string",
+        actions   = function(s)
+            context("[%s]",s)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+This time we get:
+
+\getbuffer[usage]
+
+The meaning of \type {\temp} is:
+
+\blank \showmeaning\temp \blank
+
+\stopsection
+
+\startsection[title={Basic data types}]
+
+\index {implementors+arguments}
+
+It is actually possible to write very advanced scanners but unless you're in for
+obscurity the limited subset discussed here is normally enough. The \CONTEXT\
+user interface is rather predictable, unless you want to show off with weird
+additional interfaces, for instance by using delimiters other than curly braces
+and brackets, or by using separators other than commas.
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroB",
+        public    = true,
+        arguments = { "string", "integer", "boolean", "dimen" },
+        actions   = function(s,i,b,d)
+            context("<%s> <%i> <%l> <%p>",s,i,b,d)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+This time we grab four arguments, each of a different type:
+
+\startbuffer[usage]
+\MyMacroB{foo} 123 true 45.67pt
+
+\def\temp       {oof}
+\scratchcounter 321
+\scratchdimen   76.54pt
+
+\MyMacroB\temp \scratchcounter false \scratchdimen
+\stopbuffer
+
+\typebuffer[usage]
+
+The above usage gives:
+
+\getbuffer[usage]
+
+As you can see, registers can be used as well, and the \type {\temp} macro is
+also accepted as argument. The integer and dimen arguments scan standard \TEX\
+values. If you want a \LUA\ number you can specify that as well. As our first
+example showed, when there is one argument you don't need an array to specify
+it.
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroC",
+        public    = true,
+        arguments = "number",
+        actions   = function(f)
+            context("<%.2f>",f)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroC 1.23
+\MyMacroC 1.23E4
+\MyMacroC -1.23E4
+\MyMacroC 0x1234
+\stopbuffer
+
+As you can see, hexadecimal numbers are also accepted:
+
+\typebuffer[usage]
+
+The above usage gives:
+
+\getbuffer[usage]
+
+\stopsection
+
+\startsection[title={Tables}]
+
+\index {implementors+tables}
+
+A list can be grabbed too. The individual items are separated by spaces and
+items can be bound by braces.
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroD",
+        public    = true,
+        arguments = "list",
+        actions   = function(t)
+            context("< % + t >",t)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroD { 1 2 3 4 {5 6} }
+\stopbuffer
+
+The macro call:
+
+\typebuffer[usage]
+
+results in:
+
+\getbuffer[usage]
+
+Often in \LUA\ scripts tables are uses all over the place. Picking up a table is
+also supported by the implementer.
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroE",
+        public    = true,
+        arguments = {
+            {
+                { "bar", "integer" },
+                { "foo", "dimen" },
+            }
+        },
+        actions   = function(t)
+            context("<foo : %p> <bar : %i>",t.foo,t.bar)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroE {
+    foo 12pt
+    bar 34
+}
+\stopbuffer
+
+Watch out, we don't use equal signs and commas here:
+
+\typebuffer[usage]
+
+We get:
+
+\getbuffer[usage]
+
+All the above can be combined:
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroF",
+        public    = true,
+        arguments = {
+            "string",
+            {
+                { "bar", "integer" },
+                { "foo", "dimen" },
+            },
+            {
+                { "one", "string" },
+                { "two", "string" },
+            },
+        },
+        actions   = function(s,t1,t2)
+            context("<%s> <%p> <%i> <%s> <%s>",s,t1.foo,t1.bar,t2.one,t2.two)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroF
+    {oeps}
+    { foo 12pt bar 34 }
+    { one {x} two {y} }
+\stopbuffer
+
+The following call:
+
+\typebuffer[usage]
+
+Results in one string and two table arguments.
+
+\getbuffer[usage]
+
+You can nest tables, as in:
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroG",
+        public    = true,
+        arguments = {
+            "string",
+            "string",
+            {
+                { "data", "string" },
+                { "tab", "string" },
+                { "method", "string" },
+                { "foo", {
+                    { "method", "integer" },
+                    { "compact", "number" },
+                    { "nature" },
+                    { "*" }, -- any key
+                } },
+                { "compact", "string", "tonumber" },
+                { "nature", "boolean" },
+                { "escape" },
+            },
+            "boolean",
+        },
+        actions   = function(s1, s2, t, b)
+            context("<%s> <%s>",s1,s2)
+            context("<%s> <%s> <%s>",t.data,t.tab,t.compact)
+            context("<%i> <%s> <%s>",t.foo.method,t.foo.nature,t.foo.whatever)
+            context("<%l>",b)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroG
+    {s1}
+    {s2}
+    {
+        data    {d}
+        tab     {t}
+        compact {12.34}
+        foo     { method 1 nature {n} whatever {w} }
+    }
+    true
+\relax
+\stopbuffer
+
+Although the \type {\relax} is not really needed in the next calls, I often use
+it to indicate that we're done:
+
+\typebuffer[usage]
+
+This typesets:
+
+\getbuffer[usage]
+
+\stopsection
+
+\startsection[title=Expansion]
+
+\index {implementors+expansion}
+
+When working with scanners it is important to realize that we have to do with an
+expansion engine. When \TEX\ picks up a token, it can be done as-is, that is the
+raw token, but it can also expand that token first (which can be recursive) and
+then pick up the first token that results from that. Sometimes you want that
+expansion, for instance when you pick up keywords, sometimes you don't.
+
+Expansion effects are most noticeable when we pickup a \quote {string} kind of
+value. In the implementor we have two methods for that: \type {string} and \type
+{argument}. The argument method has an expandable form (the default) and one
+that doesn't expand. Take this:
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroH",
+        public    = true,
+        arguments = {
+            "string",
+            "argument",
+            "argumentasis",
+        },
+        actions   = function(a,b,c)
+            context.type(a or "-") context.quad()
+            context.type(b or "-") context.quad()
+            context.type(c or "-") context.crlf()
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+Now take this input:
+
+\startbuffer[usage]
+\def\a{A} \def\b{B} \def\c{C}
+\MyMacroH{a}{b}{c}
+\MyMacroH{a\a}{b\b}{c\c}
+\MyMacroH\a\b\c\relax
+\MyMacroH\a xx\relax
+\stopbuffer
+
+\typebuffer[usage]
+
+We we use the string method we need a \type {\relax} (or some spacer) to end
+scanning of the string when we don't use curly braces. The last line is
+actually kind of tricky because the macro expects two arguments after
+scanning the first string.
+
+\blank {\getbuffer[usage]} \blank
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroI",
+        public    = true,
+        arguments = {
+            "argument",
+            "argumentasis",
+        },
+        actions   = function(a,b,c)
+            context.type(a or "-") context.quad()
+            context.type(b or "-") context.quad()
+            context.type(c or "-") context.crlf()
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+Here is a variant:
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\def\a{A} \def\b{B}
+\MyMacroI{a}{b}
+\MyMacroI{a\a}{b\b}
+\MyMacroI\a\b\relax
+\stopbuffer
+
+With:
+
+\typebuffer[usage]
+
+we get:
+
+\blank {\getbuffer[usage]} \blank
+
+\stopsection
+
+\startsection[title=Boxes]
+
+\index {implementors+boxes}
+
+You can pick up a box too. The value returned is a list node:
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroJ",
+        public    = true,
+        arguments = "box",
+        actions   = function(b)
+            context(b)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+The usual box specifiers are supported:
+
+\startbuffer[usage]
+\MyMacroJ \hbox        {\strut Test 1}
+\MyMacroJ \hbox to 4cm {\strut Test 2}
+\stopbuffer
+
+So, with:
+
+\typebuffer[usage]
+
+we get:
+
+\blank {\forgetall\dontcomplain\getbuffer[usage]} \blank
+
+There are three variants that don't need the box operator \type {hbox}, \type
+{vbox} and \type {vtop}:
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroL",
+        public    = true,
+        arguments = {
+            "hbox",
+            "vbox",
+        },
+        actions   = function(h,v)
+            context(h)
+            context(v)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+Again, the usual box specifiers are supported:
+
+\startbuffer[usage]
+\MyMacroL            {\strut Test 1h} to 10mm {\vfill Test 1v\vfill}
+\MyMacroL spread 1cm {\strut Test 2h} to 15mm {\vfill Test 2v\vfill}
+\stopbuffer
+
+This:
+
+\typebuffer[usage]
+
+gives:
+
+\blank {\forgetall\dontcomplain\showboxes \getbuffer[usage]} \blank
+
+\stopsection
+
+\startsection[title=Like \CONTEXT]
+
+\index {implementors+hashes}
+\index {implementors+arrays}
+
+The previously discussed scanners don't use equal signs and commas as separators,
+but you can enforce that regime in the following way:
+
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroN",
+        public    = true,
+        arguments = {
+            "hash",
+            "array",
+        },
+        actions   = function(h, a)
+            context.totable(h)
+            context.quad()
+            context.totable(a)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroN
+    [ a = 1,  b = 2 ]
+    [ 3, 4, 5, {6 7} ]
+\stopbuffer
+
+This:
+
+\typebuffer[usage]
+
+gives:
+
+\blank {\getbuffer[usage]} \blank
+
+\stopsection
+
+\startsection[title=Verbatim]
+
+\index {implementors+verbatim}
+
+There are a couple of rarely used scanners (there are more of course but these
+are pretty low level and not really used directly using implementors).
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroO",
+        public    = true,
+        arguments = "verbatim",
+        actions   = function(v)
+            context.type(v)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroO{this is \something verbatim}
+\stopbuffer
+
+There is no expansion applied in:
+
+\typebuffer[usage]
+
+so we get what we input:
+
+\blank {\getbuffer[usage]} \blank
+
+\stopsection
+
+\startsection[title=Macros]
+
+\index {implementors+macros}
+
+We can pick up a control sequence without bothering what it actually
+represents:
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroP",
+        public    = true,
+        arguments = "csname",
+        actions   = function(c)
+            context("{\\ttbf name:} {\\tttf %s}",c)
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroP\framed
+\stopbuffer
+
+The next control sequence is picked up and its name without the leading
+escape character is returned:
+
+\typebuffer[usage]
+
+So here we get:
+
+\blank {\getbuffer[usage]} \blank
+
+\stopsection
+
+\startsection[title={Token lists}]
+
+\index {implementors+token lists}
+
+If you have no clue what tokens are in the perspective of \TEX, you can skip this
+section. We can grab a token list in two ways. The most \LUA ish way is to grab
+it as a table:
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroQ",
+        public    = true,
+        arguments = "toks",
+        actions   = function(t)
+            context("%S : ",t)
+            context.sprint(t)
+            context.crlf()
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroQ{this is a {\bf token} list}
+\MyMacroQ{this is a \inframed{token} list}
+\stopbuffer
+
+\typebuffer[usage]
+
+The above sample code gives us:
+
+\blank {\getbuffer[usage]} \blank
+
+An alternative is to keep the list a user data object:
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroR",
+        public    = true,
+        arguments = "tokenlist",
+        actions   = function(t)
+            context("%S : ",t)
+            context.sprint(t)
+            context.crlf()
+        end,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroR{this is a {\bf token} list}
+\MyMacroR{this is a \inframed{token} list}
+\stopbuffer
+
+\typebuffer[usage]
+
+Now we get:
+
+\blank {\getbuffer[usage]} \blank
+
+\stopsection
+
+\startsection[title={Actions}]
+
+\index {implementors+actions}
+
+The plural \type {actions} suggests that there can be more than one and indeed
+that is the case. The next example shows a sequence of actions that are applied.
+The first one gets the arguments passes.
+
+\startbuffer[definition]
+\startluacode
+    interfaces.implement {
+        name      = "MyMacroS",
+        public    = true,
+        arguments = "string",
+        actions   = { characters.upper, context },
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+\startbuffer[usage]
+\MyMacroS{uppercase}
+\stopbuffer
+
+\typebuffer[usage]
+
+Gives: \inlinebuffer[usage]
+
+You can pass default arguments too. That way you can have multiple macros using
+the same action. Here's how to do that:
+
+\startbuffer[definition]
+\startluacode
+    local function MyMacro(a,b,sign)
+        if sign then
+            context("$%i + %i = %i$",a,b,a+b)
+        else
+            context("$%i - %i = %i$",a,b,a-b)
+        end
+    end
+
+    interfaces.implement {
+        name      = "MyMacroPlus",
+        public    = true,
+        arguments = { "integer", "integer", true },
+        actions   = MyMacro,
+    }
+
+    interfaces.implement {
+        name      = "MyMacroMinus",
+        public    = true,
+        arguments = { "integer", "integer", false },
+        actions   = MyMacro,
+    }
+\stopluacode
+\stopbuffer
+
+\typebuffer[definition] \getbuffer[definition]
+
+So,
+
+\startbuffer[usage]
+\MyMacroPlus  654 321 \crlf
+\MyMacroMinus 654 321 \crlf
+\stopbuffer
+
+\typebuffer[usage]
+
+Gives:
+
+\getbuffer[usage]
+
+
+\stopsection
+
+\startsection[title={Embedded \LUA\ code}]
+
+When you mix \TEX\ and \LUA, you can put the \LUA\ code in a \TEX\ file, for
+instance a style. In the previous sections we used this approach:
+
+\starttyping
+\startluacode
+    -- lua code
+\stopluacode
+\stoptyping
+
+This method is both reliable and efficient but you need to keep into mind that
+macros get expanded. In the next code, the second line will give an error when
+you have not defined \type {\foo} as expandable macro. When it is unexpandable it
+will get passed as it is and \LUA\ will see a \type {\f} as an escaped character.
+So, when you want a macro be passes as macro, you need to do it as in the third
+line. The fact that there is a comment trigger (\type {--}) doesn't help here.
+
+\starttyping
+\startluacode
+    context("foo")
+ -- context("\foo")
+    context("\\bar")
+\stopluacode
+\stoptyping
+
+When you use \type {\ctxlua} the same is true but there you also need to keep an
+eye on special characters. For instance a percent sign is then interpreted in the
+\TEX\ way and because all becomes one line, a \type {--} somewhere in the middle
+will make the rest of the line comment:
+
+\starttyping
+\ctxlua {
+    context("foo")
+ %  context("\foo")
+ -- context("\\bar")
+}
+\stoptyping
+
+Here, the second line goes away (\TEX\ comment) and the third line obscures all
+that follows. You can use \type {\letterpercent} to smuggle a percent sign in a
+\type {\ctxlua} call. Special characters like a hash symbol also need to be
+passed by name. Normally curly braces are no problem because \LUA\ also likes
+them properly nested.
+
+When things become too messy and complex you can always put the code in an
+external file and load that one (e.g. with \type {require}.
+
+In the examples in this chapter we put the function in the table, but for long
+ones you might want to do this:
+
+\starttyping
+\startluacode
+    local function MyMacro(s)
+        -- lots of code
+    end
+
+    interfaces.implement {
+        name      = "MyMacro",
+        public    = true,
+        arguments = "string",
+        actions   = MyMacro,
+    }
+\stopluacode
+\stoptyping
+
+It is a common mistake not to define variables and functions as local. If you
+define them global for sure there will become a time when this bites you.
+
+\stopsection
+
+\stopchapter
+
+\stopcomponent
+
+% conditional   : true|false|0=true|!0=false : too weird for users
+% bracketed     : maybe, but first I need a use case
+% bracketedasis : maybe, but first I need a use case
+% optional      : maybe, but first I need a use case
+
+\startluacode
+
+    local function Grabbed(t)
+        context(t)
+    end
+
+    interfaces.implement {
+        name      = "GrabC",
+        public    = true,
+        actions   = Grabbed,
+        arguments = "bracketed",
+    }
+    interfaces.implement {
+        name      = "GrabD",
+        public    = true,
+        actions   = Grabbed,
+        arguments = "bracketedasis",
+    }
+    interfaces.implement {
+        name      = "GrabE",
+        public    = true,
+        actions   = Grabbed,
+        arguments = "optional",
+    }
+\stopluacode
+
+\GrabC [foo {\red okay C} bar] \par
+\GrabC                         \par
+\GrabD [foo {\red okay D} bar] \par
+\GrabE [foo {\red okay E} bar] \par
+\GrabE                         \par
+
+% once stable:
+
+\startluacode
+    local random       = math.random
+    local randomseed   = math.randomseed
+
+    local scan_word    = tokens.scanners.word
+    local scan_integer = tokens.scanners.integer
+    local scan_dimen   = tokens.scanners.dimen
+    local scan_number  = tokens.scanners.float
+
+    local scan_integer  = tokens.scanners.luainteger
+    local scan_cardinal = tokens.scanners.luacardinal
+    local scan_number   = tokens.scanners.luanumber
+
+    local I = 0
+    local D = 0
+    local F = 0
+
+    interfaces.implement {
+        name      = "TestInteger",
+        public    = true,
+        actions   = function(b) if b then return I else I = scan_integer() end end,
+        valuetype = "count",
+    }
+
+    interfaces.implement {
+        name      = "TestDimension",
+        public    = true,
+        actions   = function(b) if b then return D else D = scan_dimen() end end,
+        valuetype = "dimen",
+    }
+
+    interfaces.implement {
+        name      = "TestFloat",
+        public    = true,
+        actions   = function(b)
+            if b then
+                context("%q",F)
+            else
+                F = scan_number()
+            end
+        end,
+        valuetype = "none",
+    }
+
+    interfaces.implement {
+        name      = "TestThis",
+        public    = true,
+        actions   = function(b)
+            if b then
+                return random(scan_integer(),scan_integer())
+            else
+                randomseed(scan_integer(true))
+            end
+        end,
+        valuetype = "count",
+    }
+
+\stopluacode
+
+{\tttf [set:\TestInteger           808714][get: \the\TestInteger  ]}\par % [get: \number\TestInteger  ]}\par
+{\tttf [set:\TestDimension        12.34pt][get: \the\TestDimension]}\par % [get: \number\TestDimension]}\par
+{\tttf [set:\TestFloat     12.34567890e99][get: \the\TestFloat    ]}\par % [get: \number\TestFloat    ]}\par
+
+{\tttf \TestThis 123 \dorecurse{10}{\the \TestThis 1 10 \space}}\par
+{\tttf \TestThis 123 \dorecurse{10}{\the \TestThis 1 10 \space}}\par
+{\tttf \TestThis 456 \dorecurse{10}{\the \TestThis 1 10 \space}}\par
+{\tttf               \dorecurse{10}{\the \TestThis 1 10 \space}}\par
+
+\TestFloat  .1e20\relax
+\TestFloat  -0x1.693d8e8943f17p+332\relax
+\TestFloat   0x1.693d8e8943f17p+332\relax
+
+\TestFloat  123.345E67\relax
+
+{\tttf [\the\TestFloat]}
+
+% maybe some day:
+
+% \startluacode
+%     local t1 = token.get_next()
+%     local t2 = token.get_next()
+%     local t3 = token.get_next()
+%     local t4 = token.get_next()
+%     -- watch out, we flush in sequence
+%     token.put_next { t1, t2 }
+%     -- but this one gets pushed in front
+%     token.put_next ( t3, t4 )
+% \stopluacode
+% abcd
+
+% \ctxlua{whatever = { } whatever.bf = token.get_next()}\bf
+
+\startluacode
+    local t_bf = whatever and whatever.bf or token.create("bf")
+    local put  = token.put_next
+
+    interfaces.implement {
+        name    = "whateverbfone",
+        public  = true,
+        actions = function()
+            put(t_bf)
+        end
+    }
+
+    local ctx_whateverbfxxx = context.whateverbfxxx
+
+    interfaces.implement {
+        name    = "whateverbftwo",
+        public  = true,
+        actions = function()
+            ctx_whateverbfxxx(false)
+        end
+    }
+
+    interfaces.implement {
+        name    = "whateverbfthree",
+        public  = true,
+        actions = context.core.cs.whateverbfxxx
+    }
+
+\stopluacode
+
+\let\whateverbfxxx\bf
+
+\dontleavehmode{xxx: \whateverbfxxx     xxx}\quad
+\dontleavehmode{one: \whateverbfone     one}\quad
+\dontleavehmode{two: \whateverbftwo     two}\quad
+\dontleavehmode{two: \whateverbfthree three}\par
diff --git a/doc/context/sources/general/manuals/cld/cld-specialcommands.tex b/doc/context/sources/general/manuals/cld/cld-specialcommands.tex
index 37ecf45c6..510c820f9 100644
--- a/doc/context/sources/general/manuals/cld/cld-specialcommands.tex
+++ b/doc/context/sources/general/manuals/cld/cld-specialcommands.tex
@@ -237,14 +237,14 @@ context.stepwise(function()
 end)
 \stopluacode
 \stoptext
-\starttyping
+\stoptyping
 
 Instead of the \type {testpage} function you can also play directly with
 registers, like:
 
 \starttyping
 if tex.pagegtotal + tex.count.lineheight > tex.pagetotal then
-\starttyping
+\stoptyping
 
 but often an already defined helper does a better job. Of course you will
 probably never need this kind of hacks anyway, if only because much more is going
diff --git a/doc/context/sources/general/manuals/cld/cld-variables.tex b/doc/context/sources/general/manuals/cld/cld-variables.tex
new file mode 100644
index 000000000..c9afbd44c
--- /dev/null
+++ b/doc/context/sources/general/manuals/cld/cld-variables.tex
@@ -0,0 +1,340 @@
+% language=uk
+
+\startcomponent cld-variables
+
+\environment cld-environment
+
+\startchapter[title={Variables}]
+
+\startsection[title={Introduction}]
+
+\index {variables}
+
+Sometimes a bit of experimenting and exploring the boundaries of the \TEX|-|\LUA\
+interfaces results in new mechanisms. To what extent they are really useful is
+hard to say but we just keep them around. Some are described here. This is, at
+least for the moment, a \LMTX\ specific chapter.
+
+\stopsection
+
+\startsection[title={Simple tables}]
+
+\index {variables+grouped tables}
+
+The basic \TEX\ data types are counters (integers), dimensions (kind of floating
+point variables with typographic dimensions), token lists, node lists (boxes),
+fonts, and so on, but no data organized in tables. The first mechanism that we
+discuss is one that obeys grouping. In that respect is behaves like regular
+registers.
+
+\startbuffer
+\newhashedtable\somehashtable
+
+\somehashtable{ foo = 123, bar = "foo" }
+[123 = \the\somehashtable{foo}]
+[foo = \the\somehashtable{bar}]
+
+\bgroup
+\somehashtable{ foo = 456, bar = "oof" }
+[456 = \the\somehashtable{foo}]
+[oof = \the\somehashtable{bar}]
+\egroup
+
+[123 = \the\somehashtable{foo}]
+[foo = \the\somehashtable{bar}]
+
+\bgroup
+\global\somehashtable{ foo = 456 }
+       \somehashtable{ bar = "oof" }
+[456 = \the\somehashtable{foo}]
+[oof = \the\somehashtable{bar}]
+\egroup
+
+[456 = \the\somehashtable{foo}]
+[foo = \the\somehashtable{bar}]
+\stopbuffer
+
+\typebuffer
+
+We define a hashed table, one with keys and values. The definition is global.
+We can then assign values to this table where the assignments themselves are
+hash tables. The above code generates:
+
+\getbuffer
+
+As you can see, the \type {\global} prefix makes the value persistent outside the
+group. You can mix local and global assignments.
+
+Instead of a hashed table, you can have an indexed table. This time the keys are
+numbers.
+
+\startbuffer
+\newindexedtable\someindextable
+
+\someindextable{ 123, "foo" }
+[123 = \the\someindextable 1]
+[foo = \the\someindextable 2]
+
+\bgroup
+\someindextable{ 456, "oof" }
+[456 = \the\someindextable 1]
+[oof = \the\someindextable 2]
+\egroup
+
+[123 = \the\someindextable 1]
+[foo = \the\someindextable 2]
+
+\bgroup
+\global\someindextable{ [1] = 456 }
+       \someindextable{ [2] = "oof" }
+[456 = \the\someindextable 1]
+[oof = \the\someindextable 2]
+\egroup
+
+[456 = \the\someindextable 1]
+[foo = \the\someindextable 2]
+\stopbuffer
+
+\typebuffer
+
+The outcome is the same as before:
+
+\getbuffer
+
+At the \LUA\ end you can access these tables too:
+
+\startbuffer
+\startluacode
+context("[hashed : bar = %s]",context.hashedtables.somehashtable.bar)
+context("[indexed : 2 = %s]", context.indexedtables.someindextable[2])
+\stopluacode
+\stopbuffer
+
+\typebuffer
+
+Indeed we get:
+
+\getbuffer
+
+\stopsection
+
+\startsection[title={Data tables}]
+
+\index {variables+data tables}
+
+In \LUA, tables can be more complex than in the previous section. When you need
+more complex tables, it is likely that your grouping needs are also different,
+which is why we have another mechanism. This mechanism is build on top of another
+model: data values. There are 64K integer registers in any modern \TEX\ engine
+and normally that is more than enough. However, in addition in \LUAMETATEX\ we
+have a variant integer storage unit, one that is lightweight and where the amount
+is limited by the size of the hash table. It was added as part of the low level
+cleanup up of the \LUA\ token interface (a bit more abstraction). Here is an
+example of its usage:
+
+\startbuffer
+\dorecurse {100} {
+    \setdatavalue{#1}{#1}
+}
+
+\start \tttf \darkred \raggedright \dorecurse {100} {
+    #1=\scratchcounter\getdatavalue{#1}\the\scratchcounter
+} \par \stop \blank
+
+\start \tttf \darkgreen \raggedright \dorecurse {100} {
+    #1=\thedatavalue{#1}%
+} \par \stop \blank
+\stopbuffer
+
+\typebuffer
+
+\getbuffer
+
+We define hundred values which are a simple numeric macros. Here we use two
+auxiliary macros because we prefer to use a dedicated namespace for these
+variables. However, there are also primitives that deal with these data values:
+
+\startbuffer[usage]
+\setdatavalue{my-data}{12345}%
+\letdatacode  \MyData  67890
+\thedatavalue{my-data} \the\MyData
+\stopbuffer
+
+\typebuffer[usage]
+
+gives: \inlinebuffer[usage]
+
+But, when more is needed than simple integers, tables come into view. This interface
+is different from the simple one and involves more commands. The next examples show
+about all:
+
+\startbuffer
+\newluatable\testtable
+\setluatable\testtable{ foo = 123, bar = "456", oof = "rab" }
+% \inspectluatable\testtable
+\darkcyan
+foo = \getfromluatable\testtable{foo}\par
+bar = \getfromluatable\testtable{bar}\par
+oof = \getfromluatable\testtable{oof}\par
+\bgroup
+    \useluatable\testtable
+    \setluatable\testtable{ foo = 123123, bar = "456456" }
+  % \inspectluatable\testtable
+    \darkmagenta
+    foo = \getfromluatable\testtable{foo}\par
+    bar = \getfromluatable\testtable{bar}\par
+    oof = \getfromluatable\testtable{oof}\par
+    \startluacode
+        local t = context.luatables.get("testtable")
+        context("<%s %s %s>",t.foo,t.bar,t.oof)
+    \stopluacode \par
+\egroup
+\darkyellow
+foo = \getfromluatable\testtable{foo}\par
+bar = \getfromluatable\testtable{bar}\par
+oof = \getfromluatable\testtable{oof}\par
+\startluacode
+    local t = context.luatables.get("testtable")
+    context("<%s %s %s>",t.foo,t.bar,t.oof)
+\stopluacode \par
+% \inspectluatable\testtable
+\stopbuffer
+
+\typebuffer
+
+The tables are semi|-|local: \type {\newluatable} creates a table and \type
+{\useluatable} will create a local copy that is discarded when the group ends.
+
+\startpacked
+\tttf \getbuffer
+\stoppacked
+
+Hashed and indexed tables can be used mixed, but there are additional accessors
+for indexed tables because there we expect numbers.
+
+\startbuffer
+\newluatable\moretable
+\setluatable\moretable{ 1, "foo" }
+\darkcyan
+[1] = \getfromluatable\moretable{1}\par
+[2] = \idxfromluatable\moretable 2 \par
+\bgroup
+    \useluatable\moretable
+    \setluatable\moretable{ foo = 123123, bar = "456456" }
+    \darkmagenta
+    [1] = \getfromluatable\moretable{1}\par
+    [2] = \idxfromluatable\moretable 2 \par
+    \startluacode
+        local t = context.luatables.get("moretable")
+        context("<%s %s>",t[1],t[2])
+    \stopluacode \par
+\egroup
+\darkyellow
+[1] = \getfromluatable\moretable{1}\par
+[2] = \idxfromluatable\moretable 2 \par
+\startluacode
+    local t = context.luatables.get("moretable")
+    context("<%s %s>",t[1],t[2])
+\stopluacode \par
+\stopbuffer
+
+\typebuffer
+
+\startpacked
+\tttf \getbuffer
+\stoppacked
+
+You can create more complex (nested) tables that you can handle at the \LUA\
+end in the usual way. Basically any \LUA\ that makes a table can go between
+the curly braces.
+
+\stopsection
+
+\startsection[title=Named variables]
+
+\index {variables+named}
+\index {variables+integers}
+\index {variables+cardinals}
+\index {variables+floats}
+\index {integers}
+\index {cardinals}
+\index {floats}
+
+Just because it can be done, and maybe it even has it use, we offer additional
+numbers, stored and accessible by name. The different types all live in their
+own namespace:
+
+\startbuffer[definition]
+\luainteger bar 123456
+\luafloat   bar 123.456e12
+\luainteger gnu = 0xFFFF
+\stopbuffer
+
+\startbuffer[usage]
+{\darkcyan   \the\luainteger bar}
+{\darkmagenta\the\luafloat   bar}
+{\darkyellow \the\luainteger gnu}
+\stopbuffer
+
+\typebuffer[definition,usage] \getbuffer[definition]
+
+These serialize like: \getbuffer[usage]\removeunwantedspaces , and when not set they
+default to zero. There is an extra type, cardinal:
+
+\startbuffer
+\luainteger  a  0x7FFFFFFFFFFFFFFF
+\luainteger  b -0x7FFFFFFFFFFFFFFF
+\luacardinal c  0xFFFFFFFFFFFFFFFF
+\stopbuffer
+
+\typebuffer \getbuffer
+
+We cheat a bit behind the screens because it is actually a double but we scan
+it as positive integer and serialize it as such.
+
+\startbuffer
+[\the \luainteger  a\relax]\par
+[\the \luainteger  b\relax]\par
+[\the \luacardinal c\relax]\par
+\stopbuffer
+
+\typebuffer \getbuffer
+
+You have access to the numbers at the \LUA\ end, as in:
+
+\startbuffer
+\luainteger one 123 \luafloat two 456.678
+\luaexpr{interfaces.numbers.one/1000 + interfaces.numbers.two/10000}
+\stopbuffer
+
+\typebuffer
+
+There are \type {integers}, \type {cardinals} and \type {floats} but the \type
+{numbers} one is the most generic as it tries to resolve it from one of these
+namespaces, so we get: [\inlinebuffer\space\space]. There is also a related
+expression command:
+
+\startbuffer
+(?: \luaexpression          {n.one/1000 + n.two/10000})
+(f: \luaexpression float    {n.one/1000 + n.two/10000})
+(i: \luaexpression integer  {n.one/1000 + n.two/10000})
+(c: \luaexpression cardinal {n.one/1000 + n.two/10000})
+(l: \luaexpression lua      {n.one/1000 + n.two/10000})
+\stopbuffer
+
+\typebuffer
+
+It typesets this (watch the \type {lua} variant, which is a precise
+roundtrip serialization method):
+
+\startlines
+\tttf \getbuffer
+\stoplines
+
+The \type {n} table is just a shortcut to \type {interfaces.numbers}.
+
+\stopsection
+
+\stopchapter
+
+\stopcomponent