2014-12-03 18:28:00

1 files changed, 351 insertions, 0 deletions

diff --git a/doc/context/sources/general/manuals/workflows/workflows-mkiv.tex b/doc/context/sources/general/manuals/workflows/workflows-mkiv.tex
new file mode 100644
index 000000000..fd4737296
--- /dev/null
+++ b/doc/context/sources/general/manuals/workflows/workflows-mkiv.tex
@@ -0,0 +1,351 @@
+% language=uk
+
+% \usepath[jobfile:]
+% \setupexternalfigures[directory=jobfile:]
+% \usepath[toppath:]
+% \setupexternalfigures[directory=toppath:]
+
+\usemodule
+  [abr-04]
+
+\setupbodyfont
+  [bookman,11pt]
+
+\definecolor
+  [maincolor]
+  [s=.35]
+
+\setuplayout
+  [height=middle,
+   width=middle,
+   footer=0pt]
+
+\setupwhitespace
+  [big]
+
+\setuphead
+  [chapter]
+  [style=\bfc,
+   color=maincolor,
+   header=high]
+
+\setuphead
+  [section]
+  [style=\bfb,
+   color=maincolor]
+
+\setuptyping
+  [color=maincolor]
+
+\setuptype
+  [color=maincolor]
+
+% \showframe
+
+\startdocument
+  [metadata:author=Hans Hagen,
+   metadata:title=Workflow support in context,
+   author=Hans Hagen,
+   affiliation=PRAGMA ADE,
+   location=Hasselt NL,
+   title=workflow,
+   extra=support in context,
+   support=www.contextgarden.net,
+   website=www.pragma-ade.nl]
+
+\definefontfeature[LatinModernMonoVariable][default][liga=no]
+\definefont[LatinModernMonoVariable][LMTypewriterVarWd-Regular*LatinModernMonoVariable sa 1]
+
+\startMPpage
+
+    fill fullsquare xysized(PaperWidth,PaperHeight) withcolor .4white ;
+
+    draw image (
+        fill arrowhead fullcircle scaled .5 rotated   0 scaled 10 withcolor white ;
+        fill arrowhead fullcircle scaled .5 rotated 120 scaled 10 withcolor white ;
+        fill arrowhead fullcircle scaled .5 rotated 240 scaled 10 withcolor white ;
+        fill arrowhead fullcircle scaled .5 rotated  60 scaled 10 withcolor white ;
+        fill arrowhead fullcircle scaled .5 rotated 180 scaled 10 withcolor white ;
+        fill arrowhead fullcircle scaled .5 rotated 300 scaled 10 withcolor white ;
+    ) xsized (.9PaperWidth) shifted (0,.2PaperWidth) ;
+
+    draw image (
+        fill arrowhead fullcircle scaled .5 rotated   0 scaled 10 withcolor .4red     withtransparency (1,.5) ;
+        fill arrowhead fullcircle scaled .5 rotated 120 scaled 10 withcolor .4green   withtransparency (1,.5) ;
+        fill arrowhead fullcircle scaled .5 rotated 240 scaled 10 withcolor .4blue    withtransparency (1,.5) ;
+        fill arrowhead fullcircle scaled .5 rotated  60 scaled 10 withcolor .4cyan    withtransparency (1,.5) ;
+        fill arrowhead fullcircle scaled .5 rotated 180 scaled 10 withcolor .4magenta withtransparency (1,.5) ;
+        fill arrowhead fullcircle scaled .5 rotated 300 scaled 10 withcolor .4yellow  withtransparency (1,.5) ;
+    ) xsized (.9PaperWidth) shifted (0,.2PaperWidth) ;
+
+    draw textext ("\LatinModernMonoVariable \documentvariable{title}") xsized (.9PaperWidth) shifted (0,-.425PaperWidth) withcolor white ;
+    draw textext ("\LatinModernMonoVariable \documentvariable{extra}") xsized (.9PaperWidth) shifted (0,-.575PaperWidth) withcolor white ;
+
+\stopMPpage
+
+\startfrontmatter
+
+\starttitle[title=Contents]
+
+    \placelist[chapter]
+
+\stoptitle
+
+\stopfrontmatter
+
+\startbodymatter
+
+\startchapter[title=Introduction]
+
+This manual contains some information about features that can help you to manage
+workflows or \CONTEXT\ related processes. Because we use \CONTEXT\ ourselves all
+that we need ends up in the distribution. When you discover something workflow
+related that is not yet covered here, you can tell me. I simply forget about all
+there is (especially if it's made for projects.)
+
+\startlines
+\documentvariable{author},
+\documentvariable{affiliation}
+\documentvariable{location}
+\currentdate[month,year]
+\stoplines
+
+\stopsubject
+
+\stopchapter
+
+\startchapter[title=Accessing resources]
+
+One of the benefits of \TEX\ is that you can use it in automated workflows
+where large quantities of data is involved. A document can consist of
+several files and normally also includes images. Of course there are styles
+involved too. At \PRAGMA\ normally put styles and fonts in:
+
+\starttyping
+/data/site/context/tex/texmf-project/tex/context/user/<project>/...
+/data/site/context/tex/texmf-fonts/data/<foundry>/<collection>/...
+\stoptyping
+
+alongside
+
+\starttyping
+/data/framework/...
+\stoptyping
+
+where the job management services are put, while we put resources in:
+
+\starttyping
+/data/resources/...
+\stoptyping
+
+The processing happens in:
+
+\starttyping
+/data/work/<uuid user space>/
+\stoptyping
+
+Putting styles (and resources like logos and common images) and fonts (if the
+project has specific ones not present in the distribution) in the \TEX\ tree
+makes sense because that is where such files are normally searched. Of course you
+need to keep the distributions file database upto|-|date after adding files there.
+
+Processing has to happen isolated from other runs so there we use unique
+locations. The services responsible for running also deal with regular cleanup
+of these temporary files.
+
+Resources are somewhat special. They can be stable, i.e.\ change seldom, but more
+often they are updated or extended periodically (or even daily). We're not
+talking of a few files here but of thousands. In one project we have 20 thousand
+resources, that can be combined into arbitrary books, and in another one, each
+chapter alone is about 400 \XML\ and image files. That means we can have 5000
+files per book and as we have at least 20 books, we end up with 100K files. In
+the first case accessing the resources is easy because there is a well defined
+structure (under our control) so we know exactly where each file sits in the
+resource tree. In the 100K case there is a deeper structure which is in itself
+predictable but because many authors are involved the references to these files
+are somewhat instable (and undefined). It is surprising to notice that publishers
+don't care about filenames (read: cannot control all the parties involved) which
+means that we have inconsist use of mixed case in filenames, and spaces,
+underscores and dashes creeping in. Because typesetting for paper is always at
+the end of the pipeline (which nowadays is mostly driven by (limitations) of web
+products) we need to have a robust and flexible lookup mechanism. It's a side
+effect of the click and point culture: if objects are associated (filename in
+source file with file on the system) anything you key in will work, and
+consistency completely depends on the user. And bad things then happen when files
+are copied, renamed, etc. In that stadium we can better be tolerant than try to
+get it fixed. \footnote {From what we normally receive we often conclude that
+copy|-|editing and image production companies don't impose any discipline or
+probably simply lack the tools and methods to control this. Some of our workflows
+had checkers and fixers, so that when we got 5000 new resources while only a few
+needed to be replaced we could filter the right ones.  It was not uncommon to find
+duplicates for thousands of pictures: similar or older variants.}
+
+\starttyping
+foo.jpg
+bar/foo.jpg
+images/bar/foo.jpg
+images/foo.jpg
+\stoptyping
+
+The xml files have names like:
+
+\starttyping
+b-c.xml
+a/b-c.jpg
+a/b/b-c.jpg
+a/b/c/b-c.jpg
+\stoptyping
+
+So it's sort of a mess, especially if you add arbitrary casing to this. Of course
+one can argue that a wrong (relative) location is asking for problems, it's less
+an issue here because each image has a unique name. We could flatten the resource
+tree but having tens of thousands of files on one directory is asking for
+problems when you want to manage them.
+
+The typesetting (and related services) run on virtual machines. The three
+directories:
+
+\starttyping
+/data/site
+/data/resources
+/data/work
+\stoptyping
+
+are all mounted as nfs shares on a network storage. For the styles (and binaries)
+this is no big deal as normally these files are cached, but the resources are
+another story. Scanning the complete (mounted) resource tree each run is no
+option so there we use a special mechanism in \CONTEXT\ for locating files.
+
+Already early in the development of \MKIV\ one of the locating mechanisms was
+the following:
+
+\starttyping
+tree:////data/resources/foo/**/drawing.jpg
+tree:////data/resources/foo/**/Drawing.jpg
+\stoptyping
+
+Here the tree is scanned once per run, which is normally quite okay when there
+are not that many files and when the files reside on the machine itself. For a
+more high performance approach using network shares we have a different
+mechanism. This time it looks like this:
+
+\starttyping
+dirlist:/data/resources/**/drawing.jpg
+dirlist:/data/resources/**/Drawing.jpg
+dirlist:/data/resources/**/just/some/place/drawing.jpg
+dirlist:/data/resources/**/images/drawing.jpg
+dirlist:/data/resources/**/images/drawing.jpg?option=fileonly
+dirfile:/data/resources/**/images/drawing.jpg
+\stoptyping
+
+The first two lookups are wildcard. If there is a file with that name, it will be
+found. If there are more, the first hit is used. The second and third examples
+are more selective. Here the part after the \type {**} has to match too. So here
+we can deal with multiple files named \type {drawing.jpg}. The last two
+equivalent examples are more tolerant. If no explicit match is found, a lookup
+happens without being selective. The case of a name is ignored but when found, a
+name with the right case is used.
+
+You can hook a path into the resolver for source files, for example:
+
+\starttyping
+\usepath                       [dirfile://./resources/**]
+\setupexternalfigures[directory=dirfile://./resources/**]
+\stoptyping
+
+You need to make sure that file(name)s in that location don't override ones in
+the regular \TEX\ tree. These extra paths are only used for source file lookups
+so for instance font lookups are not affected.
+
+When you add, remove or move files the tree, you need to remove the \type
+{dirlist.*} files in the root because these are used for locating files. A new
+file will be generated automatically. Don't forget this!
+
+\stopchapter
+
+\startchapter[title=Graphics]
+
+\startsection[title=Bad names]
+
+After many years of using \CONTEXT\ in workflows where large amounts of source files
+as well as graphics were involved we can safely say that it's hard for publishers to
+control the way these are named. This is probably due to the fact that in a
+click|-|and|-|point based desktop publishing workflow names don't matter as one stays on
+one machine, and names are only entered once (after that these names become abstractions and
+get cut and pasted). Proper consistent resource managament is simply not part of the flow.
+
+This means that you get names like:
+
+\starttyping
+foo_Bar_01_03-a.EPS
+foo__Bar-01a_03.eps
+foo__Bar-01a_03.eps
+foo BarA  01-03.eps
+\stoptyping
+
+Especially when a non proportional screen font is used multiple spaces can look
+like one. In fancy screen fonts upper and lowercase usage might get obscured. It
+really makes one wonder if copy|-|editing or adding labels to graphics isn't
+suffering from the same problem.
+
+Anyhow, as in an automated rendering workflow the rendering is often the last step you
+can imagine that when names get messed up it's that last step that gets blamed. It's not
+that hard to sanitize names of files on disk as well as in the files that refer to them,
+and we normally do that we have complete control. This is no option when all the resources
+are synchronzied from elsewhere. In that case the only way out is signaling potential
+issues. Say that in the source file there is a reference:
+
+\starttyping
+foo_Bar_01_03-a.EPS
+\stoptyping
+
+and that the graphic on disk has the same name, but for some reason after an update
+has become:
+
+\starttyping
+foo-Bar_01_03-a.EPS
+\stoptyping
+
+The old image is probably still there so the update is not reflected in the final
+product. This is not that uncommon when you deal with tens of thousands of files,
+many editors and graphic designers, and no strict filename policy.
+
+For this we provide the following tracing option:
+
+\starttyping
+\enabletrackers[graphics.lognames]
+\stoptyping
+
+This will put information in the log file about included graphics, like:
+
+\starttyping
+system          > graphics > start names
+
+used graphic    > asked   : cow.pdf
+used graphic    > comment : not found
+used graphic    > asked   : t:/sources/cow.pdf
+used graphic    > format  : pdf
+used graphic    > found   : t:/sources/cow.pdf
+used graphic    > used    : t:/sources/cow.pdf
+
+system          > graphics > stop names
+\stoptyping
+
+You can also information to the file itself:
+
+\starttyping
+\usemodule[s-figures-names]
+\stoptyping
+
+Of course that has to be done at the end of the document. Bad names are reported
+and suitable action can be taken.
+
+\stopsection
+
+\stopchapter
+
+\stopbodymatter
+
+\stopdocument
+
+\disabledirectives[resolvers.maxreadlevel]