path: root/doc/context/manuals/allkind/mkiv-publications.tex

% language=uk

% engine=luajittex

% criterium: all + sorttype=cite     => citex before rest
% criterium: all + sorttype=database => database order
% criterium: used
%
% numbering: label, short, indexinlist, indexused
%
% maybeyear
%
% \cite[data][whatever]

% \showframe

\usemodule[abr-02]
\usemodule[set-11]

\loadsetups[publications-en.xml] \enablemode[interface:setup:defaults]

\setupbackend
  [export=yes,
   xhtml=yes,
   css=export-example.css]

\setupexport
  [hyphen=yes,
   width=60em]

% \input publ-tmp.mkiv

\setupbodyfont
  [dejavu,10pt]

\setuphead
  [chapter]
  [header=high,
   style=\bfc,
   color=darkmagenta]

\setuplayout
  [topspace=2cm,
   bottomspace=1cm,
   header=0cm,
   width=middle,
   height=middle]

\setupwhitespace
  [big]

\setuptyping
  [color=darkmagenta]

\setuptyping
  [keeptogether=yes]

\setuptype
  [color=darkcyan]

\setupfootertexts
  [pagenumber]

\setupMPgraphics
  [mpy=\jobname.mpy]

\setupinteraction
  [state=start,
   color=darkcyan,
   contrastcolor=darkyellow]

\starttext

\startMPpage

    StartPage ;

        % input "mkiv-publications.mpy" ;

        picture pic ; pic := image (
            path pth ; pth := ((0,0) for i=1 step 2 until 20 : -- (i,1) -- (i+1,0) endfor) ;
            for i=0 upto 9 : draw pth shifted (0,2*i) ; endfor ;
        ) ;

        % picture btx ; btx := textext("\ssbf BIBTEX") ;
        % picture ctx ; ctx := textext("\ssbf THE CONTEXT WAY") ;
        picture btx ; btx := image(graphictext("\ssbf BIBTEX") withfillcolor white) ;
        picture ctx ; ctx := image(graphictext("\ssbf THE CONTEXT WAY") withfillcolor white) ;

        pic := pic shifted - llcorner pic ;
        btx := btx shifted - llcorner btx ;
        ctx := ctx shifted - llcorner ctx ;

        pic := pic xysized (PaperWidth,PaperHeight) ;
        btx := btx xsized (2PaperWidth/3) shifted (.25PaperWidth,.15PaperHeight) ;
        ctx := ctx xsized (2PaperWidth/3) shifted (.25PaperWidth,.075PaperHeight) ;

        fill Page withcolor \MPcolor{darkcyan} ;

        draw pic withcolor \MPcolor{darkmagenta} ;
        draw btx withcolor \MPcolor{lightgray} ;
        draw ctx withcolor \MPcolor{lightgray} ;

        % draw boundingbox btx ;
        % draw boundingbox ctx ;

    StopPage ;

\stopMPpage


\startfrontmatter

\starttitle[title=Contents]
    \placelist[chapter,section][color=black]
\stoptitle

\startchapter[title=Introduction]

This manual describes how \MKIV\ handles bibliographies. Support in \CONTEXT\
started in \MKII for \BIBTEX, using a module written by Taco Hoekwater. Later his
code was adapted to \MKIV, but because users demanded more, I decided that
reimplementing made more sense than patching. In particular, through the use of
\LUA, the \BIBTEX\ data files can be easily directly parsed, thus liberating
\CONTEXT\ from the dependency on an external \BIBTEX\ executable. The CritEd
project (by Thomas Schmitz, Alan Braslau, Luigi Scarso and myself) was a good
reason to undertake this rewrite. As part that project users were invited to come
up with ideas about extensions. Not all of them are (yet) honored, but the
rewrite makes more functionality possible.

This manual is dedicated to Taco Hoekwater who in a previous century implemented
the first \BIBTEX\ module and saw it morf into a \TEX||\LUA\ hybrid in this
century. The fact that there was support for bibliographies made it possible for
users to use \CONTEXT\ in an academic environment, dominated by bibliographic
databases encoded in the \BIBTEX\ format.

\startlines
Hans Hagen
PRAGMA ADE
Hasselt NL
\stoplines

\stopchapter

\stopfrontmatter

\startbodymatter

\startchapter[title=The database]

The \BIBTEX\ format is rather popular in the \TEX\ community and even with its
shortcomings it will stay around for a while. Many publication websites can
export and many tools are available to work with this database format. It is
rather simple and looks a bit like \LUA\ tables. Unfortunately the content can be
polluted with non|-|standardized \TEX\ commands which complicates pre- or
postprocessing outside \TEX. In that sense a \BIBTEX\ database is often not coded
neutrally. Some limitations, like the use of commands to encode accented
characters root in the \ASCII\ world and can be bypassed by using \UTF\ instead
(as handled somewhat in \LATEX\ through extensions such as \type {bibtex8}).

The normal way to deal with a bibliography is to refer to entries using a unique
tag or key. When a list of entries is typeset, this reference can be used for
linking purposes. The typeset list can be processed and sorted using the \type
{bibtex} program that converts the database into something more \TEX\ friendly (a
\type {.bbl} file). I never used the program myself (nor bibliographies) so I
will not go into too much detail here, if only because all I say can be wrong.

In \CONTEXT\ we no longer use the \type {bibtex} program: we just use
database files and deal with the necessary manipulations directly in \CONTEXT.
One or more such databases can be used and combined with additional entries
defined within the document. We can have several such datasets active at the same
time.

A \BIBTEX\ file looks like this:

\starttyping
@Article{sometag,
    author  = "An Author and Another One",
    title   = "A hopefully meaningful title",
    journal = maps,
    volume  = "25",
    number  = "2",
    pages   = "5--9",
    month   = mar,
    year    = "2013",
    ISSN    = "1234-5678",
}
\stoptyping

Normally a value is given between quotes (or curly brackets) but single words are
also OK (there is no real benefit in not using quotes, so we advise to always use
them). There can be many more fields and instead of strings one can use
predefined shortcuts. The title for example quite often contains \TEX\ macros.
Some fields, like \type {pages} have funny characters such as the endash
(typically as \type {--}) so we have a mixture of data and typesetting
directives. If you are covering non||english references, you often need
characters that are not in the \ASCII\ subset but \CONTEXT\ is quite happy with
\UTF. If your database file uses old|-|fashioned \TEX\ accent commands then these
will be internally converted automatically to \UTF. Commands (macros) are
converted to an indirect call, which is quite robust.

The \BIBTEX\ files are loaded in memory as \LUA\ table but can be converted to
\XML\ so that we can access them in a more flexible way, but that is a subject
for specialists.

In the old \MKII\ setup we have two kinds of entries: the ones that come from the
\BIBTEX\ run and user supplied ones. We no longer rely on \BIBTEX\ output but we
do still support the user supplied definitions. These were in fact prepared in a
way that suits the processing of \BIBTEX\ generated entries. The next variant
reflects the \CONTEXT\ recoding of the old \BIBTEX\ output.

\starttyping
\startpublication[k=Hagen:Second,t=article,a={Hans Hagen},y=2013,s=HH01]
    \artauthor[]{Hans}[H.]{}{Hagen}
    \arttitle{Who knows more?}
    \journal{MyJournal}
    \pubyear{2013}
    \month{8}
    \volume{1}
    \issue{3}
    \issn{1234-5678}
    \pages{123--126}
\stoppublication
\stoptyping

The split \type {\artauthor} fields are collapsed into a single \type {author}
field as we deal with the splitting later when it gets parsed in \LUA. The \type
{\artauthor} syntax is only kept around for backward compatibility with the
previous use of \BIBTEX.

In the new setup we support these variants as well:

\starttyping
\startpublication[k=Hagen:Third,t=article]
    \author{Hans Hagen}
    \title{Who knows who?}
    ...
\stoppublication
\stoptyping

and

\starttyping
\startpublication[tag=Hagen:Third,category=article]
    \author{Hans Hagen}
    \title{Who knows who?}
    ...
\stoppublication
\stoptyping

and

\starttyping
\startpublication
    \tag{Hagen:Third}
    \category{article}
    \author{Hans Hagen}
    \title{Who knows who?}
    ...
\stoppublication
\stoptyping

Because internally the entries are \LUA\ tables, we also support loading of \LUA\
based definitions:

\starttyping
return {
    ["Hagen:First"] = {
        author   = "Hans Hagen",
        category = "article",
        issn     = "1234-5678",
        issue    = "3",
        journal  = "MyJournal",
        month    = "8",
        pages    = "123--126",
        tag      = "Hagen:First",
        title    = "Who knows nothing?",
        volume   = "1",
        year     = "2013",
    },
}
\stoptyping

Files set up like this can be loaded too. The following \XML\ input is rather
close to this, and is also accepted as input.

\starttyping
<?xml version="2.0" standalone="yes" ?>
<bibtex>
    <entry tag="Hagen:First" category="article">
        <field name="author">Hans Hagen</field>
        <field name="category">article</field>
        <field name="issn">1234-5678</field>
        <field name="issue">3</field>
        <field name="journal">MyJournal</field>
        <field name="month">8</field>
        <field name="pages">123--126</field>
        <field name="tag">Hagen:First</field>
        <field name="title">Who knows nothing?</field>
        <field name="volume">1</field>
        <field name="year">2013</field>
    </entry>
</bibtex>
\stoptyping

{\em Todo: Add some remarks about loading EndNote and RIS formats, but first we
need to complete the tag mapping (on Alan's plate).}

So the user has a rather wide choice of formatting style for bibliography
database files.

\stopchapter

You can load more data than you actually need. Only entries that are referred to
explicitly through the \type {\cite} and \type {\nocite} commands will be shown
in lists. We will cover these details later.

\startchapter[title=Commands in entries]

One unfortunate aspect commonly found in \BIBTEX\ files is that they often
contain \TEX\ commands. Even worse is that there is no standard on what these
commands can be and what they mean, at least not formally, as \BIBTEX\ is a
program intended to be used with many variants of \TEX\ style: plain, \LATEX, and
others. This means that we need to define our use of these typesetting commands.
However, in most cases, they are just abbreviations or font switches and these
are often known. Therefore, \CONTEXT\ will try to resolve them before reporting
an issue. In the log file there is a list of commands that has been seen in the
loaded databases. For instance, loading \type {tugboat.bib} gives a long list of
commands of which we show a small set here:

\starttyping
publications > start used btx commands

publications > standard CONTEXT 1 known
publications > standard ConTeXt 4 known
publications > standard TeXLive 3 KNOWN
publications > standard eTeX    1 known
publications > standard hbox    6 known
publications > standard sltt    1 unknown

publications > stop used btxcommands
\stoptyping

You can define unknown commands, or overload existing definitions in the
following way:

\starttyping
\definebtxcommand\TUB {TUGboat}
\definebtxcommand\sltt{\tt}
\definebtxcommand\<#1>{\type{#1}}
\stoptyping

Unknown commands do not stall processing, but their names are then typeset in a
mono|-|spaced font so they probably stand out for proofreading. You can
access the commands with \type {\btxcommand {...}}, as in:

\startbuffer
commands like \btxcommand{MySpecialCommand} are handled in an indirect way
\stopbuffer

\typebuffer

As this is an undefined command we get: \quotation {\inlinebuffer}.

??

\stopchapter

\startchapter[title=Datasets]

Normally in a document you will use only one bibliographic database, whether or
not distributed over multiple files. Nevertheless we support multiple databases as well
which is why we talk of datasets instead. A dataset is loaded with the \type
{\usebtxdataset} command. Although currently it is not necessary to define a
(default) dataset you can best do this because in the future we might provide more
options. Here are some examples:

\starttyping
\definebtxdataset[standard]

\usebtxdataset[standard][tugboat.bib]
\usebtxdataset[standard][mtx-bibtex-output.xml]
\usebtxdataset[standard][test-001-btx-standard.lua]
\stoptyping

These three suffixes are understood by the loader. Here the dataset has the name
\type {standard} and the three database files are merged, where later entries having the
same tag overload previous ones. Definitions in the document source (coded in \TEX\
speak) are also added, and they are saved for successive runs. This means that if
you load and define entries, they will be known at a next run beforehand, so that
references to them are independent of when loading and definitions take place.

\showsetup{setupbtxdataset}

\showsetup{definebtxdataset}

\showsetup{usebtxdataset}

In this document we use some example databases, so let's load one of them now:

\startbuffer
\definebtxdataset[example]

\usebtxdataset[example][mkiv-publications.bib]
\stopbuffer

\typebuffer \getbuffer

You can ask for an overview of entries in a dataset with:

\startbuffer
\showbtxdatasetfields[example]
\stopbuffer

\typebuffer

this gives:

\getbuffer

You can set the current active dataset with

\starttyping
\setbtxdataset[standard]
\stoptyping

but most publication|-|related commands accept optional arguments that denote the
dataset and references to entries can be prefixed with a dataset identifier.. More
about that later.

Sometimes you want to check a database. One way of doing that is the following:

\startbuffer
\definebtxdataset[check]

\usebtxdataset[check][mkiv-publications-check.bib]

\showbtxdatasetcompleteness[check]
\stopbuffer

\typebuffer

The database like like this:

\typefile{mkiv-publications-check.bib}

The completeness check shows (with green field names) the required fields and
when one is missing this is indicated in red. In blue we show what gets
inherited.

\getbuffer

\stopchapter

\startchapter[title=Renderings]

A list of publications can be rendered at any place in the document. A database
can be much larger than needed for a document. The same is true for the fields
that make up an entry. Here is the list of fields that are currently handled, but
of course there can be additional ones:


\startalignment[flushleft,verytolerant,nothyphenated]
\startluacode
local fields = publications.tracers.fields

for i=1,#fields do
    if i > 1 then
        context(", ")
    end
    context.type(fields[i])
end
\stopluacode
\stopalignment

If you want to see what publications are in the database, the easiest way is to
ask for a complete list:

\startbuffer
\definebtxrendering
  [example]
  [dataset=example,
   method=local,
   alternative=apa]
\placelistofpublications % \placebtxrendering
  [example]
  [criterium=all]
\stopbuffer

\typebuffer

This gives:

\getbuffer

The rendering itself is somewhat complex to set up because we have not only many
different standards but also many fields that can be set up. This means that
there are several commands involved. Often there is a prescribed style to render
bibliographic descriptions, for example \type {apa}. A rendering is setup and
defined with:

\showsetup[setupbtxrendering]
%showrootvalues[btxrendering]
\showsetup[definebtxrendering]

And a list of such descriptions is generated with:

\showsetup[placebtxrendering]

A dataset can have all kind of entries:

\startalignment[flushleft,verytolerant,nothyphenated]
\startluacode
    local categories = publications.tracers.categories

    for i=1,#categories do
        if i > 1 then
            context(", ")
        end
        context.type(categories[i])
    end
\stopluacode
\stopalignment

Each has its own rendering variant. To keep things simple we have their settings
separated. However, these settings are shared for all rendering alternatives. In
practice this is seldom a problem in a publication as only one rendering
alternative will be active. If this be not sufficient, you can always group local
settings in a setup and hook that into the specific rendering.

\showsetup[setupbtxlistvariant]
%showrootvalues[btxlistvariant]
\showsetup[definebtxlistvariant]

Examples of list variants are:

\startluacode
    local variants = publications.tracers.listvariants

    for i=1,#variants do
        context.showinstancevalues( { "btxlistvariant" }, { variants[i] })
    end
\stopluacode

The exact rendering of list entries is determined by the \type {alternative} key
and defaults to \type {apa} which uses definitions from \type
{publ-imp-apa.mkiv}. If you look at that file you will see that each category has
its own setup. You may also notice that additional tests are needed to make sure
that empty fields don't trigger separators and such.

% \showsetup{setuplists}

There are a couple of accessors and helpers to get the job done. When you want to
fetch a field from the current entry you use \type {\btxfield}. In most cases
you want to make sure this field has a value, for instance because you don't want
fences or punctuation that belongs to a field.

\starttyping
\btxdoif {title} {
    \bold{\btxfield{title}},
}
\stoptyping

There are three test macros:

\starttyping
\btxdoifelse{fieldname}{action when found}{action when not found}
\btxdoif    {fieldname}{action when found}
\btxdoifnot {fieldname}                   {action when not found}
\stoptyping

An extra conditional is available for testing interactivity:

\starttyping
\btxdoifelseinteraction{action when true}{action when false}
\stoptyping

In addition there is also a conditional \type {\btxinteractive} which is
more efficient, although in practice efficiency is not so important here.

There are three commands to flush data:

\starttabulate[|||] % Funny usage here! Could not tabulate work without
                    % even specifying the number of columns?
\NC \type {\btxfield}  \NC fetch a explicit field (e.g. \type {year}) \NC \NR
\NC \type {\btxdetail} \NC fetch a derived field (e.g. \type {short}) \NC \NR
\NC \type {\btxflush}  \NC fetch a derived or explicit field \NC \NR
\stoptabulate

Normally you can use \type {\btxfield} or \type {\btxflush} as derived fields
just like analyzed author fields are flushed in a special way.

You can improve readability by using setups, for instance:

\starttyping
\btxdoifelse {author} {
    \btxsetup{btx:apa:author:yes}
} {
    \btxsetup{btx:apa:author:nop}
}
\stoptyping

Keep in mind that normally you don't need to mess with definitions like this
because standard rendering styles are provided. These styles use a few helpers
that inject symbols but also take care of leading and trailing spaces:

\starttabulate[|||]
\NC \type {\btxspace   } \NC before \btxspace    after \NC \NR
\NC \type {\btxperiod  } \NC before \btxperiod   after \NC \NR
\NC \type {\btxcomma   } \NC before \btxcomma    after \NC \NR
\NC \type {\btxlparent } \NC before \btxlparent  after \NC \NR
\NC \type {\btxrparent } \NC before \btxrparent  after \NC \NR
\NC \type {\btxlbracket} \NC before \btxlbracket after \NC \NR
\NC \type {\btxrbracket} \NC before \btxrbracket after \NC \NR
\stoptabulate

So, the previous example setup can be rewritten as:

\starttyping
\btxdoif {title} {
    \bold{\btxfield{title}}
    \btxcomma
}
\stoptyping

There is a special command for rendering a (combination) of authors:

\starttyping
\btxflushauthor{author}
\btxflushauthor{editor}
\btxflushauthor[inverted]{editor}
\stoptyping

Instead of the last one you can also use:

\starttyping
\btxflushauthorinverted{editor}
\stoptyping

You can use a (configurable) default or pass directives: Valid directives are

\starttabulate
\NC \bf conversion       \NC \bf rendering        \NC \NR
\HL
\NC \type{inverted}      \NC the Frog jr, Kermit  \NC \NR
\NC \type{invertedshort} \NC the Frog jr, K       \NC \NR
\NC \type{normal}        \NC Kermit, the Frog, jr \NC \NR
\NC \type{normalshort}   \NC K, the Frog, jr      \NC \NR
\stoptabulate

The list itself is not a list in the sense of a regular \CONTEXT\ structure related
list. We do use the list mechanism to keep track of used entries but that is mostly
because we can then reuse filtering mechanisms. The actual rendering of a reference
and entry runs on top of so called constructions (other examples of constructions are
descriptions, enumerations and notes).

\showsetup[setupbtxlist]

You need to be aware what command is used to achieve the desired result. For instance,
in order to put parentheses around a number reference you say:

\starttyping
\setupbtxlistvariant
  [num]
  [left=(,
   right=)]
\stoptyping

If you want automated width calculations, the following does the trick:

\starttyping
\setupbtxrendering
  [standard]
  [width=auto]
\stoptyping

but if you want to control it yourself you say something:

\starttyping
\setupbtxrendering
  [width=none]

\setupbtxlist
  [standard]
  [width=3cm,
   distance=\emwidth,
   color=red,
   headcolor=blue,
   headalign=flushright]
\stoptyping

In most cases the defaults will work out fine.

\stopchapter

\startchapter[title=Citations]

Citations are references to bibliographic entries that normally show up in lists
someplace in the document: at the end of a chapter, in an appendix, at the end of
an article, etc. We discussed the rendering of these lists  in the previous chapter.
A citation is normally pretty short as its main purpose is to refer uniquely to a more
detailed description. But, there are several ways to refer, which is why the citation
subsystem is configurable and extensible. Just look at the following commands:

\startbuffer
\cite[author][example::demo-003]
\cite[authoryear][example::demo-003]
\cite[authoryears][example::demo-003]
\cite[author][example::demo-003,demo-004]
\cite[authoryear][example::demo-003,demo-004]
\cite[authoryears][example::demo-003,demo-004]
\cite[author][example::demo-004,demo-003]
\cite[authoryear][example::demo-004,demo-003]
\cite[authoryears][example::demo-004,demo-003]
\stopbuffer

\typebuffer

\startlines \getbuffer \stoplines

The first argument is optional.
% What is the default? How can one set this up?

\showsetup[cite]

You can tune the way a citation shows up:

\startbuffer
\setupbtxcitevariant[author]     [sorttype=author,color=darkyellow]
\setupbtxcitevariant[authoryear] [sorttype=author,color=darkyellow]
\setupbtxcitevariant[authoryears][sorttype=author,color=darkyellow]

\cite[author][example::demo-004,demo-003]
\cite[authoryear][example::demo-004,demo-003]
\cite[authoryears][example::demo-004,demo-003]
\stopbuffer

\typebuffer

Here we sort the authors and color the citation:

\startlines \getbuffer \stoplines

For reasons of backward compatibility the \type {\cite} command is a bit picky
about spaces between the two arguments, of which the first is optional. This is
a consequence of allowing its use with the key specified between curly brackets
as is the traditional practice. (We do encourage users to adopt the more
coherent \CONTEXT\ syntax by using square brackets for keywords and reserving
curly brackets to regroup text to be typeset.)
% Just how is it picky?

The \type {\citation} command is synonymous but is more flexible with respect to
spacing of its arguments:

\starttyping
\citation[author]     [example::demo-004,demo-003]
\citation[authoryear] [example::demo-004,demo-003]
\citation[authoryears][example::demo-004,demo-003]
\stoptyping

% The first argument of cite and citation is optional. What is the default and how does one set it?

There is a whole bunch of cite options and more can be easily defined.

\startluacode
local variants = publications.tracers.citevariants

context.starttabulate { "|l|p|" }
    context.NC() context.bold("key")
    context.NC() context.bold("rendering")
    context.NC() context.NR() context.FL()
    for i=1,#variants do
        local variant = variants[i]
        context.NC() context.type(variant)
        context.NC() context.citation( { variant }, { "example::demo-005" })
        context.NC() context.NR()
    end
context.stoptabulate()
\stopluacode

Because we are dealing with database input and because we generally need to
manipulate entries, much of the work is delegated to \LUA. This makes it easier
to maintain and extend the code. Of course \TEX\ still does the rendering. The
typographic details are controlled by parameters but not all are used in all
variants. As with most \CONTEXT\ commands, it starts out with a general setup
command:

\showsetup[setupbtxcitevariant]

On top of that we can define instances that inherit either from a given parent or
from the topmost setup.

\showsetup[definebtxcitevariant]

% The default values are:

% \showrootvalues[btxcitevariant]

But, specific variants can have them overloaded:

% \showinstancevalues[setupbtxcitevariant][author]
% \showinstancevalues[setupbtxcitevariant][authornum]

\startluacode
    local variants = publications.tracers.citevariants

    for i=1,#variants do
        context.showinstancevalues( { "btxcitevariant" }, { variants[i] })
    end
\stopluacode

A citation variant is defined in several steps and if you really want to know
the dirty details, you should look into the \type {publ-imp-*.mkiv} files. Here
we stick to the concept.

\starttyping
\startsetups btx:cite:author
    \btxcitevariant{author}
\stopsetups
\stoptyping

You can overload such setups if needed, but that only makes sense when you cannot
configure the rendering with parameters. The \type {\btxcitevariant} command is
one of the build in accessors and it calls out to \LUA\ where more complex
manipulation takes place if needed. If no manipulation is known, the field with
the same name (if found) will be flushed. A command like \type {\btxcitevariant}
assumes that a dataset and specific tag has been set. This is normally done in
the wrapper macros, like \type {\cite}. For special purposes you can use these
commands

\starttyping
\setbtxdataset[example]
\setbtxentry[hh2013]
\stoptyping

But don't expect too much support for such low level rendering control.

Unless you use \type {criterium=all} only publications that are cited will end up
in the lists. You can force a citation into a list using \type {\usecitation}, for
example:

\starttyping
\usecitation[example::demo-004,demo-003]
\stoptyping

This command has two synonyms: \type {\nocite} and \type {\nocitation} so you can
choose whatever fits you best.

\showsetup[nocite]

\stopchapter

\startchapter[title=The \LUA\ view]

Because we manage data at the \LUA\ end it is tempting to access it there for
other purposes. This is fine as long as you keep in mind that aspects of the
implementation may change over time, although this is unlikely once the modules
become stable.

The entries are collected in datasets and each set has a unique name. In this
document we have the set named \type {example}. A dataset table has several
fields, and probably the one of most interest is the \type {luadata} field. Each
entry in this table describes a publication:

\startluacode
    context.tocontext(publications.datasets.example.luadata["demo-001"])
\stopluacode

This is \type {publications.datasets.example.luadata["demo-001"]}. There can be
a companion entry in the parallel \type {details} table.

\startluacode
    context.tocontext(publications.datasets.example.details["demo-001"])
\stopluacode

These details are accessed as \type
{publications.datasets.example.details["demo-001"]} and by using a separate table
we can overload fields in the original entry without losing the original.

You can loop over the entries using regular \LUA\ code combined with \MKIV\
helpers:

\startbuffer
local dataset = publications.datasets.example

context.starttabulate { "|l|l|l|" }
for tag, entry in table.sortedhash(dataset.luadata) do
    local detail = dataset.details[tag] or { }
    context.NC() context.type(tag)
    context.NC() context(detail.short)
    context.NC() context(entry.title)
    context.NC() context.NR()
end
context.stoptabulate()
\stopbuffer

\typebuffer

This results in:

\ctxluabuffer

You can manipulate a dataset after loading. Of course this assumes that you know
what kind of content you have and what you need for rendering. As example we
load a small dataset.

\startbuffer
\definebtxdataset[drumming]
\usebtxdataset[drumming][mkiv-publications.lua]
\stopbuffer

\typebuffer \getbuffer

Because we're going to do some \LUA, we could also have loaded the dataset
with:

\starttyping
publications.load("drumming","mkiv-publications.lua","lua")
\stoptyping

The dataset has three entries:

\typefile{mkiv-publications.lua}

As you can see, we can have a subtitle. We will combine the title and subtitle
into one:

\startbuffer
\startluacode
for tag, entry in next, publications.datasets.drumming.luadata do
    if entry.subtitle then
        if entry.title then
            entry.title = entry.title .. ", " .. entry.subtitle
        else
            entry.title = entry.subtitle
        end
        entry.subtitle = nil
        logs.report("btx","combining title and subtitle of entry tagged %a",tag)
    end
end
\stopluacode
\stopbuffer

\typebuffer \getbuffer

We can now typeset the entries with:

\startbuffer
\definebtxrendering[drumming][dataset=drumming,method=dataset]
\placebtxrendering[drumming]
\stopbuffer

\typebuffer

Because we just want to show the entries, and have no citations that force them
to be shown, we have to the \type {method} to \type {dataset}. \footnote {Gavin
Harrison is in my opinion one of the most creative, diverse and interesting
drummers of our time. It's also fascinating to watch him play and a welcome
distraction from writing code and manuals.}

\blank \getbuffer \blank

\stopchapter

\startchapter[title=The \XML\ view]

The \type {luadata} table can be converted into an \XML\ representation. This is
a follow up on earlier experiments with an \XML|-|only approach. I decided in the end
to stick to a \LUA\ approach and provide some simple \XML\ support in addition.

Once a dataset is accessible as \XML\ tree, you can use the regular \type {\xml...}
commands. We start with loading a dataset, in this case from just one file.

\startbuffer
\usebtxdataset[tugboat][tugboat.bib]
\stopbuffer

\typebuffer \getbuffer

The dataset has to be converted to \XML:

\startbuffer
\convertbtxdatasettoxml[tugboat]
\stopbuffer

\typebuffer \getbuffer

The tree is now accessible by its root reference \type {btx:tugboat}. If we want simple
field access we can use a few setups:

\startbuffer
\startxmlsetups btx:initialize
    \xmlsetsetup{#1}{bibtex|entry|field}{btx:*}
    \xmlmain{#1}
\stopxmlsetups

\startxmlsetups btx:field
    \xmlflushcontext{#1}
\stopxmlsetups

\xmlsetup{btx:tugboat}{btx:initialize}
\stopbuffer

\typebuffer \getbuffer

The two setups are predefined in the core already, but you might want to change them. They are
applied in for instance:

\startbuffer
\starttabulate[|||]
    \NC \type {tag}   \NC \xmlfirst {btx:tugboat}
        {/bibtex/entry[string.find(@tag,'Hagen')]/attribute('tag')}
    \NC \NR
    \NC \type {title} \NC \xmlfirst {btx:tugboat}
        {/bibtex/entry[string.find(@tag,'Hagen')]/field[@name='title']}
    \NC \NR
\stoptabulate
\stopbuffer

\typebuffer \getbuffer

\startbuffer
\startxmlsetups btx:demo
    \xmlcommand
        {#1}
        {/bibtex/entry[string.find(@tag,'Hagen')][1]}{btx:table}
\stopxmlsetups

\startxmlsetups btx:table
\starttabulate[|||]
    \NC \type {tag}   \NC \xmlatt{#1}{tag} \NC \NR
    \NC \type {title} \NC \xmlfirst{#1}{/field[@name='title']} \NC \NR
\stoptabulate
\stopxmlsetups

\xmlsetup{btx:tugboat}{btx:demo}
\stopbuffer

\typebuffer \getbuffer

Here is another example:

\startbuffer
\startxmlsetups btx:row
    \NC \xmlatt{#1}{tag}
    \NC \xmlfirst{#1}{/field[@name='title']}
    \NC \NR
\stopxmlsetups

\startxmlsetups btx:demo
    \xmlfilter {#1} {
        /bibtex
        /entry[@category='article']
        /field[@name='author' and (find(text(),'Knuth') or find(text(),'DEK'))]
        /../command(btx:row)
    }
\stopxmlsetups

\starttabulate[|||]
    \xmlsetup{btx:tugboat}{btx:demo}
\stoptabulate
\stopbuffer

\typebuffer \getbuffer

A more extensive example is the following. Of course this assumes that you
know what \XML\ support mechanisms and macros are available.

\startbuffer
\startxmlsetups btx:getkeys
    \xmladdsortentry{btx}{#1}{\xmlfilter{#1}{/field[@name='author']/text()}}
    \xmladdsortentry{btx}{#1}{\xmlfilter{#1}{/field[@name='year'  ]/text()}}
    \xmladdsortentry{btx}{#1}{\xmlatt{#1}{tag}}
\stopxmlsetups

\startxmlsetups btx:sorter
    \xmlresetsorter{btx}
  % \xmlfilter{#1}{entry/command(btx:getkeys)}
    \xmlfilter{#1}{
        /bibtex
        /entry[@category='article']
        /field[@name='author' and find(text(),'Knuth')]
        /../command(btx:getkeys)}
    \xmlsortentries{btx}
    \starttabulate[||||]
        \xmlflushsorter{btx}{btx:entry:flush}
    \stoptabulate
\stopxmlsetups

\startxmlsetups btx:entry:flush
    \NC \xmlfilter{#1}{/field[@name='year'  ]/context()}
    \NC \xmlatt{#1}{tag}
    \NC \xmlfilter{#1}{/field[@name='author']/context()}
    \NC \NR
\stopxmlsetups

\xmlsetup{btx:tugboat}{btx:sorter}
\stopbuffer

\typebuffer \getbuffer

The original data is stored in a \LUA\ table, hashed by tag. Starting with \LUA\ 5.2
each run of \LUA\ gets a different ordering of such a hash. In older versions, when you
looped over a hash, the order was undefined, but the same as long as you used the same
binary. This had the advantage that successive runs, something we often have in document
processing gave consistent results. In today's \LUA\ we need to do much more sorting of
hashes before we loop, especially when we save multi||pass data. It is for this reason
that the \XML\ tree is sorted by hash key by default. That way lookups (especially
the first of a set) give consistent outcomes.

\stopchapter

\startchapter[title=Standards]

The rendering of bibliographic entries is often standardized and prescribed by
the publisher. If you submit an article to a journal, normally it will be
reformatted (or even re|-|keyed) and the rendering will happen at the publishers
end. In that case it may not matter how entries were rendered when writing the
publication, because the publisher will do it his or her way.
This means that most users probably will stick to the standard \APA\ rules and for
them we provide some configuration. Because we use setups it is easy to overload
specifics. If you really want to tweak, best look in the files that deal with it.

Many standards exist and support for other renderings may be added to the core.
Interested users are invited to develop and to test alternate standard renderings
according to their needs.

Todo: maybe a list of categories and fields.

\stopchapter

\startchapter[title=Cleaning up]

Although the \BIBTEX\ format is reasonably well defined, in practice there are
many ways to organize the data. For instance, one can use predefined string
constants that get used (either or not combined with other strings) later on. A string
can be enclosed in curly braces or double quotes. The strings can contain \TEX\ commands
but these are not standardized. The databases often have somewhat complex
ways to deal with special characters and the use of braces in their definition is also
not normalized.

The most complex to deal with are the fields that contain names of people. At some point it
might be needed to split a combination of names into individual ones that then get split into
title, first name, optional inbetweens, surname(s) and additional: \type {Prof. Dr. Alfred
B. C. von Kwik Kwak Jr. II and P. Q. Olet} is just one example of this. The convention seems
to be not to use commas but \type {and} to separate names (often each name will be specified
as lastname, firstname).

We don't see it as challenge nor as a duty to support all kinds of messy definitions. Of
course we try to be somewhat tolerant, but you will be sure to get better results if you
use nicely setup, consistent databases.

Todo: maybe some examples of bad.

\stopchapter

\startchapter[title=Transition]

In the original bibliography support module usage was as follows (example taken
from the contextgarden wiki):

\starttyping
% engine=pdftex

\usemodule[bib]
\usemodule[bibltx]

\setupbibtex
  [database=xampl]

\setuppublications
  [numbering=yes]

\starttext
    As \cite [article-full] already indicated, bibtex is a \LATEX||centric
    program.

    \completepublications
\stoptext
\stoptyping

For \MKIV\ the modules were partly rewritten and ended up in the core so the two
{\usemodule} commands were no longer needed. The overhead associated with the
automatic loading of the bibliography macros can be neglected these days, so
standardized modules such as \type {bib} are all being moved to the core and do
not need to be explicitly loaded.

The first \type {\setupbibtex} command in this example is needed to bootstrap
the process: it tells what database has to be processed by \BIBTEX\ between
runs. The second \type {\setuppublications} command is optional. Each citation
(tagged with \type {\cite}) ends up in the list of publications.

In the new approach we no longer use \BIBTEX so we don't need to setup \BIBTEX.
Instead we define dataset(s). We also no longer set up publications with one
command, but have split that up in rendering-, list-, and cite|-|variants. The
basic \type {\cite} command remains. The above example becomes:

\starttyping
\definebtxdataset
  [document]

\usebtxdataset
  [document]
  [mybibfile.bib]

\definebtxrendering
  [document]

\setupbtxrendering
  [document]
  [numbering=yes]

\starttext
    As \cite [article-full] already indicated, bibtex is a \LATEX||centric
    program.

    \completebtxrendering[document]
\stoptext
\stoptyping

So, we have a few more commands to set up things. If you intend to use just a
single dataset and rendering, the above preamble can be simplified to:

\starttyping
\usebtxdataset
  [mybibfile.bib]

\setupbtxrendering
  [numbering=yes]
\stoptyping

But keep in mind that compared to the old \MKII\ derived method we have moved
some of the options to the rendering, list and cite setup variants.

Another difference is now the use of lists. When you define a rendering, you
also define a list. However, all entries are collected in a common list tagged
\type {btx}. Although you will normally configure a rendering you can still set
some properties of lists, but in that case you need to prefix the list
identifier. In the case of the above example this is \type {btx:document}.

\stopchapter

\startchapter[title=\MLBIBTEX]

Todo: how to plug in \MLBIBTEX\ for sorting and other advanced operations.

\stopchapter

\startchapter[title=Extensions]

As \TEX\ and \LUA\ are both open and accessible in \CONTEXT\ it is possible to
extend the functionality of the bibliography related code. For instance, you can add
extra loaders.

\starttyping
function publications.loaders.myformat(dataset,filename)
    local t = { }
    -- Load data from 'filename' and convert it to a Lua table 't' with
    -- the key as hash entry and fields conforming the luadata table
    -- format.
    loaders.lua(dataset,t)
end
\stoptyping

This then permits loading a database (into a dataset) with the command:

\starttyping
\usebtxdataset[standard][myfile.myformat]
\stoptyping

The \type {myformat} suffix is recognized automatically. If you want to use another
suffix, you can do this:

\starttyping
\usebtxdataset[standard][myformat::myfile.txt]
\stoptyping

\stopchapter

\startchapter[title=Notes]

The move from external \BIBTEX\ processing to internal processing has the
advantage that we stay within the same run. In the traditional approach we had
roughly the following steps:

\startitemize[packed]
\startitem the first run information is collected and written to file \stopitem
\startitem after that run the \BIBTEX\ program converts that file to another one \stopitem
\startitem successive runs use that data for references and producing lists \stopitem
\stopitemize

In the \MKIV\ approach the bibliographic database is loaded in memory each run
and processing also happens each run. On paper this looks less efficient but as
\LUA\ is quite fast, in practice performance is much better.

Probably most demanding is the treatment of authors as we have to analyze names,
split multiple authors and reassemble firstnames, vons, surnames and juniors.
When we sort by author sorting vectors have to be made which also has a penalty.
However, in practice the user will not notice a performance degradation. We did
some tests with a list of 500.000 authors, sorted them and typeset them as list
(producing some 5400 dense pages in a small font and with small margins). This is
typical one of these cases where using \LUAJITTEX\ saves quite time. On my
machine it took just over 100 seconds to get this done. Unfortunately not all
operating systems performed equally well: 32 bit versions worked fine, but 64 bit
\LINUX\ either crashed (stalled) the machine or ran out of memory rather fast,
while \MACOSX\ and \WINDOWS\ performed fine. In practice you will never run into
this, unless you produce massive amounts of bibliographic entries. \LUAJIT\ has
some benefits but also some drawbacks.

\stopchapter

\stopbodymatter

\stoptext

Todo:

\setuplabeltext[en][reprint=reprint]
\setuplabeltext[de][reprint=Nachdruck]

note = {\labeltext{reprint} 2004}