% language=us

% Talking of keywords: Jacob Collier, Count The People is definitely an example
% of showing keywords and no way that the fonts used there are done by tex:
%
% https://www.youtube.com/watch?v=icplHV25fqs

\environment evenmore-style

\startcomponent evenmore-keywords

\startchapter[title=Keywords]

Some primitives in \TEX\ can take one or more optional keywords and|/|or keywords
followed by one or more values. In traditional \TEX\ it concerns a handful of
primitives, in \PDFTEX\ there are plenty of backend|-|related primitives,
\LUATEX\ introduced optional keywords to some math constructs and attributes to
boxes, and \LUAMETATEX\ adds some more too. The keyword scanner in \TEX\ is
rather special. Keywords are used in cases like:

\starttyping
\hbox spread 10cm {...}
\advance\scratchcounter by 10
\vrule width 3cm height 1ex
\stoptyping

Sometimes there are multiple keywords, as with rules, in which case you can
imagine a case like:

\starttyping
\vrule width 3cm depth 1ex width 10cm depth 0ex height 1ex\relax
\stoptyping

Here we add a \type {\relax} to end the scanning. If we don't do that and the
rule specification is followed by arbitrary (read:\ unpredictable) text, the next
word might be a valid keyword and when followed by a dimension (unlikely) it will
happily be read as a directive, or when not followed by a dimension an error
message will show up. Sometimes the scanning is more restricted, as with glue
where the optional \type {plus} and \type {minus} are to come in that order, but
when missing, again a word from the text can be picked up if one doesn't
explicitly end with a \type {\relax} or some other token.

\starttyping
\scratchskip = 10pt plus 10pt minus 10pt % okay
\scratchskip = 10pt plus 10pt            % okay
\scratchskip = 10pt minus 10pt           % okay
\scratchskip = 10pt minus 10pt plus 10pt % typesets "plus 10pt"
\scratchskip = 10pt plus whatever        % an error
\stoptyping

The scanner is case insensitive, so the following specifications are all valid:

\starttyping
\hbox To 10cm {To}
\hbox TO 10cm {TO}
\hbox tO 10cm {tO}
\hbox to 10cm {to}
\stoptyping

It happens that keywords are always simple English words so the engine uses a
cheap check deep down, just offsetting to uppercase, but of course that will not
work for arbitrary \UTF-8\ (as used in \LUATEX) and it's also unrelated to the
upper- and lowercase codes as \TEX\ knows them.

The above lines scan for the keyword \type {to} and after that for a dimension.
While keyword scanning is case tolerant, dimension scanning is period tolerant:

\starttyping
\hbox to 10cm   {10cm}
\hbox to 10.0cm {10.0cm}
\hbox to .0cm   {.0cm}
\hbox to .cm    {.cm}
\hbox to 10.cm  {10.cm}
\stoptyping

These are all valid and according to the specification; even the single period is
okay, although it looks funny. It would not be hard to intercept that but I guess
that when \TEX\ was written anything that could harm performance was taken into
account. One can even argue for cases like:

\starttyping
\hbox to \first.\second cm {.cm}
\stoptyping

Here \type {\first} and|/|or \type {\second} can be empty. Most users won't
notice these side effects of scanning numbers anyway.

The reason for writing up any discussion of keywords is the following. Optional
keyword scanning is kind of costly, not so much now, but more so decades ago
(which led to some interesting optimizations, as we'll see). For instance, in the
first line below, there is no keyword. The scanner sees a \type {1} and it not
being a keyword, pushes that character back in the input.

\starttyping
\advance\scratchcounter 10
\advance\scratchcounter by 10
\stoptyping

In the case of:

\starttyping
\scratchskip 10pt plux
\stoptyping

it has to push back the four scanned tokens \type {plux}. Now, in the engine
there are lots of cases where lookahead happens and when a condition is not
satisfied, the just|-|read token is pushed back. Incidentally, when picking up
the next token triggered some expansion, it's not the original next token that
gets pushed back, but the first token seen after the expansion. Pushing back
tokens is not that inefficient, although it involves allocating a token and
pushing and popping input stacks (we're talking of a mix of reading from file,
token memory, \LUA\ prints, etc.)\ but it always takes a little time and memory.
In \LUATEX\ there are more keywords for boxes, and there we have loops too: in a
box specification one or more optional attributes are scanned before the optional
\type {to} or \type {spread}, so again there can be push back when no more \type
{attr} are seen.

\starttyping
\hbox attr 1 98 attr 2 99 to 1cm{...}
\stoptyping

In \LUAMETATEX\ there is even more optional keyword scanning, but we leave that
for now and just show one example:

\starttyping
\hbox spread 10em {\hss
    \hbox orientation 0 yoffset  1mm to 2em   {up}\hss
    \hbox                            to 2em {here}\hss
    \hbox orientation 0 xoffset -1mm to 2em {down}\hss
}
\stoptyping

Although one cannot mess too much with these low|-|level scanners there was room
for some optimization, so the penalty we pay for more keyword scanning in
\LUAMETATEX\ is not that high. (I try to compensate when adding features that
have a possible performance hit with some gain elsewhere.)

It will be no surprise that there can be interesting side effects to keyword
scanning. For instance, using the two character keyword \type {by} in an \type
{\advance} can be more efficient because nothing needs to be pushed back. The
same is true for the sometimes optional equal:

\starttyping
\scratchskip = 10pt
\stoptyping

Similar impacts on efficiency can be found in the way the end of a number is
seen, basically anything not resolving to a number (or digit). (For these, assume
a following token will terminate the number if needed; we're focusing on the
spaces here.)

\starttyping
\scratchcounter 10%          space not seen, ends \cs
\scratchcounter =10%         no push back of optional =
\scratchcounter = 10%        extra optional space gobble
\scratchcounter = 10 %       efficient ending of number scanning
\scratchcounter = 10\relax % depending on engine less efficient
\stoptyping

In the above examples scanning the number involves: skipping over spaces,
checking for an optional equal, skipping over spaces, scanning for a sign,
checking for an optional octal or hexadecimal trigger (single or double quote
character), scanning the number till a non|-|digit is seen. In the case of
dimensions there is fraction scanning as well as unit scanning too.

In any case, the equal is optional and kind of a keyword. Having an equal can be
more efficient then not having one, again due to push back in case of no equal
being seen, In the process spaces have been skipped, so add to the overhead the
scanning for optional spaces. In \LUAMETATEX\ all that has been optimized a bit.
By the way, in dimension scanning \type {pt} is actually a keyword and as there
are several dimensions possible quite some push back can happen there, but we
scan for the most likely candidates first.

All that said, we're now ready for a surprise. The keyword scanner gets a string
that it will test for, say, \type {to} in case of a box specification. It then
will fetch tokens from whatever provides the input. A token encodes a so|-|called
command and a character and can be related to a control sequence. For instance,
the character \type {t} becomes a letter command with related value \number`t.
So, we have three properties: the command code, the character code and the
control sequence code. Now, instead of checking if the command code is a letter
or other character (two checks) a fast check happens for the control sequence
code being zero. If that is the case, the character code is compared. In practice
that works out well because the characters that make up a keyword are in the
range \number"41--\number"5A\ and \number"61--\number"7A, and all other character
codes are either below that (the ones that relate to primitives where the
character code is actually a subcommand of a limited range) or much larger
numbers that, for instance, indicate an entry in some array, where the first
useful index is above the mentioned ranges.

The surprise is in the fact that there is no checking for letters or other
characters, so this is why the following code will work too: \footnote {No longer
in \LUAMETATEX\ where we do a bit more robust check.}

\starttyping
\catcode `O= 1 \hbox tO 10cm {...} % { begingroup
\catcode `O= 2 \hbox tO 10cm {...} % } endgroup
\catcode `O= 3 \hbox tO 10cm {...} % $ mathshift
\catcode `O= 4 \hbox tO 10cm {...} % & alignment
\catcode `O= 6 \hbox tO 10cm {...} % # parameter
\catcode `O= 7 \hbox tO 10cm {...} % ^ superscript
\catcode `O= 8 \hbox tO 10cm {...} % _ subscript
\catcode `O=11 \hbox tO 10cm {...} %   letter
\catcode `O=12 \hbox tO 10cm {...} %   other
\stoptyping

In the first line, if we changed the catcode of \type {T} (instead of \type {O}),
it gives an error because \TEX\ sees a begin group character (category code 1)
and starts the group, but as a second character in a keyword (\type {O}) it's
okay because \TEX\ will not look at the category code.

Of course only the cases \type {11} and \type {12} make sense in practice.
Messing with the category codes of regular letters this way will definitely give
problems with processing normal text. In a case like:

\starttyping
{\catcode `o=3 \hbox to 10cm {oeps}} % $ mathshift {oeps}
{\catcode `O=3 \hbox to 10cm {Oeps}} % $ mathshift {$eps}
\stoptyping

we have several issues: the primitive control sequence \type {\hbox} has an \type
{o} so \TEX\ will stop after \type {\hb} which can be undefined or a valid macro
and what happens next is hard to predict. Using uppercase will work but then the
content of the box is bad because there the \type {O} enters math. Now consider:

\starttyping
{\catcode `O=3 \hbox tO 10cm {Oeps Oeps}} % {$eps $eps}
\stoptyping

This will work because there are now two \type {O}'s in the box, so we have
balanced inline math triggers. But how does one explain that to a user? (Who
probably doesn't understand where an error message comes from in the first
place.) Anyway, this kind of tolerance is still not pretty, so in \LUAMETATEX\ we
now check for the command code and stick to letters and other characters. On
today's machines (and even on my by now ancient workhorse) the performance hit
can be neglected.

In fact, by intercepting the weird cases we also avoid an unnecessary case check
when we fall through the zero control sequence test. Of course that also means
that the above mentioned category code trickery doesn't work any more: only
letters and other characters are now valid in keyword scanning. Now, it can be
that some macro programmer actually used those side effects but apart from some
macro hacker being hurt because no longer mastering those details can be showed
off, it is users that we care more for, don't we?

To be sure, the abovementioned performance of keyword and equal scanning is not
that relevant in practice. But for the record, here are some timings on a laptop
with a i7-3849\cap{QM} processor using \MINGW\ binaries on a 64-bit \MSWINDOWS\
10 system. The times are the averages of five times a million such assignments
and advancements.

\starttabulate[|l|c|c|c|]
\FL
\NC one million times                    \NC terminal       \NC \LUAMETATEX\ \NC \LUATEX \NC \NR
\ML
\NC \type {\advance\scratchcounter 1}    \NC space          \NC 0.068 \NC 0.085 \NC \NR
\NC \type {\advance\scratchcounter 1}    \NC \type {\relax} \NC 0.135 \NC 0.149 \NC \NR
\NC \type {\advance\scratchcounter by 1} \NC space          \NC 0.087 \NC 0.099 \NC \NR
\NC \type {\advance\scratchcounter by 1} \NC \type {\relax} \NC 0.155 \NC 0.161 \NC \NR
\NC \type {\scratchcounter 1}            \NC space          \NC 0.057 \NC 0.096 \NC \NR
\NC \type {\scratchcounter 1}            \NC \type {\relax} \NC 0.125 \NC 0.151 \NC \NR
\NC \type {\scratchcounter=1}            \NC space          \NC 0.063 \NC 0.080 \NC \NR
\NC \type {\scratchcounter=1}            \NC \type {\relax} \NC 0.131 \NC 0.138 \NC \NR
\LL
\stoptabulate

We differentiate here between using a space as terminal or a \type {\relax}. The
latter is a bit less efficient because more code is involved in resolving the
meaning of the control sequence (which eventually boils down to nothing) but
nevertheless, these are not timings that one can lose sleep over, especially when
the rest of a decent \TEX\ run is taken into account. And yes, \LUAMETATEX\
(\LMTX) is a bit faster here than \LUATEX, but I would be disappointed if that
weren't the case.

% luametatex:

% \luaexpr{(0.068+0.070+0.069+0.067+0.068)/5} 0.068\crlf
% \luaexpr{(0.137+0.132+0.136+0.137+0.134)/5} 0.135\crlf
% \luaexpr{(0.085+0.088+0.084+0.089+0.087)/5} 0.087\crlf
% \luaexpr{(0.145+0.160+0.158+0.156+0.154)/5} 0.155\crlf
% \luaexpr{(0.060+0.055+0.059+0.055+0.056)/5} 0.057\crlf
% \luaexpr{(0.118+0.127+0.128+0.122+0.130)/5} 0.125\crlf
% \luaexpr{(0.063+0.062+0.067+0.061+0.063)/5} 0.063\crlf
% \luaexpr{(0.127+0.128+0.133+0.128+0.140)/5} 0.131\crlf

% luatex:

% \luaexpr{(0.087+0.090+0.083+0.081+0.086)/5} 0.085\crlf
% \luaexpr{(0.150+0.151+0.146+0.154+0.145)/5} 0.149\crlf
% \luaexpr{(0.100+0.092+0.113+0.094+0.098)/5} 0.099\crlf
% \luaexpr{(0.162+0.165+0.161+0.160+0.157)/5} 0.161\crlf
% \luaexpr{(0.093+0.101+0.086+0.100+0.098)/5} 0.096\crlf
% \luaexpr{(0.147+0.151+0.160+0.144+0.151)/5} 0.151\crlf
% \luaexpr{(0.076+0.085+0.088+0.073+0.078)/5} 0.080\crlf
% \luaexpr{(0.136+0.138+0.142+0.135+0.140)/5} 0.138\crlf

After the \CONTEXT\ 2020 meeting I entered another round of staring at the code.
One of the decision made at that meeting was to drop the \type {nd} and \type
{nc} units as they were never official. That made me (again) wonder of that bit
of the code could be done nicer as there is a mix of scanning units like \type
{pt}, \type {bp} and \type {cm}, fillers like \type {fi} and \type {fill}, pseudo
units like \type {ex} and \type {em}, special interception of \type {mu}, as well
as the \type {plus} and \type {minus} parsing for glue. That code was already
redone a bit so that here was less push back of tokens which had the side effect
of dimension scanning being some 50\% faster than in \LUATEX.

The same is true for scanning rule specs and scanning the box properties. In the
later case part of the optimization came from not checking properties that
already had been set, or only scanning them when for instance the \type
{orientation} flag had been set (a new option in \LUAMETATEX\ with an additional
four offset and move parameters). Also, some options, like the target dimensions,
came after scanning the new ones. Again, this was quite a bit faster than in
\LUATEX, not that it is noticeable on a normal run. All is mixed with skipping
spacers and relax tokens plus quitting at a brace.

Similar mixed scanning happens in some of the (new) math command, but these are
less critical. Actually there some new commands had to be used because for
instance \type {\over} takes any character as valid argument and keywords would
definitely be incompatible there.

Anyway, I started wondering if some could be done differently and finally decided
to use a method that I already played with years ago. The main reason for not
using it was that I wanted to remain compatible with the way traditional \TEX\
scans. However, as we have many more keyword we already are no longer compatible
in that department and the alternative implementation makes the code look nicer
and has the benefit of being (more than) twice as fast. And when I run into
issues in \CONTEXT\ I should just fix sloppy code.

The compatibility issue is not really a problem when you consider the following
cases.

\starttyping
\hbox reverse attr 123 456 orientation 4 xoffset 10pt spread 10cm { }
\hrule xoffset 10pt width 10cm depth 3mm
\hskip 3pt plus 2pt minus 1pt
\stoptyping

In the original approach these three case each have their own special side
effects. In the case of a \type {\hbox} the scanning stops at a relax or left
brace. An unknown keyword gives an error. So, there is no real benefit in pushing
back tokens here. The order matters: the \type {spread} or \type {to} comes last.

In the case of a \type {\hrule} the scanning stops when the keyword is not one of
the known. This has the side effect that when such a rule definition is hidden in
a macro and followed by for instance \type {width} without unit one gets an error
and when a unit is given the rule can come out different than expected and the
text is gone. For that reason a rule specification like this is often closed by
\type {\relax} (any command that doesn't expand to a keyword works too). Here
keywords can occur multiple times. As we have additional keyword a lookahead
becomes even more an issue (not that \type {xoffset}) is a likely candidate.

The last example is special in a different way: order matters in the sense that a
\type {minus} specifier can follow a \type {plus} but not the reverse. And only
one \type {plus} and \type {minus} can be given. Again one can best finish this
specification by a something that doesn't look like a keyword, so often one will
see a \type {\relax}.

The advantage of the new method is that the order doesn't matter any more and
that using a keyword multiple times overloads earlier settings. And this is
consistent for all commands that used keywords (with a few exceptions in math
where keywords drive later parsing and for font definitions where we need to be
compatible. We give a slightly better error message: we mention the expected
keyword. Another side effect is that any characters that is a legal start of a
known keyword will trigger further parsing and issue an error message when it
fails. Indeed, \LUAMETATEX\ has no mercy.

In practice the mentioned special effects mean that a macro package will not run
into trouble with boxes because unknown keywords make it crash and that rules and
glue is terminated in a way that prevents lookahead. The new method kind of
assumes this and one can argue that when something breaks one has to fix the
macro code. Macro writers know that one cannot predict what users come up with
and that users also don't look into the macros and therefore they take
precautions. Also, a more rigorous parsing results in hopefully a better message.

And yes, when I ran the test suite there was indeed a case where I had to add a
\type {\relax}, but I can live with that. As long as users don't notice it.

Now, one of the interesting properties of the slightly different scanning is
that we can do this:

\starttyping
\hbox to 4cm attr 123 456 reverse to 3cm {...}
\stoptyping

So, we have a less strict order and we can overload arguments too. We'll see how
this will be applied in \CONTEXT.

\stopchapter

\stopcomponent

% another nice example: \the is expanded so we get the old value

% \scratchskip = 10pt plus 1fill     \the\scratchskip  % old value
% \scratchskip = 10pt plus 1fill    [\the\scratchskip] % new value
% \scratchskip = 10pt plus 1fi l l  [\the\scratchskip] % also works
