Prototype · lexer-to-font compiler

Fontlex

The box below is a plain <textarea> with no highlighting code attached. The colors come from the font. Fontlex compiled a small JavaScript lexer into OpenType substitution rules, so the browser's text shaper runs the lexer every time it draws a line.

example.js Font lexer on Palette

Same font in a one-line input

number regex comment string keyword constant builtin

Edit anything. Turning the font lexer off disables the font's calt feature, and the same text goes back to one color. Custom colors use CSS @font-palette-values, so retheming needs no new font.

What got compiled

28DFA states

14character classes

254lexer rules emitted (132 dropped)

67keywords and builtins

11,351variant glyphs added

144 KBfont as WOFF2

Checked by shaping 20,000 random snippets with HarfBuzz, the shaper Chrome and Firefox use, and comparing every glyph to a reference implementation of the lexer: 20,000 of 20,000 matched.

The lexer, as a transition table

Each row is the state of the previous character. Each column is the class of the next one. The cell is the new state, shown in the color the character will be drawn in.

after \ next	letter	digit	dot	dq	sq	bt	backslash	slash	star	space	close	lbracket	rbracket	other
code (start)	ident	number	code	dq	sq	tpl	code	re_start	code	code	after	code	after	code
after	ident	number	code	dq	sq	tpl	code	div	code	after	after	code	after	code
ident	ident	ident	code	dq	sq	tpl	code	div	code	after	after	code	after	code
number	number	number	number	dq	sq	tpl	code	div	code	after	after	code	after	code
div	ident	number	code	dq	sq	tpl	code	line_comment	vblock	code	after	code	after	code
re_start	regex	regex	regex	regex	regex	regex	re_esc	line_comment	block	regex	regex	re_class	regex	regex
regex	regex	regex	regex	regex	regex	regex	re_esc	re_end	regex	regex	regex	re_class	regex	regex
re_esc	regex	regex	regex	regex	regex	regex	regex	regex	regex	regex	regex	regex	regex	regex
re_class	re_class	re_class	re_class	re_class	re_class	re_class	re_class_esc	re_class	re_class	re_class	re_class	re_class	regex	re_class
re_class_esc	re_class	re_class	re_class	re_class	re_class	re_class	re_class	re_class	re_class	re_class	re_class	re_class	re_class	re_class
re_end	re_flags	number	code	dq	sq	tpl	code	div	code	after	after	code	after	code
re_flags	re_flags	number	code	dq	sq	tpl	code	div	code	after	after	code	after	code
line_comment	line_comment	line_comment	line_comment	line_comment	line_comment	line_comment	line_comment	line_comment	line_comment	line_comment	line_comment	line_comment	line_comment	line_comment
block	block	block	block	block	block	block	block	block	block_star	block	block	block	block	block
block_star	block	block	block	block	block	block	block	block_end	block_star	block	block	block	block	block
block_end	ident	number	code	dq	sq	tpl	code	re_start	code	code	after	code	after	code
vblock	vblock	vblock	vblock	vblock	vblock	vblock	vblock	vblock	vblock_star	vblock	vblock	vblock	vblock	vblock
vblock_star	vblock	vblock	vblock	vblock	vblock	vblock	vblock	vblock_end	vblock_star	vblock	vblock	vblock	vblock	vblock
vblock_end	ident	number	code	dq	sq	tpl	code	div	code	after	after	code	after	code
dq	dq	dq	dq	dq_end	dq	dq	dq_esc	dq	dq	dq	dq	dq	dq	dq
dq_esc	dq	dq	dq	dq	dq	dq	dq	dq	dq	dq	dq	dq	dq	dq
dq_end	ident	number	code	dq	sq	tpl	code	div	code	after	after	code	after	code
sq	sq	sq	sq	sq	sq_end	sq	sq_esc	sq	sq	sq	sq	sq	sq	sq
sq_esc	sq	sq	sq	sq	sq	sq	sq	sq	sq	sq	sq	sq	sq	sq
sq_end	ident	number	code	dq	sq	tpl	code	div	code	after	after	code	after	code
tpl	tpl	tpl	tpl	tpl	tpl	tpl_end	tpl_esc	tpl	tpl	tpl	tpl	tpl	tpl	tpl
tpl_esc	tpl	tpl	tpl	tpl	tpl	tpl	tpl	tpl	tpl	tpl	tpl	tpl	tpl	tpl
tpl_end	ident	number	code	dq	sq	tpl	code	div	code	after	after	code	after	code

The highlighted row is the start state. It becomes the fallback rules at the end of the lookup. Faded cells give the same answer as the fallback, so the compiler emits no rule for them.

How a font can run a lexer

Every character gets a copy of its glyph for each state it can be in, named like zero.lx_number. Copies are composites that point at the original outline, so they cost a few bytes each.

A chaining contextual lookup walks the text left to right. Its backtrack context sees glyphs it already replaced, so the previous glyph's copy carries the automaton's state forward. One rule per table cell:

lookup LEX {
  …
  sub @S_after @C_slash' lookup TO_div;
  sub @S_ident @C_digit' lookup TO_ident;
  sub @S_number @C_letter' lookup TO_number;
  sub @S_div @C_slash' lookup TO_line_comment;
  sub @S_re_start @C_letter' lookup TO_regex;
  …
  # fallbacks: the start state's row
  sub @C_letter' lookup TO_ident;
  sub @C_digit' lookup TO_number;
  sub @C_dq' lookup TO_dq;
  …
} LEX;

A second lookup recolors whole identifiers that spell a keyword. The first letter of every identifier is marked, and a word's rule runs from that letter to the identifier's last, so in can't match inside inland; a last pass spreads the color back over the word. Longer words go first, so instanceof isn't read as in:

lookup START {
  ignore sub @WORDLIKE @S_ident';
  sub @S_ident' lookup TO_START;
} START;
lookup WORDS {
  ignore sub […]' @WORDLIKE;
  …
  sub \i.lxs_ident \n.lx_ident \s.lx_ident \t.lx_ident \a.lx_ident \n.lx_ident \c.lx_ident \e.lx_ident \o.lx_ident \f.lx_ident' lookup W_keyword;
  …
  sub \i.lxs_ident \n.lx_ident' lookup W_keyword;
} WORDS;

Each colored copy gets a COLR record that paints the original outline with a palette entry. The font carries a light and a dark palette.

Limits of this prototype

State lives only within one shaping run. Browsers start a new run at each line and wherever a fallback font takes over, so multi-line strings and comments lose their state. Chrome also starts one at every tab, so a tab inside a string or comment resets the lexer there. With some base fonts, Safari also starts one at a change of script; the compiler warns about those and can merge their scripts.
Coloring is decided as each character is read, with one glyph of lookahead for words and for states that ask for it. Anything that needs more context needs another pass, and a finite automaton can't track nesting.
Tested with HarfBuzz (Chrome, Firefox, Android) and Core Text (Safari, native Mac apps), shaping every case through both and comparing each glyph with a reference lexer.
The base font's own ligatures and alternates are removed, because they would replace glyphs the lexer needs to see. Its kerning and mark positioning are kept.