Prototype · lexer-to-font compiler

Fontlex

The box below is a plain <textarea> with no highlighting code attached. The colors come from the font. Fontlex compiled a small Python lexer into OpenType substitution rules, so the browser's text shaper runs the lexer every time it draws a line.

example.py Font lexer on Palette

Same font in a one-line input

number string comment decorator keyword builtin ruler

Edit anything. Turning the font lexer off disables the font's calt feature, and the same text goes back to one color. Custom colors use CSS @font-palette-values, so retheming needs no new font.

What got compiled

11DFA states

9character classes

54lexer rules emitted (42 dropped)

65keywords and builtins

4,464variant glyphs added

125 KBfont as WOFF2

Checked by shaping 20,000 random snippets with HarfBuzz, the shaper Chrome and Firefox use, and comparing every glyph to a reference implementation of the lexer: 20,000 of 20,000 matched.

The lexer, as a transition table

Each row is the state of the previous character. Each column is the class of the next one. The cell is the new state, shown in the color the character will be drawn in.

after \ next	letter	digit	dot	dq	sq	backslash	hash	at	other
code (start)	ident	number	code	dq	sq	code	comment	decorator	code
ident	ident	ident	code	dq	sq	code	comment	decorator	code
number	number	number	number	dq	sq	code	comment	decorator	code
dq	dq	dq	dq	dq_end	dq	dq_esc	dq	dq	dq
dq_esc	dq	dq	dq	dq	dq	dq	dq	dq	dq
dq_end	ident	number	code	dq	sq	code	comment	decorator	code
sq	sq	sq	sq	sq	sq_end	sq_esc	sq	sq	sq
sq_esc	sq	sq	sq	sq	sq	sq	sq	sq	sq
sq_end	ident	number	code	dq	sq	code	comment	decorator	code
comment	comment	comment	comment	comment	comment	comment	comment	comment	comment
decorator	decorator	decorator	decorator	dq	sq	code	comment	decorator	code

The highlighted row is the start state. It becomes the fallback rules at the end of the lookup. Faded cells give the same answer as the fallback, so the compiler emits no rule for them.

How a font can run a lexer

Every character gets a copy of its glyph for each state it can be in, named like zero.lx_number. Copies are composites that point at the original outline, so they cost a few bytes each.

A chaining contextual lookup walks the text left to right. Its backtrack context sees glyphs it already replaced, so the previous glyph's copy carries the automaton's state forward. One rule per table cell:

lookup LEX {
  …
  sub @S_ident @C_digit' lookup TO_ident;
  sub @S_number @C_letter' lookup TO_number;
  sub @S_dq @C_letter' lookup TO_dq;
  sub @S_dq_esc @C_letter' lookup TO_dq;
  sub @S_sq @C_letter' lookup TO_sq;
  …
  # fallbacks: the start state's row
  sub @C_letter' lookup TO_ident;
  sub @C_digit' lookup TO_number;
  sub @C_dq' lookup TO_dq;
  …
} LEX;

A second lookup recolors whole identifiers that spell a keyword. The first letter of every identifier is marked, and a word's rule runs from that letter to the identifier's last, so is can't match inside island; a last pass spreads the color back over the word. Longer words go first, so isinstance isn't read as is:

lookup START {
  ignore sub @WORDLIKE @S_ident';
  sub @S_ident' lookup TO_START;
} START;
lookup WORDS {
  ignore sub […]' @WORDLIKE;
  …
  sub \i.lxs_ident \s.lx_ident \i.lx_ident \n.lx_ident \s.lx_ident \t.lx_ident \a.lx_ident \n.lx_ident \c.lx_ident \e.lx_ident' lookup W_builtin;
  …
  sub \i.lxs_ident \s.lx_ident' lookup W_keyword;
} WORDS;

Each colored copy gets a COLR record that paints the original outline with a palette entry. The font carries a light and a dark palette.

Limits of this prototype

State lives only within one shaping run. Browsers start a new run at each line and wherever a fallback font takes over, so multi-line strings and comments lose their state. Chrome also starts one at every tab, so a tab inside a string or comment resets the lexer there. With some base fonts, Safari also starts one at a change of script; the compiler warns about those and can merge their scripts.
Coloring is decided as each character is read, with one glyph of lookahead for words and for states that ask for it. Anything that needs more context needs another pass, and a finite automaton can't track nesting.
Tested with HarfBuzz (Chrome, Firefox, Android) and Core Text (Safari, native Mac apps), shaping every case through both and comparing each glyph with a reference lexer.
The base font's own ligatures and alternates are removed, because they would replace glyphs the lexer needs to see. Its kerning and mark positioning are kept.