Skip to content

Improve lexer performance using struct-array#64

Merged
msujew merged 1 commit into
mainfrom
msujew/use-token-struct-array
Apr 27, 2026
Merged

Improve lexer performance using struct-array#64
msujew merged 1 commit into
mainfrom
msujew/use-token-struct-array

Conversation

@msujew
Copy link
Copy Markdown
Member

@msujew msujew commented Apr 25, 2026

Performs some minor and a major performance optimizations:

  1. (major) The TokenSlice uses the Token struct directly instead of using a pointer. This improves lexing performance by ~40%
  2. (minor) Adds a running tokens-per-byte average to the lexer. This ensures (1) that we rarely try to increase the slice capacity and (2) keep the capacity as low as possible. Improves performance by ~3%.
  3. (minor) changes the lookup from slices.IndexFunc to a custom loop. Does not influence performance - it seems like the Go compiler was able to inline the function call. However, this still makes everything easier to read.
  4. (minor) Do not put the utf8 conversion behind a conditional. Improves performance by 3%.

Copy link
Copy Markdown
Member

@spoenemann spoenemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

Do we need to update https://www.typefox.io/blog/fastbelt-introduction/ again 😁?

@msujew msujew force-pushed the msujew/use-token-struct-array branch from 9ef74b2 to f3ae4b2 Compare April 27, 2026 13:31
@msujew
Copy link
Copy Markdown
Member Author

msujew commented Apr 27, 2026

Do we need to update https://www.typefox.io/blog/fastbelt-introduction/ again 😁?

We could, though it's only a roughly ~4% performance increase in total.

@msujew msujew force-pushed the msujew/use-token-struct-array branch from f3ae4b2 to a70ca5f Compare April 27, 2026 13:44
@msujew msujew merged commit 890a677 into main Apr 27, 2026
4 checks passed
@msujew msujew deleted the msujew/use-token-struct-array branch April 27, 2026 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants