ATN construction by Lotes · Pull Request #65 · TypeFox/fastbelt

Lotes · 2026-04-29T13:19:52Z

There is a big Compile Time ATN for construction and a slim Runtime ATN.

I also added a Markdown generator to be able to debug the ATN visually.

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Co-authored-by: Copilot <copilot@github.com> Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

msujew · 2026-04-29T15:52:24Z

Suggestion: Since we need all of this code in other parts of the parser (not just the lookahead), it might make sense to move it to the parser parent directory.

FTM a lot of stuff is inside the generator package. Only the Runtime ATN is in the parser package.

msujew · 2026-04-30T11:42:06Z

+	"typefox.dev/fastbelt/generator"
+)
+
+func EmitMarkdownSource(pkgName string, rtn *RuntimeATN, tokenTypeNames map[int]string) generator.Node {


Suggestion: This part of the code should live in the internal part of the framework - adopters shouldn't be able to call this function IMO.

msujew · 2026-04-30T11:42:29Z

+//	pkgName  – Go package name for the generated file (e.g. "myparser")
+//	funcName – name of the generated constructor function
+//	importPath – import path of the allstar package (e.g. "typefox.dev/fastbelt/parser/allstar")
+func EmitGoSource(pkgName, funcName, importPath string, rtn *RuntimeATN) generator.Node {


Suggestion: This part of the code should live in the internal part of the framework - adopters shouldn't be able to call this function IMO.

Co-authored-by: Copilot <copilot@github.com> Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes · 2026-05-05T16:13:57Z

+	Action_Alternatives_1                ATNDecisionId = 0
+	Action_Group_2                       ATNDecisionId = 1
+	Alternatives_Group_2                 ATNDecisionId = 2
+	Alternatives_Group_3                 ATNDecisionId = 3


The counter inside the name looks broken...

The counter refers to the index of all parts in a rule. Only a few are used for the decision map.

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes · 2026-05-06T13:43:14Z

+
+func (rb *ATNRuleBuilderImpl) MakeBlock(alts []*ATNHandle) *ATNHandle {
+	for i := 0; i < len(alts)-1; i++ {
+		handle := alts[i]


@msujew Not very sure about the original implementation...
May need to redo it...

IIRC the code in the original implementation is supposed to simplify stuff like this:

In this case, all states 254-261 are not required for properly evaluating the ATN. We can go straight from 262 to 263 using the rule transitions.

MakeBlock is about concatenation, not alternatives.
Is this so much faster? I would prefer simple code.

Modelling middleware for the ATN could be a thing, that could be designed to optimize the ATN.

Right, and the idea is to skip 254-255, 256-257, etc. since they don't serve any value in the ATN. In the end, this makes the subset construction algorithm - and all other algorithm that need to traverse the ATN - slower. I would assume by quite a significant margin actually, with more gains the deeper the grammar becomes.

IMO, we should do everything, even if complicated, to improve the performance of the runtime-code in favor of more complexity in the compile-time code.
And to be honest, the simplification code isn't that difficult, though feel free to add further comments here.

Modelling middleware for the ATN could be a thing, that could be designed to optimize the ATN.

Sounds more complex than just adding the optimization in there actually ;)

True. The middleware thingy sounds more complicated, when looking only at this single optimization. When we would add more of these optimizations the code becomes more unreadable. The middleware or middleend was an idea to introduce a optimization module.
But I remember that there is also a minimization strategy for DFAs. Maybe we can do something similar here?
Middleware or a big optimization step afterwards, so or so, I like this code here to be in a simple shape. But I could reintroduce the code with a explanation and a TODO, that we can refactor in a future PR.

I'd be a bit hesitant to change the structure of the ATN too much. After all, we need it later to perform code completion and error recovery.

Then let‘s postpone the refactoring until we have the necessary features with tests.

Lotes · 2026-05-07T14:37:23Z

+		Start: Property+=Rule*;
+		Rule: Name=ID;
+	`))
+	RequireATNRecognizes(t, atn, rules, tokenTypes, "Start", []string{}, 1)


Add tests with 2 or more paths…

msujew

Looking really promising already, just a few comments, see below 👍

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes and others added 4 commits April 29, 2026 14:16

Move ATN to this branch

e1fa3a3

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Stableize ATN emission

eef71d8

Co-authored-by: Copilot <copilot@github.com> Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add markdown generator

be9ab6b

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Remove formatting

2413b92

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes commented Apr 29, 2026

View reviewed changes

Comment thread internal/grammar/atn.md

Remove newline

f6fb319

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

msujew reviewed Apr 30, 2026

View reviewed changes

Lotes and others added 12 commits April 30, 2026 17:22

Remove grammar abstraction

7638e0b

Co-authored-by: Copilot <copilot@github.com> Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Refactor ruleBuilder.Assign(...)

efee46a

Co-authored-by: Copilot <copilot@github.com> Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Reuse public methods

8c7a9cc

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Intermediate save

0df0b3c

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Intermediate save

10dbef8

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix parts of the ATN

16bc5c6

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Remove ATN to Runtime ATN conversion

eeecbe8

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix linting

65bc303

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix comments

85e106a

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Updated license headers

e50661b

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add extra config for debugging the Fastbelt ATN

501dc94

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Remove categories

6a1c19c

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes commented May 5, 2026

View reviewed changes

Comment thread parser/atn_runtime.go Outdated

Comment thread internal/grammar/atn_gen.go

Comment thread internal/generator/atn_types.go Outdated

Lotes added 3 commits May 5, 2026 17:54

Remove random ATN generation

59e8cd9

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Replace keys with enums

ba0993e

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix linting

ab8da4b

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes commented May 5, 2026

View reviewed changes

Rename symbol

b2d1014

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes commented May 6, 2026

View reviewed changes

Lotes commented May 7, 2026

View reviewed changes

msujew reviewed May 7, 2026

View reviewed changes

Comment thread internal/generator/atn.go Outdated

Comment thread internal/generator/atn.go Outdated

Comment thread internal/grammar/atn_gen.go Outdated

Comment thread parser/atn_runtime.go Outdated

Comment thread internal/generator/atn_fixture.go

msujew reviewed May 7, 2026

View reviewed changes

Comment thread internal/grammar/atn_gen.go

Resolve first comments

66d3c25

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes added 3 commits May 8, 2026 10:46

Refactor token type generation

2d38515

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Replace token ID with entire token type

5d22e1a

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Refactor

e3ac1ee

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Uh oh!

Conversation

Lotes commented Apr 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lotes May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msujew May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msujew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Lotes May 5, 2026 •

edited

Loading

msujew May 8, 2026 •

edited

Loading