Add Tool Calling and Reasoning Token Metadata to genai_config.json with Fallback Map

[sayanshaw24]

Add tool-calling and reasoning token IDs to GenAI Tokenizer

Summary

Adds four new token ID fields to the model section of genai_config.json — bot_token_id, eot_token_id, bor_token_id, eor_token_id — and exposes them on the Tokenizer (alongside bos, eos, pad), with a backward-compatible fallback for older model packages.

This establishes a new naming convention for generation-level special tokens:

Abbreviation	Expansion	Purpose
bot	beginning of tool (call)	marks start of tool-call content
eot	end of tool (call)	marks end of tool-call content
bor	beginning of reasoning	marks start of reasoning/thinking content
eor	end of reasoning	marks end of reasoning/thinking content

The naming mirrors bos/eos/pad — short, unambiguous, and follows the same access pattern (GetBotTokenId() alongside GetBosTokenId()).

Changes

Config parsing (config.h / config.cpp)

• Added to Config::Model: bot_token_id, eot_token_id, bor_token_id, eor_token_id (int, default -1)
• Parsed in Model_Element::OnValue() alongside existing token ID fields (bos, eos, pad)

Tokenizer (models/model.h / models/model.cpp)

• New private members: int32_t bot_token_id_, eot_token_id_, bor_token_id_, eor_token_id_
• New public getters: GetBotTokenId(), GetEotTokenId(), GetBorTokenId(), GetEorTokenId()
• Initialized in the Tokenizer constructor from config (identical pattern to bos/eos/pad)
• Fallback logic: if any ID is -1 after reading config, attempts to resolve it by encoding known model-family-specific token strings through the vocabulary. This provides backward compatibility for Foundry Local consuming older model packages that predate these config fields.
  • Fallback map keyed by model.type:
    • qwen2/qwen3/phi3 → <tool_call> / </tool_call>
    • gptoss → <|start|> / <|call|>
    • qwen3 → <think> / </think> (reasoning)

Public C API (ort_genai_c.h / ort_genai_c.cpp)

• OgaTokenizerGetBotTokenId(tokenizer, &token_id) — returns BOT token id or -1
• OgaTokenizerGetEotTokenId(tokenizer, &token_id) — returns EOT token id or -1
• OgaTokenizerGetBorTokenId(tokenizer, &token_id) — returns BOR token id or -1
• OgaTokenizerGetEorTokenId(tokenizer, &token_id) — returns EOR token id or -1

Same pattern as OgaTokenizerGetBosTokenId / OgaTokenizerGetPadTokenId.

C++ wrapper (ort_genai.h)

• int32_t OgaTokenizer::GetBotTokenId()
• int32_t OgaTokenizer::GetEotTokenId()
• int32_t OgaTokenizer::GetBorTokenId()
• int32_t OgaTokenizer::GetEorTokenId()

Tests (test/c_api_tests.cpp)

• TagId_Unknown — verifies gpt2 model (not in fallback map, no config IDs) returns -1 for all four
• TagId_FromConfig — creates a temp model dir with token IDs in the model section, verifies correct parsing via the tokenizer

genai_config.json format

{
  "model": {
    "type": "qwen3",
    "bos_token_id": 151643,
    "eos_token_id": 151645,
    "bot_token_id": 151657,
    "eot_token_id": 151658,
    "bor_token_id": 151659,
    "eor_token_id": 151660,
    "decoder": { ... }
  }
}

All four fields are optional. If absent, the fallback map attempts to encode the known string via the tokenizer vocabulary. If the model type isn't in the fallback map either, -1 is returned (model doesn't support tool calling / reasoning).

Motivation

The Foundry Local C++ SDK currently detects tool-call and reasoning tokens by double-decoding every token through both a normal and a special-token tokenizer stream, then doing string.find("tool_call") / string.find("think") per token. This is expensive.

With token IDs exposed by GenAI on the Tokenizer, FL can do a single integer comparison per token in the decode loop — eliminating the special stream entirely for models with configured IDs.

Also required for the Catalog v2 migration where tool/reasoning metadata is being removed from catalog.

Design Decisions

Why on the Tokenizer (not Model)?

• BOS, EOS, and PAD token IDs are already accessed from the Tokenizer
• Token IDs are a tokenizer-level concept — they describe the vocabulary
• Follows existing pattern: Tokenizer::GetBosTokenId() → Tokenizer::GetBotTokenId()
• No lazy init needed; resolved once in the constructor

Why bot/eot/bor/eor naming?

• Mirrors bos/eos/pad — short, memorable, consistent
• Avoids ambiguity with eot (end of text) since context is clearly "tool call" from the API name
• Establishes a standard that can be extended (future: boh/eoh for "harmony" channels, etc.)
• Config field names also follow this pattern: bot_token_id alongside bos_token_id

Fallback map

• Exists specifically for Foundry Local backward compatibility with older model packages
• Only triggered when config fields are not set (-1)
• Resolved eagerly in the Tokenizer constructor (no lazy cache needed since tokenizer is already available)

Related PRs

• Foundry-Local: Optimize tool/reasoning detection with token ID comparison in decode loop

[Copilot]

Pull request overview

This PR adds support for exposing model “generation tags” (tool-calling and reasoning start/end tokens) via genai_config.json, with a model-type-based fallback for older model artifacts. It extends the internal config/model layers and surfaces a generic tag getter through the public C API and C++ wrapper, plus adds C API tests to validate both fallback and explicit config parsing.

Changes:

• Extend config schema/parsing with optional tool_calling and reasoning sections (Config::ToolCalling, Config::Reasoning).
• Add Model::GetTag(tag_name) with a static fallback map keyed by model.type.
• Expose tag lookup via OgaModelGetTag (C API) and OgaModel::GetTag (C++ wrapper), and add tests covering fallback and config-driven values.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/c_api_tests.cpp	Adds tests for tag fallback behavior and tag parsing from a temp genai_config.json.
src/ort_genai.h	Adds C++ wrapper OgaModel::GetTag(...) calling the new C API.
src/ort_genai_c.h	Adds public C API declaration/docs for OgaModelGetTag(...).
src/ort_genai_c.cpp	Implements OgaModelGetTag(...) by allocating a returned string from Model::GetTag.
src/models/model.h	Declares internal Model::GetTag(...) accessor (config + fallback).
src/models/model.cpp	Implements fallback map + config accessors for tag lookup.
src/config.h	Adds Config::ToolCalling and Config::Reasoning structs to the config model.
src/config.cpp	Wires new SAX elements (ToolCalling_Element, Reasoning_Element) into root parsing.

[kunal-vaishnavi]

Why do we need an anonymous namespace? Can we remove it?

[kunal-vaishnavi]

Can all this be merged inside GetFallbackTag so that try_fallback doesn't need to exist as a method?

[kunal-vaishnavi]

Can we add the APIs to the following language bindings as well?

• C#
• Java
• Objective-C
• Python