feat: 添加更多嵌入提供商 OpenAI-兼容 embedding provider 和智谱字节ollama提供商#6642
feat: 添加更多嵌入提供商 OpenAI-兼容 embedding provider 和智谱字节ollama提供商#6642whatevertogo wants to merge 8 commits intoAstrBotDevs:masterfrom
Conversation
dosubot
bot
added
the
size:XL
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! 此拉取请求通过引入一个通用的 OpenAI 兼容 Embedding 提供商，显著扩展了 AstrBot 对嵌入服务的支持范围。它解决了现有实现中对非标准 API Base URL 处理的限制，并提供了灵活的配置选项，如 Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
dosubot
bot
added
the
area:provider
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- In
normalize_openai_compatible_embedding_api_base, a host without scheme (e.g.api.openai.com) is treated as a path and returned unchanged, which produces an invalid base URL; consider either auto-prefixinghttps://or explicitly validating and rejecting scheme-less values to avoid confusing runtime failures. - The new
send_dimensions_paramflag is parsed quite liberally viashould_send_dimensions_param(accepting various stringy truthy values); if config is only ever stored as a native boolean, you could simplify this to a direct bool read and avoid surprising behavior when unconventional strings are passed in.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `normalize_openai_compatible_embedding_api_base`, a host without scheme (e.g. `api.openai.com`) is treated as a path and returned unchanged, which produces an invalid base URL; consider either auto-prefixing `https://` or explicitly validating and rejecting scheme-less values to avoid confusing runtime failures.
- The new `send_dimensions_param` flag is parsed quite liberally via `should_send_dimensions_param` (accepting various stringy truthy values); if config is only ever stored as a native boolean, you could simplify this to a direct bool read and avoid surprising behavior when unconventional strings are passed in.
## Individual Comments
### Comment 1
<location path="tests/test_openai_compatible_embedding_source.py" line_range="72-81" />
<code_context>
assert data["status"] == "ok" and "platform" in data["data"]
+@pytest.mark.asyncio
+async def test_provider_template_exposes_openai_compatible_embedding_presets(
+ app: Quart,
</code_context>
<issue_to_address>
**suggestion (testing):** Add assertions that `terminate()` properly closes the underlying AsyncOpenAI client
Current tests call `terminate()` in `finally` blocks but never verify that the underlying client is closed. Since `_FakeAsyncOpenAI` tracks `closed`, please add assertions (e.g., `assert provider_without_dimensions.client.closed is True` or via `instances`) after `terminate()` to validate resource cleanup and catch regressions in connection handling.
</issue_to_address>
### Comment 2
<location path="docs/en/use/knowledge-base.md" line_range="25" />
<code_context>
+> `OpenAI Compatible Embedding` includes a `send_dimensions_param` switch. When enabled, AstrBot sends `embedding_dimensions` to the upstream embedding API as the `dimensions` parameter. Disable it for OpenAI-compatible services that only need the local vector size and do not support `dimensions`.
+
+> [!NOTE]
+> The Volcengine preset defaults to `doubao-embedding-vision`. AstrBot's knowledge-base pipeline is still text chunking plus text embedding only, so this integration uses the model with text input only and does not add multimodal knowledge-base support yet,although it is a multimodal embedding model.
+
+> [!NOTE]
</code_context>
<issue_to_address>
**issue (typo):** Add a space after the comma in "yet,although" to fix the typo.
Please insert a space after the comma so it reads "yet, although it is a multimodal embedding model."
```suggestion
> The Volcengine preset defaults to `doubao-embedding-vision`. AstrBot's knowledge-base pipeline is still text chunking plus text embedding only, so this integration uses the model with text input only and does not add multimodal knowledge-base support yet, although it is a multimodal embedding model.
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
本次 PR 新增了通用的 OpenAICompatibleEmbeddingProvider，并内置了智谱和火山引擎的预设，这是一个非常实用的功能增强，极大地扩展了嵌入服务的可选择范围。代码实现考虑周全，特别是对 API Base URL 的智能处理和 send_dimensions_param 开关的设计，显示了对不同服务兼容性的深入思考。同时，相关的单元测试、文档和国际化配置都一并更新，做得非常完整。
我在代码中发现了一些可以改进的地方，主要集中在新添加的 OpenAICompatibleEmbeddingProvider 中，涉及资源管理和配置解析的健壮性。具体的建议已在代码注释中提出，希望能帮助您进一步完善代码。
There was a problem hiding this comment.
Pull request overview
This PR adds a generic OpenAI-compatible embedding provider adapter (with smarter API base handling) and ships Zhipu and Volcengine embedding presets, plus corresponding Dashboard/i18n/docs and test coverage.
Changes:
- Introduce
openai_compatible_embeddingprovider with API base normalization and an opt-insend_dimensions_paramtoggle. - Add built-in provider templates/presets (OpenAI-compatible, Zhipu, Volcengine, Ollama) and expose them through the Dashboard.
- Add unit/integration tests and update docs + i18n strings.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
astrbot/core/provider/sources/openai_compatible_embedding_source.py |
New OpenAI-compatible embedding adapter (API base normalization + optional dimensions forwarding). |
astrbot/core/provider/manager.py |
Registers lazy import for the new provider type. |
astrbot/core/config/default.py |
Adds provider templates/presets and schema field for send_dimensions_param. |
dashboard/src/components/shared/AstrBotConfig.vue |
Simplifies how provider/item hints are surfaced in the config UI. |
dashboard/src/i18n/locales/zh-CN/features/config-metadata.json |
Adds hint/label strings for new embedding presets and send_dimensions_param. |
dashboard/src/i18n/locales/en-US/features/config-metadata.json |
Same i18n updates (English). |
dashboard/src/i18n/locales/ru-RU/features/config-metadata.json |
Same i18n updates (Russian). |
docs/zh/use/knowledge-base.md |
Documents new embedding options and the send_dimensions_param behavior. |
docs/en/use/knowledge-base.md |
English documentation update for the new embedding options. |
tests/test_openai_compatible_embedding_source.py |
Unit tests for API base normalization + dimensions forwarding behavior. |
tests/test_dashboard.py |
Dashboard API tests for templates exposure and embedding dim endpoint support. |
whatevertogo
changed the title
whatevertogo
changed the title
- Fix typo in docs: "yet,although" → "yet, although" - Fix resource leak: store httpx.AsyncClient as self._http_client and close in terminate() - Add timeout parsing with exception handling for invalid values - Handle API base URL without scheme (e.g., "api.openai.com") - Add test assertions to verify terminate() properly closes client
- Clarify that openai_embedding is kept for backward compatibility - Explain dimensions forwarding behavior difference between the two providers - Update hints in en-US, zh-CN, and ru-RU locales
- Explain that dimensions enables dimension reduction (not just forwarding) - List services that support it: OpenAI, Zhipu, Volcengine - Update all three locales: en-US, zh-CN, ru-RU
Replace hardcoded strings with i18n key paths to enable proper translation in the frontend for send_dimensions_param field.
whatevertogo
changed the title
Fixes #6638
Fix #3790
Fix #4226
一、PR 概述
新增通用
OpenAI Compatible Embedding提供商适配器，并内置智谱（Zhipu）和火山引擎（Volcengine）两个预设配置，让用户可以方便地接入更多嵌入向量服务。二、背景 / 动机
openai_embedding的局限：其 API base 处理逻辑会强制在所有地址后追加/v1，导致智谱/api/paas/v4、火山/api/v3等带路径的地址被破坏。三、改动内容
OpenAICompatibleEmbeddingProvider，智能处理 API base URL（纯域名自动补/v1，已有路径则保留原样）send_dimensions_param开关，控制是否将embedding_dimensions透传给上游接口（部分兼容服务不支持dimensions参数）embedding-3）和火山引擎 Embedding（doubao-embedding-vision）预设配置OpenAI Embedding替换为OpenAI Compatible Embedding作为通用选项改动类型：
四、实现说明
normalize_openai_compatible_embedding_api_base）区分纯域名和带路径的地址，再用send_dimensions_param开关解决不同服务对dimensions参数的兼容性差异。openai_embedding适配器，保持向后兼容；新增独立的openai_compatible_embedding类型send_dimensions_param默认关闭（opt-in），避免对不期望接收该参数的上游服务造成影响openai_compatible_embedding类型 + 预设配置实现，无需额外代码astrbot/core/provider/sources/openai_compatible_embedding_source.py— 新增适配器实现astrbot/core/config/default.py— 新增预设配置和 schemaastrbot/core/provider/manager.py— 注册 lazy importdashboard/src/i18n/locales/*/features/config-metadata.json— 三语 i18ndocs/zh/use/knowledge-base.md、docs/en/use/knowledge-base.md— 文档更新tests/test_openai_compatible_embedding_source.py— 单元测试tests/test_dashboard.py— Dashboard 集成测试五、行为变化（对外影响）
OpenAI Embedding配置项 ID 变更为openai_compatible_embedding，已有用户需要重新配置嵌入提供商（原openai_embedding类型仍可使用，不会被删除）六、测试与验证
自测结果：
test_openai_compatible_embedding_source.py+test_dashboard.py）测试命令：
七、Checklist
Summary by Sourcery
Introduce a generic OpenAI-compatible embedding provider and wire it into the provider manager, dashboard, and docs, with presets for major third-party services.
New Features:
Enhancements:
Documentation:
Tests: