[codex] sanitize hidden reasoning from LLM outputs

[aimdotsh]

Summary

• Strip hidden reasoning blocks such as <think>...</think> from rewritten questions and final chat answers.
• Preserve streaming answer delivery while filtering hidden reasoning chunks.
• Parse recommended questions into clean question lines and retry when malformed output yields too few questions.
• Add unit tests for reasoning-tag stripping, split streaming tags, and recommended-question parsing.

Root cause

Some LLM providers can emit hidden reasoning or formatter text despite prompt instructions. AutoFlow previously trusted these responses and stored or displayed them directly, so malformed model output could leak into chat answers, retrieval queries, and recommended questions.

Validation

• python3 -m py_compile backend/app/rag/utils.py backend/app/rag/chat/chat_flow.py backend/app/rag/chat/chat_service.py backend/tests/test_rag_utils.py
• uv run pytest -q tests/test_rag_utils.py
• uv run ruff check app/rag/utils.py app/rag/chat/chat_flow.py app/rag/chat/chat_service.py tests/test_rag_utils.py

[vercel]

@aimdotsh is attempting to deploy a commit to the pingcap Team on Vercel.

A member of the Team first needs to authorize it.