Skip to content

fix(windows): read search_code temp files as UTF-8#956

Open
Cosm1cAC wants to merge 2 commits into
DeusData:mainfrom
Cosm1cAC:fix/windows-search-code-utf8-paths
Open

fix(windows): read search_code temp files as UTF-8#956
Cosm1cAC wants to merge 2 commits into
DeusData:mainfrom
Cosm1cAC:fix/windows-search-code-utf8-paths

Conversation

@Cosm1cAC

@Cosm1cAC Cosm1cAC commented Jul 8, 2026

Copy link
Copy Markdown

Summary

search_code can miss matches on Windows when the indexed project is under a non-ASCII path, such as a CJK directory name.

Scoped search writes two temporary files: one for the pattern and one for the indexed file list. Both files are UTF-8. The Windows command builder was reading them back with plain Get-Content before handing each path to Select-String -LiteralPath.

That is fragile on Windows PowerShell 5.1. UTF-8 without a BOM is decoded with the local ANSI code page unless the encoding is explicit. A path like C:/Users/.../移动版agent终端/src/file.js is read back as mojibake, so Select-String receives a literal path that does not exist and search_code returns a clean zero-match result.

This patch makes the Windows search_code command builder read the pattern file and scoped file list with Get-Content -Encoding UTF8 -LiteralPath.

Related to #903.

Reproduction

I reproduced this on Windows PowerShell 5.1 with a project rooted under:

C:/Users/<user>/Documents/移动版agent终端

and a source file containing agentReachCommandForAction.

Before the fix:

  • Select-String -LiteralPath <actual file> -Pattern agentReachCommandForAction finds the expected lines.
  • The scoped search_code pipeline reads the UTF-8 file list through plain Get-Content.
  • The CJK path is decoded as mojibake, so Test-Path -LiteralPath <decoded path> is false.
  • search_code returns zero grep matches.

After the fix:

  • The same file list is read with Get-Content -Encoding UTF8 -LiteralPath.
  • The decoded path exists.
  • Select-String finds the expected matches.

Verification

Local command-level check on Windows PowerShell 5.1:

  • current pipeline: current_matches=0
  • patched pipeline: patched_matches=2
  • default-decoded filelist path exists: False
  • UTF-8-decoded filelist path exists: True

Also ran:

  • git diff --check

I added a Windows-only regression test that creates a scoped search_code fixture under a UTF-8 CJK project root.

I could not run the full native Windows C test suite locally because this machine does not have the MSYS2/CLANG64 GNU Make toolchain used by the Windows CI job. Visual Studio cl is installed, but that is not this project's Windows CI build path.

Checklist

  • Every commit is signed off (git commit -s)
  • Tests pass locally (make -f Makefile.cbm test)
  • Lint passes (make -f Makefile.cbm lint-ci)
  • New behavior is covered by a test (reproduce-first for bug fixes)

@Cosm1cAC Cosm1cAC requested a review from DeusData as a code owner July 8, 2026 11:41
@Cosm1cAC Cosm1cAC force-pushed the fix/windows-search-code-utf8-paths branch from 1e7f518 to 8695f3c Compare July 8, 2026 11:58
@DeusData DeusData added bug Something isn't working windows Windows-specific issues ux/behavior Display bugs, docs, adoption UX priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. labels Jul 8, 2026
@DeusData

DeusData commented Jul 8, 2026

Copy link
Copy Markdown
Owner

Thanks for covering the Windows UTF-8 temp-file path. Triage: Windows query/read bug, high priority because it affects non-ASCII projects after indexing. Review focus is PowerShell 5.1 encoding behavior, CJK path coverage, and no regression to POSIX temp-file reads.

@DeusData DeusData added this to the 0.9.1-rc milestone Jul 8, 2026
@Cosm1cAC Cosm1cAC force-pushed the fix/windows-search-code-utf8-paths branch from ac9ed81 to c5cbbcf Compare July 8, 2026 14:36
Signed-off-by: Cosm1cAC <cosm1cac@github.com>
@Cosm1cAC Cosm1cAC force-pushed the fix/windows-search-code-utf8-paths branch from c5cbbcf to 9971fab Compare July 8, 2026 14:43
Signed-off-by: Cosm1cAC <87748355+Cosm1cAC@users.noreply.github.com>
@Cosm1cAC Cosm1cAC force-pushed the fix/windows-search-code-utf8-paths branch from e4d4198 to 8e0c29a Compare July 8, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. ux/behavior Display bugs, docs, adoption UX windows Windows-specific issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants