Fix text_wrap spacing preservation for CJK-separated words - #9
Open
cha-yh wants to merge 1 commit into
Open
Conversation
Remove the text_wrap special case that suppressed spaces between adjacent tokens whenever both tokens contained CJK characters. That render-time heuristic caused explicit user input such as Korean, Chinese, and Japanese placeholder phrases with spaces to be rewritten without those spaces, changing the visible output instead of only controlling wrapping. Preserve explicit inter-word spacing consistently by using the same single-space restoration behavior for all split tokens after split_text has already separated the input on whitespace boundaries. This keeps text_wrap focused on layout and visible-width handling rather than language-specific normalization. Clean up the now-unused prev_word state and the tokenizer cjk_count import that only supported the removed branch, eliminating dead code and avoiding unnecessary string cloning in the wrapping loop. Add regression tests covering single-line output preservation and multiline wrapping behavior for Korean, Chinese, and Japanese sample phrases so future changes do not reintroduce CJK-specific space loss during wrapping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Input:
text_wrap("주문 상태를 확인해 주세요.", 80, 0, "", "", false, false)Expected output:
"주문 상태를 확인해 주세요."
Actual output before this change:
"주문상태를확인해주세요."
This fixes
text_wrapso explicit spaces between CJK words are preserved in rendered output.Previously,
text_wraptreated adjacent tokens containing CJK characters as needing no separating space. That made wrapping behave like language-specific normalization and caused user-provided spaces in Korean, Chinese, and Japanese text to disappear from terminal output.Changes
boundaries
Testing
cargo test -p streamdown-render text_wrap
Result: 12 passed.