test(lexer): cover CRLF after basic string escape and simplify test macro

[ya7010]

Summary

Follow-up to #2149, addressing two points raised in review.

1. Add CRLF regression tests

#2149 changed the tokenization of CRLF input as well as LF. For a = "\"<CRLF>b":

	InvalidBasicString	LINE_BREAK
before	span 4..8 (absorbs the \r)	span 8..9 (\n only)
after	span 4..7	span 7..9 (\r\n)

A LINE_BREAK covering only \n would mislead Cursor::line_ending detection, so add CRLF cases for both escape shapes (escaped quote and escaped backslash) to pin those token boundaries.

2. Revert the $text:expr widening in test_lex_tokens!

#2149 widened $text:literal to $text:expr so the new tests could use concat! to embed real line breaks. But Rust string literals already express line breaks directly, as the existing Token(LINE_BREAK, "\n") cases show (test_lex.rs:111,119, and many others).

Going back to literal:

• drops the concat! wrappers, so the new cases read like the rest of the file
• restores the original repetition syntax $(...,)* (the change to ),* $(,)? is no longer needed)

The substantive part of the macro extension — $item:ident plus the @item rules that allow Token and Error to be mixed — is kept as is.

Tests

cargo test -p tombi-lexer → 102 passed. cargo fmt --all and cargo clippy -p tombi-lexer --tests are clean.

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

変更はテスト強化とテスト用マクロの表記整理に限定されており、既存呼び出しとの整合も取れていて回帰リスクが低いです。

Pull request overview

tombi-lexer の lexer 回帰（基本文字列のエスケープ直後に CRLF が来るケース）をテストで固定しつつ、test_lex_tokens! マクロの引数型拡張（$text:expr）を元の literal 制約へ戻して、テスト記述の一貫性を回復する PR です。

Changes:

• \" / \\ の直後に CRLF が来る不正基本文字列ケースの回帰テストを追加し、LINE_BREAK が \r\n を一塊でトークン化することを明示。
• test_lex_tokens! の $text を expr から literal に戻し、既存のテスト記法（素直な文字列リテラル）に揃える。
• 併せて、マクロの繰り返し構文を $(...,)* へ戻して簡潔化（既存の呼び出し側は全て末尾カンマ付きで整合）。

File summaries

File	Description
crates/tombi-lexer/tests/test_lex.rs	CRLF 回帰ケースを追加し、test_lex_tokens! の $text を literal に戻してテスト表記を統一

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.