Repository navigation
Conversation
…dly punctuation mode for Markdown import and export ## Description CommonMark doesn't let `**` open or close between a letter and punctuation, so `**你好。**世界` reads with literal asterisks in both packages once meta#9430 follows CommonMark's Unicode punctuation. Import in both packages now follows the CommonMark CJK-friendly amendment (https://github.com/tats-u/markdown-cjk-friendly) by default, and export still writes Markdown for strict CommonMark/GFM readers by default. Both are selectable: - `@lexical/markdown`: the import and export functions take an options object in place of `shouldPreserveNewLines` (`MarkdownImportOptions` / `MarkdownExportOptions`) with `punctuation: 'cjk-friendly' | 'strict-commonmark'`. Flanking uses its own classifier with the amendment's CJK table (East Asian Width F/H/W without emoji, plus Hangul, from get-east-asian-width 1.7), ideographic variation selectors and the variation-selector lookback. A strict export is checked against CJK-friendly readers too. - `@lexical/mdast`: `MdastExtension` takes `importPunctuation` and `exportPunctuation`. CJK-friendly import uses `micromark-extension-cjk-friendly` and its strikethrough counterpart. CJK-friendly export writes the strict export's references beside `*`/`~` runs to CJK characters as typed. `mdast-util-gfm-strikethrough` 2.0.1 serializes `~~` as attention, so strict export now writes references beside strikethrough too. ## Changeset `.changeset/markdown-cjk-friendly-punctuation.md`, a `[feature]` patch for `@lexical/markdown` and `@lexical/mdast`. ## Test plan ### Before New tests did not exist; `**你好。**世界` imported with literal asterisks in both packages. ### After ``` $ vitest run --project unit packages/lexical-mdast packages/lexical-markdown Test Files 19 passed (19) Tests 1368 passed (1368) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
meta-cla
Bot
added
the
CLA Signed
|
AI review status for this pull request.
|
lexical
@lexical/a11y
@lexical/clipboard
@lexical/code
@lexical/code-core
@lexical/code-prism
@lexical/code-shiki
@lexical/compiler
@lexical/devtools-core
@lexical/dragon
@lexical/eslint-plugin
@lexical/extension
@lexical/fast-check
@lexical/file
@lexical/hashtag
@lexical/headless
@lexical/history
@lexical/html
@lexical/internal
@lexical/link
@lexical/list
@lexical/mark
@lexical/markdown
@lexical/mdast
@lexical/offset
@lexical/overflow
@lexical/plain-text
@lexical/react
@lexical/rich-text
@lexical/selection
@lexical/table
@lexical/tailwind
@lexical/text
@lexical/utils
@lexical/y
@lexical/yjs
commit: |
![meta-code-review[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/5334748?s=60&v=4){kind=small}
There was a problem hiding this comment.
Thanks for this. Reading **你好。**世界 the way CJK writers mean it is a real improvement, and keeping export strict by default while the positional boolean keeps working is the right shape for compatibility. One problem needs fixing before merge, plus a few optional suggestions.
Needs a fix: @lexical/mdast can't read back its own default export when a formatted run is edged by an emoji next to a letter
With the new importPunctuation: 'cjk-friendly' default (packages/lexical-mdast/src/MdastExtension.ts:291), import classifies a non-BMP emoji such as 🎉 as punctuation. Export still goes through mdast-util-to-markdown, which classifies one UTF-16 unit at a time and so writes no character reference there. A run between a letter and that emoji then can't open or close (CommonMark left-flanking rules, which the amendment keeps for non-CJK characters). With MdastGfmExtension at its defaults:
| Runs | Export | Import before this PR | Import with this PR |
|---|---|---|---|
Hello + bold 🎉 launch |
Hello**🎉 launch** |
bold | plain text with ** |
Release + bold 🚀 + now |
Release**🚀**now |
bold | plain text with ** |
สวัสดี + bold 😀 + ครับ |
สวัสดี**😀**ครับ |
bold | plain text with ** |
foo + italic 🔥hot + bar |
foo*🔥hot*bar |
italic | plain text with * |
a + strikethrough 👍ok |
a~~👍ok~~ |
strikethrough | plain text with ~~ |
exportPunctuation: 'cjk-friendly' has the same gap, since toCjkFriendly only removes references. AGENTS.md's Backwards Compatibility section asks that existing behaviour not change in ways that could break callers, and a save/load cycle that drops formatting does that. It also contradicts the new README line saying import "accepts … what Lexical has exported" (packages/lexical-mdast/README.md:199-200).
@lexical/markdown already avoids this, because its strict export checks a CJK-friendly micromark reader and writes Hello**🎉 launch**. Either give @lexical/mdast's strict export the same check (write the letter beside a *, _ or ~ run as a reference when the run's inner edge is a non-BMP punctuation or symbol), or keep importPunctuation: 'strict-commonmark' as @lexical/mdast's default until it does. Please add one of the rows above to MdastPunctuation.test.ts.
Optional, non-blocking
packages/lexical-markdown/src/utils.ts:92-94describes theCJKtable asget-east-asian-width1.7, but the lockfile resolves 1.6.0 formicromark-extension-cjk-friendly-util. The two packages then disagree on 981 recently assigned code points (for example U+18E00–U+19191). Raising the resolved version would line them up.- The README and JSDoc lines saying the strict
@lexical/mdastexport is read "the same way" by every CommonMark, GFM and CJK-friendly reader (packages/lexical-mdast/README.md:200-203,MdastExtension.ts:139) should change with the fix above. - When an options object is passed, the fifth
shouldMergeAdjacentLinesargument is ignored. An overload takingoptions?: MarkdownImportOptionswould make the TypeScript and Flow signatures say so. - The CJK-friendly strikethrough swap in
compile.tsreplaces only the built-ingfmStrikethrough()instance. A note in the punctuation-modes README section would help people who contribute their own. - The new micromark packages add about 18 KB minified (6.4 KB gzipped) to
@lexical/mdastbundles, including for apps that choose strict import. That might be worth a line in the changeset.
| config: safeCast<MdastConfig>({ | ||
| exportPunctuation: 'strict-commonmark', | ||
| exportRules: CORE_EXPORT_RULES, | ||
| importPunctuation: 'cjk-friendly', |
There was a problem hiding this comment.
With this default, import reads a non-BMP emoji as punctuation, but export (mdast-util-to-markdown) still classifies one UTF-16 unit at a time and writes no reference beside it. Hello + bold 🎉 launch exports as Hello**🎉 launch** and re-imports as plain text with literal ** (bold before this PR). Either give the strict export a reference where the CJK-friendly reader needs one, as @lexical/markdown does, or keep 'strict-commonmark' here until it does.
Comment on lines
+199
to
+200
| Import defaults to `'cjk-friendly'`, which accepts what CJK writers type and | ||
| what Lexical has exported. Export defaults to `'strict-commonmark'`: it |
There was a problem hiding this comment.
This isn't true yet for runs edged by an emoji next to a letter: Hello**🎉 launch**, foo*🔥hot*bar and a~~👍ok~~, all exported by this package, import as plain text with this default.
| } | ||
| // The amendment's CJK characters: East Asian Width F, H and W, without the | ||
| // characters shown as emoji, and the Hangul script, from the Unicode 17 data | ||
| // of `get-east-asian-width` 1.7 that `micromark-extension-cjk-friendly` reads. |
There was a problem hiding this comment.
Optional: the lockfile resolves get-east-asian-width@1.6.0 for micromark-extension-cjk-friendly-util, so with this repository's install @lexical/mdast doesn't treat 981 of these code points as CJK (U+18CD6–18CDA, U+18D1F–18D20, U+18E00–19191, U+191A0–191D2, U+1B123–1B128, U+1B168, U+1F1AE, U+1F7DA). Raising the resolved version would make the two packages agree.
… formatting beside an emoji
## Description
With the CJK-friendly import default, @lexical/mdast read a non-BMP emoji as punctuation, as CommonMark does, but its export (mdast-util-to-markdown) classifies one UTF-16 unit at a time and wrote no reference beside it, so `Hello` + bold `🎉 launch` exported as `Hello**🎉 launch**` and came back as plain text.
Export now serializes with each astral punctuation character of the text standing in as an unused BMP symbol, so mdast-util-to-markdown writes the references and picks the markers CommonMark and CJK-friendly readers need (`Hello**🎉 launch**`). Where micromark without the CJK-friendly extension would read the result differently, an emoji beside a run goes out as a reference too.
Also: get-east-asian-width resolves to 1.7.0, matching the CJK table in @lexical/markdown; the options argument of `$convertFromMarkdownString` and `$generateNodesFromMarkdownString` has its own TypeScript and Flow overloads; README notes on contributed strikethrough syntax; changeset mentions the bundle size and the new references.
## Test plan
### Before
```
$ vitest run --project unit packages/lexical-mdast/src/__tests__/unit/MdastPunctuation.test.ts
× exports [["Hello",0],["🎉 launch",1]]
```
### After
```
$ vitest run --project unit packages/lexical-mdast packages/lexical-markdown packages/lexical-playground/__tests__/unit/MarkdownTransformers.test.ts
Test Files 20 passed (20)
Tests 1383 passed (1383)
```
Random paragraphs of CJK, Latin, Thai, emoji and punctuation also round-trip through @lexical/mdast in all three of default, strict import and CJK-friendly export modes (0 failures in ~10k each).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…without changing the mdast tree
## Description
`@lexical/mdast` export put a stand-in for each emoji (or other punctuation outside the Basic Multilingual Plane) into the mdast text nodes before serializing, and never put the originals back. It chose stand-ins by searching `JSON.stringify(tree)`, which throws on `data` that is not plain JSON, from a range that includes `\p{No}` and unassigned code points that micromark reads as letters, and it replaced every copy of a stand-in in the output, including copies a contributed handler wrote itself. Where a run beside an emoji could read differently for micromark (which reads the emoji as a letter), it parsed the export twice more to decide whether to write the emoji as a reference.
Now the stand-ins are set only while the `text` handler serializes each node, then restored even if the handler throws. They are picked from punctuation only, starting with Supplemental Punctuation, avoiding every string found by walking the tree (cycles skipped). If a handler writes a stand-in itself, the output holds more of it than the text does, so the export picks again avoiding that output, and otherwise serializes without stand-ins. The re-parse is gone: with strict export, an emoji beside a run goes out as a reference whenever micromark's UTF-16 reading would make that run flank differently. The flanking classes come from micromark's own `classifyCharacter`. With CJK-friendly export, whose readers all read emoji correctly, the emoji is written as is.
`$convertFromMarkdownString` and `$generateNodesFromMarkdownString` also get an overload that takes `boolean | MarkdownImportOptions`, so a wrapper that forwards either form type-checks again.
## Test plan
### Before
The new tests in `MdastPunctuation.test.ts` fail on the previous commit. A handler that writes U+2E00/U+2E01 has those characters turned into the emoji, and an export rule that puts a cycle and a BigInt in `data` makes `$convertToMarkdownString` throw.
### After
```
$ npx vitest run packages/lexical-mdast packages/lexical-markdown packages/lexical-playground/__tests__/unit
Test Files 49 passed (49)
Tests 1658 passed (1658)
```
A throwaway round-trip fuzz over 45,000 random paragraphs (CJK, Latin, Thai, emoji, astral letters, punctuation, bold/italic/strikethrough) found no failures in the default, strict-import and CJK-friendly-export configurations. `tsc -p ./tsconfig.test.json`, Flow and `treeShakingSource.test.ts` pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-friendly punctuation
## Description
Fixes from a review of the whole branch:
- `@lexical/markdown` reads a `null` `shouldPreserveNewLines` (possible from JavaScript) as `false` again, where the options handling made it throw. A fifth `shouldMergeAdjacentLines` argument is now honored next to an options object.
- Export checks with the micromark CJK-friendly reader only when a character it reads differently (`\v`, U+2028, U+2029, U+FEFF) is involved. It no longer reads a variation selector after one of those as whitespace.
- `@lexical/mdast` export no longer writes an emoji as a reference because of an escaped `*`, `_` or `~` beside it. Stand-ins are found with a set of code points rather than substring searches, and are counted and restored in one pass.
- The CJK-friendly export finds references through micromark's events, without building an mdast tree or running contributed from-markdown extensions.
- The unused `exportSyntaxExtensions` field is removed. `micromark` is now a declared dependency. It and the CJK-friendly micromark packages stay external in the npm build, like the other micromark packages.
- The docs and changeset now say that a variation selector before a marker reads as the character it follows, as the amendment specifies.
## Test plan
### Before
The new tests fail on the previous commit. A `null` options argument throws `TypeError: Cannot read properties of null`, the fifth argument is ignored next to options, and `a\*🎉 *b*` exports as `a\*🎉 *b*`.
### After
```
$ npx vitest run packages/lexical-mdast packages/lexical-markdown packages/lexical-playground/__tests__/unit scripts/__tests__/unit/treeShakingSource.test.ts
Test Files 50 passed (50)
Tests 2038 passed (2038)
```
A throwaway round-trip fuzz over 45,000 random paragraphs (emoji, CJK, astral letters, escaped markers and stand-in characters among them) found no failures in the default, strict-import and CJK-friendly-export configurations. `tsc -p ./tsconfig.test.json` and Flow pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efore markers as CJK-friendly readers do ## Description The CJK-friendly amendment reads a variation selector (U+FE00 to U+FE0E) before a marker as the character it varies. Strict readers read it as a letter. Neither package's export reconciled the two readings in every case: - `@lexical/mdast` export did not account for variation selectors. Bold `©︎` before `2024` exported as `**©︎**2024`, which the default import reads as literal asterisks. - `@lexical/markdown` export missed the case where a piece consists only of a selector, whose base ends the previous piece (`x*︎*b`). Import missed it where a selector starts a text that was cut from the line. Now `@lexical/mdast` serializes variation selectors with stand-ins as it does emoji, and writes one as a reference where the readers' flanking differs (`**©︎**2024`). It always writes one before `~`, because micromark-extension-cjk-friendly-gfm-strikethrough 2.0.1 never lets `~~` close after a selector. A backslash before such a reference is escaped. `flankingBefore` in `@lexical/markdown` takes what lies before the text, so a leading selector reads its base there. Also: - The CJK-friendly mdast export finds references after a leading byte order mark at the right offsets. - Stand-ins are found with one walk over text and node values. - Units are classified with `micromark-util-classify-character`, as `mdast-util-to-markdown` does. - `get-east-asian-width` is required at ^1.7.0, the data the `@lexical/markdown` table comes from. ## Test plan ### Before ``` $ npx vitest run packages/lexical-mdast/src/__tests__/unit/MdastPunctuation.test.ts packages/lexical-markdown/src/__tests__/unit/MarkdownPunctuation.test.ts AssertionError: expected [ '**©︎**2024', [ [ '**©︎**2024', 0 ] ] ] to deeply equal [ '**©︎**2024', … ] AssertionError: expected 'x*︎*b' to be 'x*︎*&meta#98;' ``` ### After ``` $ npx vitest run packages/lexical-mdast packages/lexical-markdown packages/lexical-playground/__tests__/unit scripts/__tests__/unit/treeShakingSource.test.ts Test Files 50 passed (50) Tests 2045 passed (2045) ``` A throwaway round-trip fuzz with variation selectors, emoji, CJK, backslashes and markers in the text found no failures: 3×5,000 cases for `@lexical/mdast` in three configurations, and for `@lexical/markdown` in its three mode pairs. `tsc -p ./tsconfig.test.json` and Flow pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks for this. The punctuation mode is carefully built, and the mdast-util-gfm-strikethrough bump fixes the strikethrough references @lexical/mdast used to drop. One defect needs fixing before this lands, plus a few smaller suggestions.
Must fix: strikethrough ending in a variation selector no longer imports in @lexical/mdast
packages/lexical-mdast/src/compile.ts:106-116 swaps gfmStrikethrough() for gfmStrikethroughCjkFriendly() under the new cjk-friendly default. In micromark-extension-cjk-friendly-gfm-strikethrough 2.0.1 (its latest release; the 3.0.0-rc.0 prerelease has the same code), the close test compares the raw class of the character before ~~ (before$1 === constants.attentionSideAfter, dist/strikethrough.js:142) instead of the looked-back beforePrimary. So a closing ~~ after punctuation or a symbol followed by U+FE00 to U+FE0E never closes:
x ~~Done ✔︎~~ y (✔ is U+2714 followed by U+FE0E)
base @lexical/mdast plain `x ` | strikethrough `Done ✔︎` | plain ` y`
head @lexical/mdast plain `x ~~Done ✔︎~~ y`
head @lexical/markdown plain `x ` | strikethrough `Done ✔︎` | plain ` y`
~~♥︎~~ and ~~a.︀~~ (U+FE00) behave the same way, and strikethrough Done ✔︎ exported with the default configuration comes back as literal tildes. This breaks the rule the README documents (a variation selector before a marker reads as the character it follows, packages/lexical-mdast/README.md:197), and the two importers disagree here, unlike every case MarkdownPunctuation.test.ts checks. Making that comparison use beforePrimary (upstream, or as a dependency patch), or keeping gfmStrikethrough() until it does, and adding ~~Done ✔︎~~ to the both-importers table, would close it.
Suggestions (non-blocking)
@lexical/mdaststrict export beside a selector. BoldDone ✔︎beforenextexports**Done ✔︎**next. A CJK-friendly reader, including the default import, does not close that run, so the default round trip returns literal asterisks, while@lexical/markdownwrites**Done ✔︎**next. That contradicts the strict-export description inpackages/lexical-mdast/src/MdastExtension.ts:139. Extending the stand-in intoMarkdownWithStandIns(MdastExport.ts:710) to a punctuation-plus-selector pair would cover it.- Tests. Committing the random round trip the Test plan describes, with a variation selector in its alphabet, would have caught both cases. AGENTS.md (Commits and Pull Requests) also asks for the real Before output rather than a description.
@lexical/markdownexport of a lone selector. Bold U+FE00 followed by italicb?계exports**︀***b?계*, which both CJK-friendly importers read with literal markers, becauseflankingBefore(packages/lexical-markdown/src/utils.ts:323) does not look back across a marker. This shape is contrived, so it is low priority.- CJK table drift. The hand-written
CJKclass inpackages/lexical-markdown/src/utils.tsdiffers frommicromark-extension-cjk-friendly-util3.0.1 on nine new emoji (U+1F6D9, U+1FA8B to U+1FA8D, U+1FACC, U+1FADD, U+1FAEB, U+1FAF9, U+1FAFA). A unit test comparing it withisCjk(classifyCharacter(cp))would keep the two importers aligned.
| : [ | ||
| ...config.micromarkExtensions.map(extension => | ||
| extension === gfmStrikethroughSyntax | ||
| ? gfmStrikethroughCjkFriendly() |
There was a problem hiding this comment.
With this swap as the default, x ~~Done ✔︎~~ y (U+2714 U+FE0E) and ~~a.︀~~ import as literal tildes, where gfmStrikethrough() and @lexical/markdown read strikethrough. In micromark-extension-cjk-friendly-gfm-strikethrough 2.0.1, dist/strikethrough.js:142 tests before$1 === constants.attentionSideAfter (the raw class of the selector) instead of beforePrimary, so a ~~ after punctuation plus U+FE00 to U+FE0E never closes. Could this use a patched extension, or keep gfmStrikethrough() until upstream is fixed, with ~~Done ✔︎~~ added to the both-importers table?
…hrough after a variation selector
## Description
With the default CJK-friendly import, `@lexical/mdast` swaps GFM strikethrough for `micromark-extension-cjk-friendly-gfm-strikethrough`. In its 2.0.1 release (and its 3.0.0 prerelease), the close test reads the raw class of the character before `~~` instead of the character a variation selector varies. So `x ~~Done ✔︎~~ y` imported as literal tildes, where `gfmStrikethrough()` and `@lexical/markdown` read strikethrough.
The syntax is now wrapped. Before a `~` run that follows a variation selector, the tokenizer sees the character the selector varies (unless that is whitespace or `~`), as the CJK-friendly emphasis does. The export workaround that always referenced a selector before `~` is dropped, and the both-importers table gains `x ~~Done ✔︎~~ y` and `~~a.︀~~`.
## Test plan
### Before
```
$ npx vitest run packages/lexical-markdown/src/__tests__/unit/MarkdownPunctuation.test.ts
AssertionError: expected [ [ 'x ~~Done ✔︎~~ y', 0 ] ] to deeply equal [ [ 'x ', 0 ], [ 'Done ✔︎', 4 ], [ ' y', 0 ] ]
```
### After
```
$ npx vitest run packages/lexical-mdast packages/lexical-markdown scripts/__tests__/unit/treeShakingSource.test.ts
Test Files 21 passed (21)
Tests 1785 passed (1785)
```
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mporters and keep exports readable by both
## Description
Follow-ups from review:
- The `@lexical/mdast` strikethrough wrapper also handles a selector whose base is a `~` or a quotation mark that U+FE01 makes CJK punctuation. In those cases the tokenizer sees a stand-in character of the same class.
- `@lexical/markdown` reads as CJK the wide characters that Unicode 17 made emoji only where the engine doesn't know them as emoji, which is how `micromark-extension-cjk-friendly` reads them. A test compares the two classifications over every code point up to U+3FFFF. The unused direct `get-east-asian-width` dependency is removed.
- `@lexical/markdown` export reads the piece before a lone variation selector only when there is one. This takes the quadratic recursion (and the stack overflow on a few thousand one-character pieces) out of `decideReferences`.
- `@lexical/mdast` export stands in for U+FEFF, U+2028, U+2029 and `\v` too. micromark reads these as whitespace and other readers as letters, so one beside a marker goes out as a reference where the readings differ. Stand-ins also avoid characters in every string field of the tree (URLs, titles, alt text), not only node values.
- A seeded round-trip test exports random formatted text (CJK, emoji, astral letters, variation selectors, U+FEFF, stand-in characters) with both packages and reads each export back with both, in the three mode pairs. It runs 300 paragraphs in each pair in about a second.
## Test plan
### Before
```
$ npx vitest run packages/lexical-mdast/src/__tests__/unit/MdastPunctuationRoundTrip.test.ts
AssertionError: expected [ '[["︀",4]] => ~~︀~~ => [["~~︀~~",0]]', … ] to deeply equal []
```
### After
```
$ npx vitest run packages/lexical-mdast packages/lexical-markdown packages/lexical-playground/__tests__/unit scripts/__tests__/unit/treeShakingSource.test.ts
Test Files 52 passed (52)
Tests 2055 passed (2055)
```
The round-trip test also passed with 3,000 paragraphs per pair under four seeds. `tsc -p ./tsconfig.test.json` and Flow pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… read selectors that start formatted text ## Description - `@lexical/mdast` export stood in for emoji and variation selectors in an autolink's text too, then wrote one as a reference beside a `_` or `~` in the URL (`<https://x.com/_🎉>`). An autolink does not decode references, so the URL changed. Text in an autolink is now serialized as is. - `@lexical/markdown` import re-reads the content of a format, and a text match's text, without what came before it. A variation selector that starts that content therefore read as a letter instead of as the tag, or the link's `[`, it follows (`*︎__a__*`, `[︎__a__](https://x.com)`). That content now gets the character before it, as micromark-extension-cjk-friendly reads it. - `@lexical/markdown` export decides whether a selector can be written as a reference by the selector itself, not by the character it varies. - `@lexical/mdast` walks the tree's strings for stand-in collisions only when it has something to stand in for. `CompiledMdast`'s new `exportPunctuation` field is optional. - The random round-trip test only runs with `MDAST_ROUND_TRIP_CASES` set, since a random round trip is too slow for every CI run. ## Test plan ### Before ``` $ npx vitest run packages/lexical-markdown/src/__tests__/unit/MarkdownPunctuation.test.ts packages/lexical-mdast/src/__tests__/unit/MdastPunctuation.test.ts AssertionError: expected [ [ '︎__a__', 2 ] ] to deeply equal [ [ '︎', 2 ], [ 'a', 3 ] ] AssertionError: expected '<https://x.com/_🎉>' to be '<https://x.com/_🎉>' ``` ### After ``` $ npx vitest run packages/lexical-mdast packages/lexical-markdown packages/lexical-playground/__tests__/unit scripts/__tests__/unit/treeShakingSource.test.ts Test Files 51 passed | 1 skipped (52) Tests 2055 passed | 3 skipped (2058) $ MDAST_ROUND_TRIP_CASES=2000 npx vitest run packages/lexical-mdast/src/__tests__/unit/MdastPunctuationRoundTrip.test.ts Tests 3 passed (3) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elf-review round 7 ## Description - @lexical/mdast no longer stands in `\v`: a reference to it reads back as U+FFFD, so it goes out raw, as on main. - A CJK-friendly export measures table cells with the CJK references it writes as characters, so columns stay aligned. - A look ahead at a node whose handler has no `peek` (such as a link) no longer counts its text's stand-ins as written, which let a handler's own stand-in character pass the collision check and be restored as an emoji. - The stand-in restore builds its pattern once per export rather than per table cell, and the CJK-reference rewrite shares one predicate between its precheck and its pass. - A variation selector that starts formatted text reads as the last character of the opening tag. ## Test plan ### Before On deb395d the three new MdastPunctuation tests fail: ``` × exports [["x\u000b",0],["!b",1]] × counts no text that a look ahead at a link writes × aligns table cells as a CJK-friendly export writes them Expected: "x🎉 [🎉](https://x.com) ⸀" Received: "x🎉 [🎉](https://x.com) 🎉" ``` ### After `pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast packages/lexical-playground`, the tree-shaking source test and `MDAST_ROUND_TRIP_CASES=2000` round trips pass; `tsc -p ./tsconfig.test.json`, `pnpm run flow`, eslint, prettier and check-changesets are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
This looks good to me. The CJK-friendly import default with a strict export default is easy to follow, and the tests that read every case through both importers in both modes, plus the export round trips, make the behavior clear. A few optional, non-blocking suggestions, most important first:
- CJK-friendly export can throw (
packages/lexical-mdast/src/MdastExport.ts:1023-1031).rewriteCjkReferencespasses every decoded&#…;beside a*or~run toclassifyCharacter, which callsString.fromCodePoint. WithexportPunctuation: 'cjk-friendly', text�followed by boldx(which the default export writes as\�**x**) throwsRangeError: Invalid code point 9999999, and so does importing\�**x**and exporting it again. Returning the reference unchanged whencode > 0x10ffff, with a case inMdastPunctuation.test.ts, would cover it. - The pending #9430 changeset now disagrees with the default.
.changeset/markdown-unicode-punctuation-flanking.mdsays**你好。**世界and**注意：**请先登录now import with literal asterisks; with this PR both import as bold. Both entries ship in the same release, and CONTRIBUTING.md describes the changeset as the changelog entry people upgrade from, so dropping those two examples there (or qualifying them withpunctuation: 'strict-commonmark') would keep the notes consistent. - Variation selectors in the Verify note. With the new default,
**Done ✔︎**nextand**©︎**2024, which the current export writes without a reference, import with literal asterisks (they stay bold with'strict-commonmark'). That follows the amendment, and the changeset body states the rule, but a line in theVerify:paragraph of.changeset/markdown-cjk-friendly-punctuation.mdwould help anyone with stored Markdown. Parameters<>of the overloaded functions. TypeScript uses the last overload forParameters<typeof fn>, and the last overloads of$convertFromMarkdownString(packages/lexical-markdown/src/index.ts:158-163) and$generateNodesFromMarkdownString(:203-207) leave outshouldMergeAdjacentLines, so a wrapper typed that way stops accepting the old trailing argument (TS2554). AddingshouldMergeAdjacentLines?: booleanto those overloads and to the matching Flow declarations keeps the old parameter list working, in line with the backwards-compatibility section of AGENTS.md.- Round trip in CI.
MdastPunctuationRoundTrip.test.tsis skipped unlessMDAST_ROUND_TRIP_CASESis set. A small default (a few hundred cases runs in well under a second) would keep a sample in CI. - Bundle figure. Bundled and minified with esbuild, the three CJK-friendly extensions add about 17 KB minified / 6 KB gzipped to
@lexical/mdast, a little more than the 12 KB / 5 KB the changeset gives.
AGENTS.md also asks the Test plan's ### Before to paste the command's real output rather than describe it.
| return (ATTENTION_RUN.test(before) || ATTENTION_RUN.test(after)) && | ||
| before !== '_' && | ||
| after !== '_' && | ||
| isCjk(classifyCharacter(code)) && |
There was a problem hiding this comment.
Optional: classifyCharacter calls String.fromCodePoint, so a reference past U+10FFFF throws here. With exportPunctuation: 'cjk-friendly', text � before bold x (written \�**x**) makes $convertToMarkdownString throw RangeError: Invalid code point 9999999. Returning reference when code > 0x10ffff before classifying would avoid it.
| markdown: string, | ||
| transformers?: Transformer[], | ||
| node?: ElementNode, | ||
| shouldPreserveNewLines?: boolean | MarkdownImportOptions, |
There was a problem hiding this comment.
Optional: TypeScript takes this last overload for Parameters<typeof $convertFromMarkdownString>, which drops the old trailing shouldMergeAdjacentLines, so a wrapper typed that way fails with TS2554 on five arguments. Adding shouldMergeAdjacentLines?: boolean here (and in the $generateNodesFromMarkdownString and Flow counterparts) keeps the old parameter list.
|
|
||
| `@lexical/markdown`'s `$convertFromMarkdownString`, `$generateNodesFromMarkdownString`, `$convertToMarkdownString` and `$convertSelectionToMarkdownString` take an options object in place of the `shouldPreserveNewLines` argument (`MarkdownImportOptions` and `MarkdownExportOptions`, which also hold `shouldPreserveNewLines` and, for import, `shouldMergeAdjacentLines`). Its `punctuation` option is `'cjk-friendly'` (the import default) or `'strict-commonmark'` (the export default). `@lexical/mdast`'s `MdastExtension` takes the same modes as `importPunctuation` and `exportPunctuation`. With `'cjk-friendly'`, export writes such CJK text as typed (`**你好。**世界`), for readers that follow the amendment. `@lexical/mdast` now writes references beside `~~` as it does beside `**`, so that strikethrough beside punctuation reads back with a strict import too, and reads an emoji or another character outside the Basic Multilingual Plane as the punctuation CommonMark counts it as, so a letter beside a run that starts or ends with one goes out as a reference; with strict export the emoji does too where micromark, which reads it as a letter, would otherwise read the run differently (`Hello**🎉 launch**`, or `Hello**🎉 launch**` with CJK-friendly export). Both packages read a variation selector before a marker as the CJK-friendly readers do, writing it or the character beside the run as a reference where readers differ (`**©︎**2024`). `@lexical/mdast` also writes U+FEFF, U+2028 and U+2029 as a reference beside a marker where micromark, which reads them as whitespace, and other readers would read the run differently. The CJK-friendly micromark extensions add about 12 KB minified (5 KB gzipped) to `@lexical/mdast` bundles, also where import is strict. | ||
|
|
||
| Verify: Markdown such as `**你好。**世界` or `中文**「强调」**` imports as bold in both packages, as it did in `@lexical/markdown` before it followed CommonMark's Unicode punctuation; pass `punctuation: 'strict-commonmark'` (or `importPunctuation`) to read it as CommonMark does. `@lexical/mdast` export of strikethrough that starts or ends with punctuation beside a letter, and of formatting that starts or ends with an emoji beside a letter, now holds character references (`~~你好。~~世界`, `Hello**🎉 launch**`), so check export snapshots. |
There was a problem hiding this comment.
Optional: **Done ✔︎**next and **©︎**2024, which the current export writes as is, now import with literal asterisks by default; worth a line here. The pending markdown-unicode-punctuation-flanking.md Verify paragraph also still says **你好。**世界 imports literal, which this default reverses.
…eview fixes and self-review round 8 ## Description - The union overloads of `$convertFromMarkdownString` and `$generateNodesFromMarkdownString` (and their Flow declarations) keep the trailing `shouldMergeAdjacentLines`, so `Parameters<typeof fn>` still takes the old argument list. - A CJK-friendly export keeps a reference past U+10FFFF beside a marker instead of throwing. - Wrapped to-markdown handlers keep the `this` that to-markdown calls them with. - A variation selector that starts a text match's content reads against the full character before it, not a lone surrogate. - The stand-in tree walk runs once, the stand-in pick advances in one place, and table measuring skips cells with nothing to put back. - The round trip runs 200 seeded cases by default (about 0.7 s), with every bold/italic/strikethrough combination and some code. - Changesets: the CJK-friendly Verify paragraph names Markdown exported before with an emoji or a text variation selector at a run's inner edge, which now imports with literal markers, and `'strict-commonmark'` to keep the earlier reading; the meta#9430 changeset no longer says CJK examples import with literal asterisks under the default; the bundle figure is 17 KB minified (6 KB gzipped). The README matches. ## Test plan ### Before On cfb8fd9 two of the new MdastPunctuation tests fail: ``` × keeps a reference past the last code point 7ms × calls handlers with the state as this 8ms RangeError: Invalid code point 9999999 ``` ### After ``` Test Files 54 passed (54) Tests 1796 passed (1796) ``` from `pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast packages/lexical-playground`; `tsc -p ./tsconfig.test.json`, `pnpm run flow`, eslint, prettier and check-changesets are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elf-review round 9 ## Description - @lexical/mdast reads any import mode but `'cjk-friendly'` strictly, as @lexical/markdown does, so an unknown mode from untyped config reads the same in both packages. - @lexical/markdown skips the CJK and ideographic-selector tests below U+1100. - The stand-in walk again collects the strings to avoid in a second pass only when the document has a character to stand in, so documents without one allocate nothing. - The variation-selector test reuses `isNonEmojiGeneralUseVS`, and the class-bitmask helper is named `isSpaceOrPunctuationClass`. - The round trip runs a few fixed cases every time; random cases run only with `MDAST_ROUND_TRIP_CASES` set. ## Test plan ### After ``` Test Files 54 passed (54) ``` from `pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast packages/lexical-playground`, plus `MDAST_ROUND_TRIP_CASES=2000` round trips; `tsc -p ./tsconfig.test.json`, `pnpm run flow`, eslint and prettier are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks, this looks good. The CJK-friendly import reads every case in the new tables the same way micromark-extension-cjk-friendly does, and the strict export reads back through both packages' importers. The new options are additive overloads with matching Flow declarations, and the changeset has the Verify: paragraph that CONTRIBUTING.md asks for. The suggestions below are optional and non-blocking.
Optional suggestions
- Guard the strikethrough patch against a different upstream shape (
packages/lexical-mdast/src/compile.ts:37-73).cjkFriendlyStrikethroughtreatsgfmStrikethroughCjkFriendly().text[126]as a single construct. That holds for 2.0.1, butpackage.jsonallows^2.0.1and applications resolve their own version. If a later 2.x returned an array there,tokenize.callwould throw on the first~of a CJK-friendly import. Falling back to the unpatched syntax whentypeof construct?.tokenize !== 'function', or pinning2.0.1until upstream reads the selector on close, would remove that risk. - Paste the Before output in the test plan. AGENTS.md (Commits and Pull Requests) asks for the actual failing output before the change and the passing output after it. The
### Beforesection is prose, and the### Aftercounts are from an earlier round. Running the two new test files againstmainproduces that output: the CJK, variation-selector and strikethrough cases fail there. - Confirm that the new import default should re-read these shapes. With the CJK-friendly default,
**©︎**2024(in both packages) andHello**🎉 launch**(which@lexical/mdastitself exported before) now import with literal markers. That matches CommonMark 0.31.2, which counts symbols (S) as punctuation, and the amendment's variation-selector rule, and the changeset'sVerify:paragraph already describes it. This is only a flag for whoever approves the default. - Nit: the two packages write different strict Markdown for the same text. For
Hellofollowed by bold🎉 launch,@lexical/markdownwritesHello**🎉 launch**and@lexical/mdastwritesHello**🎉 launch**. Both read back correctly.isPortableinpackages/lexical-mdast/src/MdastExport.ts:951-978requires the same flanking under every reading, so the emoji becomes a reference too. Keeping it as typed would make exports easier to read. - Nit: the PR body puts the bundle cost at about 12 KB minified (5 KB gzipped), while the changeset says about 17 KB (6 KB). The changeset is what ships, so updating the body to match avoids confusion.
| */ | ||
| function cjkFriendlyStrikethrough(): MicromarkExtension { | ||
| const syntax = gfmStrikethroughCjkFriendly(); | ||
| const construct = syntax.text![126] as Construct; |
There was a problem hiding this comment.
Optional: this assumes text[126] is a single construct, which is true for micromark-extension-cjk-friendly-gfm-strikethrough 2.0.1. With ^2.0.1 in package.json, an application can resolve a later 2.x; if that returns an array here, tokenize is undefined and the first ~ of a CJK-friendly import throws. Falling back to gfmStrikethroughCjkFriendly() unpatched when typeof construct?.tokenize !== 'function' (or pinning 2.0.1) would avoid that.
| }; | ||
| // Whether the run beside `index`, toward `step`, flanks the same for every | ||
| // reading of the stand-in at `index`, and of one beyond the run. | ||
| const isPortable = (index: number, step: 1 | -1): boolean => { |
There was a problem hiding this comment.
Nit: requiring identical flanking under every reading makes Hello + bold 🎉 launch export as Hello**🎉 launch**, while @lexical/markdown writes Hello**🎉 launch** and both read back the same. Optional, but keeping the emoji as typed would make exports easier to read.
…elf-review round 10 ## Description - The random round trip's generator lost precision past 2^53 and returned 0 for nearly every draw, so its formats were almost all plain code. It is now mulberry32. With it, cases where bold-and-italic text is followed by text with only one of the two fail, because @lexical/markdown writes `***a****b*` for them; main does the same, so the generator leaves those cases out. - The stand-in text handler's look-ahead uses a contributed text handler's own `peek`. - @lexical/markdown's classify tests the code point once. ## Test plan ### Before With a working generator and `MDAST_ROUND_TRIP_CASES=1500`, each mode reported failures such as: ``` "[[\"!︎\",3],[\"ก🚀0\",18]] => ***!︎****`ก🚀0`* => [[\"!︎\",3],[\"**\",2],[\"ก🚀0\",18]]" ``` The same export on main (4d8cc44) of `[["a",3],["b",2]]`: ``` [["a",3],["b",2]] ***a****b* ``` ### After `MDAST_ROUND_TRIP_CASES=3000` round trips pass (`Tests 3 passed (3)`), as do the unit suites of both packages; `tsc -p ./tsconfig.test.json`, eslint and prettier are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…K-friendly emphasis with in-repo micromark constructs
## Description
@lexical/mdast imported through `micromark-extension-cjk-friendly`, its strikethrough package and their util, which pulled in `get-east-asian-width` and needed a wrapper for a strikethrough bug after a variation selector. Those dependencies added about 11 KB minified, more than an in-repo version.
`packages/lexical-mdast/src/cjkFriendly.ts` now holds the two tokenizers, reusing the `resolveAll` of micromark's own attention construct and of `gfmStrikethrough()`, with one classifier for the characters beside a run. The strikethrough reads a variation selector before a closing run as the character it varies, so the wrapper is gone. The CJK table that @lexical/markdown had moves to `@lexical/internal/isCjkCharacter`, which both packages bundle, and the MdastCjkTable test compares it with `get-east-asian-width`, now a root devDependency only.
A bundle of `MdastGfmExtension` with import and export (esbuild, minified, Lexical packages external):
| | minified | gzipped |
| --- | --- | --- |
| main | 136,272 | 41,730 |
| before this commit | 157,854 | 49,833 |
| after | 142,315 | 44,415 |
## Test plan
### After
```
Test Files 55 passed (55)
Tests 1799 passed (1799)
```
from `pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast packages/lexical-internal packages/lexical-playground`, including the tables that read every case through both importers in both modes; `MDAST_ROUND_TRIP_CASES=4000` round trips and the tree-shaking source test pass; `tsc -p ./tsconfig.test.json`, `pnpm run flow`, eslint, prettier and check-changesets are clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
This looks good to me. The CJK-friendly import reads the cases in the new tables the way micromark-extension-cjk-friendly does, the strict export reads back through both packages' importers, the new options are additive overloads with matching Flow declarations, and the changeset's Verify: paragraph describes the default-import and export-snapshot changes. Everything below is optional and non-blocking.
Optional suggestions
- Cover typed
@lexical/mdastshortcuts with a CJK case. The shortcut scanner parses with the same compiled micromark extensions as import (packages/lexical-mdast/src/MdastStream.ts,extensions: this.compiled.micromarkExtensions), so with this change typing中文**「强调」**applies bold and中文~~「强调」~~applies strikethrough, wheremainleaves both literal. That is the consistent result, but nothing pins it: one typed case inpackages/lexical-mdast/src/__tests__/unit/MdastShortcuts.test.tswould, and the changeset could mention that shortcuts followimportPunctuation. - Add
@lexical/internalto the changeset. The Changesets check warns thatpackages/lexical-internal/src/isCjkCharacter.ts, its Flow file and the new./isCjkCharacterexport change a published package without an entry. Adding'@lexical/internal': patchto the frontmatter of.changeset/markdown-cjk-friendly-punctuation.mdclears it, as CONTRIBUTING.md (Changesets) describes. - Consider
micromark-core-commonmark2.0.4 for the strict mode. CommonMark 0.31.2 §6.2 says a left-flanking run followed by punctuation must be preceded by whitespace or punctuation, so the opening**offoo**_bar_**bazdoes not open. The CJK-friendly default reads it that way, but'strict-commonmark'import usesmicromark-core-commonmark2.0.3 from the lockfile, which lets a*run open beside a_run and importsbarbold and italic. 2.0.4 fixed that and is outside the repository'sminimumReleaseAge; raising the lockfile (and the^2.0.3range inpackages/lexical-mdast/package.json) would make the two modes agree on text without CJK. - Refresh the PR body. AGENTS.md (Commits and Pull Requests) asks the Test plan for the real Before and After output.
### Beforeis prose, the### Aftercounts are from an earlier round (the same command now runs 22 files and 1,415 tests), and the Implementation section still describes themicromark-extension-cjk-friendlydependencies and their 12 KB, which the last commit replaced withpackages/lexical-mdast/src/cjkFriendly.ts(the changeset's 6 KB figure is the current one). - Nit: strict
@lexical/mdastexport of an emoji at a run's edge. ForHellofollowed by bold🎉 launch,@lexical/mdastwritesHello**🎉 launch**while@lexical/markdownwritesHello**🎉 launch**; both read back.isPortableinpackages/lexical-mdast/src/MdastExport.tsalso requires an opening run's ability to close to match across readings, which is what turns the emoji into a reference. Keeping it as typed would make exports easier to read, if that is worth the extra case.
| @@ -0,0 +1,12 @@ | |||
| --- | |||
| '@lexical/markdown': patch | |||
| '@lexical/mdast': patch | |||
There was a problem hiding this comment.
Optional: the Changesets check warns that @lexical/internal also changes here (src/isCjkCharacter.ts, its Flow file and the ./isCjkCharacter export). Adding '@lexical/internal': patch below this line clears the warning (CONTRIBUTING.md, Changesets).
| "mdast-util-to-markdown": "^2.2.0", | ||
| "mdast-util-to-string": "^4.0.0", | ||
| "micromark": "^4.0.2", | ||
| "micromark-core-commonmark": "^2.0.3", |
There was a problem hiding this comment.
Optional: micromark-core-commonmark 2.0.4 stops a * run from opening beside a _ run (foo**_bar_**baz), as CommonMark 0.31.2 §6.2 reads it and as the new CJK-friendly constructs already do. With 2.0.3 in the lockfile, 'strict-commonmark' import still reads bar there as bold and italic, so the two modes disagree on text without CJK. 2.0.4 is outside minimumReleaseAge.
…ndly punctuation self-review round 11 ## Description - @lexical/mdast: a reference that `mdast-util-to-markdown` writes for the last character of text after a literal backslash (`\a` before `**🎉a**`) came out as `\a`, whose backslash escapes the `&`. Such a backslash now stands in during the export and goes out as `\\` before a reference. - @lexical/markdown: a variation selector after a character written as a reference varies the reference's `;` for CJK-friendly readers, so a run after it is decided with that, and the export writes the reference it then needs (`**~~。~~0︎**︀x`). - `@lexical/internal/isCjkCharacter` reads Unicode 17 emoji the same in every engine, instead of asking the engine about the emoji Unicode 17 added. - `cjkFriendly.ts` reads the chunk before a run only after a surrogate or a selector, and its `getRunFlanking` now also serves the export's portability check, which replaces a second flanking function with another class encoding; `micromark-util-classify-character` is no longer a dependency. - The round trip's alphabet holds `\` and `&`. ## Test plan ### Before On 315700f: ``` × escapes a backslash before a reference the export writes × exports [["。",5],["0︎",1],["︀x",0]] Expected: "**~~。~~0︎**︀x" Received: "**~~。~~0︎**︀x" ``` ### After ``` Test Files 55 passed (55) Tests 1801 passed (1801) ``` from `pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast packages/lexical-internal packages/lexical-playground`; `MDAST_ROUND_TRIP_CASES=3000` round trips pass for seeds 1 to 4, as does the tree-shaking source test; `tsc -p ./tsconfig.test.json`, `pnpm run flow`, eslint, prettier and check-changesets are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elf-review round 12 ## Description - @lexical/markdown reads a piece's end as a selector varying a reference's `;` only when the piece is exactly the referenced character and one selector, not any piece with a selector after its first character. - @lexical/mdast stands in every backslash of a run before the last character of text, and writes each back escaped, so `a\\b` before bold `“x”` no longer comes out with the reference's `&` escaped. - @lexical/mdast writes a U+FEFF that starts the document as a reference, since micromark drops a leading byte order mark; the offset shift for one in the CJK-friendly rewrite is gone. - The changeset lists `@lexical/internal`, which gains `isCjkCharacter`. - Smaller: a cheap `\` check before the trailing-backslash regex, the export mode computed once, and no repeated default in compile.ts. ## Test plan ### Before On 4dc130c: ``` × keeps backslashes before a reference and a leading U+FEFF × writes a leading U+FEFF as a reference Expected: "。**你好。**世界" Received: "。**你好。**世界" ``` ### After ``` Test Files 55 passed (55) Tests 1802 passed (1802) ``` from `pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast packages/lexical-internal packages/lexical-playground`; `MDAST_ROUND_TRIP_CASES=3000` round trips pass for seeds 1, 5 and 6; `tsc -p ./tsconfig.test.json`, `pnpm run flow`, eslint, prettier and check-changesets are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… 2.0.4, CJK shortcut test
## Description
- `@lexical/mdast` now needs `micromark-core-commonmark` ^2.0.4. With 2.0.3, a `'strict-commonmark'` import let a `*` run open beside a `_` run and read `bar` in `foo**_bar_**baz` as bold and italic. CommonMark 0.31.2 §6.2 says that run doesn't open, and the CJK-friendly import already read it that way. 2.0.4 fixes this, so the two modes now agree on text without CJK.
- The lockfile now resolves a single copy each of `micromark-core-commonmark`, `micromark-util-types` and `micromark-factory-space`, so the micromark types don't conflict.
- Adds a test that typed shortcuts follow `importPunctuation`. With the default, typing `中文**「强调」**` applies bold and `中文~~「强调」~~` applies strikethrough. A strict import leaves both literal.
- The changeset mentions the shortcuts and the newer `micromark-core-commonmark`. Its bundle figure is updated: 136.3 KB to 143.3 KB minified, 41.7 KB to 44.8 KB gzipped, and 0.7 KB of the increase is the newer dependency.
## Test plan
### Before
With 2.0.3 linked in:
```
× reads a `*` run beside a `_` run alike in both import modes
- "foo**",
+ "foo",
"bar",
- 2,
+ 3,
- "**baz",
+ "baz",
```
The new shortcut test, run on main:
```
× applies emphasis beside CJK punctuation as importPunctuation reads it
AssertionError: expected '中文**「强调」**' to be '中文「强调」' // Object.is equality
```
### After
```
Test Files 55 passed (55)
Tests 1804 passed (1804)
```
The command was `vitest run --project unit packages/lexical-mdast packages/lexical-markdown packages/lexical-internal packages/lexical-playground`. `treeShakingSource.test.ts` and `tsc -p ./tsconfig.test.json` are also clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
This looks good. The options objects are additive and positional booleans keep working. Both importers read CJK emphasis the way the CommonMark CJK-friendly amendment's reference implementations do, and the changeset covers the new defaults. A few optional, non-blocking suggestions:
-
Quadratic scan in
getTrailingBackslashes(packages/lexical-mdast/src/MdastExport.ts:610). When a run of backslashes is not right before the last character,/\\+(?=[^\p{P}\p{S}\r\n]$)/urestarts at every backslash in it. Export time therefore grows with the square of the run: a text node holding 40,000 backslashes takes about 3.6 s to export on this branch, against a few ms onmain. Anchoring the start of the run, as in/(?<!\\)\\+(?=[^\p{P}\p{S}\r\n]$)/u, gives the same matches and stays linear. This is worth folding in before merge. -
Strikethrough after a variation selector (
packages/lexical-mdast/README.md:194). Both packages close~~after a selector, as inx ~~Done ✔︎~~ y.micromark-extension-cjk-friendly-gfm-strikethroughdoes not close there, whilemarkdown-it-cjk-friendlydoes. The choice is consistent with how*is read, but the README says the mode reads "asmicromark-extension-cjk-friendlydoes". A short note on this one difference would help readers. -
PR description. Two parts are out of date:
- The Implementation section still says
@lexical/mdastimports throughmicromark-extension-cjk-friendlyandmicromark-extension-cjk-friendly-gfm-strikethrough(about 12 KB). The branch now uses in-repo constructs; the changeset already says so. - The Test plan's pasted counts (20 files, 1383 tests) predate the latest commits (22 and 1,420 now). AGENTS.md asks for the real Before and After output.
Also, "reads all other text exactly as CommonMark does" does not hold for a text-presentation selector at the inner edge of a run, as in
**©︎**2024. The changeset's Verify paragraph already explains that case. - The Implementation section still says
| */ | ||
| function getTrailingBackslashes(value: string): [number, number] { | ||
| const match = value.includes('\\') | ||
| ? /\\+(?=[^\p{P}\p{S}\r\n]$)/u.exec(value) |
There was a problem hiding this comment.
Optional, but worth fixing before merge. When a run of backslashes is not right before the last character, this pattern restarts at every backslash in it and backtracks over the rest of the run each time. Export time is therefore quadratic in the run: one text node with 40,000 backslashes exports in about 3.6 s, against 7 ms on main. A lookbehind keeps the matches the same and makes it linear:
Suggested change
| ? /\\+(?=[^\p{P}\p{S}\r\n]$)/u.exec(value) | |
| ? /(?<!\\)\\+(?=[^\p{P}\p{S}\r\n]$)/u.exec(value) |
|
|
||
| - `'cjk-friendly'` follows the | ||
| [CommonMark CJK-friendly amendment](https://github.com/tats-u/markdown-cjk-friendly), | ||
| as `micromark-extension-cjk-friendly` does. It reads Markdown as CommonMark |
There was a problem hiding this comment.
Optional: one place differs from micromark-extension-cjk-friendly. Here a variation selector right before a closing ~/~~ reads as the character it varies, so x ~~Done ✔︎~~ y closes; the reference strikethrough extension leaves it open, while markdown-it-cjk-friendly closes it. Worth a sentence here so that readers comparing outputs aren't surprised.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
CommonMark doesn't let
**open or close between a letter and punctuation, so**你好。**世界reads with literal asterisks in@lexical/mdast, and in@lexical/markdownsince #9430 followed CommonMark's Unicode punctuation. This adds a punctuation mode to both packages, following the CommonMark CJK-friendly amendment thatmarkdown-it-cjk-friendlyandmicromark-extension-cjk-friendlyimplement: Markdown reads as CommonMark reads it except beside Chinese, Japanese and Korean characters and punctuation, where markers open and close as their writer meant.The mode is chosen separately for import and export:
'cjk-friendly'. It accepts what CJK writers type and what Lexical exported before, and reads all other text exactly as CommonMark does.'strict-commonmark', writing a character reference where a marker needs one (**你好。**世界), so the output reads the same in every CommonMark, GFM and CJK-friendly reader.'cjk-friendly'writes such CJK text as typed (**你好。**世界) for consumers known to follow the amendment.API:
@lexical/markdown:$convertFromMarkdownString,$generateNodesFromMarkdownString,$convertToMarkdownStringand$convertSelectionToMarkdownStringaccept an options object in place of theshouldPreserveNewLinesboolean (with its own overload in TypeScript and Flow) (MarkdownImportOptions/MarkdownExportOptions, which also carryshouldPreserveNewLinesandshouldMergeAdjacentLines), withpunctuation: 'cjk-friendly' | 'strict-commonmark'. Positional booleans keep working. Nested conversions (e.g. table cells in a transformer) inherit the outer call's mode.@lexical/mdast:MdastExtensionconfig gainsimportPunctuationandexportPunctuation.Implementation:
@lexical/markdownclassifies flanking characters itself, with the amendment's CJK set (East Asian Width F/H/W minus emoji, plus Hangul, generated from the sameget-east-asian-width1.7 data the micromark extension uses), ideographic variation selectors, and the variation-selector lookback before a marker. A strict export is also checked against CJK-friendly readers, since import now defaults to one.@lexical/mdastimports throughmicromark-extension-cjk-friendlyandmicromark-extension-cjk-friendly-gfm-strikethrough(which replaces thegfmStrikethrough()syntaxMdastStrikethroughExtensioncontributes in that mode). These add about 12 KB minified (5 KB gzipped) to@lexical/mdast.get-east-asian-widthresolves to 1.7.0, the data the@lexical/markdowntable was generated from. CJK-friendly export takes the strict export and writes the referencesmdast-util-to-markdownput beside a*or~run as the CJK characters they stand for; a CJK-friendly reader reads those runs the same either way, and references beside_, in code, or escaped are left alone.@lexical/mdastexport reads an emoji or other character outside the BMP as the punctuation CommonMark (and the amendment) count it as.mdast-util-to-markdownclassifies UTF-16 units, so it wrote no reference beside one andHello+ bold🎉 launchexported asHello**🎉 launch**, which a CJK-friendly import reads as plain text. Export now serializes with each such character standing in as an unused BMP symbol, so it writesHello**🎉 launch**; where micromark without the CJK-friendly extension would read the result differently, the emoji beside the run goes out as a reference as well.mdast-util-gfm-strikethroughis updated to 2.0.1, which serializes~~as attention, so strict export now writes references beside strikethrough as it does beside**(~~你好。~~世界used to come back as literal tildes in strict readers).Changeset
.changeset/markdown-cjk-friendly-punctuation.md, a[feature]patch for@lexical/markdownand@lexical/mdast, with aVerify:note on the import default and on mdast strikethrough references.Test plan
Before
**你好。**世界and中文**「强调」**imported with literal asterisks in both packages;@lexical/mdastexported~~你好。~~世界for strikethrough你好。before世界, which strict readers show with literal tildes.After
MarkdownPunctuation.test.tschecks that both importers read each case the same way in both modes (CJK punctuation, strikethrough, Hangul, ideographic and general variation selectors,_, non-CJK text), and that each export reads back through both importers in both modes.MdastPunctuation.test.tscovers the mdast export modes with both imports, whitespace references,_, emoji edges, code and escaped references. Random paragraphs of CJK, Latin, Thai, emoji and punctuation also round-trip through@lexical/mdastin the default, strict-import and CJK-friendly-export configurations.