• Turndown.js๋Š” HTML์„ Markdown์œผ๋กœ ๋ณ€ํ™˜ํ•˜๋Š” JavaScript ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ
  • Element/Document/DocumentFragment ๋…ธ๋“œ๋ฅผ ์ž…๋ ฅ์œผ๋กœ ๋ฐ›์•„ markdown ๋ฌธ์ž์—ด์„ ๋ฐ˜ํ™˜ํ•˜๋Š” ๋ณ€ํ™˜๊ธฐ
  • addRule() / use()๋กœ ๊ทœ์น™๊ณผ ํ”Œ๋Ÿฌ๊ทธ์ธ์„ ํ™•์žฅํ•  ์ˆ˜ ์žˆ๋Š” ๊ตฌ์กฐ

ํ•ด๋‹น ๊ฐœ๋…์ด ํ•„์š”ํ•œ ์ด์œ 

  • Readability.js๊ฐ€ ๋ณธ๋ฌธ๋งŒ ๋‚จ๊ธด HTML์„ ๋„˜๊ฒจ๋„, ๊ทธ HTML์„ ๊ทธ๋Œ€๋กœ ์ €์žฅํ•˜๋ฉด LLM-wiki Ingest flow์˜ markdown ๊ธฐ๋ฐ˜ chunk ๊ทœ์น™(heading ๊ธฐ์ค€ ๋ถ„ํ• )์— ๋งž์ง€ ์•Š๋Š”๋‹ค
  • ํ‘œ๋ฅผ ๊ธฐ๋ณธ ์˜ต์…˜์œผ๋กœ ๋ณ€ํ™˜ํ•˜๋ฉด ๊ตฌ์กฐ๊ฐ€ ๊นจ์ง€๋ฏ€๋กœ turndown-plugin-gfm ๊ฐ™์€ ํ”Œ๋Ÿฌ๊ทธ์ธ์œผ๋กœ ํ‘œ๋ฅผ ๋ณด์กดํ•ด์•ผ ํ•œ๋‹ค โ€” ์ด๊ฒŒ ๊นจ์ง€๋ฉด โ€œ์ œ3์กฐ 2ํ•ญโ€ ๊ฐ™์€ ์ •ํ™•ํ•œ ํ‚ค์›Œ๋“œ๊ฐ€ RAG, LLM wiki ๊ด€๊ณ„์˜ FTS ์ธ๋ฑ์Šค์—์„œ ์‚ฌ๋ผ์ง„๋‹ค
  • HTML Ingest ๋ ˆ์ด์–ด์™€ LLM-Wiki ์ „์ฒด ๊ตฌ์กฐ์—์„œ๋Š” ์ด ์กฐํ•ฉ์„ ๊ธฐ๋ณธ ๊ตฌํ˜„์ฒด๋กœ ์“ฐ์ง€๋งŒ, ๋ ˆ์ด์–ด ์ธํ„ฐํŽ˜์ด์Šค(HTML in โ†’ ์ •๊ทœํ™” Markdown + ํ•ด์‹œ out)๋งŒ ์ง€ํ‚ค๋ฉด trafilatura ๊ฐ™์€ ๋‹ค๋ฅธ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋กœ ๊ต์ฒดํ•  ์ˆ˜ ์žˆ๋‹ค

AS-IS

<table><tr><th>์กฐํ•ญ</th><th>๋‚ด์šฉ</th></tr><tr><td>3-2</td><td>...</td></tr></table>

๊ธฐ๋ณธ ๋ณ€ํ™˜ ์‹œ ํ‘œ ๊ตฌ์กฐ๊ฐ€ ์œ ์ง€๋˜์ง€ ์•Š๊ณ  ํ…์ŠคํŠธ๊ฐ€ ํ•œ ์ค„๋กœ ๋ญ‰๊ฐœ์งˆ ์ˆ˜ ์žˆ๋‹ค.

TO-BE

const turndownService = new TurndownService();
turndownService.use(turndownPluginGfm.gfm); // ํ‘œ, ์ทจ์†Œ์„  ๋“ฑ GFM ์ง€์›
const markdown = turndownService.turndown(article.content);

์ฃผ์š” ์˜ต์…˜

์˜ต์…˜๊ฐ’
headingStylesetext | atx
bulletListMarker- | + | *
codeBlockStyleindented | fenced
linkStyleinlined | referenced

์ฐธ๊ณ  ๋ฌธ์„œ