Convert a Website to Markdown: Tested on 20 Pages

To convert a website to Markdown, fetch the HTML, cut it down to the main content, and convert that content with an HTML-to-Markdown library. In Node.js the reliable pipeline is Mozilla Readability for the extraction and Turndown for the conversion:

npm install @mozilla/[email protected] [email protected] [email protected] [email protected] [email protected]
node to-markdown.mjs https://go.dev/doc/effective_go > effective-go.md
node to-markdown.mjs https://example.com/ --render > example.md   # for pages built by JavaScript

The pipeline is easy to get subtly wrong: a page converts without an error and quietly loses its headings or tables. So we ran it on 20 public pages, scored every output against the page itself, and fixed what broke.

A note on scope: RenderScreenshot renders pages to images and PDFs; it does not return Markdown. This tutorial uses open-source libraries. Our API appears in one section, for screenshots to keep beside each file; to go the other way, see how to render Markdown as an image.

Tested on 2026-10-18 with Node.js 22.18.0, @mozilla/readability 0.6.0, jsdom 30.1.3, turndown 7.2.4, turndown-plugin-gfm 1.0.2 and Puppeteer 25.14.0 (Chrome 155); Python 3.12.15 with requests 2.34.2, readability-lxml 0.9, markdownify 1.2.3 and trafilatura 2.3.1. macOS 27.0.1 on an Apple M1 Pro, residential connection in North America.

The pipeline in four steps

1. Fetch the HTML, or load the page in headless Chrome when JavaScript builds the content.
2. Clean the DOM: remove what confuses the extractor and unwrap what it would wrongly discard. Most of our score gains came from this step.
3. Extract the main content with Readability, the library behind Firefox Reader View.
4. Convert the article with Turndown. The GFM plugin adds tables and strikethrough; a custom rule keeps code-block languages.

Do not skip step 3 for LLM input: converting the whole <body> puts menus, footers and cookie text in every file.

The converter

// to-markdown.mjs - Convert a web page to Markdown: fetch (or render) -> clean -> Readability -> Turndown.
// Usage: node to-markdown.mjs <url> [--render] > page.md
import { JSDOM } from 'jsdom';
import { Readability } from '@mozilla/readability';
import TurndownService from 'turndown';
import { gfm } from 'turndown-plugin-gfm';

export async function fetchHtml(url) {
  const res = await fetch(url, {
    headers: { 'User-Agent': 'md-export/1.0 (+https://example.com/bot)', Accept: 'text/html' },
    redirect: 'follow',
    signal: AbortSignal.timeout(20_000),
  });
  if (!res.ok) throw new Error(`GET ${url} -> HTTP ${res.status}`);
  return { html: await res.text(), finalUrl: res.url };
}

export async function renderHtml(url, browser) {
  const page = await browser.newPage();
  try {
    await page.goto(url, { waitUntil: 'networkidle2', timeout: 30_000 });
    return { html: await page.content(), finalUrl: page.url() };
  } finally {
    await page.close();
  }
}

// Fix the DOM before Readability scores it. Each step fixes a failure seen in our 20-page run.
export function cleanDom(doc) {
  // 1. Cookie and consent banners that JavaScript injects (the rendered GOV.UK page).
  for (const el of doc.querySelectorAll('[id*="cookie" i], [class*="cookie" i], [aria-label*="cookie" i]')) el.remove();
  for (const h of doc.querySelectorAll('h1, h2, h3, h4, h5, h6')) {
    // 2. Permalink anchors inside headings: drop the pilcrow, keep the heading text.
    for (const a of h.querySelectorAll('a[href^="#"]')) {
      const text = a.textContent.trim();
      if (['', '#', '¶', '§'].includes(text)) a.remove();
      else a.replaceWith(text);
    }
    // 3. Headings wrapped in a <div> with an edit link (Wikipedia, W3C specs):
    //    Readability drops the wrapper, and the heading with it.
    const p = h.parentElement;
    if (p?.tagName === 'DIV' && p.textContent.trim().length < h.textContent.trim().length + 40) p.replaceWith(h);
  }
  // 4. Block elements inside table cells break GFM table rows (Sphinx docs).
  for (const cell of doc.querySelectorAll('td, th')) {
    for (const block of cell.querySelectorAll('p, div')) block.replaceWith(...block.childNodes, ' ');
  }
}

function absolutize(doc) {
  // JSDOM resolves .href/.src against the page URL; Turndown reads the raw attribute.
  for (const a of doc.querySelectorAll('a[href]')) a.setAttribute('href', a.href);
  for (const img of doc.querySelectorAll('img[src]')) img.setAttribute('src', img.src);
}

export function htmlToMarkdown(html, url) {
  const dom = new JSDOM(html, { url });
  const doc = dom.window.document;
  cleanDom(doc);
  absolutize(doc);
  const article = new Readability(doc).parse();
  if (!article || !article.content) return null;

  const td = new TurndownService({ headingStyle: 'atx', codeBlockStyle: 'fenced', bulletListMarker: '-' });
  td.use(gfm);
  // Keep the language of highlighted code blocks: <pre><code class="language-js">
  td.addRule('fencedWithLanguage', {
    filter: (node) => node.nodeName === 'PRE',
    replacement: (_content, node) => {
      const code = node.querySelector('code') ?? node;
      const lang = (code.className.match(/(?:language|lang)-([\w+-]+)/) ?? [])[1] ?? '';
      const text = code.textContent.replace(/\n$/, '');
      const fence = text.includes('```') ? '````' : '```';
      return `\n\n${fence}${lang}\n${text}\n${fence}\n\n`;
    },
  });
  td.remove(['script', 'style', 'noscript', 'iframe', 'form']);

  const body = td.turndown(article.content).replace(/\n{3,}/g, '\n\n').trim();
  const front = [`# ${article.title ?? ''}`.trim(), `Source: ${url}`].join('\n\n');
  return { title: article.title, markdown: `${front}\n\n${body}\n` };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const [url, flag] = process.argv.slice(2);
  if (!url) {
    console.error('usage: node to-markdown.mjs <url> [--render]');
    process.exit(2);
  }
  let browser;
  try {
    let page;
    if (flag === '--render') {
      const puppeteer = (await import('puppeteer')).default;
      browser = await puppeteer.launch();
      page = await renderHtml(url, browser);
    } else {
      page = await fetchHtml(url);
    }
    const result = htmlToMarkdown(page.html, page.finalUrl);
    if (!result) throw new Error('Readability found no article content');
    process.stdout.write(result.markdown);
  } catch (err) {
    console.error(err.message);
    process.exitCode = 1;
  } finally {
    await browser?.close();
  }
}

Three details that are easy to miss:

• Links are made absolute first. JSDOM resolves a.href, but Turndown reads the raw attribute. Without absolutize, links come out as paths like /docs/intro that break once the file leaves the site.
• The fence grows when the code contains a fence. Code that contains three backticks needs a four-backtick fence, or the block ends early.
• Errors set the exit code. A 403 or a page with no article fails loudly instead of writing an empty file that an ingestion job indexes.

What we measured: 20 pages, five pipelines

We picked 20 public pages of different kinds: reference docs (MDN, Python, PostgreSQL, Rust, Go), framework guides (React, Vue, Svelte, Angular, Tailwind), long specs (WCAG 2.2, the Markdown syntax page), an encyclopedia article, two listing pages (Hacker News, a GOV.UK browse page), a client-side app (TodoMVC), example.com and one page known to block bots (a Stack Overflow question).

Method. For each URL we loaded the page in headless Chrome and counted, inside its main or article element (or body if neither exists), the words, the h2 to h6 headings, the pre blocks and the tables. That is the reference. For each pipeline's Markdown we counted words, ATX headings of level 2 or more, fenced code blocks and GFM tables. Each measure scores min(1, markdown / reference), and the page score is the mean of the measures the reference actually has. A score of 1.00 means nothing structural was lost, not that the output is free of boilerplate; we checked that by reading the files. One run per pipeline on 2026-10-18.

The first two columns run the same Node pipeline without and with cleanDom. Pages at 0.9 or more: 7 naive, 9 cleaned, 11 cleaned with Chrome. TodoMVC scores 1.00 only because its reference has almost no text.

Timing, as medians: the static path took 0.4 to 0.5 s per page including the download (two runs). With Chrome, loading to network idle took about 1.5 s, plus about 0.1 s to convert. Rendering every page would make a crawl several times slower for a gain on two pages of 20.

Four failures and the fixes

Headings that vanished (Wikipedia kept 2 of 17, WCAG 3 of 141). Both sites wrap each heading in a div together with an edit link or a self link. Readability treats that small, link-heavy wrapper as clutter and removes it, heading included; the section text stays, so the output looks fine at a glance. Replacing the wrapper with the heading before extraction kept 15 of 17 headings on Wikipedia and all 141 on WCAG. This one fix moved WCAG from 0.51 to 1.00.

Pilcrows and self links in headings. Sphinx (Python docs) and MDN put a ¶ link or a link to #the-heading inside every heading. Turndown converts them faithfully, so you get ## json.dumps[¶](https://...). Removing anchors whose text is a symbol, and unwrapping the rest, gives clean headings, which matter when you later split the file by section.

Tables that fell apart. The Python docs put a <p> inside each table cell. GFM table rows must be one line, so Turndown's output became a table broken across many lines that no Markdown parser reads as a table: 0 of 2 tables survived. Flattening blocks inside cells to inline text fixed both tables.

A cookie banner instead of the page. The static GOV.UK page converted well enough. The rendered one did not: JavaScript inserted a consent banner, Readability picked it as the article, and the Markdown file was 59 words about cookies (score 0.12 before the fix, 0.53 after). The fix removes elements whose id, class or aria-label mention cookies. It is a blunt rule. On a recipe site, check that it does not remove the content you want; a site-specific selector list is safer when you crawl one domain. Our post on blocking cookie banners at scale covers the same problem for screenshots.

What the fixes did not solve: listing pages. Hacker News, the GOV.UK browse page and the card-heavy Angular overview are not articles, so Readability keeps one block and drops the rest. Write a site-specific extractor for those, or skip them.

Static fetch or headless Chrome?

The rendered column beats the static one on average, but look at where it helps. On example.com, a script now adds the same notice in several languages after load, so the static HTML has 30 words and the rendered page 138. On Effective Go, rendering recovered a table that the static conversion had missed. On every other page the static and rendered results scored the same, or the static one was better. On MDN, the rendered DOM had no pre elements left after the page's scripts ran, and the Markdown went from 24 code blocks to none. Our score could not see that loss, because the reference comes from the same rendered page. Reading the files did.

So the rule we use is static first, Chrome as a fallback when the result is close to empty:

// convert.mjs - Static first, headless Chrome only when the static result looks empty.
// Usage: node convert.mjs <url> [<url> ...]  (writes one .md file per URL)
import { writeFile } from 'node:fs/promises';
import puppeteer from 'puppeteer';
import { fetchHtml, renderHtml, htmlToMarkdown } from './to-markdown.mjs';

const MIN_WORDS = 200; // below this, assume the content is built by JavaScript
const wordCount = (md) => md.split(/\s+/).filter(Boolean).length;

let browser; // launched on first need, then reused
async function convert(url) {
  try {
    const page = await fetchHtml(url);
    const result = htmlToMarkdown(page.html, page.finalUrl);
    if (result && wordCount(result.markdown) >= MIN_WORDS) return { ...result, via: 'static' };
  } catch (err) {
    console.error(`static fetch failed for ${url}: ${err.message}`);
  }
  browser ??= await puppeteer.launch();
  const page = await renderHtml(url, browser);
  const result = htmlToMarkdown(page.html, page.finalUrl);
  if (!result) throw new Error(`no readable content at ${url}`);
  return { ...result, via: 'rendered' };
}

try {
  for (const url of process.argv.slice(2)) {
    try {
      const { markdown, via } = await convert(url);
      const file = `${new URL(url).hostname}${new URL(url).pathname}`.replace(/[^\w.-]+/g, '_') + '.md';
      await writeFile(file, markdown);
      console.log(`${via.padEnd(8)} ${wordCount(markdown)} words -> ${file}`);
    } catch (err) {
      console.error(`FAILED ${url}: ${err.message}`);
      process.exitCode = 1;
    }
  }
} finally {
  await browser?.close();
}

On four of our test URLs it chose the static path for MDN (4,763 words, code intact), Chrome for example.com and TodoMVC, and failed loudly on Stack Overflow. Tune MIN_WORDS to your sources. For apps that load data after the first paint, networkidle2 can be too early; the waits in our guide to screenshots of single-page applications apply here too.

Python: readability-lxml or trafilatura

We tested two Python stacks: readability-lxml with markdownify, and trafilatura, which extracts and writes Markdown in one call. trafilatura averaged 0.59 and readability-lxml 0.49, both well below the Node pipeline's 0.79 on the same static HTML. readability-lxml picked the wrong block on several large docs pages (441 words of WCAG's 22,000; 638 words of the Python json docs) and scored 0.9 or more on only 5 pages. trafilatura was steadier but lost headings on long specs and most code blocks on PostgreSQL. If you can call Node from your Python service, the Node pipeline is the better extractor. If you must stay in Python, start with trafilatura:

"""to_markdown_traf.py - Convert a web page to Markdown with trafilatura.

Usage: python to_markdown_traf.py <url> > page.md
pip install requests==2.34.2 trafilatura==2.3.1
"""
import sys

import requests
import trafilatura


def page_to_markdown(url: str) -> str:
    res = requests.get(url, headers={'User-Agent': 'md-export/1.0 (+https://example.com/bot)'}, timeout=20)
    res.raise_for_status()
    # Without a charset in Content-Type, requests decodes HTML as ISO-8859-1.
    if 'charset' not in res.headers.get('Content-Type', '').lower():
        res.encoding = res.apparent_encoding
    md = trafilatura.extract(
        res.text,
        url=res.url,
        output_format='markdown',
        include_links=True,
        include_tables=True,
        include_formatting=True,
        include_comments=False,
    )
    if not md:
        raise ValueError('no main content found')
    return f'Source: {res.url}\n\n{md.strip()}\n'


if __name__ == '__main__':
    if len(sys.argv) != 2:
        sys.exit('usage: python to_markdown_traf.py <url>')
    try:
        sys.stdout.write(page_to_markdown(sys.argv[1]))
    except (requests.RequestException, ValueError) as err:
        sys.exit(f'failed: {err}')

The encoding lines are there because of a real failure. The Rust book serves Content-Type: text/html with no charset, requests then decodes the bytes as ISO-8859-1, and every curly apostrophe in the Markdown came out as mojibake: "We’ll" became "Weâll". Setting res.encoding from the detected encoding fixed it. The same applies to readability-lxml; any Python pipeline that starts with res.text needs the check.

Preparing the Markdown for an LLM

The extractor gets you clean text. A few more steps make it useful as model input or for retrieval:

• Keep the source in the file. Both scripts write a Source: line, so you can cite and re-fetch the page.
• Split on headings, not on character counts. Chunks that start at ## keep a section's code and tables together, which is why the heading fixes matter.
• Drop citation noise. Wikipedia output keeps [\[1\]](https://...) links in most sentences; a regular expression that removes bracketed numeric links is enough.
• Respect the site. Check robots.txt and the terms, send an honest User-Agent with a contact URL, and cache what you fetched.

Keep the visual context: a screenshot next to each file

Markdown drops everything visual: charts, diagrams and the layout that shows what matters. If a model or a reviewer needs that context, store an image of the page next to the text. With RenderScreenshot that is one request per URL, and a JSON response also carries the live page's title and description for the front matter:

// snapshot.mjs - Pair a Markdown export with a screenshot and the page's own title and description.
export async function snapshot(url) {
  const res = await fetch('https://api.renderscreenshot.com/v1/screenshot', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.RS_API_KEY}`,
      'Content-Type': 'application/json',
      Accept: 'application/json', // JSON body with a CDN link instead of image bytes
    },
    body: JSON.stringify({
      url,
      viewport: { width: 1280, height: 800 },
      output: { format: 'webp' },
      cache: { ttl: 2592000 }, // keep the CDN copy for the 30-day maximum
    }),
    signal: AbortSignal.timeout(60_000),
  });
  if (!res.ok) throw new Error(`screenshot ${url}: ${res.status} ${await res.text()}`);
  const data = await res.json();
  return { image: data.image.url, title: data.metadata.title, description: data.metadata.description };
}

export function frontMatter(url, shot) {
  const q = (s) => JSON.stringify(s ?? ''); // JSON strings are valid YAML scalars
  return `---\nsource: ${q(url)}\ntitle: ${q(shot.title)}\ndescription: ${q(shot.description)}\nscreenshot: ${q(shot.image)}\n---\n\n`;
}

Call snapshot(url) next to convert(url) and prepend frontMatter(url, shot) to the Markdown before you write the file. We checked the request body and the JSON handling against a local stand-in server, not the live API. The CDN link lasts as long as the cache TTL (30 days at most, per the caching docs), so download the image if you need it for longer. The metadata fields are documented in metadata. Sites that block bots may block the screenshot as well; screenshots of sites behind Cloudflare explains what works and what does not.

Our comparison of screenshots, DOM text and the accessibility tree goes further into which form a model should get for which job. And to check a batch of conversions by eye, you can turn the Markdown back into a picture: the Markdown to image tutorial shows that with local scripts and with one API call.

Troubleshooting

Next steps

The scores above come from one run on 2026-10-18 with the versions pinned above. Sites change their markup, so run the same method on your own sources before you rely on a pipeline. To add screenshots to your own exports, create a free account and set RS_API_KEY.