How to Scrape Dynamic JavaScript Tables to Markdown with Python (No Pandas Needed)

Legal Disclaimer

The content, methodologies, and code snippets provided in this article are intended strictly for educational and informational purposes only. Web scraping may be subject to legal regulations, including but not limited to the Computer Fraud and Abuse Act (CFAA) and regional data protection laws (such as GDPR or CCPA). The author and publisher do not condone, endorse, or encourage the unauthorized extraction of data, circumvention of security measures, or violation of any website’s Terms of Service (ToS) or robots.txt directives. Readers are solely responsible for ensuring that their automated data collection practices are legally compliant and ethically sound. The author assumes no liability for any misuse of the provided code or any resulting damages or legal repercussions.

Your scraper pulls an empty table. The HTML looks fine in the browser, but requests.get() hands you a shell with no rows inside. That's not a bug in your code. It's a dynamic table, and it needs a different plan.

Most tutorials solve half the problem. They show you how to grab the data, then dump it into pandas and call it done. Fine, if you already have pandas installed and don't mind the extra 100MB+ stack dependency for a task that's really just "turn some rows into text." Here's a leaner way: find where the table actually gets its data, pull that directly when you can, and write the Markdown yourself. No pandas required.

What Is Dynamic Table Scraping?

Simply put, dynamic table scraping is the process of extracting row and column data from a web table that JavaScript builds after the page loads, rather than data that's already sitting in the raw HTML. Since the data arrives through a background request or gets injected by a script, a plain HTTP fetch of the page returns an empty shell, so you need to either intercept that background request or render the page in a browser first.

Why Skip Pandas for This

Pandas is fantastic when you're analyzing data. It's overkill when you're just converting rows into pipe-separated text for a README or a report. Every pd.read_html() call spins up a full DataFrame engine, pulls in numpy as a dependency, and still needs a second step to format the output as Markdown, since DataFrames don't emit Markdown by default without yet another package.

Having built dozens of these small extraction scripts for internal tooling, the classic mistake we see is reaching for pandas out of habit, then wondering why a "simple" scraper takes 400ms just to import. For a scheduled job or a CI step that scrapes one table and writes one file, that overhead adds up, and it's dead weight you're carrying for a task that a dictionary and a loop can handle in a few lines.

Python's footprint in exactly this kind of automation work keeps growing. According to the 2025 Stack Overflow Developer Survey, Python usage climbed to 57.9% of respondents this year, a seven-point jump from 2024, which the survey ties directly to its role in back-end and automation tasks. More of that work is happening on the web side too: the JetBrains and Python Software Foundation's 2025 developer survey found that 46% of Python developers now use it for web development, reversing a multi-year decline. Translation: more people are writing exactly this kind of scraping snippet, and fewer of them need a data-science stack to do it.

Why requests.get() Returns an Empty Table

Here's the direct answer: requests.get() only downloads the initial HTML document. If the table rows are added afterward by JavaScript, whether through a fetch call, a WebSocket, or a client-side templating library, those rows simply don't exist in the response you receive. You're looking at the blueprint, not the finished room.

Think of it like ordering food delivery and receiving only the empty takeout bag. The restaurant sent the container first and the food is still being packed. Your job is to figure out where the "food" (the actual data) is coming from, not to keep staring at an empty bag.

Finding the Hidden Data Feed

Most dynamic tables pull their data from a JSON endpoint the page calls in the background. Find it and you can skip rendering JavaScript entirely, which is faster and far less fragile than automating a browser.

Here's how to find it, step by step:

Open DevTools and filter by Fetch/XHR: load the page, open the Network tab, filter to Fetch/XHR, then reload. You're looking for a request that returns JSON matching what's on screen.
Inspect the response shape: click the request, check the Response tab. If you see an array of objects with your table's fields, you found it.
Copy the request details: right-click the request, copy it as cURL, and note the URL, headers, and any query parameters or pagination tokens.
Rebuild the request in Python: replicate the URL and headers with requests.

Method	Speed	When to use it
Hidden JSON feed	Fast, one request	Any table backed by an XHR/fetch call (most dashboards, admin panels)
Playwright fallback	Slower, full browser	Data is computed client-side or the feed is obfuscated

Here's the request-replication approach in code:

    Python
    
  

import requests

url = "https://example.com/api/employees"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept": "application/json",
}

response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()
rows = response.json()["data"]  # adjust the key to match the real payload

print(rows[0])  # confirm the shape before building the table
  

When There's No Hidden Feed: The Playwright Fallback

Sometimes the table really is built entirely in the browser, with no separate data call to intercept. Search-heavy grids that compute rows from a bundled dataset, or pages behind aggressive obfuscation, fall into this category. That's when you reach for Playwright, but only then; running a full browser for a page that has a plain JSON endpoint is like hiring a moving truck to carry one box.

    Python
    
  

from playwright.sync_api import sync_playwright

def get_rendered_rows(url, table_selector="table#data"):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")
        page.wait_for_selector(table_selector)

        rows = page.eval_on_selector_all(
            f"{table_selector} tbody tr",
            "trs => trs.map(tr => Array.from(tr.children).map(td => td.innerText.trim()))"
        )
        browser.close()
        return rows
  

Notice this pulls text directly from the DOM with eval_on_selector_all, no BeautifulSoup parsing pass needed afterward. One render, one extraction, done.

Writing the Markdown Table Without a Library

This is where most tutorials hand the job to markdownify or a similar package. That's fine for converting whole documents, but for a table alone, a formatter is maybe 15 lines, and owning those 15 lines means you control column alignment, header detection, and how you escape stray pipe characters in cell values.

    Python
    
  

def rows_to_markdown(headers, rows):
    def escape(cell):
        return str(cell).replace("|", "\\|").replace("\n", " ").strip()

    header_line = "| " + " | ".join(escape(h) for h in headers) + " |"
    divider_line = "| " + " | ".join("---" for _ in headers) + " |"
    body_lines = [
        "| " + " | ".join(escape(cell) for cell in row) + " |"
        for row in rows
    ]
    return "\n".join([header_line, divider_line] + body_lines)

headers = ["Name", "Position", "Office", "Start Date"]
data_rows = [
    ["Airi Satou", "Accountant", "Tokyo", "2008-11-28"],
    ["Angelica Ramos", "CEO", "London", "2009-10-09"],
]

print(rows_to_markdown(headers, data_rows))
  

That's the entire conversion step. No DataFrame, no extra imports beyond the standard library, and the escaping logic protects you from a broken table the moment someone's job title contains a pipe character.

Pro Tip: Always run len(row) == len(headers) as a check before formatting. Dynamic tables love to serve a ragged row when a field is empty, and a single short row silently shifts every column after it. Catching the mismatch before you write the Markdown saves you from debugging a table that "looks fine" but reads wrong three columns in.

Common Problems and How to Fix Them

Most of what goes wrong here isn't really about scraping. It's about what happens after you have the rows.

Merged or spanning cells: Markdown tables have no concept of colspan or rowspan. If the source HTML uses them, decide upfront whether to repeat the merged value across cells or flatten it into one column with a separator. Repeating it is usually the safer default since it keeps row counts consistent.

Pagination that hides most of the data: if you scrape 20 rows but the page says 500, check the JSON response for a page or offset parameter and loop through it. Most undocumented backend endpoints handle pagination similarly to the frontend UI's 'Next' button.

Race conditions in Playwright: if rows come back empty even with wait_for_selector, the table might load its data after the selector appears but before it populates. Add a second wait on the row count itself, like page.wait_for_function("document.querySelectorAll('table#data tbody tr').length > 0"), instead of guessing with a fixed sleep.

Turning This into a Reusable Automation Snippet

A script you run once is a script. A function you can drop into a scheduled job, a CI step, or a bigger pipeline is a snippet. The difference is mostly error handling and a consistent interface, and it's worth building once so you're not rewriting this same logic for the next site.

    Python
    
  

import requests
from playwright.sync_api import sync_playwright

def scrape_table_to_markdown(url, api_url=None, table_selector="table#data"):
    """Try the hidden API first, fall back to Playwright, return Markdown."""
    if api_url:
        try:
            resp = requests.get(api_url, timeout=10)
            resp.raise_for_status()
            data = resp.json()["data"]
            headers = list(data[0].keys())
            rows = [list(item.values()) for item in data]
            return rows_to_markdown(headers, rows)
        except (requests.RequestException, KeyError, IndexError):
            pass  # fall through to the browser method

    rendered = get_rendered_rows(url, table_selector)
    headers, rows = rendered[0], rendered[1:]
    return rows_to_markdown(headers, rows)
  

Save that alongside the two helper functions above and you have a self-contained snippet: point it at a page, optionally give it the API URL you found in DevTools, and get Markdown back. If the API changes or disappears, it quietly falls back to rendering the page instead of crashing your pipeline at 3 a.m.

Frequently Asked Questions

Can I scrape a dynamic table without Selenium or Playwright at all?

Yes, whenever the table pulls data from a JSON endpoint. Intercept that request with requests and you skip browser automation entirely. A headless browser is only necessary when the data is genuinely computed or assembled client-side with no separate feed to call.

Why does markdownify sometimes produce a broken table?

Markdownify converts general HTML, so it struggles with merged cells, nested tags inside <td> elements, and rows with inconsistent column counts, since none of those map cleanly to Markdown's flat grid format. Writing your own formatter lets you catch and normalize those cases before conversion instead of after.

Is it legal to scrape a table like this?

Web scraping operates in a complex legal landscape (including laws like the CFAA in the US). You must strictly adhere to the target website’s Terms of Service (ToS) and robots.txt. Do not scrape personally identifiable information (PII) or copyrighted intellectual property. The code provided is for educational purposes only.

What if the hidden API requires authentication?

If an endpoint expects a session context, ensure you are testing against your own servers or have explicit permission to utilize session cookies via requests.Session() in your automated workflows. If the token is short-lived or tied to a JavaScript-generated signature, that's usually a sign the site wants you on the Playwright path instead.

How do I handle tables that update in real time?

For a live table backed by WebSockets, snapshot the data at read time rather than trying to track every update. Connect, capture the current state once the connection confirms it's synced, then close it. A Markdown table is a snapshot format anyway, so treat it like one.

You've got the core move now: check for the hidden feed before you reach for a browser, and write your own Markdown formatter instead of installing one. Start with the DevTools Network tab on the next dynamic table you need. Which of your current scrapers is still hauling pandas around for a job this small?