---
name: crawl4ai-scraping
description: Asynchronous web crawling and structured data extraction for LLMs using Crawl4AI and Playwright.
---

# Crawl4AI Intelligent Web Scraping

High-performance, LLM-ready web crawling, markdown extraction, and structured scraping using Crawl4AI and Playwright.

## When to Use

- Extracting clean, readable content and articles from web pages without clutter (headers, footers, cookie banners, navigation menus, ads).
- Structured data extraction into Pydantic models / JSON schemas for research, affiliate product comparison, or dataset enrichment.
- Crawling JavaScript-heavy single-page applications (React, Next.js, Vue) requiring headless browser execution.
- Scraping travel itineraries, trail guides, accommodation details, and POI data for blog content.

---

## 1. Quick Setup

```bash
uv pip install crawl4ai playwright
playwright install chromium
```

---

## 2. Basic Markdown Extraction

Crawl4AI provides built-in markdown generation specifically optimized for LLM token efficiency:

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler(verbose=True) as crawler:
        result = await crawler.arun(url="https://example.com/travel-guide")
        
        # Raw markdown
        print(result.markdown)
        
        # Fit markdown (heuristically pruned of navigational boilerplate)
        print(result.fit_markdown)

if __name__ == "__main__":
    asyncio.run(main())
```

---

## 3. Structured Extraction with CSS / XPath

Extract repeated items (such as affiliate products, hotel listings, or hiking stages):

```python
import asyncio
import json
from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

schema = {
    "name": "Stage Listings",
    "baseSelector": ".stage-item",
    "fields": [
        {"name": "stage_number", "selector": ".stage-num", "type": "text"},
        {"name": "title", "selector": "h3", "type": "text"},
        {"name": "distance_km", "selector": ".distance", "type": "text"},
        {"name": "elevation_gain", "selector": ".elevation-up", "type": "text"},
        {"name": "link", "selector": "a.detail-link", "type": "attribute", "attribute": "href"}
    ]
}

async def scrape_stages(url: str):
    strategy = JsonCssExtractionStrategy(schema, verbose=True)
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url=url,
            extraction_strategy=strategy,
            bypass_cache=True
        )
        data = json.loads(result.extracted_content)
        return data

if __name__ == "__main__":
    stages = asyncio.run(scrape_stages("https://example.com/etappen"))
    print(json.dumps(stages, indent=2))
```

---

## 4. Handling Dynamic Content & JavaScript

When pages require scrolling, clicking "Load More", or waiting for specific DOM selectors:

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def scrape_dynamic():
    js_instructions = [
        "window.scrollTo(0, document.body.scrollHeight);",
        "await new Promise(r => setTimeout(r, 2000));"
    ]
    
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com/infinite-scroll-list",
            js_code=js_instructions,
            wait_for="css:.loaded-content-card",
            delay_before_return_html=1.5
        )
        return result.markdown
```

---

## 5. Respectful & Stealth Crawling Best Practices

1. **Rate Limiting & Delays:** Never hammer target servers with uncontrolled concurrent loops. Always introduce jitter (`asyncio.sleep(1.0 + random.random() * 2.0)`).
2. **User-Agent & Headers:** Configure legitimate desktop user-agent headers:
   ```python
   headers = {
       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
       "Accept-Language": "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"
   }
   ```
3. **Cache Policy:** Use Crawl4AI's cache during local testing (`bypass_cache=False`) to avoid repeated network requests.
4. **Robot Rules:** Always inspect `robots.txt` before massive bulk scraping.