> ## Documentation Index
> Fetch the complete documentation index at: https://docs.geekflare.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Web Scraping Guide
> Extract full page content from any URL as Markdown, JSON, HTML, or LLM-ready text. Supports JavaScript rendering, stealth mode, proxy routing, and structured data extraction.
The Web Scraping endpoint scrapes any URL and returns clean page content in your preferred format. It handles JavaScript sites, blocks ads, rotates proxies, and can extract structured data using CSS or XPath selectors.
**Endpoint:** `POST https://api.geekflare.com/webscraping`
Install the official SDK: `npm install @geekflare/api-node` or `pip install
geekflare-api`
Now, you can connect Web Scraping with your AI Agents or LLMs to give context. Refer to our [MCP Server](/mcp/overview) guide.
***
## Basic Scrape
Scrape a URL and get back LLM-ready content. Costs **1 credit**.
```typescript Node.js SDK theme={null}
import { GeekflareClient } from '@geekflare/api-node';
const client = new GeekflareClient({ apiKey: 'YOUR_API_KEY' });
const result = await client.webScrape({ url: 'https://example.com' });
console.log(result);
```
```python Python SDK theme={null}
from geekflare_api.client import GeekflareClient
from geekflare_api.models import WebScrapeDto
with GeekflareClient(api_key='YOUR_API_KEY') as client:
result = client.web_scrape(WebScrapeDto(url='https://example.com'))
print(result)
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
```
```json theme={null}
{
"timestamp": 1778737930991,
"apiStatus": "success",
"apiCode": 200,
"meta": {
"url": "https://example.com",
"device": "desktop",
"format": ["html-llm"],
"fileOutput": false,
"blockAds": true,
"renderJS": true,
"stealth": false,
"waitTime": 0,
"extractionMode": "default",
"test": { "id": "abc123" }
},
"data": "# Example Domain\n\nThis domain is for use in illustrative examples..."
}
```
***
## JavaScript Rendering
By default, `renderJS` is automatic: Geekflare first fetches the page without a browser, and only falls back to full JavaScript rendering if the page actually needs it. You don't need to choose between "lite" and "standard" scraping, the API detects it for you at no extra cost.
Set `renderJS: true` to always force full JavaScript rendering (skip the auto-detect and go straight to a headless browser), or `renderJS: false` to always skip rendering, even if the page would otherwise need it.
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com',
renderJS: true
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(url='https://example.com', render_js=True))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "renderJS": true}'
```
Leave `renderJS` unset for the best default experience. Only set it
explicitly when you already know a page requires (or doesn't require)
JavaScript and want to skip the auto-detection round trip.
***
## Output Formats
Choose one or more output formats. You can request up to 3 formats in a single call.
| Format | Description |
| - | - |
| `html-llm` | HTML stripped for LLM consumption (default) |
| `markdown` | Clean Markdown — recommended for most use cases |
| `html` | Raw HTML |
| `json` | Structured JSON |
| `markdown-llm` | Markdown stripped for LLM consumption |
| `text` | Plain text |
| `text-llm` | Plain text stripped for LLM consumption |
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com',
format: ['markdown', 'html', 'text']
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(
url='https://example.com',
format=['markdown', 'html', 'text']
))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "format": ["markdown", "html", "text"]}'
```
```json theme={null}
{
"timestamp": 1778737930991,
"apiStatus": "success",
"apiCode": 200,
"meta": {
"url": "https://example.com",
"format": ["markdown", "html", "text"],
"test": { "id": "abc123" }
},
"data": {
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"html": "Example Domain...",
"text": "Example Domain\n\nThis domain is for use in illustrative examples..."
}
}
```
***
## File Output
Get a CDN URL instead of inline content. Useful for large pages or when you need to store the result.
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com',
format: ['markdown'],
fileOutput: true
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(
url='https://example.com',
format=['markdown'],
file_output=True
))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "format": ["markdown"], "fileOutput": true}'
```
```json theme={null}
{
"timestamp": 1778737930991,
"apiStatus": "success",
"apiCode": 200,
"meta": {
"url": "https://example.com",
"format": ["markdown"],
"fileOutput": true,
"test": { "id": "abc123" }
},
"data": "https://cdn.geekflare.com/tests/webscraping/ZuyhINuAZPQQabbN.md"
}
```
***
## Stealth Mode
Bypass bot detection on protected pages. Slower but more reliable on heavily guarded sites.
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com',
stealth: true
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(url='https://example.com', stealth=True))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "stealth": true}'
```
***
## Wait Time
Add a delay after page load to capture lazy-loaded content or bypass bot checks.
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com',
waitTime: 2.5
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(url='https://example.com', wait_time=2.5))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "waitTime": 2.5}'
```
***
## Proxy Routing
Set `proxyCountry` to route the request through a proxy in a specific country — useful for bypassing geo-blocks or scraping region-specific content. It works on its own; you don't need to set `proxyMode`.
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com',
proxyCountry: 'gb'
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(
url='https://example.com',
proxy_country='gb'
))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "proxyCountry": "gb"}'
```
### Controlling when a proxy is used
Use the optional `proxyMode` parameter when you want to control *when* a proxy is used, rather than only where it exits:
| Mode | Behavior |
| - | - |
| `false` | Doesn't use a proxy (default when neither `proxyMode` nor `proxyCountry` is set). |
| `auto` | Tries the request without a proxy first, and automatically retries through a proxy if the site blocks the request. |
| `true` | Always routes the request through a proxy. |
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com',
proxyMode: 'auto'
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(
url='https://example.com',
proxy_mode='auto'
))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "proxyMode": "auto"}'
```
Proxy modes apply to the Web Scraping endpoint only. Other endpoints that
support proxy routing (Screenshot, Lighthouse, etc.) still use
`proxyCountry` alone — see [Using Proxies](/using-proxies).
***
## Device Emulation
Emulate a mobile device to scrape mobile-specific content.
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com',
device: 'mobile'
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(url='https://example.com', device='mobile'))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "device": "mobile"}'
```
***
## Structured Extraction — Templates
All structured extraction modes below return JSON automatically — you don't
need to set `format` to `json` when you set `extractionMode`.
For common data types, skip writing a custom CSS/XPath schema entirely. Set `extractionMode: 'template'` and pick a `template` to get structured JSON out of the box:
| Template | Extracts |
| - | - |
| `product` | Title, brand, category, description, ratings, and per-variant pricing/availability/images |
| `contact` | Company name, locations/addresses, phone numbers, emails, contact forms, and social profiles |
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://testingurl.dev/scraping/ecommerce/product/1',
extractionMode: 'template',
template: 'product'
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(
url='https://testingurl.dev/scraping/ecommerce/product/1',
extraction_mode='template',
template='product'
))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://testingurl.dev/scraping/ecommerce/product/1", "extractionMode": "template", "template": "product"}'
```
```json theme={null}
{
"timestamp": 1786985781147,
"apiStatus": "success",
"apiCode": 200,
"meta": {
"url": "https://testingurl.dev/scraping/ecommerce/product/1",
"device": "desktop",
"format": ["json"],
"extractionMode": "template",
"template": "product",
"test": { "id": "22c8fcbf-dcf9-42e5-aaec-a0a3df4b334a" }
},
"data": {
"json": {
"product": {
"title": "Vertex Air Laptop 107",
"brand": "Vertex",
"category": "laptops",
"url": "https://testingurl.dev/scraping/ecommerce/product/1",
"description": "Designed with a minimalist aesthetic.",
"aggregateRating": {
"ratingValue": 2,
"bestRating": 5,
"reviewCount": 19
},
"variants": [
{
"sku": "TU-1",
"price": { "amount": 86, "currency": "USD" },
"availability": { "inStock": true, "text": "https://schema.org/InStock" },
"images": [{ "url": "https://testingurl.dev/assets/placeholder-product.svg" }]
}
]
}
}
}
}
```
```json theme={null}
{
"timestamp": 1786985696914,
"apiStatus": "success",
"apiCode": 200,
"meta": {
"url": "https://example.com/contact",
"device": "desktop",
"format": ["json"],
"extractionMode": "template",
"template": "contact",
"test": { "id": "8f3a07aa-8b8d-4f21-b0d6-706d3fea5dc0" }
},
"data": {
"json": {
"contact": {
"companyName": "Example Inc.",
"locations": [
{
"id": "london",
"label": "London",
"address": "221B Baker Street",
"mapUrl": "https://www.google.com/maps/search/?api=1&query=221B%20Baker%20Street",
"phones": [{ "value": "+1-555-0102" }],
"emails": [
{ "value": "hello@example.com", "label": "general" },
{ "value": "sales@example.com", "label": "sales" }
]
}
],
"contactChannels": {
"emails": [
{ "value": "hello@example.com", "label": "general" },
{ "value": "sales@example.com", "label": "sales" }
],
"phones": [{ "value": "+1-555-0102" }],
"forms": ["https://example.com/contact"]
},
"socialProfiles": [
{ "platform": "github", "url": "https://github.com/example/example" }
],
"extractionMeta": {
"fieldsFound": ["companyName", "locations", "address", "phones", "emails", "forms", "socialProfiles"],
"fieldsMissing": ["hours", "chatUrl", "contactPersons"]
}
}
}
}
}
```
`extractionMeta.fieldsFound` / `fieldsMissing` in the contact template's
response tell you which fields the page actually had data for, so you can
handle partial matches gracefully.
***
## Structured Extraction — CSS Schema
Extract specific fields from a page using CSS selectors. Returns structured JSON.
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com/products',
extractionMode: 'cssSchema',
extractionSchema: {
name: 'Product Schema',
baseSelector: '.product',
fields: [
{ name: 'title', selector: 'h1.product-title', type: 'text' },
{ name: 'price', selector: '.price', type: 'text' },
{ name: 'link', selector: 'a.product-link', type: 'attr', attribute: 'href' }
]
}
});
```
```python Python SDK theme={null}
from geekflare_api.models import WebScrapeDto, ExtractionSchemaDto, SelectorExtractionFieldDto
result = client.web_scrape(WebScrapeDto(
url='https://example.com/products',
extraction_mode='cssSchema',
extraction_schema=ExtractionSchemaDto(
name='Product Schema',
base_selector='.product',
fields=[
SelectorExtractionFieldDto(name='title', selector='h1.product-title', type='text'),
SelectorExtractionFieldDto(name='price', selector='.price', type='text'),
]
)
))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/products",
"extractionMode": "cssSchema",
"extractionSchema": {
"name": "Product Schema",
"baseSelector": ".product",
"fields": [
{"name": "title", "selector": "h1.product-title", "type": "text"},
{"name": "price", "selector": ".price", "type": "text"}
]
}
}'
```
```json theme={null}
{
"timestamp": 1778737930991,
"apiStatus": "success",
"apiCode": 200,
"meta": {
"url": "https://example.com/products",
"format": ["json"],
"extractionMode": "cssSchema",
"test": { "id": "abc123" }
},
"data": [
{ "title": "Running Shoe Pro", "price": "$129.99", "link": "/products/running-shoe-pro" },
{ "title": "Trail Runner X", "price": "$89.99", "link": "/products/trail-runner-x" }
]
}
```
***
## Structured Extraction — XPath Schema
Use XPath expressions for more precise extraction.
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com/articles',
extractionMode: 'xpathSchema',
extractionSchema: {
name: 'Article Schema',
baseSelector: "//div[@class='article']",
fields: [
{ name: 'title', selector: './/h1/text()', type: 'text' },
{ name: 'author', selector: ".//span[@class='author']/text()", type: 'text' }
]
}
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(
url='https://example.com/articles',
extraction_mode='xpathSchema',
extraction_schema=ExtractionSchemaDto(
name='Article Schema',
base_selector="//div[@class='article']",
fields=[
SelectorExtractionFieldDto(name='title', selector='.//h1/text()', type='text'),
SelectorExtractionFieldDto(name='author', selector=".//span[@class='author']/text()", type='text'),
]
)
))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/articles",
"extractionMode": "xpathSchema",
"extractionSchema": {
"name": "Article Schema",
"baseSelector": "//div[@class='\''article'\'']",
"fields": [
{"name": "title", "selector": ".//h1/text()", "type": "text"}
]
}
}'
```
```json theme={null}
{
"timestamp": 1778737930991,
"apiStatus": "success",
"apiCode": 200,
"meta": {
"url": "https://example.com/articles",
"format": ["json"],
"extractionMode": "xpathSchema",
"test": { "id": "abc123" }
},
"data": [
{ "title": "How to Run Faster", "author": "Jane Smith" },
{ "title": "Best Trails in 2025", "author": "John Doe" }
]
}
```
***
## Default Extraction — Static Fields
Inject static metadata fields alongside scraped content.
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com',
extractionMode: 'default',
extractionSchema: {
name: 'Quick Fields',
fields: [
{ title: 'Category', value: 'Electronics' },
{ title: 'Country', value: 'India' }
]
}
});
```
```python Python SDK theme={null}
from geekflare_api.models import DefaultExtractionFieldDto
result = client.web_scrape(WebScrapeDto(
url='https://example.com',
extraction_mode='default',
extraction_schema=ExtractionSchemaDto(
name='Quick Fields',
fields=[
DefaultExtractionFieldDto(title='Category', value='Electronics'),
DefaultExtractionFieldDto(title='Country', value='India'),
]
)
))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"extractionMode": "default",
"extractionSchema": {
"name": "Quick Fields",
"fields": [
{"title": "Category", "value": "Electronics"},
{"title": "Country", "value": "India"}
]
}
}'
```
***
## AI Extraction
Ask AI to answer questions, extract structured data, or analyze the scraped page — summaries, sentiment, keywords, contact info, and more. AI requests always run against Markdown content; any `format` you set is ignored when `aiPrompt` is present. Costs **+7 credits** on top of the base scraping cost (8 total).
```typescript Node.js SDK theme={null}
const result = await client.webScrape({
url: 'https://example.com/products/wireless-headphones',
aiPrompt: {
type: 'prompt',
query: 'What is the return policy?'
}
});
```
```python Python SDK theme={null}
result = client.web_scrape(WebScrapeDto(
url='https://example.com/products/wireless-headphones',
ai_prompt={'type': 'prompt', 'query': 'What is the return policy?'}
))
```
```bash cURL theme={null}
curl -X POST https://api.geekflare.com/webscraping \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/products/wireless-headphones",
"aiPrompt": {"type": "prompt", "query": "What is the return policy?"}
}'
```
AI Extraction supports 8 modes — open-ended questions, custom JSON Schema
extraction, product data, category listings, summaries, contact info,
sentiment analysis, and keyword/entity extraction. See the full [AI
Extraction](/ai-extraction-web-scraping) guide for
request/response examples of every mode.
***
## All Parameters
| Parameter | Type | Default | Description |
| - | - | - | - |
| `url` | string | required | Target URL |
| `device` | `desktop` \| `mobile` | `desktop` | Device to emulate |
| `format` | array | `["html-llm"]` | Output format(s). Up to 3. Ignored if `aiPrompt` is set. |
| `renderJS` | boolean | auto | Execute JavaScript. Left unset, Geekflare auto-detects whether the page needs it. Set `true`/`false` to force rendering on or off. |
| `blockAds` | boolean | `true` | Block ads during scrape |
| `stealth` | boolean | `false` | Bypass bot detection |
| `waitTime` | number | `0` | Seconds to wait after page load |
| `fileOutput` | boolean | `false` | Return CDN URL instead of inline data |
| `proxyMode` | `false` \| `auto` \| `true` | `false` | Controls when a proxy is used. Optional — see [Proxy Routing](#proxy-routing). |
| `proxyCountry` | string | — | Route through a proxy in this country (ISO code, e.g. `us`, `gb`). Works on its own — `proxyMode` is not required. |
| `extractionMode` | `default` \| `cssSchema` \| `xpathSchema` \| `template` | `default` | Extraction mode. JSON output is used automatically when an extraction mode is requested — no need to set `format`. |
| `template` | `product` \| `contact` | — | Extraction template to use when `extractionMode` is `template`. See [Templates](#structured-extraction-templates). |
| `extractionSchema` | object | — | Schema for structured extraction (used when `extractionMode` is `cssSchema`, `xpathSchema`, or `default`; not used for `template`) |
| `aiPrompt` | object | — | Ask AI to extract/analyze the page. See [AI Extraction](/ai-extraction-web-scraping). Adds +7 credits. |
## Credits
| Mode | Credits |
| - | - |
| Web Scraping | 1 |
| Web Scraping + AI Extraction (`aiPrompt`) | 8 |
`npm install @geekflare/api-node`
`pip install geekflare-api`
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.