The Real Problem
Scraping web pages, running headless browsers, and rotating residential proxies incur real per-request proxy and compute costs. Developers offering scraping APIs need to charge per extracted URL without incurring subscription overhead.
Real-World Example: For example, a developer hosting a headless browser scraping API could charge $0.01 per extracted HTML payload, allowing scrapers or AI agents to pay per URL.
How aipp Solves It
aipp handles the 402 challenge negotiation. When a client requests a scraped page URL, your server returns HTTP 402. Upon payment verification, your headless browser executes the page parse and returns data.
Step-by-Step Payment Flow
STEP 01
Scrape Request
Client submits target URL to scrape.
STEP 02
402 Challenge
API returns HTTP 402 invoice header.
STEP 03
Payment Settlement
Client pays via Lightning or Base USDC.
STEP 04
Parsed Data Delivered
Server verifies receipt, executes scraper, and returns JSON/Markdown.
Scope & Boundaries
What aipp Does
- ✓ Provides 402 payment verification layer for data extraction services
- ✓ Enables AI scrapers and agents to pay per extracted page programmatically
- ✓ Auto-forwards net proceeds to merchant wallet
What aipp Does Not Do
- ✗ Does not scrape websites or manage proxy pools
- ✗ Does not bypass anti-bot protections or CAPTCHAs
- ✗ Does not authorize scraping that violates terms of service
Implementation Pattern
Below is the minimal implementation pattern using standard aipp endpoints / SDK:
app.post('/api/scrape', l402Paywall({
client: aipp,
amountUsd: 0.01
}), async (req, res) => {
const content = await fetchPageMarkdown(req.body.url);
res.json({ markdown: content });
});
Settlement Architecture Note:
• Lightning Rail: In hosted Lightning flows, payments route through aipp's Lightning gateway before net proceeds are automatically forwarded to your configured wallet address.
• Base USDC Rail: USDC transfers settle directly on-chain to your designated EVM wallet address.
• No Merchant Spending Balance: aipp does not maintain a spendable merchant balance or require manual withdrawal steps.
Requirements & Limitations
- Merchant operates the scraping infrastructure and proxy network
- Does not authorize scraping of prohibited or illegal targets
- Funds auto-forward to merchant wallet
Frequently Asked Questions
Is this suitable for AI crawler agents?
Yes. AI crawler agents can parse the 402 response, pay the Lightning invoice, and consume structured markdown directly.
Related Use Cases
Gate Paid Web Scraping & Data Extraction Queries with aipp
Require payment verification before executing data queries or research synthesis.
Open Studio Console →