---
# === IDENTITY ===
id: business/lead-generation/web-scraping-for-lead-lists/2026
canonical_question: "How do I scrape lead lists from directories, conferences, G2 reviewers, GitHub contributors — with legal constraints?"
aliases:
  - "Web scraping for B2B lead lists from public directories and communities"
  - "How to build lead lists from conference attendees and G2 reviews"
  - "Scraping GitHub contributors and open-source community leads"
entity_type: execution_recipe
domain: business > lead-generation > web scraping for lead lists
region: global
jurisdiction: global
temporal_scope: 2024-2026

# === VERIFICATION ===
last_verified: 2026-03-11
confidence: 0.86
version: 1.0
first_published: 2026-03-11

# === TEMPORAL VALIDITY ===
temporal_validity:
  status: volatile
  last_breaking_change: null
  next_review: 2026-09-07
  change_sensitivity: high

# === CONSTRAINTS ===
constraints:
  - "GDPR: scraping personal data of EU residents requires legitimate interest basis — public visibility does not remove protection"
  - "CCPA: California residents can request deletion; maintain suppression list"
  - "CNIL fined KASPR EUR 240,000 in Dec 2024 for scraping LinkedIn contacts"
  - "Respect robots.txt — increasingly enforced as contractual obligation under GDPR and DSA"
  - "GitHub API rate limit: 5,000 requests/hour (authenticated), 60/hour (unauthenticated)"
  - "G2 and most review sites prohibit automated scraping in their ToS"

# === SKIP CONDITIONS ===
skip_this_unit_if:
  - condition: "Have budget for data providers like Apollo or ZoomInfo"
    use_instead: "business/lead-generation/apollo-zoominfo-api-lead-pull/2026"
  - condition: "Only need LinkedIn leads"
    use_instead: "business/lead-generation/linkedin-sales-navigator-scraping-workflow/2026"
  - condition: "Need more than 5,000 leads with verified emails"
    use_instead: "business/lead-generation/apollo-zoominfo-api-lead-pull/2026"

# === AGENT HINTS ===
inputs_needed:
  - key: source_type
    question: "Which sources should be scraped?"
    type: choice
    options: ["directories (Clutch, Crunchbase)", "conferences (attendee lists)", "communities (GitHub, G2)", "all available"]
  - key: technical_skill
    question: "What is the user's technical skill level?"
    type: choice
    options: ["non-technical (no-code only)", "semi-technical (can edit code)", "developer (can write code)"]
  - key: budget_for_tools
    question: "What's the tool/API budget?"
    type: choice
    options: ["free tier only", "up to $50/month", "up to $200/month"]
  - key: scale
    question: "How many leads needed?"
    type: choice
    options: ["50-200 leads", "200-1000 leads", "1000-5000 leads"]

# === EXECUTION METADATA ===
execution:
  required_inputs:
    - name: "ICP definition"
      source: "agents/startup/persona-builder"
      format: "structured data"
    - name: "Target source URLs"
      source: "manual"
      format: "list of URLs or source names"
  outputs:
    - name: "Raw scraped lead list"
      format: "CSV"
      description: "Names, titles, companies, URLs from public sources — pre-enrichment"
    - name: "Legal compliance log"
      format: "JSON"
      description: "robots.txt check results, source ToS notes, consent basis documentation"
  tools_required:
    - name: "Python (BeautifulSoup/Scrapy)"
      purpose: "Web scraping engine"
      tier: "free"
      cost: "$0"
      alternatives: ["Apify ($49/mo)", "Octoparse (free tier)", "Browse AI ($39/mo)"]
    - name: "GitHub API"
      purpose: "Extract contributor profiles from target repositories"
      tier: "free"
      cost: "$0 (5,000 req/hr authenticated)"
      alternatives: ["GitScraper", "manual browsing"]
  credentials_needed:
    - service: "GitHub"
      type: "Personal access token"
      where_to_get: "https://github.com/settings/tokens"
      free_tier_limits: "5,000 requests/hour"
  estimated_duration: "1-3 hours for 500 leads"
  estimated_cost: "$0 (free tools) to $50/month (Apify/Browse AI)"

# === DISTRIBUTION ===
canonical_source: "https://knowledgelib.io/business/lead-generation/web-scraping-for-lead-lists/2026"
suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-03-11)"

# === RELATED UNITS ===
related_kos:
  depends_on:
    - id: "business/customer-research/ideal-customer-profile-framework/2026"
      label: "Building an ICP — firmographic (B2B) or demographic (B2C) criteria, behavioral signals, disqualification criteria"
  feeds_into:
    - id: "business/lead-generation/lead-enrichment-pipeline/2026"
      label: "Scraped leads need email and firmographic enrichment"
    - id: "business/lead-generation/email-verification-workflow/2026"
      label: "Verify any discovered emails before outreach"
  related_to:
    - id: "business/lead-generation/linkedin-sales-navigator-scraping-workflow/2026"
      label: "LinkedIn-specific scraping alternative"
  alternative_to:
    - id: "business/lead-generation/apollo-zoominfo-api-lead-pull/2026"
      label: "Paid data provider alternative"

# === SOURCES ===
sources:
  - id: src1
    title: "Web Scraping in 2025: The EUR 20 Million GDPR Mistake"
    author: Dragos-Constantin Tugui
    url: https://medium.com/deep-tech-insights/web-scraping-in-2025-the-20-million-gdpr-mistake-you-cant-afford-to-make-07a3ce240f4f
    type: technical_blog
    published: 2025-03-15
    reliability: high
  - id: src2
    title: "Web Scraping Legal Issues: 2025 Enterprise Compliance Guide"
    author: GroupBWT
    url: https://groupbwt.com/blog/is-web-scraping-legal/
    type: industry_report
    published: 2025-06-10
    reliability: high
  - id: src3
    title: "The State of Web Scraping in the EU"
    author: IAPP
    url: https://iapp.org/news/a/the-state-of-web-scraping-in-the-eu
    type: industry_report
    published: 2025-04-20
    reliability: authoritative
  - id: src4
    title: "Web Scraping Compliance: GDPR, CCPA and AI Laws"
    author: X-Byte
    url: https://www.xbyte.io/the-future-of-web-scraping-compliance-navigating-gdpr-ccpa-and-ai-laws-in-2025/
    type: technical_blog
    published: 2025-07-05
    reliability: high
  - id: src5
    title: "GitHub REST API Documentation"
    author: GitHub
    url: https://docs.github.com/en/rest
    type: official_docs
    published: 2026-01-01
    reliability: authoritative
  - id: src6
    title: "CNIL Guidelines on Web Scraping for AI Development"
    author: Hogan Lovells
    url: https://www.hoganlovells.com/en/publications/development-of-an-ai-system-cnil-issues-guidelines-regarding-collection-of-data-via-web-scraping
    type: industry_report
    published: 2025-08-15
    reliability: authoritative
---

# Web Scraping for Lead Lists

## Purpose

This recipe produces a raw lead list of 200-2,000 contacts scraped from public web sources — business directories (Clutch, Crunchbase), conference attendee pages, G2 review profiles, and GitHub contributor lists — with full legal compliance documentation. Output requires enrichment pipeline processing before outreach.

## Prerequisites

- [ ] **ICP definition** available from `persona-builder` agent — [ICP Definition](/business/customer-research/ideal-customer-profile-definition/2026)
- [ ] **Python 3.10+** with `requests`, `beautifulsoup4`, `pandas` installed
- [ ] **GitHub personal access token** (if scraping contributors) — from [github.com/settings/tokens](https://github.com/settings/tokens)
- [ ] **Legal review** — confirm legitimate interest basis for data collection in target jurisdictions

## Constraints

- GDPR: Public visibility does not remove data protection. Scraping personal data of EU residents requires a documented legitimate interest basis. Fines up to EUR 20 million or 4% of global revenue. [src1]
- CNIL fined KASPR EUR 240,000 in Dec 2024 for scraping 160 million LinkedIn contacts without adequate legal basis. [src6]
- Robots.txt is increasingly treated as a binding contract under GDPR and the EU Digital Services Act. Always check and respect it. [src2]
- CCPA: California residents have right to deletion — maintain a suppression list and honor opt-out requests within 45 days. [src4]
- GitHub API: 5,000 requests/hour authenticated, 60/hour unauthenticated. Each contributor list page returns 30 results. [src5]
- G2, Clutch, and most review platforms explicitly prohibit automated scraping in their Terms of Service. Proceed with legal counsel. [src2]

## Tool Selection Decision

```
Which path?
├── User is non-technical AND any budget
│   └── PATH A: No-Code — Browse AI or Octoparse visual scraper
├── User is semi-technical AND budget = free
│   └── PATH B: Python + BeautifulSoup for simple pages
├── User is developer AND budget = free
│   └── PATH C: Scrapy + GitHub API for scale
└── User is developer AND budget > $0
    └── PATH D: Apify actors + custom scrapers
```

| Path | Tools | Cost | Speed | Output Quality |
|------|-------|------|-------|---------------|
| A: No-Code | Browse AI / Octoparse | $0-$39/mo | 1 hr/100 leads | Basic — may miss structured data |
| B: Python Simple | BeautifulSoup + requests | $0 | 1 hr/200 leads | Good — custom parsing |
| C: Scrapy + API | Scrapy + GitHub API | $0 | 30 min/500 leads | High — structured + pagination |
| D: Apify | Apify marketplace actors | $49/mo | 15 min/1000 leads | Highest — pre-built scrapers |

## Execution Flow

### Step 1: Legal Compliance Check

**Duration**: 15 minutes
**Tool**: Browser + documentation

Before scraping any source, check legal compliance for each target.

```python
import requests

def check_robots_txt(base_url):
    """Check if scraping is permitted by robots.txt"""
    robots_url = f"{base_url}/robots.txt"
    response = requests.get(robots_url, timeout=10)
    if response.status_code == 200:
        return {"url": base_url, "robots_txt": response.text, "accessible": True}
    return {"url": base_url, "robots_txt": None, "accessible": True}

# Check each target source
sources = [
    "https://www.g2.com",
    "https://clutch.co",
    "https://www.crunchbase.com",
    "https://github.com"
]
compliance_log = [check_robots_txt(s) for s in sources]
```

**Verify**: Document legitimate interest basis for each source. Record which paths are disallowed by robots.txt.
**If failed**: If robots.txt disallows scraping target paths, do not proceed with that source. Use alternative data providers.

### Step 2: Scrape GitHub Contributors

**Duration**: 15-30 minutes
**Tool**: GitHub REST API
**Rate limit**: 5,000 requests/hour (authenticated). [src5]

```python
import requests
import time

GITHUB_TOKEN = "your_token_here"
headers = {"Authorization": f"token {GITHUB_TOKEN}"}

def get_contributors(owner, repo, max_pages=10):
    """Extract contributors from a GitHub repository"""
    contributors = []
    for page in range(1, max_pages + 1):
        response = requests.get(
            f"https://api.github.com/repos/{owner}/{repo}/contributors",
            headers=headers,
            params={"per_page": 100, "page": page}
        )
        if response.status_code != 200 or not response.json():
            break
        for user in response.json():
            # Get full profile for each contributor
            profile = requests.get(
                f"https://api.github.com/users/{user['login']}",
                headers=headers
            ).json()
            contributors.append({
                "name": profile.get("name"),
                "company": profile.get("company"),
                "email": profile.get("email"),
                "bio": profile.get("bio"),
                "location": profile.get("location"),
                "github_url": profile.get("html_url"),
                "public_repos": profile.get("public_repos"),
                "followers": profile.get("followers"),
            })
            time.sleep(0.8)  # Stay under rate limit
    return contributors

# Example: scrape contributors from popular repos in your ICP's tech stack
leads = get_contributors("vercel", "next.js", max_pages=5)
```

**Expected output**: List of dicts with name, company, email (if public), bio, location, GitHub URL.
**Verify**: At least 50% of contributors have a name and company. Email coverage will be low (~15-25%) since many GitHub users don't publish emails.
**If failed**: If rate-limited (403), wait for `X-RateLimit-Reset` header timestamp.

### Step 3: Scrape Directory Listings

**Duration**: 30-60 minutes
**Tool**: Python + BeautifulSoup (or Apify)

```python
from bs4 import BeautifulSoup
import requests
import pandas as pd

def scrape_directory_page(url, selectors):
    """Generic directory scraper — customize selectors per site"""
    response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
    soup = BeautifulSoup(response.text, "html.parser")
    entries = []
    for item in soup.select(selectors["listing"]):
        entries.append({
            "company": item.select_one(selectors["company"]).text.strip() if item.select_one(selectors["company"]) else None,
            "location": item.select_one(selectors["location"]).text.strip() if item.select_one(selectors["location"]) else None,
            "url": item.select_one(selectors["link"])["href"] if item.select_one(selectors["link"]) else None,
        })
    return entries
```

**Verify**: At least 80% of entries have company name and URL populated.
**If failed**: If response is 403/429, the site has anti-scraping protection. Use Apify actors or respect rate limits.

### Step 4: Merge, Deduplicate, and Tag Sources

**Duration**: 10-15 minutes
**Tool**: Python (pandas)

```python
# Combine all sources
all_leads = pd.DataFrame(github_leads + directory_leads + conference_leads)

# Deduplicate by name + company
all_leads = all_leads.drop_duplicates(subset=["name", "company"], keep="first")

# Remove entries without name
all_leads = all_leads.dropna(subset=["name"])

# Tag data source for compliance tracking
all_leads["data_source"] = all_leads.apply(tag_source, axis=1)
all_leads["scrape_date"] = "2026-03-11"
all_leads["consent_basis"] = "legitimate_interest_b2b"

all_leads.to_csv("scraped_leads_raw.csv", index=False)
print(f"Final: {len(all_leads)} unique leads from {all_leads['data_source'].nunique()} sources")
```

**Verify**: Duplicate rate < 15% across combined sources.

### Step 5: Generate Compliance Report

**Duration**: 5 minutes

```python
import json

compliance_report = {
    "scrape_date": "2026-03-11",
    "sources_checked": len(compliance_log),
    "robots_txt_respected": True,
    "legal_basis": "legitimate_interest_b2b",
    "gdpr_applicable": True,
    "suppression_list_applied": True,
    "total_leads": len(all_leads),
    "sources": compliance_log
}

with open("compliance_report.json", "w") as f:
    json.dump(compliance_report, f, indent=2)
```

**Output files**:
- `scraped_leads_raw.csv` — Raw lead list with source tags
- `compliance_report.json` — Legal compliance documentation

## Output Schema

```json
{
  "output_type": "raw_lead_list",
  "format": "CSV",
  "columns": [
    {"name": "name", "type": "string", "description": "Full name", "required": true},
    {"name": "company", "type": "string", "description": "Company name", "required": false},
    {"name": "title", "type": "string", "description": "Job title if available", "required": false},
    {"name": "email", "type": "string", "description": "Email if publicly available", "required": false},
    {"name": "location", "type": "string", "description": "Geographic location", "required": false},
    {"name": "source_url", "type": "string", "description": "URL where data was found", "required": true},
    {"name": "data_source", "type": "string", "description": "Source type (github/directory/conference)", "required": true},
    {"name": "consent_basis", "type": "string", "description": "Legal basis for processing", "required": true}
  ],
  "expected_row_count": "200-2000",
  "sort_order": "data_source, company ascending",
  "deduplication_key": "name + company"
}
```

## Quality Benchmarks

| Quality Metric | Minimum Acceptable | Good | Excellent |
|---------------|-------------------|------|-----------|
| Name completeness | > 80% | > 90% | > 98% |
| Company name available | > 50% | > 70% | > 85% |
| Email available (pre-enrichment) | > 10% | > 25% | > 40% |
| Compliance documentation | Complete | Complete | Complete + legal review |
| Duplicate rate (cross-source) | < 15% | < 8% | < 3% |

**If below minimum**: Improve CSS selectors for directory scraping. Filter out non-person entries. Add more source URLs.

## Error Handling

| Error | Likely Cause | Recovery Action |
|-------|-------------|----------------|
| 403 Forbidden | Anti-scraping protection triggered | Add delays, rotate User-Agent, use Apify |
| 429 Rate Limited | Too many requests | Wait for retry-after header, reduce frequency |
| CAPTCHA presented | Bot detection | Cannot bypass — switch to manual or API alternative |
| Empty results | CSS selectors outdated | Inspect page source, update selectors |
| Legal cease-and-desist | ToS violation or GDPR complaint | Stop immediately, delete data, consult legal counsel |

## Cost Breakdown

| Component | Free Tier | Paid Tier | At Scale |
|-----------|-----------|-----------|----------|
| Python + BeautifulSoup | $0 | $0 | $0 |
| GitHub API | $0 (5K req/hr) | $0 | $0 |
| Apify (optional) | $5/mo (free tier) | $49/mo | $499/mo |
| Proxy rotation (optional) | $0 | $30/mo | $100+/mo |
| **Total for 500 leads** | **$0** | **$49/mo** | **$599/mo** |

## Anti-Patterns

### Wrong: Scraping personal data without legal basis documentation
Assuming public data is free to collect. CNIL fined KASPR EUR 240,000 for this exact assumption applied to LinkedIn contacts. [src6]

### Correct: Document legitimate interest basis before scraping
Create compliance report per source. Only scrape B2B professional data. Maintain suppression lists. Honor opt-out requests. [src1]

### Wrong: Ignoring robots.txt and site Terms of Service
Robots.txt is increasingly enforced like a contract under GDPR and DSA. Ignoring it undermines any legitimate interest defense. [src2]

### Correct: Check robots.txt first, document compliance
Programmatically check robots.txt before scraping each source. Skip disallowed paths. Log compliance status for each URL.

## When This Matters

Use when the agent needs to build a lead list from public web sources without paying for data providers. Best for small batches (200-1,000 leads) from niche sources like conference attendees, open-source contributors, or industry directory listings. Always pair with enrichment pipeline for email/phone discovery.

## Related Units

- [ICP Definition](/business/customer-research/ideal-customer-profile-definition/2026)
- [Lead Enrichment Pipeline](/business/lead-generation/lead-enrichment-pipeline/2026)
- [Apollo/ZoomInfo API Lead Pull](/business/lead-generation/apollo-zoominfo-api-lead-pull/2026)
- [Email Verification Workflow](/business/lead-generation/email-verification-workflow/2026)
- [LinkedIn Sales Navigator Scraping](/business/lead-generation/linkedin-sales-navigator-scraping-workflow/2026)