Firecrawl
First met in aicp-c3-basecamp-03 · Scraping & data sources
FirecrawlFirst met in aicp-c3-basecamp-03 · Scraping & data sources
Scraping built for AI — hand it a URL and get clean markdown or structured JSON back, with an LLM doing the extraction.
You need page content in a shape a model can use, and you don't want to write or maintain selectors.
AI extraction: give it a prompt and a JSON schema and it returns structured fields, not HTML
Native n8n nodes with quick-connect, plus 100k free credits on the hobby plan
Open source — self-host it when the credits run out
Scrapes single pages (/scrape) and crawls sites, returning markdown and/or JSON
Extracts to a schema you define using a prompt
Offers proxy and stealth options for sites that resist
Runs as native nodes inside n8n with a credential connect flow
Doesn't handle every hostile target — geo-blocked and heavily defended sites want a proxy provider like Bright Data
Doesn't stay free forever, though self-hosting is the escape hatch
Doesn't do the platform-specific structured feeds Apify specializes in
The ingestion end of any content workflow: scrape → structure → process with an agent → publish.
Scraping & data sources
The pipeline reads local files, not the web — but the LMS-facing fetch scripts are the same problem shape.
If Outskill's LMS ever needs scraping for materials the API doesn't expose, this is the tool, and its schema-extraction mode maps neatly onto the manifest structure.
current
apify: Apify for platform-specific scrapers (LinkedIn, Instagram); Firecrawl for general pages
bright-data: geo-proxies in front of it for ban-resistant scraping
n8n: native nodes; the BC4 newsletter workflow's front end
| Session | Coverage | Moment | What happened |
|---|---|---|---|
| Basecamp 3: Introduction to n8n — I | demonstrated | 0:54:14 | Crawl action node early; then native Firecrawl nodes in the Hacker News scraper with live credential connect |
| Basecamp 4: Introduction to n8n — II | demonstrated | 0:47:14 | /scrape with JSON+markdown formats, AI-extraction prompt + schema, proxy/stealth options, n8n quick-connect (hobby plan + 100k free credits), self-hosting the open-source repo when credits end |
| Session 9: Introduction to Cursor & its principles | demonstrated | 1:17:52 | Blog/site scraper MCP; auto-invoked by the agent to rescue the design extraction |
| Session 19: Content Strategy Pipeline — the YouTube Growth Stack, Built by an Agent Team of Chats | demonstrated | Cloud key (1,000 free credits) for the demo; standing advice to self-host and never pay | |
| Session 20: Complete LinkedIn Agency Automation — Carousels + Newsletter from One Content System | demonstrated | v2 search across 10 themes, deduped + credibility-ranked; API route chosen over self-host for the demo | |
| Session 21: Advanced Data Extraction from PDFs — Parsing, OCR, and the Token-Free Product | demonstrated | PDF Inspector + AnyDoc repos are the build's extraction spine; MIT licensing the deciding factor | |
| Session 23: Automating AEO/GEO — Audits, the File Layer, and Evidence-Led AI Search | demonstrated | The crawler for the external audit tool; scope = the front-end surface engines see | |
| AI Sprint: The Hermes Agent — Day 1 (Always-On Agents, the Employee-on-Day-One System, and the Self-Improving Ghostwriter) | demonstrated | Pre-scraped outskill.com brand page reused to feed the color scheme when screenshots overflowed the image API | |
| AI Sprint: OpenClaw — Session 2 (Token Anatomy, Claude Code Inside the Server, Skills, and Multi-Agent Workspaces) | demonstrated | The live skill install: proposal → approval → skill.md + Python script + .env key wiring | |
| AI Sprint: The Open Source Playbook — Day 1 (Licenses, Open Weights, and Why Open Isn't Free) | demonstrated | Scrape-with-branding-format as the brand-capture step | |
| Session 6: Building an AI-Powered Lead Gen Machine | mentioned | 1:31:37 | Cohort's answer to 'what scrapes websites' — superseded here by giving the domain to an LLM directly |