Most developers discover these two Claude Code capabilities through frustration rather than documentation. They hit a task that requires live web data — competitor pricing, documentation from an unstable URL, structured product information across dozens of pages — and realise Claude Code has no native way to reach it. Or they give Claude Code a genuinely complex architectural refactor and watch it produce plausible-looking code that silently breaks four adjacent files, because the model generated fast rather than reasoned slow. ScrapeGraph solves the first problem. Extended Thinking's Scratchpad solves the second. This article explains both in full: how they work, what they cost, when to use them, and how the two capabilities combine into something more powerful than either delivers alone.
I work at the intersection of SEO infrastructure and AI tooling — building data pipelines, scraping competitor intelligence, and running Claude Code across research, content, and technical workflows for clients in healthcare, legal services, hospitality, and e-commerce. Both features landed in my regular workflow at different points in 2026, and understanding their real limits (not just their marketing descriptions) separates practitioners who use them effectively from those who end up frustrated with token bills and unreliable outputs. This article gives you the complete picture of both.
Part One — ScrapeGraph: Giving Claude Code Live Web Access
The Problem It Solves — Why Claude Code Can't Scrape Natively
Claude Code operates on content you provide. It reads files in your project, executes shell commands, and calls APIs — but it cannot autonomously fetch arbitrary web pages, parse HTML from live URLs, or monitor web pages for changes on its own. When you ask Claude Code to "pull current pricing from competitor.com" or "extract the structured data from these 50 product pages," without a scraping layer the model simply can't do it. This is the architectural gap ScrapeGraph fills.
- 🔴 Fetch live HTML from arbitrary URLs on demand
- 🔴 Extract structured JSON from web pages through natural language prompts
- 🔴 Monitor pages for changes over time without an external pipeline
- 🔴 Crawl multi-page documentation sites automatically
- 🔴 Search the web for URLs and then scrape the results in a single workflow
- 🟢 Scrape any URL on demand and return clean markdown, HTML, JSON, screenshot, or summary
- 🟢 Extract structured JSON from messy pages using natural language schema prompts
- 🟢 Set up recurring page-change monitors that alert when content shifts
- 🟢 Crawl entire documentation sites and ingest the output directly into context
- 🟢 Search the web for URLs without a prior address, then scrape the findings
What ScrapeGraph Is — The Architecture
ScrapeGraph AI is an AI-powered scraping platform that exposes its capabilities through just-scrape, an official CLI tool designed specifically for AI agents. The Claude Code skill wraps that CLI into a tight, trigger-aware workflow that Claude understands when and how to invoke — eliminating the need to write Beautiful Soup scripts, manage selectors, or manually coordinate API calls every time you need live web data.
As of July 2026, just-scrape holds the highest install count of all 30 qualified web scraping skills indexed by claudeskills.info — ahead of Firecrawl's own official skills (firecrawl-scrape and firecrawl-crawl), which round out the top three. The install lead reflects a combination of breadth (eight output formats, search plus scrape plus monitor in one skill) and ease of trigger (Claude recognises natural language cues without requiring specific command syntax from the user).
The Five just-scrape Commands — What Each One Does
| Command | What It Does | When Claude Uses It |
|---|---|---|
just-scrape extract
|
Pulls structured JSON from any URL using a natural language extraction prompt — no selector writing required | "Extract product name, price, rating, and availability from this URL as JSON" |
just-scrape search
|
Searches the web and returns structured results — useful when you don't have a URL yet | "Search for the three largest e-commerce players in [niche] and extract their pricing pages" |
just-scrape scrape
|
Fetches a page in any of eight formats: markdown, HTML, screenshot, branding, links, images, summary, or JSON | "Scrape this page as clean markdown" or "grab a screenshot of this competitor's homepage" |
just-scrape crawl
|
Crawls an entire site or documentation set, following links and returning bulk content | "Crawl this documentation site and build me a full index of its pages and headings" |
just-scrape monitor
|
Sets up a recurring change tracker that alerts when a specific page's content changes | "Monitor this competitor's pricing page and tell me when it changes" |
Installing and Configuring the ScrapeGraph Skill
Install the just-scrape CLI
The skill requires Node.js and npm. Install the CLI globally:
# Install the just-scrape CLI globally npm install -g just-scrape # Verify installation just-scrape --version
Get Your ScrapeGraph API Key
Create a free account at the ScrapeGraph AI dashboard and generate an API key. The CLI accepts the key through an environment variable, a config file, or an interactive first-run prompt — no manual copying into code required.
# Set your API key via environment variable export SGAI_API_KEY="your-api-key-here" # Or add to ~/.scrapegraphai/config.json { "api_key": "your-api-key-here" }
Install the Claude Code Skill
Add the skill to your Claude Code project using the skills protocol:
# Install just-scrape skill into your project's .claude/skills directory npx skills add scrapegraphai/just-scrape --skill just-scrape --agent claude-code # The skill file teaches Claude when and how to invoke the CLI # No additional configuration required after this step
Use Natural Language Triggers — No Syntax Required
Once installed, Claude Code recognises trigger phrases automatically. You describe what you want in plain English; the skill maps it to the correct just-scrape command without requiring you to specify the command syntax directly.
# These natural language prompts all trigger just-scrape automatically: "Scrape this page and return it as clean markdown:" "Extract product name, price, and SKU from this URL as JSON" "Crawl the docs at docs.example.com and give me the full structure" "Search for the top 5 SaaS tools in [category] and grab their pricing pages" "Monitor competitor.com/pricing — alert me when it changes"
The Honest Limits — What ScrapeGraph Still Cannot Solve Alone
ScrapeGraph handles the AI extraction layer brilliantly. It does not handle the infrastructure layer that modern anti-bot systems specifically target. Sites protected by Cloudflare Enterprise, sophisticated browser fingerprinting, CAPTCHA-as-a-service, or residential IP detection require a separate stealth infrastructure layer beneath the scraping skill — typically a service like Bright Data, Scrapfly, or a purpose-built stealth browser layer. As of July 2026, no Claude Code skill delivers stealth scraping as a free, open-source layer. For targets with moderate anti-bot protection, ScrapeGraph's built-in stealth mode (the --asp flag) handles most cases. For heavy Cloudflare targets, pair ScrapeGraph with a stealth proxy layer.
Plain scrapes cost fewer ScrapeGraph credits. The anti-bot (--asp) and JavaScript rendering (--render-js) flags carry a credit multiplier. For bulk crawl tasks — hundreds or thousands of pages — the credit cost accumulates quickly. Build credit monitoring into any production pipeline using just-scrape credits to track consumption, and start without the anti-bot flags, adding them only when a response signals that protection has blocked the plain request.
Part Two — The Scratchpad: Giving Claude Code Time to Think
The Problem It Solves — Why Fast Answers Are Sometimes Wrong Answers
Claude Code's default behaviour optimises for speed: receive a prompt, generate a response, emit code. For simple, well-defined tasks, this is exactly right. For genuinely complex problems — multi-file architectural refactors, debugging subtle logic errors across a large codebase, designing a system that must satisfy conflicting constraints — fast generation without deliberate reasoning produces confident-sounding code that breaks in non-obvious ways. The Scratchpad (Extended Thinking) changes this. It gives Claude Code a private working space to reason through problems step by step before committing to any code output.
Think of the Scratchpad as the whiteboard a senior developer uses before opening their editor. The whiteboard takes time. It costs mental energy. But for hard problems, the code that follows from whiteboard-first thinking is qualitatively different from code that comes from immediately typing whatever first comes to mind. Extended Thinking gives Claude Code a private whiteboard — and the discipline to use it before generating output.
How the Scratchpad Actually Works — The Technical Mechanism
When Extended Thinking is active, Claude generates internal reasoning tokens — a private working space called the thinking block — before producing the visible response. This thinking block is distinct from the text block that contains the actual answer. Both appear in the API response, but the thinking block contains the model's step-by-step deliberation: exploring approaches, mapping dependencies, catching errors in its own reasoning, and discarding dead ends before they reach the output.
Adaptive Thinking — How Claude Code Manages the Scratchpad in 2026
The original Extended Thinking API required developers to specify a fixed budget_tokens — essentially guessing how much reasoning the problem needed. Claude 4 models (Opus 4.6, Sonnet 4.6) replaced this with adaptive thinking: the model evaluates the complexity of each prompt and dynamically decides how much to think, rather than burning a fixed token budget regardless of whether the problem actually needs it.
| Approach | How It Works | Status in 2026 |
|---|---|---|
| Fixed budget_tokens | Developer specifies exact token count to allocate for reasoning (e.g., 5,000 tokens). Model uses that budget whether the task is trivial or complex. | Deprecated on claude-opus-4.6 and claude-sonnet-4.6 |
| Adaptive thinking (effort parameter) | Developer sets an effort level: low, medium, high, or max. Model decides dynamically how many tokens the specific task actually warrants. | Current default on Claude 4 models |
| ultrathink keyword | Legacy keyword that previously triggered maximum thinking budget (31,999 tokens) in Claude Code. Removed as Claude Code evolved its built-in thinking management. | Deprecated — use effort: "max" instead |
Claude Code uses adaptive thinking by default. For most coding sessions, you don't need to configure anything — the model applies appropriate reasoning depth automatically based on what it perceives as the complexity of each task. You can steer thinking depth using the effort parameter when you need explicit control: low for speed-sensitive tasks, medium for the practical default, high for genuinely complex problems where reasoning quality matters more than speed, and max to replace the deprecated ultrathink keyword for maximum deliberation.
When Extended Thinking Actually Improves Output — And When It Doesn't
Extended thinking is not universally better. Research shows it can hurt performance by up to 36% on certain task types — analogous to how humans perform worse when they overthink intuitive tasks. The productivity-maximising position is precise task matching: use the Scratchpad specifically for tasks where multi-step reasoning genuinely changes the output quality, and skip it for tasks where fast generation is already sufficient.
- 🔴 Simple code generation with clear, well-specified requirements
- 🔴 Summarisation, classification, or basic Q&A tasks
- 🔴 Text reformatting or straightforward string manipulation
- 🔴 Any task where speed matters more than reasoning depth
- 🔴 High-frequency, low-complexity API calls where token cost compounds fast
- 🟢 Multi-file architectural refactors where changes cascade across the codebase
- 🟢 Debugging subtle logic errors with multiple interacting failure modes
- 🟢 System design with conflicting constraints that require genuine trade-off analysis
- 🟢 Security-sensitive code where missing an edge case carries real cost
- 🟢 Novel algorithm design or complex data structure reasoning
The Cost Structure — What Thinking Tokens Actually Cost
Thinking tokens are billed as output tokens at the model's standard output rate. They are not free, and a single call with heavy reasoning can spend 3 to 10 times the tokens of a standard completion. The economics only justify the cost when the reasoning genuinely changes the output in ways that matter to your use case.
| Thinking Token Range | Typical Use | Approximate Latency Impact |
|---|---|---|
| 1,024 – 5,000 tokens | Light reasoning — moderate complexity tasks, constraint checking, simple architectural decisions | +2–8 seconds per call |
| 5,000 – 20,000 tokens | Standard reasoning range — most complex coding tasks, multi-file refactors, debugging with context | +5–20 seconds per call |
| 20,000 – 128,000 tokens | Maximum deliberation — novel algorithm design, large-codebase architectural analysis, highest-stakes reasoning tasks | +20–90+ seconds per call |
"I use both features regularly, but for genuinely different purposes that don't overlap. ScrapeGraph entered my workflow when I needed to build a competitor intelligence pipeline for an e-commerce client — pulling structured pricing data from 40 product URLs weekly, without maintaining a selector-based scraper that broke every time any of those sites updated their frontend. The extraction prompt approach — 'extract product name, price, promotional label, and availability as JSON from this URL' — produced reliable structured output across layouts that would have broken any CSS-selector approach within two sprints. The Scratchpad became essential when I hit a genuinely complex codebase task: reorganising a multi-file content pipeline that had grown organically and had implicit dependencies nobody had fully documented. The first time I ran that task without Extended Thinking, the generated code was plausible but missed three implicit assumptions that only surfaced in testing. The second time, with high effort thinking enabled, the model surfaced those assumptions in the thinking block before generating a single line — and flagged two of them as requiring clarification before it could proceed safely. That difference — catching the hidden assumption before the code ships — is exactly what the Scratchpad is for."
Using Both Together — The Combined Workflow
The most powerful Claude Code sessions combine both capabilities in sequence: ScrapeGraph first to bring live, structured data into context, Extended Thinking second to reason carefully about what to do with it. This combination addresses both the data-access problem and the reasoning-quality problem simultaneously.
Use ScrapeGraph to Fetch and Structure Live Data
Start any task requiring external data with a ScrapeGraph extraction step. Extract structured JSON from the relevant URLs, clean markdown from documentation pages, or a crawled site map from a competitor's documentation. This step brings real, current data into Claude Code's context window in a format it can reason about directly — replacing the manual copy-paste workflow or the brittle selector pipeline.
Pass the Structured Output Directly Into a Reasoning Task
With the scraped data available as a variable or file in context, give Claude Code a complex task that requires reasoning about that data — building a comparison report, generating code that integrates the extracted data into an existing pipeline, or designing a monitoring system around the observed page structure. Let the extracted JSON serve as the grounding data source rather than requiring the model to work from assumptions.
Enable High Effort Thinking for the Reasoning Step
For tasks where the reasoning step is genuinely complex — multiple files affected, implicit dependencies in the existing code, conflicting constraints in the requirements — explicitly set the effort level to high or max. This gives the model enough deliberation time to surface hidden assumptions and evaluate multiple approaches before committing to the one it generates in the visible output.
Review the Thinking Summary Before Accepting Generated Code
On Sonnet 4.6, the condensed thinking summary appears in the response before the text block. Read it before accepting the generated code. The thinking summary reveals what assumptions the model made, what approaches it considered and rejected, and what uncertainty remains — information that is invisible in the final code output but critical for catching misunderstandings before they reach production.
Frequently Asked Questions
effort: "max" in API calls — which achieves the same maximum deliberation intent through the supported, forward-compatible parameter rather than a detected keyword that could be removed without notice. Claude Code's default of adaptive thinking also means most tasks no longer require any explicit trigger at all — the model applies appropriate depth automatically.The Bottom Line
ScrapeGraph and the Scratchpad solve the two most consequential limitations in Claude Code's default capability set: lack of live web data access, and the tendency toward fast-but-wrong reasoning on genuinely complex problems. ScrapeGraph's just-scrape skill — the most-installed web scraping skill for Claude Code as of July 2026 — gives the model structured access to any URL in eight formats, through natural language triggers, without requiring selector maintenance or manual HTML processing. The Scratchpad (Extended Thinking) gives the model a private reasoning space to think before it generates, catching hidden assumptions and evaluating competing approaches before they reach the code output. Both run most effectively when matched to the right task: ScrapeGraph for any workflow requiring live, structured web data; Extended Thinking for tasks where reasoning quality determines whether the generated code actually works in a complex system. Used together — scrape first, reason carefully about the structured result second — they produce the kind of Claude Code output that earns real trust in production workflows.
.webp)
Driven by advanced SEO expertise, deep marketing analytics, high-impact content strategy
With 5+ years of hands-on experience, I specialize in holistic search strategies that don’t just rank—they drive real, measurable business growth. I’ve worked across industries including healthcare, hospitality, legal, e-commerce, and professional services, helping brands dominate their target markets. My approach bridges the gap between raw data and creative execution. Every strategy I build is rooted in rigorous market analysis, structured SEO frameworks, and tailored content ecosystems—no templates, no shortcuts. Whether you’re a single-location brand or scaling across multiple cities, I create data-driven marketing systems designed to compound results and grow with you.
Want Better Visibility in Claude & AI Search?
Get a free AI SEO audit from DigitalArka. We'll optimise your website for AI search by improving crawlability, technical SEO, structured data, page speed, internal linking, entity signals, and content quality—helping search engines and AI platforms better understand and discover your content.
Get Your Free AI SEO Audit →