Scrapling 全解析从单次请求到大规模爬取的自适应 Web 抓取框架【免费下载链接】Scrapling️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!项目地址: https://gitcode.com/GitHub_Trending/sc/Scrapling本文基于 Scrapling 仓库的德语主 READMEdocs/README_DE.md整理成文系统讲解 Scrapling 的三大核心能力——自适应解析器、多模式 Fetcher 与 Spider 爬虫框架并结合同仓库源码逐一印证关键实现自适应元素的auto_save/adaptive参数、Spider 的并行与断点续爬配置、scrapling install的浏览器依赖安装流程以及 CLIextract子命令的真实定义帮助你从安装、请求、解析到完整 Crawl 全链路掌握该框架的实战用法。项目概览Scrapling 是一个自适应 Web 抓取框架覆盖从单条请求到大规模站点爬取的全部场景。它的三个支柱在 docs/README_DE.md 中的定位如下自适应解析器Parser能从网站变更中学习当页面结构更新时自动重新定位已保存的元素Fetcher 家族内置的抓取器可直接绕过 Cloudflare Turnstile 等反爬体系支持静态 HTTP、Playwright 浏览器自动化与 Stealth 隐身浏览器三种模式Spider 框架支持并行多会话 Crawl、暂停/恢复Pause Resume与自动代理轮换并提供实时统计与流式输出。最小示例来自 README 顶部from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher StealthyFetcher.adaptive True p StealthyFetcher.fetch(https://example.com, headlessTrue, network_idleTrue) # 隐身抓取 products p.css(.product, auto_saveTrue) # 保存可抗页面改版的选择结果 products p.css(.product, adaptiveTrue) # 页面结构变化后用 adaptiveTrue 重新找回元素或者直接升级为完整 Crawlfrom scrapling.spiders import Spider, Response class MySpider(Spider): name demo start_urls [https://example.com/] async def parse(self, response: Response): for item in response.css(.product): yield {title: item.css(h2::text).get()} MySpider().start()从源码结构看这两个入口分别对应两个独立子包scrapling/fetchers/请求层惰性导入避免重型依赖与scrapling/spiders/爬虫引擎层互不强制绑定这正是 README 安装章节强调「基础包只含 Parser」的原因。核心特性总览Spider 爬虫框架README 将 Spider 部分列为第一特性要点包括Scrapy 风格的start_urls 异步parse回调、可配置并行度与域级限流、多会话统一接口、基于 Checkpoint 的暂停恢复、async for item in spider.stream()流式输出、阻塞请求自动检测重试、AutoThrottle 自适应限速、可选robots_txt_obey合规、开发模式缓存回放、以及CrawlSpider/SitemapSpider/ShopifySpider等现成模板。这些能力在当前代码中都有明确落点scrapling/spiders/spider.py 中Spider基类L65 起直接定义了全部可调属性# robots.txt 合规 robots_txt_obey: bool False # 开发模式响应落盘缓存重复运行回放 development_mode: bool False development_cache_dir: Optional[str] None # 并发设置 concurrent_requests: int 4 concurrent_requests_per_domain: int 0 download_delay: float 0.0 max_blocked_retries: int 3 # AutoThrottle 设置 autothrottle_enabled: bool False autothrottle_start_delay: float 5.0 autothrottle_max_delay: float 60.0 autothrottle_target_concurrency: Optional[float] None autothrottle_block_backoff: bool True几个值得注意的源码细节阻塞判定同文件 L16 定义BLOCKED_CODES {401, 403, 407, 429, 444, 500, 502, 503, 504}配合max_blocked_retries 3实现「自动检测并重试被阻塞的请求」Pause Resume构造函数__init__(self, crawldirNone, interval300.0)中crawldir决定 Checkpoint 目录interval默认 300 秒是周期落盘间隔——这解释了为什么文档说「CtrlC 受控暂停后进度自动保存」流式模式scrapling/spiders/spider.py#L304 的async def stream(self)以异步生成器逐条产出Dict结果与 README 描述的async for item in spider.stream()一致模板全家桶scrapling/spiders/init.py 还额外导出了XMLFeedSpider、CSVFeedSpider与LinkExtractor、Scheduler、CrawlerEngine、SessionManager等构件比 README 提到的模板更多。上图为仓库自带的 Spider 架构图docs/assets/spider_architecture.png展示了 Spider → Scheduler → Crawler Engine → Session Manager → Output 的数据流以及 Checkpoint 系统「从上次检查点恢复」的虚线路径与上面源码中的crawldir/interval机制一一对应。多模式 Fetcher 与 SessionFetcher 部分能力清单HTTP 伪装请求、动态加载、反 Bot 绕过、Session 管理、代理轮换、域/广告拦截、DNS 泄漏防护、远程浏览器、XHR 后台捕获、全异步在 scrapling/fetchers/init.py 中体现为一套惰性导入映射# scrapling/fetchers/__init__.py L11-L21节选 _LAZY_IMPORTS { Fetcher: (scrapling.fetchers.requests, Fetcher), AsyncFetcher: (scrapling.fetchers.requests, AsyncFetcher), FetcherSession: (scrapling.fetchers.requests, FetcherSession), DynamicFetcher: (scrapling.fetchers.chrome, DynamicFetcher), DynamicSession: (scrapling.fetchers.chrome, DynamicSession), AsyncDynamicSession: (scrapling.fetchers.chrome, AsyncDynamicSession), StealthyFetcher: (scrapling.fetchers.stealth_chrome, StealthyFetcher), StealthySession: (scrapling.fetchers.stealth_chrome, StealthySession), AsyncStealthySession: (scrapling.fetchers.stealth_chrome, AsyncStealthySession), }即三种引擎各有独立的模块文件requestsHTTP基于 TLS 指纹伪装、chromePlaywright/Chrome 全自动化、stealth_chromeStealth 隐身浏览器。惰性导入__getattr__按需加载意味着from scrapling.fetchers import Fetcher不会在导入期拉起 Playwright 等重依赖。关于 README 提到的参数scrapling/fetchers/stealth_chrome.py 的 docstring 给出了官方语义disable_resources丢弃非必要资源请求以提速network_idle等待页面至少 500ms 无网络活动solve_cloudflare在返回响应前解决所有类型的 Cloudflare Turnstile/Interstitial 挑战cdp_url则用于连接已运行的浏览器而非新起实例。代理轮换由 scrapling/engines/toolbelt/proxy_rotation.py 的ProxyRotatorL39 起实现线程安全、支持循环轮换默认与可插拔自定义策略同时接受字符串 URL 与 Playwright 风格 dict 两种代理格式。广告拦截数据则存放在 scrapling/engines/toolbelt/ad_domains.py约 3500 个域名条目与 README「~3.500 个已知广告/跟踪域名」的说法吻合。自适应解析与 AI 集成自适应能力的核心参数可以直接在 scrapling/parser.py 中确认。Selector类以adaptive: Optional[bool] False和storage参数全局启用该功能L89 起而css()/xpath()方法L566、L626接受adaptive: bool False—— 若元素曾被保存则尝试重新定位identifier—— 用于在存储中保存/检索元素数据的字符串键auto_save: bool False—— 自动保存新元素供后续adaptive使用percentage—— 相似度匹配的最低接受阈值。源码中还有一个细节校验若构造时未启用adaptive却传入auto_save会发出「auto_save将被忽略」的警告scrapling/parser.py#L659-L683这解释了为什么顶部示例必须先写StealthyFetcher.adaptive True。README 提到的其余解析特性find_similar()相似元素、below_elements()下方元素、find_by_text()文本查找均可在 scrapling/parser.py 中找到对应方法定义L390、L1013、L1075 起。AI 集成方面仓库提供 agent-skill/Scrapling-Skill/SKILL.md 作为安装即用的 Agent SkillMCP 服务器能力见 docs/ai/mcp-server.md 与 docs/api-reference/mcp-server.md。架构与工程实践README 声明的基础依赖与工程配置可从 pyproject.toml 核实Python3.10核心依赖仅lxml、cssselect、orjsonREADME「比标准库快 10 倍」的 JSON 序列化即来自 orjson、tld、w3lib、typing_extensions——轻量依赖面支撑了「Speichereffizient内存高效」的定位。快速上手基础用法HTTP 请求支持会话与一次性请求两种风格from scrapling.fetchers import Fetcher, FetcherSession with FetcherSession(impersonatechrome) as session: # 使用最新版 Chrome 的 TLS 指纹 page session.get(https://quotes.toscrape.com/, stealthy_headersTrue) quotes page.css(.quote .text::text).getall() # 或一次性请求 page Fetcher.get(https://quotes.toscrape.com/) quotes page.css(.quote .text::text).getall()隐身模式Stealth 浏览器可解 Cloudflare 挑战from scrapling.fetchers import StealthyFetcher, StealthySession with StealthySession(headlessTrue, solve_cloudflareTrue) as session: # 浏览器保持打开直到完成 page session.fetch(https://nopecha.com/demo/cloudflare, google_searchFalse) data page.css(#padded_content a).getall() # 或一次性请求风格为该请求开启浏览器结束后关闭 page StealthyFetcher.fetch(https://nopecha.com/demo/cloudflare) data page.css(#padded_content a).getall()完整浏览器自动化Playwright Chromium / Google Chromefrom scrapling.fetchers import DynamicFetcher, DynamicSession with DynamicSession(headlessTrue, disable_resourcesFalse, network_idleTrue) as session: page session.fetch(https://quotes.toscrape.com/, load_domFalse) data page.xpath(//span[classtext]/text()).getall() # 亦支持 XPath page DynamicFetcher.fetch(https://quotes.toscrape.com/) data page.css(.quote .text::text).getall()Spider并行、多会话与断点续爬带并行度与翻页的完整 Crawlerfrom scrapling.spiders import Spider, Request, Response class QuotesSpider(Spider): name quotes start_urls [https://quotes.toscrape.com/] concurrent_requests 10 async def parse(self, response: Response): for quote in response.css(.quote): yield { text: quote.css(.text::text).get(), author: quote.css(.author::text).get(), } next_page response.css(.next a) if next_page: yield response.follow(next_page[0].attrib[href]) result QuotesSpider().start() print(f{len(result.items)} Zitate gescrapt) result.items.to_json(quotes.json)一个 Spider 内混用多种会话类型按 ID 路由请求from scrapling.spiders import Spider, Request, Response from scrapling.fetchers import FetcherSession, AsyncStealthySession class MultiSessionSpider(Spider): name multi start_urls [https://example.com/] def configure_sessions(self, manager): manager.add(fast, FetcherSession(impersonatechrome)) manager.add(stealth, AsyncStealthySession(headlessTrue), lazyTrue) async def parse(self, response: Response): for link in response.css(a::attr(href)).getall(): if protected in link: yield Request(link, sidstealth) # 受保护页面走隐身会话 else: yield Request(link, sidfast, callbackself.parse)长任务断点续爬QuotesSpider(crawldir./crawl_data).start()CtrlC 受控暂停后进度自动保存再次以相同crawldir启动即从断点继续。对应的导出能力在 scrapling/spiders/result.py 的CrawlResult上实现to_json()L40、to_jsonl()L55、to_csv()L67支持自定义字段与分隔符、to_xml()L89支持自定义根/子标签。模板类可跳过自写爬取逻辑例如拉取整个 Shopify 商店目录from scrapling.spiders import ShopifySpider class MyStore(ShopifySpider): target_website example.com result MyStore().start() # 商店中每个产品每个变体一个 Item模板的源码实现位于 scrapling/spiders/templates/crawler.py、scrapling/spiders/templates/sitemap.py 与 scrapling/spiders/templates/shopify.py并有配套测试 tests/spiders/test_templates.py。高级解析与导航from scrapling.fetchers import Fetcher page Fetcher.get(https://quotes.toscrape.com/) # 多种选择方法 quotes page.css(.quote) # CSS quotes page.xpath(//div[classquote]) # XPath quotes page.find_all(div, {class: quote}) # BeautifulSoup 风格 quotes page.find_all(div, class_quote) quotes page.find_all([div], class_quote) quotes page.find_all(class_quote) quotes page.find_by_text(quote, tagdiv) # 按文本内容查找 # 导航 quote_text page.css(.quote)[0].css(.text::text).get() quote_text page.css(.quote).css(.text::text).getall() # 选择器链式调用 first_quote page.css(.quote)[0] author first_quote.next_sibling.css(.author::text) parent_container first_quote.parent # 元素关系与相似性 similar_elements first_quote.find_similar() below_elements first_quote.below_elements()不需要抓取网页时也可以直接使用解析器from scrapling.parser import Selector page Selector(html.../html) # 用法与页面对象完全一致异步会话管理import asyncio from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession async with FetcherSession(http3True) as session: # 上下文感知sync/async 均可用 page1 session.get(https://quotes.toscrape.com/) page2 session.get(https://quotes.toscrape.com/, impersonatefirefox135) # 异步会话 async with AsyncStealthySession(max_pages2) as session: tasks [] urls [https://example.com/page1, https://example.com/page2] for url in urls: tasks.append(session.fetch(url)) print(session.get_pool_stats()) # 浏览器 Tab 池状态忙碌/空闲/错误 results await asyncio.gather(*tasks) print(session.get_pool_stats())get_pool_stats()反映的正是 README 所述「浏览器 Tab 池」机制隐身会话在有限数量的标签页内复用浏览器实例而非逐请求冷启动。CLI 与交互式 ShellScrapling 提供完整的命令行入口入口点定义于 pyproject.tomlscrapling scrapling.cli:main另有scrapling-mcp scrapling.cli:mcp。命令行实现集中在 scrapling/cli.py。启动交互式 Web 抓取 ShellIPython 集成含 Curl 转 Scrapling 请求等工具scrapling shell不写代码直接抓取页面到文件scrapling/cli.py#L217 的extract命令组。输出格式由扩展名决定.txt提取纯文本、.md生成 Markdown、.html保留原始 HTML默认提取body内容scrapling extract get https://example.com content.md scrapling extract get https://example.com content.txt --css-selector #fromSkipToProducts --impersonate chrome scrapling extract fetch https://example.com content.md --css-selector #fromSkipToProducts --no-headless scrapling extract stealthy-fetch https://nopecha.com/demo/cloudflare captchas.html --css-selector #padded_content a --solve-cloudflare从 scrapling/cli.py 可确认extract下实际实现了get/post/put/deleteHTTP与fetch/stealthy-fetch浏览器两组子命令HTTP 组共享--headers、--cookies、--timeout、--proxy、--css-selector、--impersonate、--stealthy-headers等公共选项_common_http_optionsL227 起POST/PUT 另带--data/--json。MCP 服务器通过独立的scrapling mcp即scrapling-mcp脚本启动支持 stdio 与 streamable-http 两种传输并带--auth-token、--allowed-host防 DNS 重绑定等安全选项。性能基准README 给出两组基准均为 100 次循环的平均值方法学见 benchmarks.py可用 tests/ 中的测试用例交叉验证行为文本提取速度5000 个嵌套元素#库时间 (ms)相对 Scrapling1Scrapling1.991.0x2Parsel/Scrapy2.061.035x3Raw Lxml2.561.286x4PyQuery23.98~12x5Selectolax197.02~99x6MechanicalSoup1545.15~776.5x7BS4 Lxml1562.1~785.0x8BS4 html5lib3412.73~1714.9x元素相似度与文本搜索库时间 (ms)相对 ScraplingScrapling2.31.0xAutoScraper12.585.47x安装与可选依赖要求 Python 3.10pyproject.toml 中requires-python 3.10元数据声明支持 3.10–3.13pip install scrapling重要基础安装仅包含 Parser 引擎及其依赖lxml/cssselect/orjson 等不含 Fetcher 与命令行依赖。此时from scrapling.fetchers import ...或from scrapling.spiders import ...会抛出ModuleNotFoundError。Fetcher 与浏览器依赖pip install scrapling[fetchers] scrapling install # 正常安装 scrapling install --force # 强制重装scrapling install的底层动作在 scrapling/cli.py#L120依次执行python -m playwright install chromium、python -m playwright install-deps chromium并刷新 tld 库数据成功后写入标记文件.scrapling_dependencies_installed以避免重复安装。也可以从代码内调用from scrapling.cli import install install([], standalone_modeFalse) # 正常安装 install([--force], standalone_modeFalse) # 强制重装其他 extras定义于 pyproject.toml 的[project.optional-dependencies]scrapling[fetchers]click、curl_cffi、playwright、patchright、browserforge、apify-fingerprint-datapoints、msgspec、anyio、protegopip install scrapling[ai]MCP 服务器mcp、markdownify含 fetcherspip install scrapling[shell]交互式 Shell 与extract命令IPython、markdownify含 fetcherspip install scrapling[all]以上全部。注意安装任何 extras 后如尚未执行过仍需运行scrapling install落地浏览器依赖。Docker 方式镜像随每次发布与主干分支经 CI 自动构建推送docker pull pyd4vinci/scrapling # 或 docker pull ghcr.io/d4vinci/scrapling:latest法律、许可与致谢README 以「免责声明」收尾该库仅供教育与研究目的使用者须遵守当地及国际数据抓取与隐私法律并尊重目标网站的条款与 robots.txt。项目采用BSD-3-Clause许可见 LICENSE并致谢了适配自 ParselBSD 许可的 translator 子模块。小结Scrapling 的 README德语版 docs/README_DE.md英文版见 README.md勾勒的「Parser Fetcher Spider」三层能力在源码中均有清晰对应自适应选择逻辑在 scrapling/parser.py三种引擎在 scrapling/fetchers/爬虫引擎与会话管理在 scrapling/spiders/CLI 与安装器在 scrapling/cli.py。配套的中文文档入口在 docs/README_CN.md选择方法、Fetcher 选型、Spider 架构等主题在 docs/parsing/selection.md、docs/fetching/choosing.md、docs/spiders/architecture.md 中均有展开可结合本文继续深入。【免费下载链接】Scrapling️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!项目地址: https://gitcode.com/GitHub_Trending/sc/Scrapling创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 Scrapling 全解析：从单次请求到大规模爬取的自适应 Web 抓取框架
Scrapling 全解析从单次请求到大规模爬取的自适应 Web 抓取框架【免费下载链接】Scrapling️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!项目地址: https://gitcode.com/GitHub_Trending/sc/Scrapling本文基于 Scrapling 仓库的德语主 READMEdocs/README_DE.md整理成文系统讲解 Scrapling 的三大核心能力——自适应解析器、多模式 Fetcher 与 Spider 爬虫框架并结合同仓库源码逐一印证关键实现自适应元素的auto_save/adaptive参数、Spider 的并行与断点续爬配置、scrapling install的浏览器依赖安装流程以及 CLIextract子命令的真实定义帮助你从安装、请求、解析到完整 Crawl 全链路掌握该框架的实战用法。项目概览Scrapling 是一个自适应 Web 抓取框架覆盖从单条请求到大规模站点爬取的全部场景。它的三个支柱在 docs/README_DE.md 中的定位如下自适应解析器Parser能从网站变更中学习当页面结构更新时自动重新定位已保存的元素Fetcher 家族内置的抓取器可直接绕过 Cloudflare Turnstile 等反爬体系支持静态 HTTP、Playwright 浏览器自动化与 Stealth 隐身浏览器三种模式Spider 框架支持并行多会话 Crawl、暂停/恢复Pause Resume与自动代理轮换并提供实时统计与流式输出。最小示例来自 README 顶部from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher StealthyFetcher.adaptive True p StealthyFetcher.fetch(https://example.com, headlessTrue, network_idleTrue) # 隐身抓取 products p.css(.product, auto_saveTrue) # 保存可抗页面改版的选择结果 products p.css(.product, adaptiveTrue) # 页面结构变化后用 adaptiveTrue 重新找回元素或者直接升级为完整 Crawlfrom scrapling.spiders import Spider, Response class MySpider(Spider): name demo start_urls [https://example.com/] async def parse(self, response: Response): for item in response.css(.product): yield {title: item.css(h2::text).get()} MySpider().start()从源码结构看这两个入口分别对应两个独立子包scrapling/fetchers/请求层惰性导入避免重型依赖与scrapling/spiders/爬虫引擎层互不强制绑定这正是 README 安装章节强调「基础包只含 Parser」的原因。核心特性总览Spider 爬虫框架README 将 Spider 部分列为第一特性要点包括Scrapy 风格的start_urls 异步parse回调、可配置并行度与域级限流、多会话统一接口、基于 Checkpoint 的暂停恢复、async for item in spider.stream()流式输出、阻塞请求自动检测重试、AutoThrottle 自适应限速、可选robots_txt_obey合规、开发模式缓存回放、以及CrawlSpider/SitemapSpider/ShopifySpider等现成模板。这些能力在当前代码中都有明确落点scrapling/spiders/spider.py 中Spider基类L65 起直接定义了全部可调属性# robots.txt 合规 robots_txt_obey: bool False # 开发模式响应落盘缓存重复运行回放 development_mode: bool False development_cache_dir: Optional[str] None # 并发设置 concurrent_requests: int 4 concurrent_requests_per_domain: int 0 download_delay: float 0.0 max_blocked_retries: int 3 # AutoThrottle 设置 autothrottle_enabled: bool False autothrottle_start_delay: float 5.0 autothrottle_max_delay: float 60.0 autothrottle_target_concurrency: Optional[float] None autothrottle_block_backoff: bool True几个值得注意的源码细节阻塞判定同文件 L16 定义BLOCKED_CODES {401, 403, 407, 429, 444, 500, 502, 503, 504}配合max_blocked_retries 3实现「自动检测并重试被阻塞的请求」Pause Resume构造函数__init__(self, crawldirNone, interval300.0)中crawldir决定 Checkpoint 目录interval默认 300 秒是周期落盘间隔——这解释了为什么文档说「CtrlC 受控暂停后进度自动保存」流式模式scrapling/spiders/spider.py#L304 的async def stream(self)以异步生成器逐条产出Dict结果与 README 描述的async for item in spider.stream()一致模板全家桶scrapling/spiders/init.py 还额外导出了XMLFeedSpider、CSVFeedSpider与LinkExtractor、Scheduler、CrawlerEngine、SessionManager等构件比 README 提到的模板更多。上图为仓库自带的 Spider 架构图docs/assets/spider_architecture.png展示了 Spider → Scheduler → Crawler Engine → Session Manager → Output 的数据流以及 Checkpoint 系统「从上次检查点恢复」的虚线路径与上面源码中的crawldir/interval机制一一对应。多模式 Fetcher 与 SessionFetcher 部分能力清单HTTP 伪装请求、动态加载、反 Bot 绕过、Session 管理、代理轮换、域/广告拦截、DNS 泄漏防护、远程浏览器、XHR 后台捕获、全异步在 scrapling/fetchers/init.py 中体现为一套惰性导入映射# scrapling/fetchers/__init__.py L11-L21节选 _LAZY_IMPORTS { Fetcher: (scrapling.fetchers.requests, Fetcher), AsyncFetcher: (scrapling.fetchers.requests, AsyncFetcher), FetcherSession: (scrapling.fetchers.requests, FetcherSession), DynamicFetcher: (scrapling.fetchers.chrome, DynamicFetcher), DynamicSession: (scrapling.fetchers.chrome, DynamicSession), AsyncDynamicSession: (scrapling.fetchers.chrome, AsyncDynamicSession), StealthyFetcher: (scrapling.fetchers.stealth_chrome, StealthyFetcher), StealthySession: (scrapling.fetchers.stealth_chrome, StealthySession), AsyncStealthySession: (scrapling.fetchers.stealth_chrome, AsyncStealthySession), }即三种引擎各有独立的模块文件requestsHTTP基于 TLS 指纹伪装、chromePlaywright/Chrome 全自动化、stealth_chromeStealth 隐身浏览器。惰性导入__getattr__按需加载意味着from scrapling.fetchers import Fetcher不会在导入期拉起 Playwright 等重依赖。关于 README 提到的参数scrapling/fetchers/stealth_chrome.py 的 docstring 给出了官方语义disable_resources丢弃非必要资源请求以提速network_idle等待页面至少 500ms 无网络活动solve_cloudflare在返回响应前解决所有类型的 Cloudflare Turnstile/Interstitial 挑战cdp_url则用于连接已运行的浏览器而非新起实例。代理轮换由 scrapling/engines/toolbelt/proxy_rotation.py 的ProxyRotatorL39 起实现线程安全、支持循环轮换默认与可插拔自定义策略同时接受字符串 URL 与 Playwright 风格 dict 两种代理格式。广告拦截数据则存放在 scrapling/engines/toolbelt/ad_domains.py约 3500 个域名条目与 README「~3.500 个已知广告/跟踪域名」的说法吻合。自适应解析与 AI 集成自适应能力的核心参数可以直接在 scrapling/parser.py 中确认。Selector类以adaptive: Optional[bool] False和storage参数全局启用该功能L89 起而css()/xpath()方法L566、L626接受adaptive: bool False—— 若元素曾被保存则尝试重新定位identifier—— 用于在存储中保存/检索元素数据的字符串键auto_save: bool False—— 自动保存新元素供后续adaptive使用percentage—— 相似度匹配的最低接受阈值。源码中还有一个细节校验若构造时未启用adaptive却传入auto_save会发出「auto_save将被忽略」的警告scrapling/parser.py#L659-L683这解释了为什么顶部示例必须先写StealthyFetcher.adaptive True。README 提到的其余解析特性find_similar()相似元素、below_elements()下方元素、find_by_text()文本查找均可在 scrapling/parser.py 中找到对应方法定义L390、L1013、L1075 起。AI 集成方面仓库提供 agent-skill/Scrapling-Skill/SKILL.md 作为安装即用的 Agent SkillMCP 服务器能力见 docs/ai/mcp-server.md 与 docs/api-reference/mcp-server.md。架构与工程实践README 声明的基础依赖与工程配置可从 pyproject.toml 核实Python3.10核心依赖仅lxml、cssselect、orjsonREADME「比标准库快 10 倍」的 JSON 序列化即来自 orjson、tld、w3lib、typing_extensions——轻量依赖面支撑了「Speichereffizient内存高效」的定位。快速上手基础用法HTTP 请求支持会话与一次性请求两种风格from scrapling.fetchers import Fetcher, FetcherSession with FetcherSession(impersonatechrome) as session: # 使用最新版 Chrome 的 TLS 指纹 page session.get(https://quotes.toscrape.com/, stealthy_headersTrue) quotes page.css(.quote .text::text).getall() # 或一次性请求 page Fetcher.get(https://quotes.toscrape.com/) quotes page.css(.quote .text::text).getall()隐身模式Stealth 浏览器可解 Cloudflare 挑战from scrapling.fetchers import StealthyFetcher, StealthySession with StealthySession(headlessTrue, solve_cloudflareTrue) as session: # 浏览器保持打开直到完成 page session.fetch(https://nopecha.com/demo/cloudflare, google_searchFalse) data page.css(#padded_content a).getall() # 或一次性请求风格为该请求开启浏览器结束后关闭 page StealthyFetcher.fetch(https://nopecha.com/demo/cloudflare) data page.css(#padded_content a).getall()完整浏览器自动化Playwright Chromium / Google Chromefrom scrapling.fetchers import DynamicFetcher, DynamicSession with DynamicSession(headlessTrue, disable_resourcesFalse, network_idleTrue) as session: page session.fetch(https://quotes.toscrape.com/, load_domFalse) data page.xpath(//span[classtext]/text()).getall() # 亦支持 XPath page DynamicFetcher.fetch(https://quotes.toscrape.com/) data page.css(.quote .text::text).getall()Spider并行、多会话与断点续爬带并行度与翻页的完整 Crawlerfrom scrapling.spiders import Spider, Request, Response class QuotesSpider(Spider): name quotes start_urls [https://quotes.toscrape.com/] concurrent_requests 10 async def parse(self, response: Response): for quote in response.css(.quote): yield { text: quote.css(.text::text).get(), author: quote.css(.author::text).get(), } next_page response.css(.next a) if next_page: yield response.follow(next_page[0].attrib[href]) result QuotesSpider().start() print(f{len(result.items)} Zitate gescrapt) result.items.to_json(quotes.json)一个 Spider 内混用多种会话类型按 ID 路由请求from scrapling.spiders import Spider, Request, Response from scrapling.fetchers import FetcherSession, AsyncStealthySession class MultiSessionSpider(Spider): name multi start_urls [https://example.com/] def configure_sessions(self, manager): manager.add(fast, FetcherSession(impersonatechrome)) manager.add(stealth, AsyncStealthySession(headlessTrue), lazyTrue) async def parse(self, response: Response): for link in response.css(a::attr(href)).getall(): if protected in link: yield Request(link, sidstealth) # 受保护页面走隐身会话 else: yield Request(link, sidfast, callbackself.parse)长任务断点续爬QuotesSpider(crawldir./crawl_data).start()CtrlC 受控暂停后进度自动保存再次以相同crawldir启动即从断点继续。对应的导出能力在 scrapling/spiders/result.py 的CrawlResult上实现to_json()L40、to_jsonl()L55、to_csv()L67支持自定义字段与分隔符、to_xml()L89支持自定义根/子标签。模板类可跳过自写爬取逻辑例如拉取整个 Shopify 商店目录from scrapling.spiders import ShopifySpider class MyStore(ShopifySpider): target_website example.com result MyStore().start() # 商店中每个产品每个变体一个 Item模板的源码实现位于 scrapling/spiders/templates/crawler.py、scrapling/spiders/templates/sitemap.py 与 scrapling/spiders/templates/shopify.py并有配套测试 tests/spiders/test_templates.py。高级解析与导航from scrapling.fetchers import Fetcher page Fetcher.get(https://quotes.toscrape.com/) # 多种选择方法 quotes page.css(.quote) # CSS quotes page.xpath(//div[classquote]) # XPath quotes page.find_all(div, {class: quote}) # BeautifulSoup 风格 quotes page.find_all(div, class_quote) quotes page.find_all([div], class_quote) quotes page.find_all(class_quote) quotes page.find_by_text(quote, tagdiv) # 按文本内容查找 # 导航 quote_text page.css(.quote)[0].css(.text::text).get() quote_text page.css(.quote).css(.text::text).getall() # 选择器链式调用 first_quote page.css(.quote)[0] author first_quote.next_sibling.css(.author::text) parent_container first_quote.parent # 元素关系与相似性 similar_elements first_quote.find_similar() below_elements first_quote.below_elements()不需要抓取网页时也可以直接使用解析器from scrapling.parser import Selector page Selector(html.../html) # 用法与页面对象完全一致异步会话管理import asyncio from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession async with FetcherSession(http3True) as session: # 上下文感知sync/async 均可用 page1 session.get(https://quotes.toscrape.com/) page2 session.get(https://quotes.toscrape.com/, impersonatefirefox135) # 异步会话 async with AsyncStealthySession(max_pages2) as session: tasks [] urls [https://example.com/page1, https://example.com/page2] for url in urls: tasks.append(session.fetch(url)) print(session.get_pool_stats()) # 浏览器 Tab 池状态忙碌/空闲/错误 results await asyncio.gather(*tasks) print(session.get_pool_stats())get_pool_stats()反映的正是 README 所述「浏览器 Tab 池」机制隐身会话在有限数量的标签页内复用浏览器实例而非逐请求冷启动。CLI 与交互式 ShellScrapling 提供完整的命令行入口入口点定义于 pyproject.tomlscrapling scrapling.cli:main另有scrapling-mcp scrapling.cli:mcp。命令行实现集中在 scrapling/cli.py。启动交互式 Web 抓取 ShellIPython 集成含 Curl 转 Scrapling 请求等工具scrapling shell不写代码直接抓取页面到文件scrapling/cli.py#L217 的extract命令组。输出格式由扩展名决定.txt提取纯文本、.md生成 Markdown、.html保留原始 HTML默认提取body内容scrapling extract get https://example.com content.md scrapling extract get https://example.com content.txt --css-selector #fromSkipToProducts --impersonate chrome scrapling extract fetch https://example.com content.md --css-selector #fromSkipToProducts --no-headless scrapling extract stealthy-fetch https://nopecha.com/demo/cloudflare captchas.html --css-selector #padded_content a --solve-cloudflare从 scrapling/cli.py 可确认extract下实际实现了get/post/put/deleteHTTP与fetch/stealthy-fetch浏览器两组子命令HTTP 组共享--headers、--cookies、--timeout、--proxy、--css-selector、--impersonate、--stealthy-headers等公共选项_common_http_optionsL227 起POST/PUT 另带--data/--json。MCP 服务器通过独立的scrapling mcp即scrapling-mcp脚本启动支持 stdio 与 streamable-http 两种传输并带--auth-token、--allowed-host防 DNS 重绑定等安全选项。性能基准README 给出两组基准均为 100 次循环的平均值方法学见 benchmarks.py可用 tests/ 中的测试用例交叉验证行为文本提取速度5000 个嵌套元素#库时间 (ms)相对 Scrapling1Scrapling1.991.0x2Parsel/Scrapy2.061.035x3Raw Lxml2.561.286x4PyQuery23.98~12x5Selectolax197.02~99x6MechanicalSoup1545.15~776.5x7BS4 Lxml1562.1~785.0x8BS4 html5lib3412.73~1714.9x元素相似度与文本搜索库时间 (ms)相对 ScraplingScrapling2.31.0xAutoScraper12.585.47x安装与可选依赖要求 Python 3.10pyproject.toml 中requires-python 3.10元数据声明支持 3.10–3.13pip install scrapling重要基础安装仅包含 Parser 引擎及其依赖lxml/cssselect/orjson 等不含 Fetcher 与命令行依赖。此时from scrapling.fetchers import ...或from scrapling.spiders import ...会抛出ModuleNotFoundError。Fetcher 与浏览器依赖pip install scrapling[fetchers] scrapling install # 正常安装 scrapling install --force # 强制重装scrapling install的底层动作在 scrapling/cli.py#L120依次执行python -m playwright install chromium、python -m playwright install-deps chromium并刷新 tld 库数据成功后写入标记文件.scrapling_dependencies_installed以避免重复安装。也可以从代码内调用from scrapling.cli import install install([], standalone_modeFalse) # 正常安装 install([--force], standalone_modeFalse) # 强制重装其他 extras定义于 pyproject.toml 的[project.optional-dependencies]scrapling[fetchers]click、curl_cffi、playwright、patchright、browserforge、apify-fingerprint-datapoints、msgspec、anyio、protegopip install scrapling[ai]MCP 服务器mcp、markdownify含 fetcherspip install scrapling[shell]交互式 Shell 与extract命令IPython、markdownify含 fetcherspip install scrapling[all]以上全部。注意安装任何 extras 后如尚未执行过仍需运行scrapling install落地浏览器依赖。Docker 方式镜像随每次发布与主干分支经 CI 自动构建推送docker pull pyd4vinci/scrapling # 或 docker pull ghcr.io/d4vinci/scrapling:latest法律、许可与致谢README 以「免责声明」收尾该库仅供教育与研究目的使用者须遵守当地及国际数据抓取与隐私法律并尊重目标网站的条款与 robots.txt。项目采用BSD-3-Clause许可见 LICENSE并致谢了适配自 ParselBSD 许可的 translator 子模块。小结Scrapling 的 README德语版 docs/README_DE.md英文版见 README.md勾勒的「Parser Fetcher Spider」三层能力在源码中均有清晰对应自适应选择逻辑在 scrapling/parser.py三种引擎在 scrapling/fetchers/爬虫引擎与会话管理在 scrapling/spiders/CLI 与安装器在 scrapling/cli.py。配套的中文文档入口在 docs/README_CN.md选择方法、Fetcher 选型、Spider 架构等主题在 docs/parsing/selection.md、docs/fetching/choosing.md、docs/spiders/architecture.md 中均有展开可结合本文继续深入。【免费下载链接】Scrapling️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!项目地址: https://gitcode.com/GitHub_Trending/sc/Scrapling创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考