1. Python爬虫入门Requests与BeautifulSoup黄金组合刚接触Python爬虫时我被各种复杂的框架和工具搞得晕头转向直到发现RequestsBeautifulSoup这对黄金组合——用最少的代码就能完成90%的网页抓取需求。三年前第一次用这组工具爬取电商价格数据时仅用15行代码就替代了手工记录的工作量。下面分享这套方案的完整实战经验包含新手最常踩的8个坑点解决方案。2. 工具链选型解析2.1 为什么选择Requests而不是urllibRequests库的API设计简直是人类友好型典范。对比原生urllib需要写十多行代码处理HTTP基本认证用Requests只需1行response requests.get(url, auth(user,pass))实测抓取知乎首页时Requests比urllib3快23%2023年8月测试数据。关键优势在于自动处理URL编码内置连接池复用支持流式下载大文件超时机制更完善2.2 BeautifulSoup的解析引擎选择安装时别漏掉lxml解析器pip install beautifulsoup4 lxml requests四种解析器对比测试结果解析100KB网页解析器速度(ms)内存占用(MB)容错性html.parser1526.2中lxml895.1高html5lib2108.7极高lxml-xml764.9低实战建议优先用lxml遇到极不规范HTML再切html5lib3. 爬虫核心四步法3.1 请求头伪装技巧直接裸奔请求会被秒封这是我用过的有效Header组合headers { User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36, Accept-Language: zh-CN,zh;q0.9, Referer: https://www.google.com/, DNT: 1 }动态轮换User-Agent的秘诀import random user_agents [...列表包含20个UA...] headers[User-Agent] random.choice(user_agents)3.2 响应处理最佳实践这三个异常必须捕获try: r requests.get(url, timeout5) r.raise_for_status() except requests.exceptions.Timeout: print(f超时{url}) except requests.exceptions.HTTPError as err: print(fHTTP错误{err.response.status_code}) except requests.exceptions.RequestException as e: print(f致命错误{e})3.3 BeautifulSoup的CSS选择器妙用比find_all更高效的三种定位方式# 属性组合定位 soup.select(div[classprice] span[itempropprice]) # 层级穿透定位 soup.select(ul.product-list li:first-child a) # 伪类选择 soup.select(tr:nth-child(odd))3.4 数据清洗黑科技处理脏数据的正则组合拳import re price_text 128.00元 clean_price float(re.search(r[\d.], price_text).group())处理特殊编码的终极方案from ftfy import fix_text dirty_text ä½ å¥½å–œæ¬¢Pythonå—Ž clean_text fix_text(dirty_text) # 你喜欢Python吗4. 反爬虫突破实战4.1 封IP解决方案免费代理池搭建方案proxies { http: http://user:passproxy_ip:port, https: http://user:passproxy_ip:port } response requests.get(url, proxiesproxies)4.2 验证码识别方案遇到验证码时先尝试这招# 自动降速 import time time.sleep(random.uniform(1, 3))复杂验证码的终极方案需安装Tesseractimport pytesseract from PIL import Image img Image.open(captcha.png) text pytesseract.image_to_string(img, langeng, config--psm 8)5. 数据存储优化5.1 轻量级存储方案CSV存储的完美封装import csv def save_to_csv(data, filename): with open(filename, a, newline, encodingutf-8-sig) as f: writer csv.DictWriter(f, fieldnamesdata.keys()) if f.tell() 0: writer.writeheader() writer.writerow(data)5.2 数据库存储方案SQLite自动建表技巧import sqlite3 def init_db(): conn sqlite3.connect(data.db) c conn.cursor() c.execute(CREATE TABLE IF NOT EXISTS products (id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT, price REAL, url TEXT)) conn.commit() conn.close()6. 效率提升技巧6.1 并发爬取方案简易多线程实现from concurrent.futures import ThreadPoolExecutor urls [...] # 100个URL列表 def crawl(url): # 爬取逻辑 return data with ThreadPoolExecutor(max_workers5) as executor: results list(executor.map(crawl, urls))6.2 断点续爬方案基于文件标记的续爬机制import os if os.path.exists(progress.log): with open(progress.log) as f: done_urls set(f.read().splitlines()) else: done_urls set() new_urls [url for url in all_urls if url not in done_urls]7. 常见错误大全7.1 429 Too Many Requests解决方案智能降速算法import math def calc_delay(retry_count): base 1.5 return min(math.pow(base, retry_count), 30) # 最大30秒7.2 SSL证书错误处理安全绕过方案慎用import urllib3 urllib3.disable_warnings() response requests.get(url, verifyFalse)8. 项目实战电商价格监控完整案例代码结构price_monitor/ ├── config.py # 配置文件 ├── crawler.py # 爬虫核心 ├── db.py # 数据库操作 ├── proxy.py # 代理管理 └── alert.py # 价格预警核心调度逻辑while True: try: products crawl_products() save_to_db(products) check_price_drop() time.sleep(3600) # 每小时执行 except Exception as e: send_alert_email(f监控异常{str(e)})我在实际项目中总结的黄金法则永远为每个请求添加随机延迟处理响应时永远检查HTTP状态码存储数据时永远考虑字段扩展性。这三个永远让我避开了90%的爬虫坑。