安裝

urllib庫更適合寫爬蟲文件，scrapy更適合做爬蟲項目。

步驟：

先更改pip源，國外的太慢了，參考：https://www.jb51.net/article/159167.htm
升級pip：python -m pip install --upgrade pip
pip install wheel
pip install lxml
pip install Twisted
pip install scrapy

常用命令

核心目錄

新建項目：scrapy startproject mcq
運行獨立的爬蟲文件（不是項目）：比如

然后輸入命令scrapy runspider gg.py

獲取設(shè)置信息：cd到項目，比如scrapy settings --get BOT_NAME
交互式爬?。簊crapy shell http://www.baidu.com，可以使用python代碼
scrapy版本信息：scrapy version
爬取并且在瀏覽器顯示：scrapy view http://news.1152.com，將網(wǎng)頁下載到本地打開
測試本地硬件性能：scrapy bench ,每分鐘可以爬取多少頁面
依據(jù)模板創(chuàng)建爬蟲文件：scrapy genspider -l ,有以下模板

選擇basic，scrapy genspider -t basic haha baidu.com （注意：這里填可爬取的域名，域名是不以www、edu……開頭的）

測試爬蟲文件是否合規(guī)：scrapy check haha
運行爬蟲項目下的文件：scrapy crawl haha
不顯示中間的日志信息：scrapy crawl haha --nolog
查看當(dāng)前項目下可用的爬蟲文件：scrapy list
指定某個爬蟲文件獲取url：
F:\scrapy項目\mcq>scrapy parse --spider=haha http://www.baidu.com

XPath表達(dá)式

XPath與正則簡單對比：

XPath表達(dá)式效率會高一點
正則表達(dá)式功能強(qiáng)一點
一般來說，優(yōu)先選擇XPath，但是XPath解決不了的問題我們就選正則去解決

/：逐層提取

text()提取標(biāo)簽下面的文本

如要提取標(biāo)題:/html/head/title/text()

//標(biāo)簽名：提取所有名為……的標(biāo)簽

如提取所有的div標(biāo)簽：//div

//標(biāo)簽名[@屬性='屬性值']：提取屬性為……的標(biāo)簽

@屬性表示取某個屬性值

使用scrapy做當(dāng)當(dāng)網(wǎng)商品爬蟲

新建爬蟲項目：F:\scrapy項目>scrapy startproject dangdang

F:\scrapy項目>cd dangdang

F:\scrapy項目\dangdang>scrapy genspider -t basic dd dangdang.com

修改items.py：

          
            # -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class DangdangItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title=scrapy.Field() #商品標(biāo)題
    link=scrapy.Field() #商品鏈接
    comment=scrapy.Field() #商品評論

我們翻一下頁，分析兩個鏈接：

http://category.dangdang.com/pg2-cid4008154.html

http://category.dangdang.com/pg3-cid4008154.html

可以找到初始鏈接：http://category.dangdang.com/pg1-cid4008154.html

分析頁面源碼，可以從name="itemlist-title" 下手，因為這個正好有48個結(jié)果，即一頁商品的數(shù)量。

ctrl+f 條評論，可以發(fā)現(xiàn)正好有48條記錄。

dd.py：

          
            # -*- coding: utf-8 -*-
import scrapy
from dangdang.items import DangdangItem
from scrapy.http import Request
class DdSpider(scrapy.Spider):
    name = 'dd'
    allowed_domains = ['dangdang.com']
    start_urls = ['http://category.dangdang.com/pg1-cid4008154.html']

    def parse(self, response):
        item=DangdangItem()
        item["title"]=response.xpath("http://a[@name='itemlist-title']/@title").extract()
        item["link"]=response.xpath("http://a[@name='itemlist-title']/@href").extract()
        item["comment"]=response.xpath("http://a[@name='itemlist-review']/text()").extract()
        # print(item["title"])
        yield item
        for i in range(2,11): #爬取2~10頁
            url='http://category.dangdang.com/pg'+str(i)+'-cid4008154.html'
            yield Request(url, callback=self.parse)

對于dd里的Request：

url: 就是需要請求，并進(jìn)行下一步處理的url
callback: 指定該請求返回的Response，由那個函數(shù)來處理。

先把settings.py的robots改為False:

settings.py:

          
            # -*- coding: utf-8 -*-

# Scrapy settings for dangdang project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'dangdang'

SPIDER_MODULES = ['dangdang.spiders']
NEWSPIDER_MODULE = 'dangdang.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'dangdang (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'dangdang.middlewares.DangdangSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'dangdang.middlewares.DangdangDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html

ITEM_PIPELINES = {
   'dangdang.pipelines.DangdangPipeline': 300,
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

運行：F:\scrapy項目\dangdang>scrapy crawl dd --nolog

去settings.py將pipeline開啟：

pipelines.py:

          
            # -*- coding: utf-8 -*-
import pymysql
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


class DangdangPipeline(object):
    def process_item(self, item, spider):
        conn=pymysql.connect(host='127.0.0.1',user="root",passwd="123456",db="dangdang")
        cursor = conn.cursor()
        for i in range(len(item["title"])):
            title=item["title"][i]
            link=item["link"][i]
            comment=item["comment"][i]
            # print(title+":"+link+":"+comment)
            sql="insert into goods(title,link,comment) values('%s','%s','%s')"%(title,link,comment)
            # print(sql)
            try:
                cursor.execute(sql)
                conn.commit()
            except Exception as e:
                print(e)
        conn.close()
        return item

登錄mysql，創(chuàng)建一個數(shù)據(jù)庫：mysql> create database dangdang;

mysql> use dangdang

mysql> create table goods(id int(32) auto_increment primary key,title varchar(100),link varchar(100) unique,comment varchar(100));

最后運行 scrapy crawl dd --nolog

每頁48條，48*10=480，爬取成功！

完整項目源代碼參考我的github

scrapy模擬登陸實戰(zhàn)

以這個網(wǎng)站為例http://edu.iqianyue.com/，我們不爬取內(nèi)容，只模擬登陸，所以不需要寫item.py

點擊登陸，用fiddler查看真正的登陸網(wǎng)址：http://edu.iqianyue.com/index_user_login

修改login.py：

          
            # -*- coding: utf-8 -*-
import scrapy
from scrapy.http import FormRequest, Request


class LoginSpider(scrapy.Spider):
    name = 'login'
    allowed_domains = ['iqianyue.com']
    start_urls = ['http://iqianyue.com/']
    header={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"}
    #編寫start_request()方法，第一次會默認(rèn)調(diào)取該方法中的請求
    def start_requests(self):
        #首先爬一次登錄頁，然后進(jìn)入回調(diào)函數(shù)parse()
        return [Request("http://edu.iqianyue.com/index_user_login",meta={"cookiejar":1},callback=self.parse)]
    def parse(self, response):
        #設(shè)置要傳遞的post信息，此時沒有驗證碼字段
        data={
            "number":"swineherd",
            "passwd":"123",
        }
        print("登錄中……")
        #通過ForRequest.from_response()進(jìn)行登錄
        return FormRequest.from_response(response,
                                          #設(shè)置cookie信息
                                          meta={"cookiejar":response.meta["cookiejar"]},
                                          #設(shè)置headers信息模擬成瀏覽器
                                          headers=self.header,
                                          #設(shè)置post表單中的數(shù)據(jù)
                                          formdata=data,
                                          #設(shè)置回調(diào)函數(shù)
                                          callback=self.next,
                                          )
    def next(self,response):
        data=response.body
        fp=open("a.html","wb")
        fp.write(data)
        fp.close()
        print(response.xpath("/html/head/title/text()").extract())
        #登錄后訪問
        yield Request("http://edu.iqianyue.com/index_user_index",callback=self.next2,meta={"cookiejar":1})
    def next2(self,response):
        data=response.body
        fp=open("b.html","wb")
        fp.write(data)
        fp.close()
        print(response.xpath("/html/head/title/text()").extract())

scrapy新聞爬蟲實戰(zhàn)

目標(biāo)：爬取百度新聞首頁所有新聞

F:>cd scrapy項目

F:\scrapy項目>scrapy startproject baidunews

F:\scrapy項目>cd baidunews

F:\scrapy項目\baidunews>scrapy genspider -t basic n1 baidu.com

抓包分析

找到j(luò)son文件：

idle查看一下

首頁ctrl+f:

在首頁往下拖觸發(fā)所有新聞，在fiddler中找到存儲url、title等等的js文件（并不是每一個js文件都有用）

發(fā)現(xiàn)不止js文件有新聞信息，還有別的，要細(xì)心在fiddler找！

http://news.baidu.com/widget?id=LocalNews&ajax=json&t=1566824493194

http://news.baidu.com/widget?id=civilnews&t=1566824634139

http://news.baidu.com/widget?id=InternationalNews&t=1566824931323

http://news.baidu.com/widget?id=EnterNews&t=1566824931341

http://news.baidu.com/widget?id=SportNews&t=1566824931358

http://news.baidu.com/widget?id=FinanceNews&t=1566824931376

http://news.baidu.com/widget?id=TechNews&t=1566824931407

http://news.baidu.com/widget?id=MilitaryNews&t=1566824931439

http://news.baidu.com/widget?id=InternetNews&t=1566824931456

http://news.baidu.com/widget?id=DiscoveryNews&t=1566824931473

http://news.baidu.com/widget?id=LadyNews&t=1566824931490

http://news.baidu.com/widget?id=HealthNews&t=1566824931506

http://news.baidu.com/widget?id=PicWall&t=1566824931522

我們可以發(fā)現(xiàn)真正影響新聞信息的是widget?后面的id值

寫個腳本把id提取出來：

兩種不同的鏈接的源代碼的url也不同：

items.py:

          
            # -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class BaidunewsItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title=scrapy.Field()
    link=scrapy.Field()
    content=scrapy.Field()

n1.py:

          
            # -*- coding: utf-8 -*-
import scrapy
from baidunews.items import BaidunewsItem #從核心目錄
from scrapy.http import Request
import re
import time
class N1Spider(scrapy.Spider):
    name = 'n1'
    allowed_domains = ['baidu.com']
    start_urls = ["http://news.baidu.com/widget?id=LocalNews&ajax=json"]
    allid=['LocalNews', 'civilnews', 'InternationalNews', 'EnterNews', 'SportNews', 'FinanceNews', 'TechNews', 'MilitaryNews', 'InternetNews', 'DiscoveryNews', 'LadyNews', 'HealthNews', 'PicWall']
    allurl=[]
    for k in range(len(allid)):
        thisurl="http://news.baidu.com/widget?id="+allid[k]+"&ajax=json"
        allurl.append(thisurl)

    def parse(self, response):
        while True: #每隔5分鐘爬一次
            for m in range(len(self.allurl)):
                yield Request(self.allurl[m], callback=self.next)
                time.sleep(300) #單位為秒
    cnt=0
    def next(self,response):
        print("第" + str(self.cnt) + "個欄目")
        self.cnt+=1
        data=response.body.decode("utf-8","ignore")
        pat1='"m_url":"(.*?)"'
        pat2='"url":"(.*?)"'
        url1=re.compile(pat1,re.S).findall(data)
        url2=re.compile(pat2,re.S).findall(data)
        if(len(url1)!=0):
            url=url1
        else :
            url=url2
        for i in range(len(url)):
            thisurl=re.sub("\\\/","/",url[i])
            print(thisurl)
            yield Request(thisurl,callback=self.next2)
    def next2(self,response):
        item=BaidunewsItem()
        item["link"]=response.url
        item["title"]=response.xpath("/html/head/title/text()")
        item["content"]=response.body
        print(item)
        yield item

將settings的pipeline開啟：

將robots改為False，scrapy crawl n1 --nolog即可運行

scrapy豆瓣網(wǎng)登錄爬蟲

要在settings里加上：

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.54 Safari/536.5'

關(guān)于scrapy.http.FormRequest和scrapy.http.FormRequest.from_response的用法區(qū)別參考這篇博客：https://blog.csdn.net/qq_33472765/article/details/80958820

d1.py:

          
            # -*- coding: utf-8 -*-
import scrapy
from scrapy.http import Request, FormRequest


class D1Spider(scrapy.Spider):
    name = 'd1'
    allowed_domains = ['douban.com']
    # start_urls = ['http://douban.com/']
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"}

    def start_requests(self):
        # 首先爬一次登錄頁，然后進(jìn)入回調(diào)函數(shù)parse()
        print("開始：")
        return [Request("https://accounts.douban.com/passport/login",meta={"cookiejar":1},callback=self.login)]

    def login(self, response):
        #判斷驗證碼
        captcha=response.xpath("http://")
        data = {
            "ck": "",
            "name": "***",
            "password": "***",
            "remember": "false",
            "ticket": ""
        }
        print("登陸中……")
        return FormRequest(url="https://accounts.douban.com/j/mobile/login/basic",
                                         # 設(shè)置cookie信息
                                         meta={"cookiejar": response.meta["cookiejar"]},
                                         # 設(shè)置headers信息模擬成瀏覽器
                                         headers=self.headers,
                                         # 設(shè)置post表單中的數(shù)據(jù)
                                         formdata=data,
                                         # 設(shè)置回調(diào)函數(shù)
                                         callback=self.next,
                                         )
    def next(self,response):
        #跳轉(zhuǎn)到個人中心
        yield Request("https://www.douban.com/people/202921494/",meta={"cookiejar":1},callback=self.next2)
    def next2(self, response):
        title = response.xpath("/html/head/title/text()").extract()
        print(title)

現(xiàn)在的豆瓣是滑塊驗證碼，對于現(xiàn)在的我這個菜雞還不會處理。

在urllib中使用XPath表達(dá)式

先安裝lxml模塊：pip install lxml，然后將網(wǎng)頁數(shù)據(jù)通過lxml下的etree轉(zhuǎn)化為treedata的形式。

          
            import urllib.request
from lxml import etree
data=urllib.request.urlopen("http://www.baidu.com").read().decode("utf-8","ignore")
treedata=etree.HTML(data)
title=treedata.xpath("http://title/text()")
print(title)

更多文章、技術(shù)交流、商務(wù)合作、聯(lián)系博主

微信掃碼或搜索：z360901061

微信掃一掃加我為好友

QQ號聯(lián)系： 360901061

您的支持是博主寫作最大的動力，如果您喜歡我的文章，感覺我的文章對您有幫助，請用微信掃描下面二維碼支持博主2元、5元、10元、20元等您想捐的金額吧，狠狠點擊下面給點支持吧，站長非常感激您！手機(jī)微信長按不能支付解決辦法：請將微信支付二維碼保存到相冊，切換到微信，然后點擊微信右上角掃一掃功能，選擇支付二維碼完成支付。

【本文對您有幫助就好】元

2元

5元

10元

20元

自定義