Python爬蟲要經(jīng)歷爬蟲、爬蟲被限制、爬蟲反限制的過程。當(dāng)然后續(xù)還要網(wǎng)頁爬蟲限制優(yōu)化，爬蟲再反限制的一系列道高一尺魔高一丈的過程。爬蟲的初級(jí)階段，添加headers和ip代理可以解決很多問題。
本人自己在爬取豆瓣讀書的時(shí)候,就以為爬取次數(shù)過多,直接被封了IP.后來就研究了代理IP的問題.
(當(dāng)時(shí)不知道什么情況,差點(diǎn)心態(tài)就崩了…),下面給大家介紹一下我自己代理IP爬取數(shù)據(jù)的問題,請(qǐng)大家指出不足之處.
問題
這是我的IP被封了,一開始好好的,我還以為是我的代碼問題了

從網(wǎng)上查找了一些關(guān)于爬蟲代理IP的資料,得到下面的思路
爬取一些IP,過濾掉不可用.
在requests的請(qǐng)求的proxies參數(shù)加入對(duì)應(yīng)的IP.
繼續(xù)爬取.
收工
好吧,都是廢話,理論大家都懂,上面直接上代碼…
思路有了,動(dòng)手起來.
運(yùn)行環(huán)境
Python 3.7, Pycharm
這些需要大家直接去搭建好環(huán)境…
準(zhǔn)備工作
爬取IP地址的網(wǎng)站(國內(nèi)高匿代理)
校驗(yàn)IP地址的網(wǎng)站
你之前被封IP的py爬蟲腳本…
上面的網(wǎng)址看個(gè)人的情況來選取
爬取IP的完整代碼
PS:簡單的使用bs4獲取IP和端口號(hào),沒有啥難度,里面增加了一個(gè)過濾不可用IP的邏輯
關(guān)鍵地方都有注釋了
#!/usr/bin/env python3

- - coding: utf-8 - -

@Time : 2018/11/22

@Author : liangk

@Site :

@File : auto_archive_ios.py

@Software: PyCharm

import requests
from bs4 import BeautifulSoup
import json

class GetIp(object):
“”“抓取代理IP”""
def init (self):
“”“初始化變量”""
self.url = ‘http://www.xicidaili.com/nn/’
self.check_url = ‘https://www.ip.cn/’
self.ip_list = []
@staticmethod
def get_html(url):
“”“請(qǐng)求html頁面信息”""
header = {
‘User-Agent’: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36’
}
try:
request = requests.get(url=url, headers=header)
request.encoding = ‘utf-8’
html = request.text
return html
except Exception as e:
return ‘’
def get_available_ip(self, ip_address, ip_port):
“”“檢測(cè)IP地址是否可用”""
header = {
‘User-Agent’: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36’
}
ip_url_next = ‘/’ + ip_address + ‘:’ + ip_port
proxies = {‘http’: ‘http’ + ip_url_next, ‘https’: ‘https’ + ip_url_next}
try:
r = requests.get(self.check_url, headers=header, proxies=proxies, timeout=3)
html = r.text
except:
print(‘fail-%s’ % ip_address)
else:
print(‘success-%s’ % ip_address)
soup = BeautifulSoup(html, ‘lxml’)
div = soup.find(class_=‘well’)
if div:
print(div.text)
ip_info = {‘a(chǎn)ddress’: ip_address, ‘port’: ip_port}
self.ip_list.append(ip_info)
def main(self):
“”“主方法”""
web_html = self.get_html(self.url)
soup = BeautifulSoup(web_html, ‘lxml’)
ip_list = soup.find(id=‘ip_list’).find_all(‘tr’)
for ip_info in ip_list:
td_list = ip_info.find_all(‘td’)
if len(td_list) > 0:
ip_address = td_list[1].text
ip_port = td_list[2].text
# 檢測(cè)IP地址是否有效
self.get_available_ip(ip_address, ip_port)
# 寫入有效文件
with open(‘ip.txt’, ‘w’) as file:
json.dump(self.ip_list, file)
print(self.ip_list)

程序主入口

if name == ‘ main ’:
get_ip = GetIp()
get_ip.main()
使用方法完整代碼
PS: 主要是通過使用隨機(jī)的IP來爬取,根據(jù)request_status來判斷這個(gè)IP是否可以用.
為什么要這樣判斷?
主要是雖然上面經(jīng)過了過濾,但是不代表在你爬取的時(shí)候是可以用的,所以還是得多做一個(gè)判斷.
#!/usr/bin/env python3

- - coding: utf-8 - -

@Time : 2018/11/22

@Author : liangk

@Site :

@File : get_douban_books.py

@Software: PyCharm

from bs4 import BeautifulSoup
import datetime
import requests
import json
import random
ip_random = -1
article_tag_list = []
article_type_list = []

def get_html(url):
header = {
‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36’
}
global ip_random
ip_rand, proxies = get_proxie(ip_random)
print(proxies)
try:
request = requests.get(url=url, headers=header, proxies=proxies, timeout=3)
except:
request_status = 500
else:
request_status = request.status_code
print(request_status)
while request_status != 200:
ip_random = -1
ip_rand, proxies = get_proxie(ip_random)
print(proxies)
try:
request = requests.get(url=url, headers=header, proxies=proxies, timeout=3)
except:
request_status = 500
else:
request_status = request.status_code
print(request_status)
ip_random = ip_rand
request.encoding = ‘gbk’
html = request.content
print(html)
return html

def get_proxie(random_number):
with open(‘ip.txt’, ‘r’) as file:
ip_list = json.load(file)
if random_number == -1:
random_number = random.randint(0, len(ip_list) - 1)
ip_info = ip_list[random_number]
ip_url_next = ‘/’ + ip_info[‘a(chǎn)ddress’] + ‘:’ + ip_info[‘port’]
proxies = {‘http’: ‘http’ + ip_url_next, ‘https’: ‘https’ + ip_url_next}
return random_number, proxies

程序主入口

if name == ‘ main ’:
“”“只是爬取了書籍的第一頁,按照評(píng)價(jià)排序”""
start_time = datetime.datetime.now()
url = ‘https://book.douban.com/tag/?view=type&icn=index-sorttags-all’
base_url = ‘https://book.douban.com/tag/’
html = get_html(url)
soup = BeautifulSoup(html, ‘lxml’)
article_tag_list = soup.find_all(class_=‘tag-content-wrapper’)
tagCol_list = soup.find_all(class_=‘tagCol’)
for table in tagCol_list:
“”" 整理分析數(shù)據(jù) “”"
sub_type_list = []
a = table.find_all(‘a(chǎn)’)
for book_type in a:
sub_type_list.append(book_type.text)
article_type_list.append(sub_type_list)
for sub in article_type_list:
for sub1 in sub:
title = ‘ ’ + sub1 + ' ’
print(title)
print(base_url + sub1 + ‘?start=0’ + ‘&type=S’)
with open(‘book.text’, ‘a(chǎn)’, encoding=‘utf-8’) as f:
f.write(’\n’ + title + ‘\n’)
f.write(url + ‘\n’)
for start in range(0, 2):
# (start * 20) 分頁是0 20 40 這樣的
# type=S是按評(píng)價(jià)排序
url = base_url + sub1 + ‘?start=%s’ % (start * 20) + ‘&type=S’
html = get_html(url)
soup = BeautifulSoup(html, ‘lxml’)
li = soup.find_all(class_=‘subject-item’)
for div in li:
info = div.find(class_=‘info’).find(‘a(chǎn)’)
img = div.find(class_=‘pic’).find(‘img’)
content = ‘書名:<%s>’ % info[‘title’] + ’ 書本圖片:’ + img[‘src’] + ‘\n’
print(content)
with open(‘book.text’, ‘a(chǎn)’, encoding=‘utf-8’) as f:
f.write(content)
end_time = datetime.datetime.now()
print('耗時(shí): ', (end_time - start_time).seconds)
為什么選擇國內(nèi)高匿代理，因?yàn)槭褂眠@樣簡單的代理IP,基本上就可以應(yīng)付在爬爬爬著被封IP的情況了.而且沒有使用自己的IP,間接的保護(hù)