user-agent與cookie查詢與配置
用戶代理（user-agent）是瀏覽器客戶端與服務器交互時的重要信息之一，用于幫助網站識別請求用戶的瀏覽器類別，以便于網站發送相應的網頁數據。用戶代理數據包括：操作系統標識、加密等級標識和瀏覽器語言。如今越來越多的網絡自動搜索工具出現，尤其是網絡爬蟲，為了做到更好地、流暢地網絡搜索，用戶代理信息是不可或缺的。
開發人員工具 ——> network ——> 大小排序
![[Python爬蟲] user-agent與cookie查詢與配置_第1張圖片](https://img.it610.com/image/info8/b2e3cb70f1554aa39407a29ad23cf13c.jpg)
![[Python爬蟲] user-agent與cookie查詢與配置_第2張圖片](https://img.it610.com/image/info8/2249803c9544483a9bcfe63d5fb2e027.jpg)
常用user-agent
class
UA
(
)
:
USER_AGENTS
=
[
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60'
,
'Opera/8.0 (Windows NT 5.1; U; en)'
,
'Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.50'
,
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 9.50'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0'
,
'Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36'
,
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11'
,
'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER'
,
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)'
,
'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0'
,
'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; SE 2.X MetaSr 1.0)'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 UBrowser/4.0.3214.0 Safari/537.36'
]
def
get_proxy
(
self
)
:
self
.
page
+=
1
headers
=
{
"User-Agent"
:
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
}
request
=
urllib2
.
Request
(
"https://www.kuaidaili.com/free/inha/"
+
str
(
self
.
page
)
,
headers
=
headers
)
html
=
urllib2
.
urlopen
(
request
)
.
read
(
)
爬蟲headers格式：
headers
=
{
"Host"
:
"######"
,
"Connection"
:
"keep-alive"
,
"Cache-Control"
:
"max-age=0"
,
"Upgrade-Insecure-Requests"
:
"1"
,
"User-Agent"
:
user_agent
,
"Accept-Encoding"
:
"gzip, deflate, br"
,
"Accept-Language"
:
"zh-CN,zh;q=0.9"
,
"Cookie"
:
"######"
}
更多文章、技術交流、商務合作、聯系博主
微信掃碼或搜索：z360901061
微信掃一掃加我為好友
QQ號聯系： 360901061
您的支持是博主寫作最大的動力，如果您喜歡我的文章，感覺我的文章對您有幫助，請用微信掃描下面二維碼支持博主2元、5元、10元、20元等您想捐的金額吧，狠狠點擊下面給點支持吧，站長非常感激您！手機微信長按不能支付解決辦法：請將微信支付二維碼保存到相冊，切換到微信，然后點擊微信右上角掃一掃功能，選擇支付二維碼完成支付。
【本文對您有幫助就好】元
