bricks
开发指南
2.3 请求篇
2.3.1 Request 对象

2.3.1 Request 对象

Request对象是 bricks 在运行过程中的请求对象，通过这个对象可以定义请求方式，进行请求下载并获取响应等，Request对象存放于bricks.lib.request.Request

实例化

参数名参数类型参数描述默认值
urlstr请求的 URL无默认值，必传
methodstr请求方式，大小写方式均可"GET"
paramsOptional[dict]请求 URL 参数None
bodyOptional[Union[str, dict]]请求 Body，传入为字典的时候, 具体编码方式看 headers 里面的 Content-TypeNone
headersUnion[Header, dict]请求头None
cookiesDict[str, str]请求 cookiesNone
optionsdict额外参数，一般针对下载器的一些额外参数都放在这里None
timeoutint请求超时时间...
allow_redirectsbool是否允许自动重定向True
proxiesOptional[str]请求的代理，如 http://127.0.0.1:7890, 理解为 current proxyNone
proxyOptional[Union[dict, str, List[Union[dict, str]]]]代理 Key, 理解为 proxy from empty
okOptional[Union[str, Dict[str, Union[type(signals.Signal), Callable]]]]判断成功动态脚本, 字符串形式, 如通过 403 状态码可以写为: 200 <= response.status_code < 400 or response.status_code == 403empty， 不设置为spider.parse
retryint当前重试次数1
max_retry[int, float]最大重试次数5
use_session[int, float]是否使用下载器的 session 模式False
from bricks.lib.request import Request
 
req = Request(
    url='https://www.baidu.com/',
    headers={
        'Connection': 'keep-alive',
        'Pragma': 'no-cache',
        'Cache-Control': 'no-cache',
        'sec-ch-ua': '" Not;A Brand";v="99", "Microsoft Edge";v="91", "Chromium";v="91"',
        'sec-ch-ua-mobile': '?0',
        'DNT': '1',
        'Upgrade-Insecure-Requests': '1',
        'User-Agent': 'user_agent.firefox(device="pc")',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
        'Sec-Fetch-Site': 'none',
        'Sec-Fetch-Mode': 'navigate',
        'Sec-Fetch-User': '?1',
        'Sec-Fetch-Dest': 'document',
        'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,pt;q=0.5',
    }
)

from_curl

如果你需要将一个 Curl 转换为一个 Request 对象，你可以使用以下方式：

 
from bricks.lib.request import Request
from bricks.utils import convert
 
curl = '''
curl 'https://www.baidu.com/sugrec?prod=pc_his&from=pc_web&json=1&sid=&hisdata=%5B%7B%22time%22%3A1628471166%2C%22kw%22%3A%22%2Fphpmyadmin%2F%22%7D%2C%7B%22time%22%3A1628471207%2C%22kw%22%3A%22%2Fboaform%2Fadmin%2Fformlogin%22%2C%22fq%22%3A2%7D%2C%7B%22time%22%3A1628473507%2C%22kw%22%3A%22add_argument()%E4%B8%AD%E7%9A%84%E5%8F%82%E6%95%B0%22%7D%2C%7B%22time%22%3A1628477075%2C%22kw%22%3A%22fastapi%20%E8%B0%83%E7%94%A8%E5%86%85%E9%83%A8%E8%A7%86%E5%9B%BE%22%7D%2C%7B%22time%22%3A1628477991%2C%22kw%22%3A%22fastapi%20%E8%B0%83%E7%94%A8api%22%7D%2C%7B%22time%22%3A1628484995%2C%22kw%22%3A%22crontab%20%E6%AF%8F%E5%B0%8F%E6%97%B6%22%7D%2C%7B%22time%22%3A1628485024%2C%22kw%22%3A%22crontab%20%E6%AF%8F%E5%B0%8F%E6%97%B6%E6%95%B4%E7%82%B9%22%2C%22fq%22%3A2%7D%2C%7B%22time%22%3A1628487556%2C%22kw%22%3A%22%E8%8E%B7%E5%8F%96%20self%20%E7%9A%84%E5%8F%98%E9%87%8F%E5%90%8D%22%7D%2C%7B%22time%22%3A1628487592%2C%22kw%22%3A%22%E6%A0%B9%E6%8D%AE%E5%8F%98%E9%87%8F%E5%80%BC%E8%8E%B7%E5%8F%96%E5%8F%98%E9%87%8F%E5%90%8D%22%7D%2C%7B%22time%22%3A1628488959%2C%22kw%22%3A%22python%20%E8%AE%BE%E7%BD%AE%E7%8E%AF%E5%A2%83%E5%8F%98%E9%87%8F%22%2C%22fq%22%3A2%7D%5D&_t=1644977838770&req=2&bs=1&csor=0' \
-H 'Connection: keep-alive' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache' \
-H 'sec-ch-ua: " Not;A Brand";v="99", "Microsoft Edge";v="97", "Chromium";v="97"' \
-H 'Accept: application/json, text/javascript, */*; q=0.01' \
-H 'DNT: 1' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36 Edg/97.0.1072.76' \
-H 'sec-ch-ua-platform: "macOS"' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Dest: empty' \
-H 'Referer: https://www.baidu.com/s?tn=44004473_48_oem_dg&ie=utf-8&wd=1' \
-H 'Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,pt;q=0.5' \
-H 'Cookie: BDUSS=241MUc2a35sY0ZhVFZpSERqT1RHTERiUk5xWXhHUXpTT0FxTTUtV0p2bnF6R2hoRVFBQUFBJCQAAAAAAAAAAAEAAAChYIhqS0tLS0tLS0VNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOo~QWHqP0FhV; BDUSS_BFESS=241MUc2a35sY0ZhVFZpSERqT1RHTERiUk5xWXhHUXpTT0FxTTUtV0p2bnF6R2hoRVFBQUFBJCQAAAAAAAAAAAEAAAChYIhqS0tLS0tLS0VNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOo~QWHqP0FhV; __yjs_duid=1_a84547a403081c40e13d1eb63b5320dc1631840700133; BIDUPSID=80506F5A0D6432B19F1C8543597DDCD7; PSTM=1633615692; BAIDUID=B20E1955C785681DA9CBC05B171FC1E6:FG=1; BDORZ=FFFB88E999055A3F8A630C64834BD6D0; H_WISE_SIDS=107317_110085_127969_174441_179346_181536_184716_185637_186844_188333_189037_189755_190247_190792_191068_191246_191288_191371_192206_192388_193284_194085_194520_195328_195342_196426_196514_196527_197286_197471_197711_197782_197829_197955_198068_198256_198649_199023_199082_199156_199176_199305_199490_199579_199598_199752_199847_200029_200048_200090_200273_200350_200763_200959_200960_200993_201055_201185_201232_201328_201445_201538_201554_201581_201700_201733_201820_201825_201948_201979_201996_202177_202298_202477_202554_202561_202893_202915_203196_203251_203538_203684; H_PS_PSSID=; BD_UPN=123253; BDRCVFR[tLeyU5FIL-6]=mk3SLVN4HKm; delPer=0; BD_CK_SAM=1; PSINO=7; sugstore=1; H_PS_645EC=11d1%2BH7MDi2Py0X75NSXY1op7oLPB5X8CupEHj948YL%2FH%2FW%2BQiJSbjAcbS687n0Wf7bZNbnsNUSG; BA_HECTOR=aha4058g2k21ag2k541h0onle0r' \
--compressed
 
 
'''
req = Request.from_curl(curl)
resp = convert.req2resp(req)  # 使用内置的 convert 方法来消耗 request
print(resp)

图 0

put_options / get_options

为 request 设置额外的属性，一般下载器经常用到

from bricks.lib.request import Request
 
req = Request(
    url='https://www.baidu.com/',
    headers={
        'Connection': 'keep-alive',
        'Pragma': 'no-cache',
        'Cache-Control': 'no-cache',
        'sec-ch-ua': '" Not;A Brand";v="99", "Microsoft Edge";v="91", "Chromium";v="91"',
        'sec-ch-ua-mobile': '?0',
        'DNT': '1',
        'Upgrade-Insecure-Requests': '1',
        'User-Agent': 'user_agent.firefox(device="pc")',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
        'Sec-Fetch-Site': 'none',
        'Sec-Fetch-Mode': 'navigate',
        'Sec-Fetch-User': '?1',
        'Sec-Fetch-Dest': 'document',
        'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,pt;q=0.5',
    }
)
 
req.put_options("name", "kem")
print(req.get_options("name"))

图 1

curl

将 request 转为 curl 对象，用于 postman 之类的软件调试

 
from bricks.lib.request import Request
 
req = Request(
    url='https://www.baidu.com/',
    headers={
        'Connection': 'keep-alive',
        'Pragma': 'no-cache',
        'Cache-Control': 'no-cache',
        'sec-ch-ua': '" Not;A Brand";v="99", "Microsoft Edge";v="91", "Chromium";v="91"',
        'sec-ch-ua-mobile': '?0',
        'DNT': '1',
        'Upgrade-Insecure-Requests': '1',
        'User-Agent': 'user_agent.firefox(device="pc")',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
        'Sec-Fetch-Site': 'none',
        'Sec-Fetch-Mode': 'navigate',
        'Sec-Fetch-User': '?1',
        'Sec-Fetch-Dest': 'document',
        'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,pt;q=0.5',
    }
)
 
print(req.curl)

图 2

2.2.4 启动参数2.3.2 代码开发