标准化爬虫,接口验证

目标网站

不确定请求头,不确定请求体格式,进行接口验证,标准化爬虫

  1. 找到数据接口(XHR抓包搜索)
  2. 用接口的路径断点调试,找到请求对象的发送位置
  3. 补充请求头,更改data…
1
2
3
4
5
6
7
8
9
10
11
12
import requests

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"Accept": "application/json, text/plain, */*",
"Content-Type": "application/json;charset=utf-8",
"X-Requested-With": "XMLHttpRequest"}
url = "http://www.whggzy.com/portal/category"
data = "{\"pageNo\":1,\"pageSize\":15,\"categoryCode\":\"GovernmentProcurement\",\"_t\":1719906086000}"

resp = requests.post(url=url, headers=headers, data=data)

print(resp.json())