Requests And Json
Webscraping with XHR
Inspect page
Navigate to Network > XHR > Name
Open in new tab or curl
Figure out pagination pattern from json link
Check if get or post (Copy > Copy Request Headers
Add parameters ()
import re
resp = requests.post(
"https://data.fcm.ca/dynamics/projects",
data={
"f": "gmfprojects",
"p": "%",
"p1": page_num,
"c": "en"
}
)
resp = requests.get(
"https://data.fcm.ca/dynamics/projects",
params={
"f": "gmfprojects",
"p": "%",
"p1": page_num,
"c": "en"
}
)
Extract JSON Data
for element in data[index]['name']['name']....:
print(element)
Last updated
Was this helpful?