Quantcast
Viewing all articles
Browse latest Browse all 14011

Replicating Get request fails to scrape data [closed]

I've been trying to access data from this website:

https://livedragon.vdsc.com.vn/hsx/hsxInit.rv?groupId=VN30&sectorId=0

I looked at DeveloperTools and saw a Get request:hsxInit.rv?groupId=VN30&sectorId=0I tried to replicate the Get request and never got it to work. One of the versions is as follows:

def scrape_rv():    url = 'https://livedragon.vdsc.com.vn/hsx/hsxInit.rv?groupId=VN30&sectorId=0'    headers = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7','Accept-Encoding': 'gzip, deflate, br','Accept-Language': 'en-US,en;q=0.9,vi-VN;q=0.8,vi;q=0.7,fr-FR;q=0.6,fr;q=0.5','Cache-Control': 'max-age=0','Connection': 'keep-alive','Host': 'livedragon.vdsc.com.vn','Sec-Fetch-Dest': 'document','Sec-Fetch-Mode': 'navigate','Sec-Fetch-Site': 'none','Sec-Fetch-User': '?1','Upgrade-Insecure-Requests': '1','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36','sec-ch-ua': '"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"','sec-ch-ua-mobile': '?0','sec-ch-ua-platform': '"Windows"'}    params = {'groupId': 'VN30', 'sectorId': 0}        rck = requests.get(url, headers= headers, params = params)    # to mimic the behavior of loading the website twice     # just to get cookie?    ck = rck.headers['Set-Cookie']        h2 = headers    h2['Cookie'] = ck    r = requests.get(url, headers= h2, params = params)    return r.text

One thing I noticed is that when I open it with a normal browser (Google Chrome), I have to open it twice. First time, it returns a page with:{"success":false,"groupType":1,"grids":[],"message":"exception"}then I refresh the page - and boom, I get a lot of data on the page.

I could scrape the data with selenium + beautifulsoup from my local desktop. but would love to make it work with just a simple Get request. Please help.Thanks.


Viewing all articles
Browse latest Browse all 14011

Trending Articles