I'm trying to run a Python script for web scraping that was originally created to run locally. I made the necessary modifications to adapt it to the Colab environment, but the script can't retrieve results from the website, although it works normally when run locally.
The modified script:
!pip install selenium!apt-get update!apt-get install -y libgconf-2-4!apt-get install -y libxss1 libappindicator1 libindicator7!apt-get install xvfbfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECfrom selenium import webdriverfrom selenium.webdriver.common.by import Byimport osimport time# Set up Chrome optionsoptions = webdriver.ChromeOptions()options.add_argument('--headless')options.add_argument('--no-sandbox')options.add_argument('--disable-dev-shm-usage')# Set up the Chrome driverdriver = webdriver.Chrome(options=options)driver.get("https://www.virustotal.com/gui/home/search")wait = WebDriverWait(driver, 15)search_input = wait.until(EC.element_to_be_clickable((driver.execute_script("return document.querySelector('home-view').shadowRoot.querySelector('vt-ui-search-bar').shadowRoot.querySelector('input#searchInput')"))))search_input.send_keys("759b1fcbdf4ad54bb8adc2c060df8ab5")time.sleep(10)result = wait.until(EC.element_to_be_clickable((driver.execute_script("return document.querySelector('file-view').shadowRoot.querySelector('vt-ui-main-generic-report').shadowRoot.querySelector('vt-ui-detections-widget').shadowRoot.querySelector('div.positives')"))))result_text = str(result.text)print(result_text)driver.quit()The received error :
JavascriptException: Message: javascript error: Cannot read properties of null (reading 'shadowRoot') (Session info: chrome-headless-shell=120.0.6099.71)Stacktrace:#0 0x55599bde0d33 <unknown>#1 0x55599ba9df87 <unknown>#2 0x55599baa4533 <unknown>#3 0x55599baa6e44 <unknown>#4 0x55599bb36e43 <unknown>#5 0x55599bb17342 <unknown>#6 0x55599bb36297 <unknown>#7 0x55599bb170e3 <unknown>#8 0x55599badf044 <unknown>#9 0x55599bae044e <unknown>#10 0x55599bda5861 <unknown>#11 0x55599bda9785 <unknown>#12 0x55599bd93285 <unknown>#13 0x55599bdaa41f <unknown>#14 0x55599bd7720f <unknown>#15 0x55599bdce028 <unknown>#16 0x55599bdce1f7 <unknown>#17 0x55599bddfed4 <unknown>#18 0x791291309ac3 <unknown>