I am trying to scrape the following site travelocity.com.
My code should do the following:
- enter the site
- scroll down to the bottom of the page
- click button "Show more" (which loads more results)
- scroll down to the bottom of the page
- click button "Show more" (which loads more results)
- etc.
But when it click the "Show more" button it does not update more results, it just keeps clicking on it and nothing happens. What could be the issue? I presume the site detects I am a bot.
It is this button at the bottom of the page:
My current code for clicking and releasing the button is the following:
try: show_more_button = driver.find_element(By.XPATH, "//button[contains(text(), 'Show more')]") action_chains = ActionChains(driver) action_chains.click_and_hold(show_more_button).perform() time.sleep(1) action_chains.release().perform() time.sleep(5)except NoSuchElementException: print("All results are loaded.") breakI have also tried:
show_more_button = driver.find_element(By.XPATH, "//button[contains(text(), 'Show more')]")show_more_button.click()but it didn't work as well.
Here is the full code:
import timefrom selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.common.exceptions import NoSuchElementExceptionfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.common.action_chains import ActionChainsURL = 'https://www.travelocity.com/Hotel-Search?destination=San%20Francisco%20%28and%20vicinity%29%2C%20California%2C%20United%20States%20of%20America®ionId=178305&latLong=37.7874%2C-122.4082&flexibility=0_DAY&d1=2024-05-08&startDate=2024-05-08&d2=2024-05-22&endDate=2024-05-22&adults=2&rooms=1&theme=&userIntent=&semdtl=&useRewards=false&sort=RECOMMENDED'driver = webdriver.Chrome()driver.get(URL)driver.maximize_window()time.sleep(4)while True: try: driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") show_more_button = driver.find_element(By.XPATH, "//button[contains(text(), 'Show more')]") action_chains = ActionChains(driver) action_chains.click_and_hold(show_more_button).perform() time.sleep(0.5) action_chains.release().perform() time.sleep(3) except NoSuchElementException: print("All results are loaded.") breakdriver.quit()
