good day dear coding ppl.
well i have issues to get this runned in google-colab
However, i did some checks - and tried using the cloudscraper module.
For example:
import cloudscraperimport pandas as pdfrom bs4 import BeautifulSoupfrom tabulate import tabulateheaders = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS Xyz) ""AppleWebKit/537.36 (KHTML, like Gecko) ""Chrome/113.0.0.0 Safari/537.36 Edg/xyz",}scraper = cloudscraper.create_scraper()response = scraper.get("https://clutch.co/il/it-services", headers=headers)soup = BeautifulSoup(response.content, "html.parser")company_names = soup.select(".directory-list div.provider-info--header .company_info a")locations = soup.select(".locality")company_names_list = [name.get_text(strip=True) for name in company_names]locations_list = [location.get_text(strip=True) for location in locations]data = {"Company Name": company_names_list, "Location": locations_list}df = pd.DataFrame(data)df.index += 1print(tabulate(df, headers="keys", tablefmt="psql"))df.to_csv("it_services_data.csv", index=False)what i get back is this
+----------------+------------+| Company Name | Location ||----------------+------------|+----------------+------------+