I'm trying to write a script that will automatically parse product cards. I managed to cope with one page. How can I make the script automatically go to another page?
I've seen several answers where selenium helped. But I couldn't figure it out.
Here's the code:
import randomimport stringimport csv import requestsfrom bs4 import BeautifulSoupurl = "https://game29.ru/products?category=926"response = requests.get(url)html = response.textmulti_class = {'class': ['row'], 'style': 'border: 2px solid #898989;border-radius: 7px;padding: 2px;margin-top: -2px;'}soup = BeautifulSoup(html, "html.parser")products = soup.find_all("div", {"class":"row"})identifaer = "".join([random.choice(string.ascii_letters + string.digits) for n in range(32)])ad_status = "Free"category = "Игры, приставкиипрограммы"goods_type = "Игрыдляприставок"ad_type = "Продаюсвоё"adress = ""discription = ""condition = "Новое"data_begin = "2024-04-03"data_end = "2024-05-03" allow_email = "Нет"contact_phone = ""contact_method = "Потелефонуивсообщениях"all_products = []for product in products: if product.attrs == multi_class: identifaer image ="https://www.game29.ru" + product.find("img")["src"] if image != "https://game29.ru/zaglushka.png": title = product.find("div", {"class":"cart-item-name"}).text price = product.find("div", {"class": "cart-item-price"}).text.strip().replace("руб.", "") all_products.append([identifaer, ad_status, category, goods_type, ad_type, adress, title, discription, condition, price, data_begin, data_end, allow_email, contact_phone, image, contact_method])# names = ["Id", "AdStatus", "Category", "GoodsType", "Adtype", "Adress", "Title", "Discription", "Condition", "Price", "DataBegin", "DataEnd", "AllowEmail", "ContactPhone","ImageUrls", "ContactMethod"]with open("data.csv", "a", newline='') as csv.file: writer = csv.writer(csv.file, delimiter=',') # writer.writerow(names) for product in all_products: writer.writerow(product)I really thought selenium would help me. And I think that the answer is there, but unfortunately, I don't understand it yet, and I don’t have much time. I would be glad if you could help me.