Code isn't working. I'm trying to scrape multiple pages from differents brands (categories). The page that I'm using have a list of all the brands. The brands are arranged in groups, which are classified by the initial letter of that brand. And inside the page of this brands, there are multiple pages os differents products that the brand has.
Tried to do a code that takes the brand using a counter, and if there are no brands in this specific first-letter-group, it goes to the next gorup. (The request is fine, the problem is in the code. The scrape by itself is working, only when I try this pagination that the code fails).
import scrapyfrom scrapy import Requestclass MlSpider(scrapy.Spider): name = "ml" def start_requests(self): yield scrapy.Request('https://lista.mercadolivre.com.br/produtos-cabelo') def parse(self, response, **kwargs): cgroup = 1 cbrand = 1 num_group = response.xpath(f'//div[@class="ui-search-search-modal-filter-group"][{cgroup}]').get() for m in num_group: link_marca = m.xpath(f'.//a[@class="ui-search-search-modal-filter ui-search-link"][{cbrand}]/@href').get() if link_marca: yield scrapy.Request(url=link_marca) for i in response.xpath('.//div[@class="ui-search-result__content"]'): marca = i.xpath('.//span[@class="ui-search-item__brand-discoverability ui-search-item__group__element"]/text()').get() title = i.xpath('.//h2/text()').get() real = i.xpath('.//span[@class="andes-money-amount ui-search-price__part ui-search-price__part--medium andes-money-amount--cents-superscript"]//span[@class="andes-money-amount__fraction"]/text()').get() centavo = i.xpath('//span[@class="andes-money-amount ui-search-price__part ui-search-price__part--medium andes-money-amount--cents-superscript"]//span[@class="andes-money-amount__cents andes-money-amount__cents--superscript-24"]/text()').get() value = f'R$ {real},{centavo}' link = i.xpath('.//a/@href').get() yield {'marca': marca,'title': title,'value': value,'link': link } next_page = response.xpath('//a[contains(@title,"Seguinte")]/@href').get() if next_page: yield scrapy.Request(url=next_page, callback=self.parse) cbrand += 1 else: cgroup += 1