Scrapefy is a Node.js scraping toolkit built to extract structured product data from five major Albanian and Kosovar e-commerce platforms — Gjirafa50, Foleja, Buton KS, Point Computers, and StarTech24 — each with its own DOM structure, pagination scheme, and anti-bot quirks. Rather than crawling by page number, which most of these sites cap or throttle, Scrapefy walks each site's product categories directly, sidestepping pagination limits entirely and giving more predictable, complete coverage per store.
The scraper runs categories and batches concurrently, with adaptive rate limiting and randomized delays built in to avoid tripping anti-scraping defenses. A multi-retry system with exponential backoff absorbs transient failures without losing progress, and a resume system checkpoints completed work so a long-running job can recover from a crash or manual stop without re-scraping from scratch. Real-time progress statistics surface as the job runs, so long scraping sessions across thousands of products stay observable.
Each extracted product record includes title, canonical URL, price, image, and store-specific metadata, normalized into a consistent output format despite the five source sites having nothing in common structurally.
Example output:
[
{
"title": "Laptop Lenovo NB LOQ 15ARP9, 15.6\", AMD Ryzen 5...",
"url": "https://gjirafa50.com/laptop-lenovo-nb-loq-15arp9...",
"price": "599.50 €",
"image": "https://iqq6kf0xmf.gjirafa.net/images/10c87e6c...",
"categoryId": "1179",
"scraped_at": "2024-01-15T10:30:00.000Z"
}
]