webcrawling

HTTP API for Scrapy spiders

Raspagem de dados para iniciante usando Scrapy e outras libs básicas

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

Web-scraping script that writes the data of all players from FutHead and FutBin to a CSV file or a DB

Automates the process of repeatedly searching for a website via scraped proxy IP and search keywords

API to parse tibia.com content into python objects.

Crawl4DeepSeek = Crawl4AI + DeepSeek 🚀 Smart, efficient, and built for deep web exploration! 🌐🤖

This program aims to check active targets by saving screenshots in a project.

WebDiver is a versatile Python script for crawling websites, extracting internal and external links, titles, and descriptions. It's useful for tasks such as web analysis, OSINT (Open Source Intelligence) gathering, and competitive analysis.

Example frontera project

Python async data gathering

Find open position on ponisha (Freelancering job offer website)

A Web Crawler developed in Python.

This is an automatic message fowarder bot within WhatsApp using Python and Selenium

👻Web Crawling and Convert to Executable with Pyinstaller

Easy to use web page analyzer

Cross-platform Python crawler that finds and verifies downloadable media, documents, and other files, then creates wget-ready URL lists for fast bulk downloading. It also scans sitemap trees and generates validated text or XML sitemaps, with HTTP, HTTPS, FTP, persistent SQLite history, resumable crawls, robots support, and no pip dependencies.

Scrapes attendance and marks related data from AURIS (Ahmedabad University Resource Information System) and notifies the user without him having to check his data repeatedly