webcrawler

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Open-source Enterprise Grade Search Engine Software

Bot para monitoramento de promoções no fórum do Hardmob http://www.hardmob.com.br/promocoes/

Java web crawling library

A ZAPROXY Add-on that allows testing of web application vulnerabilities by recording complex multi-step sequences. You can test applications that need to access pages in a specific order, such as shopping carts or registration of member information.

SearchEngine and Web Crawler

Sample MVP project uses jsoup-web-crawl like API

A simple ZhiHu Crawler using WebMagic

Multi-threaded Java web crawler library with flexible search strategies and configurable persistence options. Published on Maven Central, features intelligent crawling with politeness controls, custom filtering, and extensible architecture using Strategy design pattern.

An async web crawler for ads.txt project

A simple Web Crawler built with Spring Boot.

A web crawler that implements breadth first search algorithm and built with maven.

This project provides a REST API that allows users to submit URLs for crawling. The app internally uses RabbitMQ to publish the URLs, and then listens back to fetch the contents of the URLs using Jsoup. The app also scrapes links and indexes the content using Apache Lucene.

Web-crawler - fetches available links from the given website and provides keyword search

自动爬虫

Cosmos is a WebCrawler + SearchEngine written in Java

spring-boot-webcrawler-rest-demo-using-completable-futures

simple web crawler java 11 lts