mirror of
https://github.com/YGGverse/aquatic-crawler.git
synced 2026-03-31 17:15:35 +00:00
56 lines
2.5 KiB
Markdown
56 lines
2.5 KiB
Markdown
# aquatic-crawler
|
|
|
|
|
|
[](https://deps.rs/repo/github/YGGverse/aquatic-crawler)
|
|
[](https://crates.io/crates/aquatic-crawler)
|
|
|
|
SSD-friendly FS crawler for the [Aquatic](https://github.com/greatest-ape/aquatic) BitTorrent tracker, based on the [librqbit](https://github.com/ikatson/rqbit/tree/main/crates/librqbit) API
|
|
|
|
> [!NOTE]
|
|
> * requires PR#233 or the [info-hash-api](https://github.com/YGGverse/aquatic/tree/info-hash-api) branch implementation
|
|
> * compatible with any other `--infohash` source in `hash1hash2...` binary format
|
|
> * use [βtracker](https://github.com/YGGverse/btracker/) for the frontend
|
|
> * see the project [Wiki](https://github.com/YGGverse/aquatic-crawler/wiki) for details
|
|
|
|
## Conception
|
|
|
|
```
|
|
torrent client > aquatic_udp > infohash.bin < aquatic-crawler > * /preload/info-hash.torrent
|
|
torrent client <-----------------------| * /preload/info-hash/data
|
|
<-------------------------------------| * /preload/.info-hash/tmp
|
|
```
|
|
|
|
## Install
|
|
|
|
> [!TIP]
|
|
> You may want to install some [system dependencies](https://github.com/YGGverse/aquatic-crawler/wiki/Dependencies)
|
|
|
|
1. `git clone https://github.com/YGGverse/aquatic-crawler.git && cd aquatic-crawler`
|
|
2. `cargo build --release`
|
|
3. `sudo install target/release/aquatic-crawler /usr/local/bin/aquatic-crawler`
|
|
|
|
## Usage
|
|
|
|
> [!TIP]
|
|
> * prepend `RUST_LOG=debug` to debug, append `NO_COLOR=1` to disable fmt
|
|
> * use `--preload-*` arguments group to avoid preloading everything (by default)
|
|
> * as the crawler is running in a single thread, use `--peer-read-write-timeout` (in seconds) to skip long-lasting connections. Content preload will be restored in the next crawl cycle after resolving other torrents from active peers in the queue.
|
|
|
|
``` bash
|
|
aquatic-crawler --infohash /path/to/info-hash-ipv4.bin\
|
|
--infohash /path/to/info-hash-ipv6.bin\
|
|
--infohash /path/to/another-source.bin\
|
|
--tracker udp://host1:port\
|
|
--tracker udp://host2:port\
|
|
--preload /path/to/directory\
|
|
--preload-max-filesize=50000\
|
|
--preload-max-filecount=1\
|
|
--preload-regex="\.(png|gif|jpeg|jpg|webp)$"\
|
|
--peer-read-write-timeout=10
|
|
```
|
|
|
|
### Options
|
|
|
|
``` bash
|
|
aquatic-crawler --help
|
|
```
|