aquatic-crawler

SSD-friendly FS crawler for the Aquatic BitTorrent tracker, based on the librqbit API

Note

• requires PR#233 or the info-hash-api branch implementation
• compatible with any other --infohash source in hash1hash2... binary format
• see also the βtracker as the web frontend for preload storage
• visit project Wiki for details

Conception

torrent client > aquatic_udp > infohash.bin < aquatic-crawler > * /preload/info-hash.torrent
torrent client               <-----------------------|          * /preload/info-hash/data
               <-------------------------------------|          * /preload/.info-hash/tmp

Install

Tip

You may want to install some system dependencies

1. git clone https://github.com/YGGverse/aquatic-crawler.git && cd aquatic-crawler
2. cargo build --release
3. sudo install target/release/aquatic-crawler /usr/local/bin/aquatic-crawler

Usage

Tip

• prepend RUST_LOG=debug to debug, append NO_COLOR=1 to disable fmt
• use --preload-* arguments group to avoid preloading everything (by default)
• as the crawler is running in a single thread, use --peer-read-write-timeout (in seconds) to skip long-lasting connections. Content preload will be restored in the next crawl cycle after resolving other torrents from active peers in the queue.

aquatic-crawler --infohash /path/to/info-hash-ipv4.bin\
                --infohash /path/to/info-hash-ipv6.bin\
                --infohash /path/to/another-source.bin\
                --tracker  udp://host1:port\
                --tracker  udp://host2:port\
                --preload  /path/to/directory\
                --preload-max-filesize=50000\
                --preload-max-filecount=1\
                --preload-regex="\.(png|gif|jpeg|jpg|webp)$"\
                --peer-read-write-timeout=10

Options

aquatic-crawler --help