YGGverse/Yo
1
0
Fork 0
mirror of https://github.com/YGGverse/Yo.git synced 2026-03-31 17:55:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
Micro Web Crawler in PHP & Manticore
24 commits 2 branches 0 tags 249 KiB
Find a file
ghost da365c1ab1 fix total condition
2023-11-25 04:59:40 +02:00
example implement FTP snaps 2023-11-25 03:19:54 +02:00
src fix total condition 2023-11-25 04:59:40 +02:00
.gitignore fix gitignore 2023-11-25 00:17:06 +02:00
composer.json implement FTP snaps 2023-11-25 03:19:54 +02:00
LICENSE Initial commit 2023-11-19 20:07:17 +02:00
README.md implement yggo db migration cli tool 2023-11-25 04:44:07 +02:00

README.md

Yo! Micro Web Crawler in PHP & Manticore

Next generation of YGGo! project with goal to reduce server requirements and make deployment process simpler

  • Index model changed to distributed cluster model, and now oriented to aggregate search results from network instances trough API
  • Refactored data exchange model where drop all internal keys dependencies
  • Snaps now using tar.gz compression to reduce storage requirements and still supporting remote mirrors, FTP including
  • Minimalism everywhere

Implementation

Engine written in PHP 8 and uses Manticore on backend.

Default build adapted for Yggdrasil but could be used to make internet search portal.

Components

  • CLI tools for index operations
  • JS-less frontend to make search web portal
  • API tools to make search index distributed

Features

  • MIME-based crawler with flexible filter settings
  • Page snap history with local and remote mirrors support

Install

  1. Install composer, php and manticore
  2. Grab latest Yo version git clone https://github.com/YGGverse/Yo.git
  3. Run composer update inside the project directory
  4. Copy and customize config file cp example/config.json config.json
  5. Make sure storage folder writable
  6. Run indexes init script php src/cli/index/init.php
  7. Add new URL php src/cli/document/add.php URL
  8. Run crawler php src/cli/document/crawl.php
  9. Get search results php src/cli/document/search.php '*'

Web UI

  1. cd src/webui
  2. php -S 127.0.0.1:8080
  3. now open 127.0.0.1:8080 in your browser!

Documentation

CLI

Index

Init

Create initial index

php src/cli/index/init.php [reset]
  • reset - optional, reset existing index

Document

Add
php src/cli/document/add.php URL
  • URL - add new URL to the crawl queue
Crawl
php src/cli/document/crawl.php
Search
php src/cli/document/search.php '@title "*"' [limit]
  • query - required
  • limit - optional search results limit
Migration
YGGo

Import index from YGGo database

php src/cli/yggo/import.php 'host' 'port' 'user' 'password' 'database' [unique]

Source DB fields required:

  • host
  • port
  • user
  • password
  • database
  • unique - optional, check for unique URL (takes more time)