YGGverse/Yo
1
0
Fork 0
mirror of https://github.com/YGGverse/Yo.git synced 2026-04-01 02:05:29 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

init gemini protocol implementation

Browse source
This commit is contained in:
yggverse 2024-04-03 17:07:28 +03:00
parent 8418a56617
commit 1f96ca8a2c
11 changed files with 204 additions and 2422 deletions
Download patch file Download diff file Expand all files Collapse all files

63
README.md
View file

@ -2,26 +2,24 @@
Micro Web Crawler in PHP & Manticore
Yo! is the super thin layer for Manticore search server that extends official [manticoresearch-php](https://github.com/manticoresoftware/manticoresearch-php) client with CLI tools and simple JS-less WebUI.
Yo! Gemini is the super thin layer for Manticore search server that extends official [manticoresearch-php](https://github.com/manticoresoftware/manticoresearch-php) client with CLI tools and Gemini protocol UI.
This branch contain implementation for [Gemini Protocol](https://geminiprotocol.net).
To use `HTTP` version, please checkout [main branch](https://github.com/YGGverse/Yo)!
## Features
* MIME-based crawler with flexible filter settings by regular expressions, selectors, external links etc
* Page snap history with local and remote mirrors support (including FTP protocol)
* CLI tools for index administration and crontab tasks
* JS-less frontend to run local or public search web portal
* API tools to make search index distributed
* Gemini Protocol UI (coming soon)
## Components
* [Manticore Server](https://github.com/manticoresoftware/manticoresearch)
* [PHP library for Manticore](https://github.com/manticoresoftware/manticoresearch-php)
* [Symfony DOM crawler](https://github.com/symfony/dom-crawler)
* [Symfony CSS selector](https://github.com/symfony/css-selector)
* [FTP client for snap mirrors](https://github.com/YGGverse/ftp-php)
* [Hostname ident icons](https://github.com/dmester/jdenticon-php)
* [Captcha](https://github.com/Gregwar/Captcha)
* [Bootstrap icons](https://icons.getbootstrap.com/)
### Install
@ -32,22 +30,23 @@ Yo! is the super thin layer for Manticore search server that extends official [m
* `wget https://repo.manticoresearch.com/manticore-repo.noarch.deb`
* `dpkg -i manticore-repo.noarch.deb`
* `apt update`
* `apt install git composer manticore manticore-extra php-fpm php-curl php-mbstring php-gd`
* `apt install git composer manticore manticore-extra php-fpm php-mbstring`
Yo search engine uses Manticore as the primary database. If your server sensitive to power down,
change default [binlog flush strategy](https://manual.manticoresearch.com/Logging/Binary_logging#Binary-flushing-strategies) to `binlog_flush = 1`
#### Deployment
Project in development, to create new search project, use `dev-main` branch:
* `composer create-project yggverse/yo:dev-main`
* `git clone https://github.com/YGGverse/Yo.git`
* `cd Yo`
* `git checkout gemini`
* `composer update`
#### Development
* `git clone https://github.com/YGGverse/Yo.git`
* `cd Yo`
* `composer update`
* `git checkout gemini`
* `git checkout -b pr-branch`
* `git commit -m 'new fix'`
* `git push`
@ -69,11 +68,9 @@ Project in development, to create new search project, use `dev-main` branch:
* `php src/cli/document/crawl.php`
* `php src/cli/document/search.php '*'`
#### Web UI
#### Gemini UI
1. `cd src/webui`
2. `php -S 127.0.0.1:8080`
3. open `http://127.0.0.1:8080` in browser
Coming soon..
## Documentation
@ -134,27 +131,6 @@ php src/cli/document/search.php '@title "*"' [limit]
* `query` - required
* `limit` - optional search results limit
##### Migration
###### YGGo
Import index from YGGo database
```
php src/cli/yggo/import.php 'host' 'port' 'user' 'password' 'database' [unique=off] [start=0] [limit=100]
```
Source DB fields required:
* `host`
* `port`
* `user`
* `password`
* `database`
* `unique` - optional, check for unique URL (takes more time)
* `start` - optional, offset to start queue
* `limit` - optional, limit queue
### Backup
#### Logical
@ -171,13 +147,4 @@ Better for infrastructure administration and includes original data binaries.
## Instances
### [Yggdrasil](https://github.com/yggdrasil-network)
* `http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yo/` - IPv6 `0200::/7` addresses only | [index](http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yo/index.sql)
### [Alfis DNS](https://github.com/Revertron/Alfis)
* `http://yo.ygg` - `.ygg` domain zone search only | [index](http://yo.ygg/index.sql)
* `http://ygg.yo.index` - alias of `http://yo.ygg` | [index](http://ygg.yo.index/index.sql)
_*`*.yo.index` reserved for domain-oriented instances e.g. `.btn`, `.conf`, `.mirror` - feel free to request the address_
Coming soon..