YGGverse/aquatic
1
0
Fork 0
mirror of https://github.com/YGGverse/aquatic.git synced 2026-03-31 09:45:31 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
1611 commits 3 branches 0 tags 5.1 MiB
Commit graph

55 commits

Author SHA1 Message Date
Joakim Frostegård
2e67f11caf
udp: add experimental io_uring implementation (#131) 
* WIP: add udp uring support

* WIP: fix udp uring address parsing

* WIP: udp uring: resubmit recv when needed

* WIP: udp uring: add OutMessageStorage, send swarm responses

* WIP: udp uring: increase ring entries to 1024

* WIP: udp uring: add constants

* WIP: udp uring: use sqpoll, avoid kernel calls

* WIP: udp uring: disable sqpoll

* WIP: udp uring: use VecDeque for local responses

* udp uring: enable setup_coop_taskrun

* udp uring: add RecvMsgStorage

* udp: improve split of uring and mio implementations

* udp uring: clean up

* udp uring: initial ipv6 support

* udp uring: improve helper structs

* udp uring: clean up, use constants for important data

* udp: share create_socket fn between implementations

* udp uring: improve send buffer free index finding

* udp uring: work on SendBuffers.try_add

* udp uring: split into modules

* udp uring: Rename RecvMsgMultiHelper to RecvHelper

* udp uring: improve SendBuffers

* udp uring: fix copyright attribution in buf_ring module

* udp uring: stop always consuming 100% cpu

* udp uring: clean up

* udp uring: add handle_recv_cqe

* udp uring: move local_responses into SocketWorker

* udp uring: move timeout_timespec into SocketWorker

* Update TODO

* udp: make io-uring optional

* Update TODO

* udp uring: enqueue timeout before sends

* udp uring: move likely empty buffer tracking logic into SendBuffers

* udp uring: improve error handling and logging

* udp uring: keep one timeout submitted at a time

* udp uring: update pending_scrape_valid_until

* udp uring: add second timeout for cleaning

* Update TODO

* udp uring: store resubmittable squeue entries in a Vec

* udp uring: add comment, remove a log statement

* Update TODO

* Update TODO

* udp: io_uring: fall back to mio if io_uring support not recent enough

* udp: uring: add bytes_received statistics

* udp: uring: add bytes_sent statistics

* udp: uring: add more statistics

* Update TODO

* udp: uring: improve SendBuffers code

* udp: uring: remove unneeded squeue sync calls

* udp: uring: replace buf_ring impl with one from tokio-uring

* udp: uring: store ring in TLS so it can be used in Drop impls

* udp: uring: store BufRing in SocketWorker

* udp: uring: silence buf_ring dead code warnings, improve comment

* Update TODO

* udp: uring: improve CurrentRing docs, use anonymous struct field

* udp: uring: improve ring setup

* udp: uring: get ipv6 working

* udp: uring: make ring entry count configurable, use more send entries

* udp: uring: log number of pending responses (info level)

* udp: uring: improve comment on send_buffer_entries calculation

* udp: improve config comments

* udp: uring: add to responses stats when they are confirmed as sent

* Update TODO

* udp: uring: enable IoUring setup_submit_all

* Update README
 2023-03-07 19:01:37 +01:00
Joakim Frostegård
c6a134683c Toggle prometheus feature in scripts/run-aquatic-http.sh 2023-01-18 21:50:10 +01:00
Joakim Frostegård
3ac12b947f ws: add prometheus support (active connections, requests, responses) 2023-01-17 20:27:43 +01:00
Joakim Frostegård
417ea68701 scripts/bench-udp-handlers.sh: use release-debug profile 2022-10-26 20:05:39 +02:00
Joakim Frostegård
ce9fdc3887 Fix env-native-cpu-without-avx-512 issue 2022-09-25 12:03:47 +02:00
Joakim Frostegård
8beb13db29 In scripts, explicitly choose what workspace member to build 
Otherwise, building aquatic_udp fails on other OS:es than Linux
 2022-09-18 18:05:21 +02:00
Joakim Frostegård
44b50204e2 Switch to thin LTO; add Cargo profile release-debug 2022-04-04 12:32:42 +02:00
Joakim Frostegård
11cba4eb12 Add scripts/run-aquatic-http-private.sh 2022-04-03 16:31:45 +02:00
Joakim Frostegård
32a6285215 Update scripts/gen-tls.sh 2022-03-31 12:41:06 +02:00
Joakim Frostegård
cb563ee37e Adjust scripts/run-aquatic-ws.sh now that there is no mio version 2022-03-18 15:29:08 +01:00
Joakim Frostegård
222fac0e09
Improve aquatic_ws glommio implementation, fixing memory leak (#37) 
* ws: upgrade tungstenite and async-tungstenite to 0.16

* ws: use latest glommio

* ws: glommio: add config key connection_cleaning_interval

* ws: glommio: limit socket worker local channel size

Quick fix for memory leak

* ws: glommio: use prioritized task queue for sending responses

This is a start on trying to fix the large amount of responses
being dropped due to local response channel being full

* scripts/watch-threads.sh: don't highlight changes

* ws: glommio: await when sending responses from shared to local

* ws: glommio: limit ws message queue; add lots of ::log::warn for debug

* ws: glommio: add timeout to send_out_message, maybe pinpointing issue

* ws: glommio: clean up, tweak channel sizes; update TODO

* ws: glommio: set send timeout to 10s, add backpressure, accept drops

* ws: glommio: yield if needed in ConnectionReader backoff loop

* ws load test: add config key connection_creation_interval_ms

* ws load test: don't print "run connection" when opening connection

* ws load test: rename config num_connection to num_connections_per_worker

* ws load test config: improve order of code in file

* ws: glommio: info-level log for socket send timeout

* ws glommio: clean idle connections, improve cleaning code

* ws: glommio: make ConnectionWriter::send_error_response fallible; fmt

* ws: glommio: store JoinHandle in conn reference, cancel task on clean

* add scripts/heaptrack.sh

* Update TODO
 2021-12-11 18:50:05 +01:00
Joakim Frostegård
c1f2d036c0 add scripts/watch-threads.sh for showing runtime stats 
Includes per-thread cpu affinity (PSR)
 2021-11-19 00:14:46 +01:00
Joakim Frostegård
138ae710ef udp: remove io_uring version, it is slower than mio version 2021-11-18 22:42:37 +01:00
Joakim Frostegård
4641dd29f2 udp: remove glommio implementation 2021-11-18 22:42:36 +01:00
Joakim Frostegård
ce1c0b24c3 udp: fix build failure; fix scripts/run-aquatic-udp.sh 2021-11-18 22:42:36 +01:00
Joakim Frostegård
03192d2afb scripts: improve run-aquatic-udp.sh and run-aquatic-ws.sh 2021-11-07 13:30:42 +01:00
Joakim Frostegård
1c8da337a1 scripts/run-aquatic-ws: support running either mio or glommio impl 2021-11-07 11:51:23 +01:00
Joakim Frostegård
2bed6ccdc5 WIP: ws: split into features, other fixes 2021-11-05 13:25:19 +01:00
Joakim Frostegård
79bbf957c0 add scripts/gen-tls.sh for generating snakeoil tls cert & key 2021-10-26 19:39:05 +02:00
Joakim Frostegård
de2b7cb187 Add GitHub Docker action for testing UDP and HTTP file transfers 
The Dockerfile works when run locally. The GitHub action
plumbing may or not work in this state.
 2020-08-17 00:49:12 +02:00
Joakim Frostegård
beea0e7d97 add scripts/build-aquatic.sh 2020-08-13 00:15:43 +02:00
Joakim Frostegård
9efc1fc66a add aquatic crate with master executable, refactor cli_helpers 2020-08-13 00:13:01 +02:00
Joakim Frostegård
cff9b44a7f move RUSTFLAGS gen to own file instead of setting in each script 2020-08-12 00:28:03 +02:00
Joakim Frostegård
b3caa7ab7b Update TODO 2020-08-12 00:20:00 +02:00
Joakim Frostegård
f2d697011b Don't use target-cpu native etc in scripts/test.sh 2020-08-11 23:04:28 +02:00
Joakim Frostegård
56bc028a44 aquatic_ws: OutMessage and InMessage simd-json fixes 
* use simd-json in OutMessage deserialization
* take ownership in InMessage deserialization to avoid bugs
* Update benchmark results, benchmark was broken
* Compile simd-json even if simd is not on, so that rust-analyzer
  works
 2020-08-11 23:02:19 +02:00
Joakim Frostegård
5d40954936 aquatic_ws: deserialize InMessage with simd-json for performance 
From 18% to 13% CPU time during load test run.

Criterion benchmark of deserialization got about 90% improvement.
 2020-08-11 06:32:51 +02:00
Joakim Frostegård
99724c7e5d aquatic_ws: add bench for InMessage::from_ws_message, take ws msg by ref 2020-08-11 05:36:29 +02:00
Joakim Frostegård
118c2e446b scripts: don't print RUSTFLAGS, causes issues when printing configs 2020-08-01 23:23:10 +02:00
Joakim Frostegård
a5108f813d WIP: aquatic_ws: add half-crappy load test, fix infinite loop bug 2020-08-01 03:35:00 +02:00
Joakim Frostegård
0d8b6f6cc4 build scripts: disable avx512, since it decreases performance 2020-07-31 01:34:13 +02:00
Joakim Frostegård
32402a4dca criterion announce-response-to-bytes script: ask about saving result 2020-07-21 17:24:17 +02:00
Joakim Frostegård
c1b2a30803 criterion request-from-bytes script: ask user about saving result 2020-07-21 15:59:32 +02:00
Joakim Frostegård
b3154c5bcf aquatic http protocol: delete obsoleted request-from-path benchmark 
Now there is a request-from-bytes benchmark that supersedes
request-from-path.

Final comparison of gains made since first benchmark:

request-from-path
    time:   [540.01 ns 541.51 ns 543.15 ns]
    change: [-90.786% -90.741% -90.696%] (p = 0.00 < 0.01)
    Performance has improved.
 2020-07-21 14:25:47 +02:00
Joakim Frostegård
ea4b560faa aquatic_http_protocol: add benchmark for request from bytes 2020-07-20 20:27:42 +02:00
Joakim Frostegård
6471e4a8a9 Add scripts/run-load-test-http.sh 2020-07-20 15:21:25 +02:00
Joakim Frostegård
4caf174da5 aquatic_http: add criterion benchmark for announce response to bytes 
Speedup of custom implementation compared to using bendy:

announce-response-to-bytes:

time:   [413.77 ns 415.34 ns 417.08 ns]
change: [-93.074% -93.021% -92.968%] (p = 0.00 < 0.01)
Performance has improved.
 2020-07-19 16:56:13 +02:00
Joakim Frostegård
2fe42e7235 update criterion script 2020-07-19 13:26:17 +02:00
Joakim Frostegård
ca8d278a03 aquatic_http: parse_key_value_pairs_memchr: small code improvement 2020-07-19 13:06:39 +02:00
Joakim Frostegård
a8e355f22a aquatic_http: add criterion benchmarks for request from path parsing 
About 35% improvement by using both memchr methods
 2020-07-19 01:58:41 +02:00
Joakim Frostegård
a487347a0d aquatic_http: continue implementation work 2020-07-02 13:21:39 +02:00
Joakim Frostegård
7512b4b44f setup-bench-on-debian.sh: roll nginx into other apt-get line 
Not installing nginx means script is aborted because of
"set -e", and that setting serves a purpose
 2020-06-30 00:04:01 +02:00
Joakim Frostegård
4c2c6634c3 setup-bench-on-debian.sh: install git 2020-06-30 00:03:21 +02:00
Joakim Frostegård
3addab07f0 move plot_pareto from aquatic_udp_bench crate into own crate 2020-05-25 19:38:53 +02:00
Joakim Frostegård
1b8d74e26d rename aquatic to aquatic_udp, same for bench and load test crates 2020-05-11 16:55:46 +02:00
Joakim Frostegård
ab178b3189 add scripts/run-aquatic-ws.sh 2020-05-11 15:41:46 +02:00
Joakim Frostegård
cae534d82d scripts/test.sh: add comment about quickcheck constants 2020-05-04 17:10:55 +02:00
Joakim Frostegård
8effcb8d91 setup-bench-on-debian.sh: add rustc build flag for aquatic 2020-05-04 17:10:39 +02:00
Joakim Frostegård
b77af78483 Add scripts for setting up benchmarks on linux 2020-04-19 10:31:33 +02:00
Joakim Frostegård
f6ed47fec7 add aquatic_load_test for benchmarking aquatic over the network 2020-04-19 10:21:12 +02:00