Commit graph

1138 commits

Author SHA1 Message Date
Jason Rhinelander bf6d6d6973 Work around Session not exposing ed25519 pubkeys
Session ids are, unfortunately, x25519 pubkeys rather than ed25519, and
there isn't a one-to-one conversion from x to ed, so we have to have the
client send along the ed25519 pubkey *along with* the 05-prefixed x25519
pubkey, then we also check that the given ed25519 converts to the x25519
pubkey when checking client signatures.
2021-06-20 23:10:40 -03:00
Jason Rhinelander b43c0ab337 More test suite 2021-06-20 11:11:34 -03:00
Jason Rhinelander 7f148b4d6e Fix last_hash parsing to accept blake2b+b64 hashes 2021-06-20 11:09:24 -03:00
Jason Rhinelander de01ba092b Fix pubkey encoding in https header 2021-06-20 09:13:55 -03:00
Jason Rhinelander 323c459df8 Add basic swarm testnet test suite 2021-06-20 00:47:57 -03:00
Jason Rhinelander 8d459ff7d2 get_swarm: +port_https/port_omq; deprecate "port"
Not only is it only providing one of the ports, it also stringifies it
for no good reason.
2021-06-20 00:03:48 -03:00
Jason Rhinelander 1090dabd09 Update oxend endpoint for reachability reporting 2021-06-19 11:03:25 -03:00
Jason Rhinelander 5048751ca7 typedef the clock, not the clock's time 2021-06-19 11:03:09 -03:00
Jason Rhinelander 9b5a8739fd Fix duplicate failing enqueuing
Nodes were being removed from `failing` while queued for testing, which
could result in multiple tests of the same node getting queued.
2021-06-19 10:58:52 -03:00
Jason Rhinelander 4299ef14ee Remove testnet exceptions to HF guards 2021-06-17 21:33:19 -03:00
Jason Rhinelander b529b962a6 Re-add seed1/seed3 to seed node list 2021-06-17 21:29:55 -03:00
Jason Rhinelander f9d6932554 Add support for minor forks (spoons? sporks? snode revisions?) 2021-06-17 21:21:44 -03:00
Jason Rhinelander eab06e985e Version-guard blake2b hash 2021-06-17 14:43:07 -03:00
Jason Rhinelander 8e05625d5a HF version gating & store distribution fix
- Improve serialization versioning constants

- Use constants for version gating when `store` becomes recursive

- Don't do a background message distribution when we are using recursive
stores.
2021-06-17 13:00:38 -03:00
Jason Rhinelander 988979df0e If we get a recursive failure put the message in "reason" 2021-06-17 12:30:39 -03:00
Jason Rhinelander fc0eefc783 Fix store response not being set in swarms 2021-06-17 12:26:38 -03:00
Jason Rhinelander 4ea0748509 Use 500 status if all answers are failures 2021-06-17 12:23:18 -03:00
Jason Rhinelander 3be368d994 Fix variant error on string proxy callback 2021-06-17 12:14:21 -03:00
Jason Rhinelander c24849e653 DB migration fixes 2021-06-17 12:09:37 -03:00
Jason Rhinelander b7db30e857 SQLiteCpp: Disable column metadata
We don't use it, and it isn't always available in sqlite3.
2021-06-17 12:02:59 -03:00
Jason Rhinelander ef8492668c Print swarmid info message in hex 2021-06-17 11:46:12 -03:00
Jason Rhinelander 0d5b61ad4f Add swarm (as hex string) into swarm members response 2021-06-17 11:41:30 -03:00
Jason Rhinelander d58f195eef Rename badly named hex_to_u64, document it, and test it. 2021-06-16 22:29:38 -03:00
Jason Rhinelander 6a3ec7dee9 Remove unused/uncommitted header 2021-06-16 20:22:12 -03:00
Jason Rhinelander c9a3aadf45 Remove _t suffix from concrete structs
The `_t` suffix stands for `typedef` and is deceptive when used for a
concrete type.
2021-06-16 19:37:33 -03:00
Jason Rhinelander e7dc6e352a C++ improvements
Remove superfluous `this->`.  This is C++, not Python/Rust/etc. that
require `self` boilerplate.

Use optional/pair for optional multi-value return rather than mutable
lvalue arguments with a bool return (which wasn't even checked in one
place).
2021-06-16 19:37:33 -03:00
Jason Rhinelander 1727053980 Bump version to 2.2.0 2021-06-16 19:37:33 -03:00
Jason Rhinelander bcce006013 Test suite fixes for recent changes 2021-06-16 19:37:33 -03:00
Jason Rhinelander 3f1e69e79b Make unit test logger go via Catch2 UNSCOPED_INFO
Hides log spam while still allowing Catch2 to show logs leading up to a
failure when a test failure occurs.
2021-06-16 19:37:33 -03:00
Jason Rhinelander 1453d9e816 Make store recursive
This allows a client to store on *any* swarm member, have the swarm
member distribute, and then verify the result of the store (if it wishes
to).
2021-06-16 19:37:33 -03:00
Jason Rhinelander 314894d4b1 Database redesign, refactor & migration
Redesigns the database to be a more appropriate, less duplicative design
using "owners" and "messages" with a foreign key between them.

Rewrites all the database code using SQLiteCpp which substantially
reduces the amount of boilerplate, duplicate code for query handling.

Makes the statement handlers thread_local for better thread safety; this
also allows the actual query to be where it is executed, rather than
having all the prepared queries in one place nowhere close to where they
are actually used.
2021-06-16 19:37:33 -03:00
Jason Rhinelander b7b0d75d4f SS refactor/cleanups
- Eliminated storage::Item because it just duplicates message_t for no
useful reason.

- Always pass user pubkeys as user_pubkey_t, never as std::strings.
Pubkeys are not strings, and the same pubkey has multiple string
representations.

- Properly split up the pubkey as [type][pubkey] where type = byte (5
for Session) and pubkey is a type-dependent key (currently only
supporting 32-byte Ed25519 keys).

- Allow pubkeys to be loaded from either hex (66) or binary (33) values.

- Allow pubkey prefixes on testnet.  A 32-byte key will be interpreted
as if the key was prefixed with a 0.  Thus there is now just one
"proper" pubkey size of 33 bytes or 66 bytes, and we silently accept the
missing-prefix for testnet (but can do away with the proper sizes being
different).

- Expose pubkey stringification in different ways (prefixed/unprefixed,
hex/raw).

- Simplify time_point-to-integer conversion with new time.hpp functions
from_epoch_ms and to_epoch_ms.

- Restore old message hash format so that we can keep using it until the
mandatory transition point.

- Added a much more network efficient serialization format, using
standard b-encoding rather than NIH custom encoding.  After the
mandatory upgrade we can remove the old format.

- Refactor swarm bootstrap code to be more efficient
2021-06-16 19:37:33 -03:00
Jason Rhinelander ef2e454d05 Fix/update boost mirror URL
boost's mirror moved.
2021-06-16 18:59:17 -03:00
Jason Rhinelander 4a514a107e Add SQLiteCpp
This is a nice C++ wrapper that cleans up the data interface
considerably over using the C sqlite3 API.

I also evaluated (and started implementing) sqlite_orm for this, but ran
into considerable obstacles: the orm components get in the way without
being good enough to really solve anything (and essentially just making
you write queries in C++ code that is much less elegant than straight
queries), plus it fundamentally doesn't support threaded operation,
which sucks.
2021-06-16 18:59:17 -03:00
Jason Rhinelander e39317239c Fix: valid message hash 2021-06-16 18:59:17 -03:00
Jason Rhinelander 02074c082e Include timestamp in retrieve request 2021-06-16 18:59:17 -03:00
Jason Rhinelander a28eeabda0 Fix message hash acceptance 2021-06-16 18:59:17 -03:00
Jason Rhinelander b284679745 Add method name to signature generation
Without this you could submit an expiry request and someone could replay
it later as a delete request (when the expiry timestamp arrives).

With the method include this isn't possible.
2021-06-16 18:59:17 -03:00
Jason Rhinelander 8772aeaaeb Wire up remaining delete/expiry endpoints
- Further documents each endpoint

- Changes the expiry update endpoints to just return the hashes of
messages rather than hashes + timestamps as its quite a bit simpler
(especially for signing).

- implements delete_msgs/delete_before/expire_all/expire_msgs endpoint
processing logic.
2021-06-16 18:59:17 -03:00
Jason Rhinelander 9ac5305e06 Check request being for us before replying 2021-06-16 18:59:17 -03:00
Jason Rhinelander b93ff6fa4a Reduce swarm propagation timeout to 5s 2021-06-16 18:59:17 -03:00
Jason Rhinelander facd068675 Change message hash to blake2b-256 + base64
SHA512+hex is *way* longer than needed for a message identifier and is
just wasteful.
2021-06-16 18:59:17 -03:00
Jason Rhinelander 71bddf96bd Fix "Store relayer's own result" 2021-06-16 18:59:17 -03:00
Jason Rhinelander 3430fa41b7 Document response codes 2021-06-16 18:59:17 -03:00
Jason Rhinelander f7a15f6e97 Fix forward request handling 2021-06-16 18:59:17 -03:00
Jason Rhinelander 339ad4a3c8 Make get_results return # rows, -1 on error
This lets it be used for queries that might (legitimately) return 0
rows.
2021-06-16 18:59:07 -03:00
Jason Rhinelander 0e8b364c9c Remove unused constant 2021-06-16 18:59:07 -03:00
Jason Rhinelander 42f7862d46 Store relayer's own result in "swarm" dict when recursive
Otherwise it's a bit weird to have one result at the top-level and the
others under "swarm".
2021-06-16 18:59:07 -03:00
Jason Rhinelander 95e12c7682 Fix pending, add request debugging 2021-06-16 18:59:07 -03:00
Jason Rhinelander 7f0a6c311d Don't use sodium piecewise ed25519 construction
It doesn't work as expected: rather it creates an "ed25519ph"
signature, which is a signature over a pre-hashed value, rather than a
regular Ed25519 signature.

Instead this now uses an allocation-minimized conversion/concatenation
of arguments to build the proper Ed25519 signature message.
2021-06-16 18:59:07 -03:00