Originally the message hash included the timestamp and expiry of the
message when building the hash.
This breaks things in a couple of ways:
- the hash isn't reproducible because the message expiry is no longer
fixed.
- libsession-util needs de-duplication of identical messages even if the
timestamps differ.
This removes the use of timestamp/expiry starting at HF19.3.
The nearest-swarm logic was rather broken when handling ids in the
wrapping space (i.e. > max swarm id or < min swarm id). This fixes the
issues, as well as optimizing it.
Issues:
- all the wrapped-around values trying to compute distances were off by
*2*:
- The first off-by-one was because, even though swarm ids cannot be ==
max-u64, swarm space values (from pubkeys) *can* be, so all the
logic trying to make it wrap at MAX_SWARM_ID was invalid.
- The wrapping at MAX_SWARM_ID was *also* wrong because, even ignoring
the above point, distances are % (MAX_SWARM_ID+1), not %
(MAX_SWARM_ID).
- Because of the above, a session ID that maps to 0xff..ff in swarm
space would compute the distance to swarm_id 0 as max-u64 and thus
always go to the top swarm_id, even when 0xff..fe would map to
swarm_id 0.
- Values in the space between 0 and the minimum swarm id were rounding
to the *right* side node, while everything else rounds to the left
edge.
All of this is due to bad logic and insufficient test cases that
unfortunately was not properly reviewed when originally added to storage
server.
This commit changes the behaviour. This means old and new nodes *will*
slightly disagree for a couple of edge cases:
- we *do* have a swarm 0 on mainnet, so a session id that maps to
exactly 0xff..ff will break while the network has mixed versions.
This, however, is extremely rare (~1 in 2^64), and so is unlikely to
matter in practice.
- Values in the [0, lowest_swarm_id) that are exactly at the midpoint
change to now prefer the left node (highest_swarm_id) rather than the
right (lowest_swarm_id). This makes it consistent with the other
cases. On mainnet, however, this case doesn't actually exist between
lowest_swarm_id equals 0.
This also changes the code to use a binary search, rather than linear
scan, because swarms are guaranteed to be sorted. This also
significantly simplifies the code. The linear scan was just
unoptimized.
Swarm replacement had a bug in updating IPs: it would try to *preserve*
the IP from the old data when loading new swarm data.
This was completely wrong: it meant we would refuse IP changes, which
would almost certainly lead to storage server connectivity problems when
an IP changes.
This fixes it to only preserve IP/port data *if* the new data has it as
0s (which could happen if, for example, oxend is resyncing and hasn't
received proofs yet, but we got an initial data from a bootstrap node).
The code path was also rather inefficient with multiple unnecessary
vector copies, and an oversived map allocation.
There was yet another oddity where we were returning a reference to a
dummy value, rather than returning a nullptr on failure; this changes it
to be a nullable pointer instead (which also cascaded into returning an
optional swarm where that pointer was being used).
- Moves all source code under oxenss/
- Removes .../include, .../src setup
- Use include subdirectories and subnamespaces
- Various small code cleanups in common/utils code, eliminating or
splitting up unrelated functionality.
boost::po has terrible syntax, various weird bugs, and is the last
remaining boost dependency in the package.
This also serves as a bit of a proof-of-concept for switching to CLI11
in lokinet and oxen-core.
This changes how we compute hashes to better match how data is now
stored (i.e. expiry instead of ttl, binary data instead of base64).
This results in different hashes across upgraded/non-upgraded storage
servers, however Session currently handles that properly (and will
detect the duplicate) so while this will temporarily result in some
double storage during the migration period, it shouldn't cause any
other issues.
Refactors storage rpc requests to abstract parsing and then wires them
up so that all of direct, onion-requested, and omq can now take the same
codepaths for requests.
The new OMQ requests, in particular, are publicly accessable at
`storage.whatever` endpoints (e.g. storage.store).
This also relaxes/changes some of the argument parsing:
- allow `pubkey` to be passed as that (currently `pubKey` is required,
and continues to work)
`last_hash` (AKA `lastHash`) is no longer required; omitting it is now
the same as providing an empty one.
Data is now stored as binary in the database; previously we were storing
the base64-encoded value received from the client. (This will, however,
break any client that expected to be able to send random data).
Also adds an `info` storage endpoint that returns the current version &
timestamp.
TODO: need a database migration to convert existing (base64) data. (We
need a db migration for other reasons, as well).
Currently we have an awkward storage of timestamp + ttl + expiry in the
database, and timestamp + ttl passed in from the client (as strings!).
Althis is awkward because we want to be able to shorten the expiry, but
that would mess up TTL. Additionally everything is stored as
`uint64_t`s, which is messy and not type safe.
This commit makes these changes:
- ttl and timestamp can now be sent by the client as integers (in
addition to the current string value). NB: this is not reliable until
the entire SN network is on the next SS release.
- ttl, timestamp, and expiry are now type-safe std::chrono types instead
of raw integers (milliseconds for ttl and system_clock::time_points for
the other two).
- ttl is no longer stored: instead we just store timestamp + expiry.
(This will let us update expiry later without worrying about TTL).
- ttl/timestamp value validation is moved out of `common/oxen_common.h`
(which was a very odd place for it) and into request_handler.
- serialization no longer supports message_t; rather message_t is *only*
for holding the value the client gives us. SS now uses storage::Item
everywhere other than incoming client data.
- Removed TTL/Nonce storage and retrieval
- Fully specify the queries columns instead of using `SELECT *`
- Don't use "`" for identifier quoting inside sqlite. It's non-standard
MySQL garbage that sqlite3 supported only for MySQL compatibility.
- Replace boost beast http(s) client code with cpr (same as used in
oxen-core and lokinet)
- Replace bootstrap http requests with encrypted/authenticated omq rpc
requests to the bootstrap nodes (rather than unauthenticated,
unencrypted http connections).
- Remove boost::asio
- Remove boost circular buffer; a regular map with a two-line trimming
code is simpler for block hash cache, and a limit on stored snodes
doesn't seem necessary for the rate limiter.
- Make rate_limiter clean itself periodically; currently it only cleans
once it reaches max clients (i.e. 10k).
- Make rate_limiter thread safe, plus various code cleanups/DRYing.
- Remove ip_utils; we now longer really need a check for public ip for
proxy requests because we are sufficiently restrictive on the proxy url
target that we are unlikely to be able to reach anything sensitive (and
we also don't allow redirects).
- Replace /swarms/ping_test_v1 with /ping_test/v1; this new request now
returns the remote pubkey in a header, and no longer includes an SSL
cert signature (so that we can drop the SSL cert signatures after HF19).
The old one will still be used until HF19.
- Add OMQ endpoint for storage tests; it gets used instead of the https
one at HF19.
- Refactor storage test retries into request_handler (needed so both
legacy and omq endpoints can use it).
- Fix HF18 swarms/storage_test endpoint which was missing the snode
signature required for current SS versions to accept the response.
- Move HTTPS server-specific code (validate_snode_signature) out of
RequestHandler into HTTPSServer
- Make onion proxy-to-url timeout a bit less than the onion request
timeout so that the client has a better chance of actually getting a
timeout error.
- Miscellaneous cleanups
- Remove pre-HF18 compatibility code.
- Remove `process_lns_request` onion request target: it was accidentally
broken in HF18; clients are now using an oxend rpc request proxy to
`ons_resolve` instead.
- Shorten timeout values for ping and storage tests (5s and 15s,
respectively) and bootstrap connections (now 10s instead of 60s).
Database has a dependency on boost::asio so that it can set up a timer,
but this is awkward as it couples the Database class with an
implementation detail of the Database user.
Fix this by removing it, making the cleanup timer callback the
responsibility of the caller.
This also fixes some spurious failures due to race conditions between
the threads in the storage test code.
The size before the string was being serialized as a size_t, which
differs on different arches; fixed it to a uin64_t so that it matches
amd64.
Also remove an undesirable `reserve()`: reserving the amount needed for
an intermediate result tends to result in *more* allocations happening
rather than less. Plus the reserve size was wrong (allocating only 4
bytes for the 8 byte size_t value).
This alters the derived key hash to reverse the pubkey order when
replying so that the same key in each direction is used, i.e. for a
message from client C to snode S the key is:
H(cS || C || S)
before this commit, the would use an encryption key for return messages
of:
H(sC || S || C)
and, while the client can still decrypt that, it means the client has
*two* derived keys to worry about. With this change, server swap the
order so it puts itself *second*:
H(sC || C || S)
which will yield the same shared key as the client derived for the
original message.
- Do the json parsing as part of the payload parsing rather than
allocating a string and then making the caller do it (there's no case
where the caller *doesn't* want to do it).
- Modernize code to use structured bindings, allowing both cleaner code
and reduction in the number of moves/copies.
This replaces the HTTP RPC interface for communicating with the local
oxend with OxenMQ-based RPC requests. This has advantages:
- Persistent, single connection rather than opening new connections
every time we need anything.
- We don't have to poll for new blocks every 1s: rather we can use
`sub.block` to have oxend push notifications to us when new blocks are
added; previously we had excessive (1s) polling which we needed to be
sure we noticed new blocks; now we avoid all of that *and* learn about
new blocks faster (since we get pushed notifications).
- Allows better host security (by protecting access to the unix socket
rather than needing a localhost port)
- Matches lokinet, which moved to use OMQ RPC in the last hard fork
(0.8).
Currently we still need HTTP JSON RPC requests to contact the bootstrap
nodes but that will eventually change as well.
