The recent PR that revamped the connection IDs missed a case when
connecting to service nodes where we store the SN pubkey in peers, but
then fail to find the peer when we look it up by connection id.
This adds the required tracking to fix that case (and adds a test that
fails without the fix here).
- Allow up to 200ms (instead of 100ms) for the things we are waiting on
to become available, to prevent occasional spurious failures.
- Add unscoped info for how long we waited.
- Avoid calling into oxenmq with the catch lock held in the "hey google"
tests (because this will deadlock if the oxenmq call invokes any
logging).
- Replace an old std::cerr logger with the updated catch2 logger.
This commit adds support for listening on new ports after startup. This
will make things easier in storage server, in particular, where we want
to delay listening on public ports until we have an established
connection and initial block status update from oxend.
I realized after merging the previous PR that it is difficult to
correctly pass ownership into a timer, because something like:
TimerID x = omq.add_timer([&] { omq.cancel_timer(x); }, 5ms);
doesn't work when the timer job needs to outlive the caller. My next
approach was:
auto x = std::make_shared<TimerID>();
*x = omq.add_timer([&omq, x] { omq.cancel_timer(*x); }, 5ms);
but this has two problems: first, TimerID wasn't default constructible,
and second, there is no guarantee that the assignment to *x happens
before (and is visible to) the access for the cancellation.
This commit fixes both issues: TimerID is now default constructible, and
an overload is added that takes the lvalue reference to the TimerID to
set rather than returning it (and guarantees that it will be set before
the timer is created).
Updates `add_timer` to return a new opaque TimerID object that can later
be passed to `cancel_timer` to cancel an existing timer.
Also adds timer tests, which was omitted (except for one in the tagged
threads section), along with a new test for timer deletion.
This provides an interface for sending a reply to a message later (i.e.
after the Message& itself is no longer valid) by using a new
`send_later()` method of the Message instance that returns an object
that can properly route replies (and can outlive the Message it was
called on).
Intended use is:
run_this_lambda_later([send=msg.send_later()] {
send.reply("content");
});
which is equivalent to:
run_this_lambda_later([&msg] {
msg.send_reply("content");
});
except that it works properly even if the lambda is invoked beyond the
lifetime of `msg`.
Decoding into a std::byte output iterator was not working because the
`*out++ = val` assignment doesn't work when the output is std::byte and
val is a char/unsigned char/uint8_t. Instead we need to explicitly
cast, but figuring out what we have to cast to is a little bit tricky.
This PR makes it work (and bumps the version for this and the is_hex
fix).
`is_hex()` is a bit misleading as `from_hex()` requires an even-length
hex string, but `is_hex()` also allows odd-length hex strings, which
means currently callers should be doing `if (lokimq::is_hex(str) &&
str.size() % 2 == 0)`, but probably aren't.
Since the main point of `lokimq/hex.h` is for byte<->hex conversions it
doesn't make much sense to allow `is_hex()` to return true for something
that can't be validly decoded via `from_hex()`, thus this PR changes it
to return false.
If someone *really* wants to test for an odd-length hex string (though
I'm skeptical that there is a need for this), this also exposes
`is_hex_digit` so that they could use:
bool all_hex = std::all_of(str.begin(), str.end(), lokimq::is_hex_digit<char>)
Apple, in particular, often fails tests with an address already in use
if attempt to reuse a port that the process just closed, because it is a
wonderful OS.
Add var::get/var::visit implementations of std::get/std::visit that get
used if compiling for an old macos target, and use those.
The issue is that on a <10.14 macos target Apple's libc++ is missing
std::bad_variant_access, and so any method that can throw it (such as
std::get and std::visit) can't be used. This workaround is ugly, but
such is life when you want to support running on Apple platforms.
On the wire they are just lists, but this lets you put tuples onto and
pull tuples off of the wire. (Also supports std::pair).
Supports direct serialization (via bt_serialize()/bt_deserialize()),
list/dict consumer deserialization, and conversion from a bt_value or
bt_list via a new bt_tuple() function.
data_parts() wasn't currently used anywhere, and was broken: it was
calling bt_deserialize which was just wrong.
This repurposes it to take iterators over strings (or string-like types)
and append those parts as message parts.
Also adds tests for it.
This allows mixing some outside task into the lokimq job queue for a
category (queued up with native LMQ requests for that category) for use
when there is some external process that is able to generate messages.
For example, the most immediate use for this is to allow an HTTP server
to handle incoming RPC requests and, as soon as they arrive, inject them
into LokiMQ's queue for the "rpc" category so that native LMQ rpc
requests and HTTP rpc requests share the same thread pool and queue.
These injected jobs bypass all of LokiMQ's authentication and response
mechanisms: that's up to the invoked callback itself to manage.
Injected tasks are somewhat similar to batch jobs, but unlike batch jobs
the are queued and prioritized as ordinary external LokiMQ requests.
(Batch jobs, in contrast, have a higher scheduling priority, no queue
limits, and typically a larger available thread pool).
Deprecates the existing connect_remote() that takes remote addr and
pubkey as separate strings, just taking a `address` instead (into which
the caller can set pubkey/curve data as desired).
Also slightly changes how `connect_remote()` works when called with a
string remote but no pubkey: that string is now an augmented
lokimq::address string so that it can use the various formats supported
by `lokimq::address`.
(This was meant to be included in the PR that added `address` but
apparently didn't get implemented.)
There can be a spurious failure here if the backdoor_details element
hasn't been added yet, so lock & check it when waiting for the test
conditions.
The weirdest thing about this error is that it can fail but then when
expanding values they expand to *correct* values, i.e. so you get:
FAILED:
REQUIRE( backdoor_details == all_the_things )
with expansion:
{ "Alaska", "I'm the luckiest man in the world", "Loretta", "because all my
life are belong to Google", "moustache hatred", "photos", "scallops",
"snorted when she laughed", "tickled pink" }
==
{ "Alaska", "I'm the luckiest man in the world", "Loretta", "because all my
life are belong to Google", "moustache hatred", "photos", "scallops",
"snorted when she laughed", "tickled pink" }
The init function doesn't seem all that useful and makes the interface a
bit more complicated, so drop it.
Also addresses a race condition that can happen with tagged thread
startup when the proxy tries to talk to a tagged thread but the tagged
thread hasn't connected yet (which then aborts the proxy because it
assumes workers are always routable).
This renames the class to make it clearer what it does, and drops the
.name attribute from it so that it can cheaply be passed around. This
then means it can be cheaply passed by value (using std::optionals)
rather than by pointer when specifying a thread.
This adds to ability to have lokimq manage specific threads to which
jobs (individual, batch jobs, batch completions, or timers) can be
directed to. This allows dedicating a thread to some slow or
thread-unsafe action where you can dump jobs to the tagged thread as
a method of lockless job queuing.
We call libsodium functions which require a sodium_init() call; this is
usually a no-op (zmq will have already called it for us), but in case
zmq is built with tweetnacl instead of sodium we need to call it before
we call it directly in the LokiMQ ctor and the test suite.
These (...) lambdas are va_arg which cause corruption under clang on
both linux and macos. They weren't supposed to be va_args -- switch
them to the intended universal references, which fixes the crash.
Various small C++17 code improvements.
Replace mapbox::variant with std::variant.
Remove the bt_u64 type wrapper; instead we know have `bt_value` which
wraps a variant holding both int64_t and uint64_t, and has contructors
to send signed/unsigned integer types into the appropriate one.
lokimq::get_int checks both as appropriate during extraction.
As a side effect this means we no longer do the uint64_t -> int64_t
conversion on the wire, ever, without needing the wrapper; although this
can break older versions sending large positive integers (i.e. larger
than int64_t max) those weren't actually working completely reliably
with mapbox variant anyway, and the one place using such a value in loki
core (in a checksum) is already fully upgraded across the network
(currently using bt_u64, but always sending a positive value on the
wire).
Previously you could only generate a string from a string_view, or could
manage the string yourself and pass input iterators plus an output
iterator.
This commit adds an intermediate version that creates a string from a
pair of input iterators.
Removes lokimq::string_view (the type alias is still provided for
backwards compat, but now is always std::string_view).
Bump version (on dev branch) to 1.2.0
This class extends the basic ZMQ addresses with addresses that handle
parsing and generating of addresses with embedded curve pubkeys of
various forms, along with a QR-friendly address generator.
ConnectionIDs weren't comparing their routes, which meant that if
external code stored one in a map or set *all* incoming connections on
the same listener would be considered the same connection.
This fixes it by considering route for equality/hashing, and strips
route off internally where we need to map it to a socket.
When we hit the limit on the number of workers the proxy thread would
stop processing incoming messages, sending it into an infinite loop of
death. The check was supposed to use `active_workers()` rather than
`workers.size()`, but even that isn't quite right: we want to *always*
pull all incoming messages off and queue them internally since different
categories have their own queue sizes (and so we have to pull it off to
know whether we want to keep it -- if spare category queue room -- or
drop it).
This replaces the recognition of SN status to be checked per-command
invocation rather than on connection. As this breaks the API quite
substantially, though doesn't really affect the functionality, it seems
suitable to bump the minor version.
This requires a fundamental shift in how the calling application tells
LokiMQ about service nodes: rather than using a callback invoked on
connection, the application now has to call set_active_sns() (or the
more efficient update_active_sns(), if changes are readily available) to
update the list whenever it changes. LokiMQ then keeps this list
internally and uses it when determining whether to invoke.
This release also brings better request responses on errors: when a
request fails, the data argument will now be set to the failure reason,
one of:
- TIMEOUT
- UNKNOWNCOMMAND
- NOT_A_SERVICE_NODE (the remote isn't running in SN mode)
- FORBIDDEN (auth level denies the request)
- FORBIDDEN_SN (SN required and the remote doesn't see us as a SN)
Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were
already sent by remotes, but there was no connection to a request and so
they would log a warning, but the request would have to time out.
These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command
is a request but didn't include a reply tag) are also sent in response
to regular commands, but they simply result in a log warning showing the
error type and the command that caused the failure when received.
We really don't *ever* want send to block, no matter how it is called,
since the send is always in the proxy thread. This makes the actual
send call always non-blocking, and adds callbacks that we can invoke on
send failures: either on queue full errors (which might be recoverable),
or both full queue and hard failures (which are generally not
recoverable). These callbacks are both optional: they have to be passed
in using `send_option::queue_full` (if you just want queue full
notifies) or `send_option::queue_failure` (if you want queue full
notifies *and* other send exceptions).
For example:
lmq.request(conn, "some.method", callback, lokimq::request_timeout{5s});
will result in the callback being called with a failure if the response
doesn't arrive within 5s. (If it still arrives, but after the failure
callback, it gets dropped).
This removes two superfluous erases that occur during connection closing
(the proxy_close_connection just above them already removes the element
from `peers`), and also short-circuits the incoming message loop if our
pollitems becomes stale so that we don't try to use a closed connection.
It also fixes a bug in the outgoing connection index that was
decrementing the wrong connection indices, leading to failures when
trying to send on an existing connection after a disconnect.
Also adds a test case (which fails before the changes in this commit) to
test this.
The existing code was largely set up for SN-to-SN or client-to-SN
communications, where messages can always get to the right place because
we can always send by pubkey.
This doesn't work when we want general communications with a random
remote address.
This commit overhauls the way loki-mq handles communication in a few
important ways:
- Listening instances no longer pass bind addresses into the
constructor; instead they call `listen_curve()` or `listen_plain()`
before invoking `start()`.
- `listen_curve()` is equivalent to the existing bind support: it
listens on a socket and accepts encrypted handshaked connections from
anyone who already knows the server's public key.
- `listen_plain()` is all new: it sets up a plain text listening socket
over which random clients can connect and talk. End-points aren't
verified, and it isn't encrypted, but if you don't know who you are
talking to then encryption isn't doing anything anyway.
- Connecting to a remote now connections in CURVE encryption or NULL
(plain-text) encryption based on whether you provide a remote_pubkey.
For CURVE, the connection will fail if the pubkey does not match.
- `ConnectionID` objects are now returned when connecting to a remote
address; this object is then passed in to send/request/etc. to direct
the message. For SN communication, ConnectionID's can be created
implicitly from SN pubkey strings, so the existing interface of
`lmq.send(pubkey, ...)` will still work in most cases.
- A ConnectionID is now passed to the ConnectSuccess and ConnectFailure
callbacks. This can be used to uniquely identify which connection
succeeded or failed, and can determine whether the remote is a service
node (`.sn()`) and/or the pubkey (`.pubkey()`). (Obviously the service
node status is only available when the client can do service node
lookups, and the pubkey() is only non-empty for encrypted connections).
Jason Rhinelander