This allows mixing some outside task into the lokimq job queue for a
category (queued up with native LMQ requests for that category) for use
when there is some external process that is able to generate messages.
For example, the most immediate use for this is to allow an HTTP server
to handle incoming RPC requests and, as soon as they arrive, inject them
into LokiMQ's queue for the "rpc" category so that native LMQ rpc
requests and HTTP rpc requests share the same thread pool and queue.
These injected jobs bypass all of LokiMQ's authentication and response
mechanisms: that's up to the invoked callback itself to manage.
Injected tasks are somewhat similar to batch jobs, but unlike batch jobs
the are queued and prioritized as ordinary external LokiMQ requests.
(Batch jobs, in contrast, have a higher scheduling priority, no queue
limits, and typically a larger available thread pool).
The init function doesn't seem all that useful and makes the interface a
bit more complicated, so drop it.
Also addresses a race condition that can happen with tagged thread
startup when the proxy tries to talk to a tagged thread but the tagged
thread hasn't connected yet (which then aborts the proxy because it
assumes workers are always routable).
This adds to ability to have lokimq manage specific threads to which
jobs (individual, batch jobs, batch completions, or timers) can be
directed to. This allows dedicating a thread to some slow or
thread-unsafe action where you can dump jobs to the tagged thread as
a method of lockless job queuing.
Removes lokimq::string_view (the type alias is still provided for
backwards compat, but now is always std::string_view).
Bump version (on dev branch) to 1.2.0
When doing an optional send that gets declined (because we aren't
connected) the "sending would block" warning would still be printed, but
shouldn't be.
ConnectionIDs weren't comparing their routes, which meant that if
external code stored one in a map or set *all* incoming connections on
the same listener would be considered the same connection.
This fixes it by considering route for equality/hashing, and strips
route off internally where we need to map it to a socket.
When we hit the limit on the number of workers the proxy thread would
stop processing incoming messages, sending it into an infinite loop of
death. The check was supposed to use `active_workers()` rather than
`workers.size()`, but even that isn't quite right: we want to *always*
pull all incoming messages off and queue them internally since different
categories have their own queue sizes (and so we have to pull it off to
know whether we want to keep it -- if spare category queue room -- or
drop it).
ZMQ's default reconnection time is 100ms, indefinitely, which seems far
too aggressive, particularly where we have some potential for hundreds
or thousands of connections.
This changes the default to be slightly slower (250ms instead of 100ms)
on the first attempt, and to use exponential backoff doubling the time
between each failed connection attempt up to a max of 5s between
reconnection attempts to calm things down.
This replaces the recognition of SN status to be checked per-command
invocation rather than on connection. As this breaks the API quite
substantially, though doesn't really affect the functionality, it seems
suitable to bump the minor version.
This requires a fundamental shift in how the calling application tells
LokiMQ about service nodes: rather than using a callback invoked on
connection, the application now has to call set_active_sns() (or the
more efficient update_active_sns(), if changes are readily available) to
update the list whenever it changes. LokiMQ then keeps this list
internally and uses it when determining whether to invoke.
This release also brings better request responses on errors: when a
request fails, the data argument will now be set to the failure reason,
one of:
- TIMEOUT
- UNKNOWNCOMMAND
- NOT_A_SERVICE_NODE (the remote isn't running in SN mode)
- FORBIDDEN (auth level denies the request)
- FORBIDDEN_SN (SN required and the remote doesn't see us as a SN)
Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were
already sent by remotes, but there was no connection to a request and so
they would log a warning, but the request would have to time out.
These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command
is a request but didn't include a reply tag) are also sent in response
to regular commands, but they simply result in a log warning showing the
error type and the command that caused the failure when received.
SS wants this, in particular, to be able to do reachability tests.
(Using connect_remote for this was bad with pubkey-based routing ids
because the second connection could replace an existing connection).
Downgrades a bunch of not-useful-at-info-level debug messages from info
-> debug. This makes `info` a more useful value for a client that wants
messages about startup/shutdown but not random non-serious connection
related messages.
We really don't *ever* want send to block, no matter how it is called,
since the send is always in the proxy thread. This makes the actual
send call always non-blocking, and adds callbacks that we can invoke on
send failures: either on queue full errors (which might be recoverable),
or both full queue and hard failures (which are generally not
recoverable). These callbacks are both optional: they have to be passed
in using `send_option::queue_full` (if you just want queue full
notifies) or `send_option::queue_failure` (if you want queue full
notifies *and* other send exceptions).
When we fail to send to a SN but can retry (e.g. because we had an
incoming connection which no longer works, but can retry an outgoing
connection) we were recursing, but this was resulting in a double-free
of the request callback (since we'd try to take ownership of the
incoming serialized pointer twice).
Rewrite the code to use a loop with single ownership instead.
This also changes the request callback behaviour to fire a failure
callback immediately if we can't send a request; previously you'd have
to wait for a timeout, but that is pointless if we couldn't get the
request out.
For example:
lmq.request(conn, "some.method", callback, lokimq::request_timeout{5s});
will result in the callback being called with a failure if the response
doesn't arrive within 5s. (If it still arrives, but after the failure
callback, it gets dropped).
lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp
into multiple smaller cpp files by logical purpose for better parallel
compilation ability. It also splits up the lokimq.h header slightly by
moving the ConnectionID and Message types into their own headers.
Jason Rhinelander
Jeff Becker