Commit Graph

35 Commits

Author SHA1 Message Date
Jason Rhinelander e4f93afafa Fix trace IP address 2020-02-29 16:34:37 -04:00
Jason Rhinelander 922435ca3b Add auth and ConnectionID routing tests 2020-02-29 16:06:55 -04:00
Jason Rhinelander 1ad68b2605 Wrap HI response in try/catch
It's possible that the client has gone away, in which case we can get an
exception raised from an unroutable request.
2020-02-29 16:06:08 -04:00
Jason Rhinelander 3be632e73e Fix auth_level for remote connections
The wrong key was being set here (deserialization expects auth_level).
2020-02-29 16:04:43 -04:00
Jason Rhinelander 09c487f327 Add ability to use random routing ids for outgoing 2020-02-29 16:03:25 -04:00
Jason Rhinelander ece8870896 Move routing prefix into ConnectionID
This allows storing a ConnectionID received in a message callback and
using it later to send another message along the connection without
worrying about a routing id: the ConnectionID will have it if it is
required.  Previously you would have had to store the ConnectionID *and*
the routing prefix, and then specified the route as a
send_option::route{}, which was annoying and cumbersome.
2020-02-29 16:01:47 -04:00
Jason Rhinelander 28e36a3eaf Make AuthLevel stream printable 2020-02-29 15:16:58 -04:00
Jason Rhinelander 2743e576b2 Distinguish between batch jobs and reply jobs
This adds a separate category (and reserve count) for "reply jobs",
which are jobs triggered by receiving a reply to a request, or after a
successful connect or unsuccessful timeout.  Previously these were
scheduled as regular batch jobs; this schedules them as a new "reply
jobs" category with its own reserved threads count.

This also changes the defaults for batch jobs and reply jobs to be based
on the specified general workers count rather than directly on hardware
concurrency, so that if you are on a 16-thread CPU but override general
workers from its default of 16 to 4 and don't change batch workers you
now get reserved batch workers set to 2 rather than 8 which constrains
the typical parallel batch jobs to 4 (i.e. the general worker limit)
rather than exceeding it with the batch job limit.

Similarly for reply jobs, which is now ceil(general/8) by default.
2020-02-28 17:54:00 -04:00
Jason Rhinelander 57f0ca74da Added support for general (non-SN) communications
The existing code was largely set up for SN-to-SN or client-to-SN
communications, where messages can always get to the right place because
we can always send by pubkey.

This doesn't work when we want general communications with a random
remote address.

This commit overhauls the way loki-mq handles communication in a few
important ways:

- Listening instances no longer pass bind addresses into the
constructor; instead they call `listen_curve()` or `listen_plain()`
before invoking `start()`.

- `listen_curve()` is equivalent to the existing bind support: it
listens on a socket and accepts encrypted handshaked connections from
anyone who already knows the server's public key.

- `listen_plain()` is all new: it sets up a plain text listening socket
over which random clients can connect and talk.  End-points aren't
verified, and it isn't encrypted, but if you don't know who you are
talking to then encryption isn't doing anything anyway.

- Connecting to a remote now connections in CURVE encryption or NULL
(plain-text) encryption based on whether you provide a remote_pubkey.
For CURVE, the connection will fail if the pubkey does not match.

- `ConnectionID` objects are now returned when connecting to a remote
address; this object is then passed in to send/request/etc. to direct
the message.  For SN communication, ConnectionID's can be created
implicitly from SN pubkey strings, so the existing interface of
`lmq.send(pubkey, ...)` will still work in most cases.

- A ConnectionID is now passed to the ConnectSuccess and ConnectFailure
callbacks.  This can be used to uniquely identify which connection
succeeded or failed, and can determine whether the remote is a service
node (`.sn()`) and/or the pubkey (`.pubkey()`).  (Obviously the service
node status is only available when the client can do service node
lookups, and the pubkey() is only non-empty for encrypted connections).
2020-02-28 00:16:43 -04:00
Jason Rhinelander e4d371b026 Fixed string_view c++17 compatibility
string_view isn't supposed to be implicitly convertible to std::string
and code would break compiling under c++17 (when our local string_view
is simply a std::string_view typedef).
2020-02-24 22:20:56 -04:00
Jason Rhinelander 3c1a0c4280 Adding missing test file 2020-02-23 23:56:26 -04:00
Jason Rhinelander c589599892 Fix off-by-one `remotes` access 2020-02-23 23:50:47 -04:00
Jason Rhinelander 99f4333b18 Add gcc 5 constexpr workaround 2020-02-23 16:28:22 -04:00
Jason Rhinelander f2ee3d9b41 constexpr string_view fixes
Pre-C++17 char_traits::compare isn't constexpr so we can't constexpr the
find/rfind methods that use it.

begin() etc, however, can be constexpr (and need to be for some of the
other constexpr methods here that use them).
2020-02-22 23:27:21 -04:00
Jason Rhinelander da96c1ec79 Make string_view a full std::string_view implementation
Adds the missing pieces plus adds a test script.
2020-02-22 21:59:07 -04:00
Jason Rhinelander 03827ac1f7 Properly remove pending requests after they are invoked 2020-02-20 20:42:44 -04:00
Jason Rhinelander ed9af92411 Fix reply_tag off-by-one 2020-02-20 20:23:16 -04:00
Jason Rhinelander de0a5842af Added debugging around pending request creation/removal 2020-02-20 20:13:29 -04:00
Jason Rhinelander 5d30846cee Disable non-working self-connection inproc socket
This was meant to be an optimization but doesn't actually work because
we don't do a ZAP request at all when connecting inproc sockets, so the
metadata we need never gets set.

Remove it so a SN can still connect to itself.
2020-02-17 17:59:01 -04:00
Jason Rhinelander 58e45db996 Fix truncated pubkey
Fix bug from quorumnet transition; quorumnet set the user id to
something like "S:abc..." or "C:abc..." to indicate SN or non-SN, but
now that gets carried in a separate message property but the +2 on the
copying position was still erroneously being used.
2020-02-16 22:42:03 -04:00
Jason Rhinelander b11d2870bd Fix verify pubkey initialization 2020-02-13 00:53:43 -04:00
Jason Rhinelander 98a4aed68f Fix bad string access for pubkey verification 2020-02-13 00:39:00 -04:00
Jason Rhinelander 1b6f38fc07 Add LMQ_TRACE macro instead of LMQ_LOG(trace
LMQ_TRACE becomes nothing under a release build, which is good because
many traces are in the proxy hot path.

Also fixes some confusing log level comparison logic by flipping the
order of log levels.  Previous trace < debug < info < warn, which was
confusing: now that order is reversed which makes way more sense (i.e.
larger LogLevel means more logging).
2020-02-12 22:19:18 -04:00
Jason Rhinelander 13e7953bd7 Updated headers and remove dead code 2020-02-12 22:05:15 -04:00
Jason Rhinelander ede91341da Add c++14 requirement to tests target 2020-02-11 20:39:13 -04:00
Jason Rhinelander 63e70f9912 Fix delayed proxy-scheduled batch jobs
Batch jobs scheduled by the proxy thread itself were delayed to the next
poll timeout (because nothing ever gets sent on a socket).  Add a
variable to bypass the next poll to handle this case.
2020-02-11 19:08:19 -04:00
Jason Rhinelander ccfb6d080b Add request/reply abstraction
This allows making RPC requests with a callback that gets called when
the response comes back.  The is essentially a wrapper around doing it
yourself (i.e. by setting up a server-side "request" and client-side
"reply" command where "request" responds with a "reply" command), but
abstracted into lokimq itself as it is likely to be very useful when
integrating client/server connections rather than peer-to-peer
connections.
2020-02-11 02:30:07 -04:00
Jason Rhinelander 061bdee0a8 Add zmq timer support 2020-02-11 02:29:00 -04:00
Jason Rhinelander 7393e8422c Improve on-the-fly bt deserialization interface
This separates the dict traversal into `next_...` functions that return
a key-value pair, and `consume_...` that return just the value.  The
latter is particularly useful when using `skip_until` to position
yourself at a known key.
2020-02-11 02:18:14 -04:00
Jason Rhinelander 3ff66490ad Add ability to run a batch completion job in the proxy thread
For very small jobs (i.e. just setting a flag or something) this is
going to be faster than dispatching to a thread.
2020-02-11 02:17:12 -04:00
Jason Rhinelander 63c71396be Add initial test suite with some batch job tests 2020-02-06 18:10:26 -04:00
Jason Rhinelander 03ea49167c Various small optimizations 2020-02-06 00:50:31 -04:00
Jason Rhinelander f75b6cf221 Added batch job implementation + various improvements
This overhauls the proposed batch implementation (described in the
README but previously not implemented) and implements it.

Various other minor improvements and code restructuring.

Added a proposed "request" type to the README to be implemented; this is
like a command, but always expects a ["REPLY", TAG, ...] response.  The
request invoker provides a callback to invoke when such a REPLY arrives.
2020-02-05 20:21:27 -04:00
Jason Rhinelander 8d97ba31ad Added missing CMakeLists.txt and LICENSE 2020-02-05 20:21:02 -04:00
Jason Rhinelander f3d583c520 Initial LokiMQ release
This library is adapted from lokid's existing quorumnet code (added in
6.x) used for SN-to-SN communication for quorum voting but generalized
to be usable both there and as a basis for other communication channels
with loki projects (for example: wallet-to-lokid communication; loki-ss
and lokinet internal communication with lokid; loki-ss to loki-ss
communication and message passing; perhaps eventually loki p2p traffic).

This initial release compiles but likely has a few warts and bugs that
need ironing out in the implementation before it is production ready.
Some tests will follow.
2020-02-02 22:39:26 -04:00