oxen-io
/
oxen-mq
mirror of https://github.com/oxen-io/oxen-mq.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
oxen-mq/oxenmq/connections.cpp

430 lines
17 KiB
C++
Raw Normal View History

Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
#include "oxenmq.h"
#include "oxenmq-internal.h"
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
#include <oxenc/hex.h>
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
#include <optional>
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
#ifdef OXENMQ_USE_EPOLL
extern "C" {
#include <sys/epoll.h>
#include <unistd.h>
}
#endif
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
namespace oxenmq {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
std::ostream& operator<<(std::ostream& o, const ConnectionID& conn) {
adds custom formatter for OMQ structs that have to_string member
2022-07-28 04:02:29 +02:00
return o << conn.to_string();
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
#ifdef OXENMQ_USE_EPOLL
void OxenMQ::rebuild_pollitems() {
if (epoll_fd != -1)
close(epoll_fd);
epoll_fd = epoll_create1(0);
struct epoll_event ev;
ev.events = EPOLLIN | EPOLLET;
ev.data.u64 = EPOLL_COMMAND_ID;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, command.get(zmq::sockopt::fd), &ev);
ev.data.u64 = EPOLL_WORKER_ID;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, workers_socket.get(zmq::sockopt::fd), &ev);
ev.data.u64 = EPOLL_ZAP_ID;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, zap_auth.get(zmq::sockopt::fd), &ev);
for (auto& [id, s] : connections) {
ev.data.u64 = id;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, s.get(zmq::sockopt::fd), &ev);
}
connections_updated = false;
}
#else // !OXENMQ_USE_EPOLL
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
namespace {
void add_pollitem(std::vector<zmq::pollitem_t>& pollitems, zmq::socket_t& sock) {
pollitems.emplace_back();
auto &p = pollitems.back();
p.socket = static_cast<void *>(sock);
p.fd = 0;
p.events = ZMQ_POLLIN;
}
} // anonymous namespace
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
void OxenMQ::rebuild_pollitems() {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
pollitems.clear();
add_pollitem(pollitems, command);
add_pollitem(pollitems, workers_socket);
add_pollitem(pollitems, zap_auth);
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
for (auto& [id, s] : connections)
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
add_pollitem(pollitems, s);
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
connections_updated = false;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
#endif // OXENMQ_USE_EPOLL
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
void OxenMQ::setup_external_socket(zmq::socket_t& socket) {
cppzmq 4.7+ compatibility Updates bundled cppzmq to 4.7.1, and replaces deprecated functions with new API.
2020-11-10 22:47:31 +01:00
socket.set(zmq::sockopt::reconnect_ivl, (int) RECONNECT_INTERVAL.count());
socket.set(zmq::sockopt::reconnect_ivl_max, (int) RECONNECT_INTERVAL_MAX.count());
socket.set(zmq::sockopt::handshake_ivl, (int) HANDSHAKE_TIME.count());
socket.set(zmq::sockopt::maxmsgsize, MAX_MSG_SIZE);
Disable IPv6 by default libzmq's IPv6 support is buggy when also using DNS hostname: in particular, if you try to connect to a DNS name that has an IPv6 address, then zmq will *only* try an IPv6 connection, even if the local client has no IPv6 connectivity, and even if the remote is only listening on its IPv4 address. This is much too unreliable to enable by default.
2021-12-02 23:58:57 +01:00
if (IPV6)
socket.set(zmq::sockopt::ipv6, 1);
Add ZMTP heartbeating (enabled by default) ZMTP heartbeating should help keep the connection alive, and should result in earlier detection of connection failures.
2020-04-14 21:06:01 +02:00
if (CONN_HEARTBEAT > 0s) {
cppzmq 4.7+ compatibility Updates bundled cppzmq to 4.7.1, and replaces deprecated functions with new API.
2020-11-10 22:47:31 +01:00
socket.set(zmq::sockopt::heartbeat_ivl, (int) CONN_HEARTBEAT.count());
Add ZMTP heartbeating (enabled by default) ZMTP heartbeating should help keep the connection alive, and should result in earlier detection of connection failures.
2020-04-14 21:06:01 +02:00
if (CONN_HEARTBEAT_TIMEOUT > 0s)
cppzmq 4.7+ compatibility Updates bundled cppzmq to 4.7.1, and replaces deprecated functions with new API.
2020-11-10 22:47:31 +01:00
socket.set(zmq::sockopt::heartbeat_timeout, (int) CONN_HEARTBEAT_TIMEOUT.count());
Add ZMTP heartbeating (enabled by default) ZMTP heartbeating should help keep the connection alive, and should result in earlier detection of connection failures.
2020-04-14 21:06:01 +02:00
}
Use slower and exponential backoff in reconnection ZMQ's default reconnection time is 100ms, indefinitely, which seems far too aggressive, particularly where we have some potential for hundreds or thousands of connections. This changes the default to be slightly slower (250ms instead of 100ms) on the first attempt, and to use exponential backoff doubling the time between each failed connection attempt up to a max of 5s between reconnection attempts to calm things down.
2020-04-17 21:09:53 +02:00
}
Rename PUBKEY_BASED_ROUTING_ID to EPHEMERAL_ROUTING_ID And similarly for the connect_option
2021-04-15 20:42:04 +02:00
void OxenMQ::setup_outgoing_socket(zmq::socket_t& socket, std::string_view remote_pubkey, bool use_ephemeral_routing_id) {
Use slower and exponential backoff in reconnection ZMQ's default reconnection time is 100ms, indefinitely, which seems far too aggressive, particularly where we have some potential for hundreds or thousands of connections. This changes the default to be slightly slower (250ms instead of 100ms) on the first attempt, and to use exponential backoff doubling the time between each failed connection attempt up to a max of 5s between reconnection attempts to calm things down.
2020-04-17 21:09:53 +02:00
setup_external_socket(socket);
if (!remote_pubkey.empty()) {
cppzmq 4.7+ compatibility Updates bundled cppzmq to 4.7.1, and replaces deprecated functions with new API.
2020-11-10 22:47:31 +01:00
socket.set(zmq::sockopt::curve_serverkey, remote_pubkey);
socket.set(zmq::sockopt::curve_publickey, pubkey);
socket.set(zmq::sockopt::curve_secretkey, privkey);
Use slower and exponential backoff in reconnection ZMQ's default reconnection time is 100ms, indefinitely, which seems far too aggressive, particularly where we have some potential for hundreds or thousands of connections. This changes the default to be slightly slower (250ms instead of 100ms) on the first attempt, and to use exponential backoff doubling the time between each failed connection attempt up to a max of 5s between reconnection attempts to calm things down.
2020-04-17 21:09:53 +02:00
}
Add ZMTP heartbeating (enabled by default) ZMTP heartbeating should help keep the connection alive, and should result in earlier detection of connection failures.
2020-04-14 21:06:01 +02:00
Rename PUBKEY_BASED_ROUTING_ID to EPHEMERAL_ROUTING_ID And similarly for the connect_option
2021-04-15 20:42:04 +02:00
if (!use_ephemeral_routing_id) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
std::string routing_id;
routing_id.reserve(33);
routing_id += 'L'; // Prefix because routing id's starting with \0 are reserved by zmq (and our pubkey might start with \0)
routing_id.append(pubkey.begin(), pubkey.end());
cppzmq 4.7+ compatibility Updates bundled cppzmq to 4.7.1, and replaces deprecated functions with new API.
2020-11-10 22:47:31 +01:00
socket.set(zmq::sockopt::routing_id, routing_id);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
// else let ZMQ pick a random one
}
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
void OxenMQ::setup_incoming_socket(zmq::socket_t& listener, bool curve, std::string_view pubkey, std::string_view privkey, size_t bind_index) {
setup_external_socket(listener);
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
listener.set(zmq::sockopt::zap_domain, oxenc::bt_serialize(bind_index));
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
if (curve) {
listener.set(zmq::sockopt::curve_server, true);
listener.set(zmq::sockopt::curve_publickey, pubkey);
listener.set(zmq::sockopt::curve_secretkey, privkey);
}
listener.set(zmq::sockopt::router_handover, true);
listener.set(zmq::sockopt::router_mandatory, true);
}
Add connect_option::*; allow per-connection pubkey-based routing setting Storage server, in particular, needs to disable pubkey-based routing on its connection to oxend (because it is sharing oxend's own keys), but wants it by default for SS-to-SS connections. This allows the oxend connection to turn it off so that we don't have oxend omq connections replacing each other.
2021-04-15 20:09:49 +02:00
// Deprecated versions:
ConnectionID OxenMQ::connect_remote(std::string_view remote, ConnectSuccess on_connect,
ConnectFailure on_failure, AuthLevel auth_level, std::chrono::milliseconds timeout) {
return connect_remote(address{remote}, std::move(on_connect), std::move(on_failure),
auth_level, connect_option::timeout{timeout});
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
}
Add connect_option::*; allow per-connection pubkey-based routing setting Storage server, in particular, needs to disable pubkey-based routing on its connection to oxend (because it is sharing oxend's own keys), but wants it by default for SS-to-SS connections. This allows the oxend connection to turn it off so that we don't have oxend omq connections replacing each other.
2021-04-15 20:09:49 +02:00
ConnectionID OxenMQ::connect_remote(std::string_view remote, ConnectSuccess on_connect,
ConnectFailure on_failure, std::string_view pubkey, AuthLevel auth_level,
std::chrono::milliseconds timeout) {
return connect_remote(address{remote}.set_pubkey(pubkey), std::move(on_connect),
std::move(on_failure), auth_level, connect_option::timeout{timeout});
Take lokimq::address as connect_remote argument Deprecates the existing connect_remote() that takes remote addr and pubkey as separate strings, just taking a `address` instead (into which the caller can set pubkey/curve data as desired). Also slightly changes how `connect_remote()` works when called with a string remote but no pubkey: that string is now an augmented lokimq::address string so that it can use the various formats supported by `lokimq::address`. (This was meant to be included in the PR that added `address` but apparently didn't get implemented.)
2020-06-29 20:39:11 +02:00
}
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
void OxenMQ::disconnect(ConnectionID id, std::chrono::milliseconds linger) {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
detail::send_control(get_control_socket(), "DISCONNECT", oxenc::bt_serialize<oxenc::bt_dict>({
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
{"conn_id", id.id},
{"linger_ms", linger.count()},
{"pubkey", id.pk},
}));
}
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
std::pair<zmq::socket_t *, std::string>
Rename PUBKEY_BASED_ROUTING_ID to EPHEMERAL_ROUTING_ID And similarly for the connect_option
2021-04-15 20:42:04 +02:00
OxenMQ::proxy_connect_sn(std::string_view remote, std::string_view connect_hint, bool optional, bool incoming_only, bool outgoing_only, bool use_ephemeral_routing_id, std::chrono::milliseconds keep_alive) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
ConnectionID remote_cid{remote};
auto its = peers.equal_range(remote_cid);
peer_info* peer = nullptr;
for (auto it = its.first; it != its.second; ++it) {
if (incoming_only && it->second.route.empty())
continue; // outgoing connection but we were asked to only use incoming connections
Add `send_option::outgoing` to force a send on an outgoing connection SS wants this, in particular, to be able to do reachability tests. (Using connect_remote for this was bad with pubkey-based routing ids because the second connection could replace an existing connection).
2020-04-03 06:31:45 +02:00
if (outgoing_only && !it->second.route.empty())
continue;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
peer = &it->second;
break;
}
if (peer) {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_TRACE("proxy asked to connect to ", oxenc::to_hex(remote), "; reusing existing connection");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (peer->route.empty() /* == outgoing*/) {
if (peer->idle_expiry < keep_alive) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "updating existing outgoing peer connection idle expiry time from ",
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
peer->idle_expiry.count(), "ms to ", keep_alive.count(), "ms");
peer->idle_expiry = keep_alive;
}
peer->activity();
}
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
return {&connections[peer->conn_id], peer->route};
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
} else if (optional || incoming_only) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "proxy asked for optional or incoming connection, but no appropriate connection exists so aborting connection attempt");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return {nullptr, ""s};
}
// No connection so establish a new one
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(debug, "proxy establishing new outbound connection to ", oxenc::to_hex(remote));
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
std::string addr;
bool to_self = false && remote == pubkey; // FIXME; need to use a separate listening socket for this, otherwise we can't easily
// tell it wasn't from a remote.
if (to_self) {
// special inproc connection if self that doesn't need any external connection
addr = SN_ADDR_SELF;
} else {
addr = std::string{connect_hint};
if (addr.empty())
addr = sn_lookup(remote);
else
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "using connection hint ", connect_hint);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (addr.empty()) {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(error, "peer lookup failed for ", oxenc::to_hex(remote));
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return {nullptr, ""s};
}
}
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(debug, oxenc::to_hex(pubkey), " (me) connecting to ", addr, " to reach ", oxenc::to_hex(remote));
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
std::optional<zmq::socket_t> socket;
Catch connect failures socket.connect() can throw, e.g. if given an invalid connection address; catch this, log the error, and return a failure condition.
2020-03-29 19:40:21 +02:00
try {
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
socket.emplace(context, zmq::socket_type::dealer);
setup_outgoing_socket(*socket, remote, use_ephemeral_routing_id);
socket->connect(addr);
Catch connect failures socket.connect() can throw, e.g. if given an invalid connection address; catch this, log the error, and return a failure condition.
2020-03-29 19:40:21 +02:00
} catch (const zmq::error_t& e) {
// Note that this failure cases indicates something serious went wrong that means zmq isn't
// even going to try connecting (for example an unparseable remote address).
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(error, "Outgoing connection to ", addr, " failed: ", e.what());
Catch connect failures socket.connect() can throw, e.g. if given an invalid connection address; catch this, log the error, and return a failure condition.
2020-03-29 19:40:21 +02:00
return {nullptr, ""s};
}
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
auto& p = peers.emplace(std::move(remote_cid), peer_info{})->second;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
p.service_node = true;
p.pubkey = std::string{remote};
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
p.conn_id = next_conn_id++;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
p.idle_expiry = keep_alive;
p.activity();
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
connections_updated = true;
Fix messages coming back on an outgoing connection The recent PR that revamped the connection IDs missed a case when connecting to service nodes where we store the SN pubkey in peers, but then fail to find the peer when we look it up by connection id. This adds the required tracking to fix that case (and adds a test that fails without the fix here).
2021-07-01 05:37:55 +02:00
outgoing_sn_conns.emplace_hint(outgoing_sn_conns.end(), p.conn_id, ConnectionID{remote});
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
auto it = connections.emplace_hint(connections.end(), p.conn_id, *std::move(socket));
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
return {&it->second, ""s};
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
std::pair<zmq::socket_t *, std::string> OxenMQ::proxy_connect_sn(oxenc::bt_dict_consumer data) {
Require C++17 Removes lokimq::string_view (the type alias is still provided for backwards compat, but now is always std::string_view). Bump version (on dev branch) to 1.2.0
2020-05-12 20:33:59 +02:00
std::string_view hint, remote_pk;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
std::chrono::milliseconds keep_alive;
Rename PUBKEY_BASED_ROUTING_ID to EPHEMERAL_ROUTING_ID And similarly for the connect_option
2021-04-15 20:42:04 +02:00
bool optional = false, incoming_only = false, outgoing_only = false, ephemeral_rid = EPHEMERAL_ROUTING_ID;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
// Alphabetical order
Rename PUBKEY_BASED_ROUTING_ID to EPHEMERAL_ROUTING_ID And similarly for the connect_option
2021-04-15 20:42:04 +02:00
if (data.skip_until("ephemeral_rid"))
ephemeral_rid = data.consume_integer<bool>();
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (data.skip_until("hint"))
Fix storing reference to temporary consume_string returns a temporary string; we wnat consume_string_view which returns a view into the data being consumed.
2020-04-17 19:06:00 +02:00
hint = data.consume_string_view();
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (data.skip_until("incoming"))
incoming_only = data.consume_integer<bool>();
Rename keep-alive -> keep_alive in internal serialization This makes it consistent with other internal parameter names.
2020-03-24 02:28:23 +01:00
if (data.skip_until("keep_alive"))
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
keep_alive = std::chrono::milliseconds{data.consume_integer<uint64_t>()};
if (data.skip_until("optional"))
optional = data.consume_integer<bool>();
Add `send_option::outgoing` to force a send on an outgoing connection SS wants this, in particular, to be able to do reachability tests. (Using connect_remote for this was bad with pubkey-based routing ids because the second connection could replace an existing connection).
2020-04-03 06:31:45 +02:00
if (data.skip_until("outgoing_only"))
outgoing_only = data.consume_integer<bool>();
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (!data.skip_until("pubkey"))
throw std::runtime_error("Internal error: Invalid proxy_connect_sn command; pubkey missing");
Fix storing reference to temporary consume_string returns a temporary string; we wnat consume_string_view which returns a view into the data being consumed.
2020-04-17 19:06:00 +02:00
remote_pk = data.consume_string_view();
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Rename PUBKEY_BASED_ROUTING_ID to EPHEMERAL_ROUTING_ID And similarly for the connect_option
2021-04-15 20:42:04 +02:00
return proxy_connect_sn(remote_pk, hint, optional, incoming_only, outgoing_only, ephemeral_rid, keep_alive);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
/// Closes outgoing connections and removes all references. Note that this will call `erase()`
/// which can invalidate iterators on the various connection containers - if you don't want that,
/// delete it first so that the container won't contain the element being deleted.
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
void OxenMQ::proxy_close_connection(int64_t id, std::chrono::milliseconds linger) {
auto it = connections.find(id);
if (it == connections.end()) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "internal error: connection to close (", id, ") doesn't exist!");
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
return;
}
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Closing conn ", id);
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
it->second.set(zmq::sockopt::linger, linger > 0ms ? (int) linger.count() : 0);
connections.erase(it);
connections_updated = true;
Fix messages coming back on an outgoing connection The recent PR that revamped the connection IDs missed a case when connecting to service nodes where we store the SN pubkey in peers, but then fail to find the peer when we look it up by connection id. This adds the required tracking to fix that case (and adds a test that fails without the fix here).
2021-07-01 05:37:55 +02:00
outgoing_sn_conns.erase(id);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
void OxenMQ::proxy_expire_idle_peers() {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
for (auto it = peers.begin(); it != peers.end(); ) {
auto &info = it->second;
if (info.outgoing()) {
Fix broken idle expiry timeout Idle time was being calculated as the negative of what it should have been, so a connection idle for 30s was idle for "-30s", and since -30 is not greater than whatever the idle time is, it would never expire and get closed. This was resulting in SNs keeping connections open forever, which was very likely not helping with connectivity (and probably also responsible for some of the connection rushes triggering ISP DDOS warnings).
2020-04-17 21:06:54 +02:00
auto idle = std::chrono::steady_clock::now() - info.last_activity;
if (idle > info.idle_expiry) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Closing outgoing connection to ", it->first, ": idle time (",
Drop idle connections regularly The check here on "only if we have some idle workers" fails catastrophically with one worker because that worker is always occupied when this code gets called because of how the loop works and so connections don't get expired at all.
2020-04-18 07:55:12 +02:00
std::chrono::duration_cast<std::chrono::milliseconds>(idle).count(), "ms) reached connection timeout (",
info.idle_expiry.count(), "ms)");
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
proxy_close_connection(info.conn_id, CLOSE_LINGER);
it = peers.erase(it);
Fix broken idle expiry timeout Idle time was being calculated as the negative of what it should have been, so a connection idle for 30s was idle for "-30s", and since -30 is not greater than whatever the idle time is, it would never expire and get closed. This was resulting in SNs keeping connections open forever, which was very likely not helping with connectivity (and probably also responsible for some of the connection rushes triggering ISP DDOS warnings).
2020-04-17 21:06:54 +02:00
} else {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(trace, "Not closing ", it->first, ": ", std::chrono::duration_cast<std::chrono::milliseconds>(idle).count(),
Drop idle connections regularly The check here on "only if we have some idle workers" fails catastrophically with one worker because that worker is always occupied when this code gets called because of how the loop works and so connections don't get expired at all.
2020-04-18 07:55:12 +02:00
"ms <= ", info.idle_expiry.count(), "ms");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
++it;
continue;
}
} else {
++it;
}
}
}
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
void OxenMQ::proxy_conn_cleanup() {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("starting proxy connections cleanup");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Drop idle connections regularly The check here on "only if we have some idle workers" fails catastrophically with one worker because that worker is always occupied when this code gets called because of how the loop works and so connections don't get expired at all.
2020-04-18 07:55:12 +02:00
// Drop idle connections (if we haven't done it in a while)
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("closing idle connections");
Drop idle connections regularly The check here on "only if we have some idle workers" fails catastrophically with one worker because that worker is always occupied when this code gets called because of how the loop works and so connections don't get expired at all.
2020-04-18 07:55:12 +02:00
proxy_expire_idle_peers();
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
auto now = std::chrono::steady_clock::now();
// FIXME - check other outgoing connections to see if they died and if so purge them
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("Timing out pending outgoing connections");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
// Check any pending outgoing connections for timeout
for (auto it = pending_connects.begin(); it != pending_connects.end(); ) {
auto& pc = *it;
if (std::get<std::chrono::steady_clock::time_point>(pc) < now) {
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
auto id = std::get<int64_t>(pc);
job([cid = ConnectionID{id}, callback = std::move(std::get<ConnectFailure>(pc))] { callback(cid, "connection attempt timed out"); });
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
it = pending_connects.erase(it); // Don't let the below erase it (because it invalidates iterators)
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
proxy_close_connection(id, CLOSE_LINGER);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
} else {
++it;
}
}
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("Timing out pending requests");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
// Remove any expired pending requests and schedule their callback with a failure
for (auto it = pending_requests.begin(); it != pending_requests.end(); ) {
auto& callback = it->second;
if (callback.first < now) {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(debug, "pending request ", oxenc::to_hex(it->first), " expired, invoking callback with failure status and removing");
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
job([callback = std::move(callback.second)] { callback(false, {{"TIMEOUT"s}}); });
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
it = pending_requests.erase(it);
} else {
++it;
}
}
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("done proxy connections cleanup");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
};
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
void OxenMQ::proxy_connect_remote(oxenc::bt_dict_consumer data) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
AuthLevel auth_level = AuthLevel::none;
long long conn_id = -1;
ConnectSuccess on_connect;
ConnectFailure on_failure;
std::string remote;
std::string remote_pubkey;
std::chrono::milliseconds timeout = REMOTE_CONNECT_TIMEOUT;
Rename PUBKEY_BASED_ROUTING_ID to EPHEMERAL_ROUTING_ID And similarly for the connect_option
2021-04-15 20:42:04 +02:00
bool ephemeral_rid = EPHEMERAL_ROUTING_ID;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (data.skip_until("auth_level"))
auth_level = static_cast<AuthLevel>(data.consume_integer<std::underlying_type_t<AuthLevel>>());
if (data.skip_until("conn_id"))
conn_id = data.consume_integer<long long>();
Rename PUBKEY_BASED_ROUTING_ID to EPHEMERAL_ROUTING_ID And similarly for the connect_option
2021-04-15 20:42:04 +02:00
if (data.skip_until("connect"))
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
on_connect = detail::deserialize_object<ConnectSuccess>(data.consume_integer<uintptr_t>());
Rename PUBKEY_BASED_ROUTING_ID to EPHEMERAL_ROUTING_ID And similarly for the connect_option
2021-04-15 20:42:04 +02:00
if (data.skip_until("ephemeral_rid"))
ephemeral_rid = data.consume_integer<bool>();
if (data.skip_until("failure"))
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
on_failure = detail::deserialize_object<ConnectFailure>(data.consume_integer<uintptr_t>());
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (data.skip_until("pubkey")) {
remote_pubkey = data.consume_string();
assert(remote_pubkey.size() == 32 || remote_pubkey.empty());
}
if (data.skip_until("remote"))
remote = data.consume_string();
if (data.skip_until("timeout"))
timeout = std::chrono::milliseconds{data.consume_integer<uint64_t>()};
if (conn_id == -1 || remote.empty())
throw std::runtime_error("Internal error: CONNECT_REMOTE proxy command missing required 'conn_id' and/or 'remote' value");
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(debug, "Establishing remote connection to ", remote,
remote_pubkey.empty() ? " (NULL auth)" : " via CURVE expecting pubkey " + oxenc::to_hex(remote_pubkey));
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
std::optional<zmq::socket_t> sock;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
try {
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
sock.emplace(context, zmq::socket_type::dealer);
setup_outgoing_socket(*sock, remote_pubkey, ephemeral_rid);
sock->connect(remote);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
} catch (const zmq::error_t &e) {
proxy_schedule_reply_job([conn_id, on_failure=std::move(on_failure), what="connect() failed: "s+e.what()] {
on_failure(conn_id, std::move(what));
});
return;
}
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
auto &s = connections.emplace_hint(connections.end(), conn_id, std::move(*sock))->second;
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
connections_updated = true;
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Opened new zmq socket to ", remote, ", conn_id ", conn_id, "; sending HI");
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
send_direct_message(s, "HI");
pending_connects.emplace_back(conn_id, std::chrono::steady_clock::now() + timeout,
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
std::move(on_connect), std::move(on_failure));
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
auto& peer = peers.emplace(ConnectionID{conn_id, remote_pubkey}, peer_info{})->second;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
peer.pubkey = std::move(remote_pubkey);
peer.service_node = false;
peer.auth_level = auth_level;
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
peer.conn_id = conn_id;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
peer.idle_expiry = 24h * 10 * 365; // "forever"
peer.activity();
}
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
void OxenMQ::proxy_disconnect(oxenc::bt_dict_consumer data) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
ConnectionID connid{-1};
std::chrono::milliseconds linger = 1s;
if (data.skip_until("conn_id"))
connid.id = data.consume_integer<long long>();
if (data.skip_until("linger_ms"))
linger = std::chrono::milliseconds(data.consume_integer<long long>());
if (data.skip_until("pubkey"))
connid.pk = data.consume_string();
if (connid.sn() && connid.pk.size() != 32)
throw std::runtime_error("Error: invalid disconnect of SN without a valid pubkey");
proxy_disconnect(std::move(connid), linger);
}
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
void OxenMQ::proxy_disconnect(ConnectionID conn, std::chrono::milliseconds linger) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("Disconnecting outgoing connection to ", conn);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
auto pr = peers.equal_range(conn);
for (auto it = pr.first; it != pr.second; ++it) {
auto& peer = it->second;
if (peer.outgoing()) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Closing outgoing connection to ", conn);
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
proxy_close_connection(peer.conn_id, linger);
peers.erase(it);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return;
}
}
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Failed to disconnect ", conn, ": no such outgoing connection");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
adds custom formatter for OMQ structs that have to_string member
2022-07-28 04:02:29 +02:00
std::string ConnectionID::to_string() const {
if (!pk.empty())
return (sn() ? std::string("SN ") : std::string("non-SN authenticated remote ")) + oxenc::to_hex(pk);
else
return std::string("unauthenticated remote [") + std::to_string(id) + "]";
}
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}