oxen-io
/
oxen-mq
mirror of https://github.com/oxen-io/oxen-mq.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
oxen-mq/oxenmq/proxy.cpp

844 lines
35 KiB
C++
Raw Normal View History

Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
#include "oxenmq.h"
#include "oxenmq-internal.h"
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
#include <oxenc/hex.h>
Propagate proxy thread startup exceptions Currently if the proxy thread fails to start (typically because a bind fails) the exception happens in the proxy thread which is uncatchable by the caller (and aborts the program). This makes it nicer by transporting startup exceptions back to the start() call.
2021-11-30 18:57:53 +01:00
#include <exception>
#include <future>
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Add missing header for BSD pthread naming
2020-07-06 02:21:06 +02:00
#if defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__)
extern "C" {
#include <pthread.h>
#include <pthread_np.h>
}
#endif
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
#ifdef OXENMQ_USE_EPOLL
#include <sys/epoll.h>
#endif
Add support for setting umask when binding This is needed to be able to control the permissions of any created ipc sockets.
2020-05-06 19:52:41 +02:00
#ifndef _WIN32
extern "C" {
#include <sys/stat.h>
allow an IPC socket to set which group it is owned by using SOCKET_GID similar to how STARTUP_UMASK is done.
2020-09-08 19:15:11 +02:00
#include <sys/types.h>
#include <unistd.h>
Add support for setting umask when binding This is needed to be able to control the permissions of any created ipc sockets.
2020-05-06 19:52:41 +02:00
}
#endif
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
namespace oxenmq {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
void OxenMQ::proxy_quit() {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Received quit command, shutting down proxy thread");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
assert(std::none_of(workers.begin(), workers.end(), [](auto& worker) { return worker.worker_thread.joinable(); }));
Tagged threads for jobs, batches, and timers This adds to ability to have lokimq manage specific threads to which jobs (individual, batch jobs, batch completions, or timers) can be directed to. This allows dedicating a thread to some slow or thread-unsafe action where you can dump jobs to the tagged thread as a method of lockless job queuing.
2020-06-05 04:32:28 +02:00
assert(std::none_of(tagged_workers.begin(), tagged_workers.end(), [](auto& worker) { return std::get<0>(worker).worker_thread.joinable(); }));
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
cppzmq 4.7+ compatibility Updates bundled cppzmq to 4.7.1, and replaces deprecated functions with new API.
2020-11-10 22:47:31 +01:00
command.set(zmq::sockopt::linger, 0);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
command.close();
{
C++17 class template deduction
2020-05-12 20:59:22 +02:00
std::lock_guard lock{control_sockets_mutex};
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
proxy_shutting_down = true; // To prevent threads from opening new control sockets
}
workers_socket.close();
int linger = std::chrono::milliseconds{CLOSE_LINGER}.count();
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
for (auto& [id, s] : connections)
cppzmq 4.7+ compatibility Updates bundled cppzmq to 4.7.1, and replaces deprecated functions with new API.
2020-11-10 22:47:31 +01:00
s.set(zmq::sockopt::linger, linger);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
connections.clear();
peers.clear();
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Proxy thread teardown complete");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
void OxenMQ::proxy_send(oxenc::bt_dict_consumer data) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
// NB: bt_dict_consumer goes in alphabetical order
Require C++17 Removes lokimq::string_view (the type alias is still provided for backwards compat, but now is always std::string_view). Bump version (on dev branch) to 1.2.0
2020-05-12 20:33:59 +02:00
std::string_view hint;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
std::chrono::milliseconds keep_alive{DEFAULT_SEND_KEEP_ALIVE};
Make request timeout configurable For example: lmq.request(conn, "some.method", callback, lokimq::request_timeout{5s}); will result in the callback being called with a failure if the response doesn't arrive within 5s. (If it still arrives, but after the failure callback, it gets dropped).
2020-03-24 02:30:53 +01:00
std::chrono::milliseconds request_timeout{DEFAULT_REQUEST_TIMEOUT};
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
bool optional = false;
Add `send_option::outgoing` to force a send on an outgoing connection SS wants this, in particular, to be able to do reachability tests. (Using connect_remote for this was bad with pubkey-based routing ids because the second connection could replace an existing connection).
2020-04-03 06:31:45 +02:00
bool outgoing = false;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
bool incoming = false;
bool request = false;
bool have_conn_id = false;
ConnectionID conn_id;
std::string request_tag;
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
ReplyCallback request_callback;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (data.skip_until("conn_id")) {
conn_id.id = data.consume_integer<long long>();
if (conn_id.id == -1)
throw std::runtime_error("Invalid error: invalid conn_id value (-1)");
have_conn_id = true;
}
if (data.skip_until("conn_pubkey")) {
if (have_conn_id)
throw std::runtime_error("Internal error: Invalid proxy send command; conn_id and conn_pubkey are exclusive");
conn_id.pk = data.consume_string();
conn_id.id = ConnectionID::SN_ID;
} else if (!have_conn_id)
throw std::runtime_error("Internal error: Invalid proxy send command; conn_pubkey or conn_id missing");
if (data.skip_until("conn_route"))
conn_id.route = data.consume_string();
if (data.skip_until("hint"))
hint = data.consume_string_view();
if (data.skip_until("incoming"))
incoming = data.consume_integer<bool>();
Rename keep-alive -> keep_alive in internal serialization This makes it consistent with other internal parameter names.
2020-03-24 02:28:23 +01:00
if (data.skip_until("keep_alive"))
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
keep_alive = std::chrono::milliseconds{data.consume_integer<uint64_t>()};
if (data.skip_until("optional"))
optional = data.consume_integer<bool>();
Add `send_option::outgoing` to force a send on an outgoing connection SS wants this, in particular, to be able to do reachability tests. (Using connect_remote for this was bad with pubkey-based routing ids because the second connection could replace an existing connection).
2020-04-03 06:31:45 +02:00
if (data.skip_until("outgoing"))
outgoing = data.consume_integer<bool>();
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (data.skip_until("request"))
request = data.consume_integer<bool>();
if (request) {
if (!data.skip_until("request_callback"))
throw std::runtime_error("Internal error: received request without request_callback");
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
All sends use dontwait; add send failure callbacks We really don't *ever* want send to block, no matter how it is called, since the send is always in the proxy thread. This makes the actual send call always non-blocking, and adds callbacks that we can invoke on send failures: either on queue full errors (which might be recoverable), or both full queue and hard failures (which are generally not recoverable). These callbacks are both optional: they have to be passed in using `send_option::queue_full` (if you just want queue full notifies) or `send_option::queue_failure` (if you want queue full notifies *and* other send exceptions).
2020-03-29 20:03:04 +02:00
request_callback = detail::deserialize_object<ReplyCallback>(data.consume_integer<uintptr_t>());
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (!data.skip_until("request_tag"))
throw std::runtime_error("Internal error: received request without request_name");
request_tag = data.consume_string();
Make request timeout configurable For example: lmq.request(conn, "some.method", callback, lokimq::request_timeout{5s}); will result in the callback being called with a failure if the response doesn't arrive within 5s. (If it still arrives, but after the failure callback, it gets dropped).
2020-03-24 02:30:53 +01:00
if (data.skip_until("request_timeout"))
request_timeout = std::chrono::milliseconds{data.consume_integer<uint64_t>()};
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
if (!data.skip_until("send"))
throw std::runtime_error("Internal error: Invalid proxy send command; send parts missing");
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
oxenc::bt_list_consumer send = data.consume_list_consumer();
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
All sends use dontwait; add send failure callbacks We really don't *ever* want send to block, no matter how it is called, since the send is always in the proxy thread. This makes the actual send call always non-blocking, and adds callbacks that we can invoke on send failures: either on queue full errors (which might be recoverable), or both full queue and hard failures (which are generally not recoverable). These callbacks are both optional: they have to be passed in using `send_option::queue_full` (if you just want queue full notifies) or `send_option::queue_failure` (if you want queue full notifies *and* other send exceptions).
2020-03-29 20:03:04 +02:00
send_option::queue_failure::callback_t callback_nosend;
if (data.skip_until("send_fail"))
callback_nosend = detail::deserialize_object<decltype(callback_nosend)>(data.consume_integer<uintptr_t>());
send_option::queue_full::callback_t callback_noqueue;
if (data.skip_until("send_full_q"))
callback_noqueue = detail::deserialize_object<decltype(callback_noqueue)>(data.consume_integer<uintptr_t>());
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
// Now figure out which socket to send to and do the actual sending. We can repeat this loop
// multiple times, if we're sending to a SN, because it's possible that we have multiple
// connections open to that SN (e.g. one out + one in) so if one fails we can clean up that
// connection and try the next one.
Silence spurious warning on optional send failure When doing an optional send that gets declined (because we aren't connected) the "sending would block" warning would still be printed, but shouldn't be.
2020-04-29 19:54:54 +02:00
bool retry = true, sent = false, nowarn = false;
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
while (retry) {
retry = false;
zmq::socket_t *send_to;
if (conn_id.sn()) {
Rename PUBKEY_BASED_ROUTING_ID to EPHEMERAL_ROUTING_ID And similarly for the connect_option
2021-04-15 20:42:04 +02:00
auto sock_route = proxy_connect_sn(conn_id.pk, hint, optional, incoming, outgoing, EPHEMERAL_ROUTING_ID, keep_alive);
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
if (!sock_route.first) {
Silence spurious warning on optional send failure When doing an optional send that gets declined (because we aren't connected) the "sending would block" warning would still be printed, but shouldn't be.
2020-04-29 19:54:54 +02:00
nowarn = true;
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
if (optional)
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Not sending: send is optional and no connection to ",
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
oxenc::to_hex(conn_id.pk), " is currently established");
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
else
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(error, "Unable to send to ", oxenc::to_hex(conn_id.pk), ": no valid connection address found");
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
break;
}
send_to = sock_route.first;
conn_id.route = std::move(sock_route.second);
} else if (!conn_id.route.empty()) { // incoming non-SN connection
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
auto it = connections.find(conn_id.id);
if (it == connections.end()) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Unable to send to ", conn_id, ": incoming listening socket not found");
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
break;
}
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
send_to = &it->second;
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
} else {
auto pr = peers.equal_range(conn_id);
if (pr.first == peers.end()) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Unable to send: connection id ", conn_id, " is not (or is no longer) a valid outgoing connection");
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
break;
}
auto& peer = pr.first->second;
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
auto it = connections.find(peer.conn_id);
if (it == connections.end()) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Unable to send: peer connection id ", conn_id, " is not (or is no longer) a valid outgoing connection");
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
break;
}
send_to = &it->second;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
try {
All sends use dontwait; add send failure callbacks We really don't *ever* want send to block, no matter how it is called, since the send is always in the proxy thread. This makes the actual send call always non-blocking, and adds callbacks that we can invoke on send failures: either on queue full errors (which might be recoverable), or both full queue and hard failures (which are generally not recoverable). These callbacks are both optional: they have to be passed in using `send_option::queue_full` (if you just want queue full notifies) or `send_option::queue_failure` (if you want queue full notifies *and* other send exceptions).
2020-03-29 20:03:04 +02:00
sent = send_message_parts(*send_to, build_send_parts(send, conn_id.route));
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
} catch (const zmq::error_t &e) {
if (e.num() == EHOSTUNREACH && !conn_id.route.empty() /*= incoming conn*/) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Incoming connection is no longer valid; removing peer details");
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
auto pr = peers.equal_range(conn_id);
if (pr.first != peers.end()) {
if (!conn_id.sn()) {
peers.erase(pr.first);
} else {
bool removed;
for (auto it = pr.first; it != pr.second; ) {
auto& peer = it->second;
if (peer.route == conn_id.route) {
peers.erase(it);
removed = true;
break;
}
}
// The incoming connection to the SN is no longer good, but we can retry because
// we may have another active connection with the SN (or may want to open one).
if (removed) {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(debug, "Retrying sending to SN ", oxenc::to_hex(conn_id.pk), " using other sockets");
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
retry = true;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
}
}
}
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
if (!retry) {
Don't warn when we can't send back to an non-SN incoming connection This happens commonly when the remote has gone away and we shouldn't be warning about it.
2021-04-16 01:16:44 +02:00
if (!conn_id.sn() && !conn_id.route.empty()) { // incoming non-SN connection
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Unable to send message to incoming connection ", conn_id, ": ", e.what(),
Don't warn when we can't send back to an non-SN incoming connection This happens commonly when the remote has gone away and we shouldn't be warning about it.
2021-04-16 01:16:44 +02:00
"; remote has probably disconnected");
} else {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Unable to send message to ", conn_id, ": ", e.what());
Don't warn when we can't send back to an non-SN incoming connection This happens commonly when the remote has gone away and we shouldn't be warning about it.
2021-04-16 01:16:44 +02:00
}
Silence spurious warning on optional send failure When doing an optional send that gets declined (because we aren't connected) the "sending would block" warning would still be printed, but shouldn't be.
2020-04-29 19:54:54 +02:00
nowarn = true;
All sends use dontwait; add send failure callbacks We really don't *ever* want send to block, no matter how it is called, since the send is always in the proxy thread. This makes the actual send call always non-blocking, and adds callbacks that we can invoke on send failures: either on queue full errors (which might be recoverable), or both full queue and hard failures (which are generally not recoverable). These callbacks are both optional: they have to be passed in using `send_option::queue_full` (if you just want queue full notifies) or `send_option::queue_failure` (if you want queue full notifies *and* other send exceptions).
2020-03-29 20:03:04 +02:00
if (callback_nosend) {
job([callback = std::move(callback_nosend), error = e] { callback(&error); });
callback_nosend = nullptr;
}
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
}
}
}
if (request) {
if (sent) {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(debug, "Added new pending request ", oxenc::to_hex(request_tag));
Avoid segfault on retried SN connection request When we fail to send to a SN but can retry (e.g. because we had an incoming connection which no longer works, but can retry an outgoing connection) we were recursing, but this was resulting in a double-free of the request callback (since we'd try to take ownership of the incoming serialized pointer twice). Rewrite the code to use a loop with single ownership instead. This also changes the request callback behaviour to fire a failure callback immediately if we can't send a request; previously you'd have to wait for a timeout, but that is pointless if we couldn't get the request out.
2020-03-27 18:49:02 +01:00
pending_requests.insert({ request_tag, {
std::chrono::steady_clock::now() + request_timeout, std::move(request_callback) }});
} else {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Could not send request, scheduling request callback failure");
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
job([callback = std::move(request_callback)] { callback(false, {{"TIMEOUT"s}}); });
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
}
All sends use dontwait; add send failure callbacks We really don't *ever* want send to block, no matter how it is called, since the send is always in the proxy thread. This makes the actual send call always non-blocking, and adds callbacks that we can invoke on send failures: either on queue full errors (which might be recoverable), or both full queue and hard failures (which are generally not recoverable). These callbacks are both optional: they have to be passed in using `send_option::queue_full` (if you just want queue full notifies) or `send_option::queue_failure` (if you want queue full notifies *and* other send exceptions).
2020-03-29 20:03:04 +02:00
if (!sent) {
if (callback_nosend)
job([callback = std::move(callback_nosend)] { callback(nullptr); });
else if (callback_noqueue)
job(std::move(callback_noqueue));
Silence spurious warning on optional send failure When doing an optional send that gets declined (because we aren't connected) the "sending would block" warning would still be printed, but shouldn't be.
2020-04-29 19:54:54 +02:00
else if (!nowarn)
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Unable to send message to ", conn_id, ": sending would block");
All sends use dontwait; add send failure callbacks We really don't *ever* want send to block, no matter how it is called, since the send is always in the proxy thread. This makes the actual send call always non-blocking, and adds callbacks that we can invoke on send failures: either on queue full errors (which might be recoverable), or both full queue and hard failures (which are generally not recoverable). These callbacks are both optional: they have to be passed in using `send_option::queue_full` (if you just want queue full notifies) or `send_option::queue_failure` (if you want queue full notifies *and* other send exceptions).
2020-03-29 20:03:04 +02:00
}
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
void OxenMQ::proxy_reply(oxenc::bt_dict_consumer data) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
bool have_conn_id = false;
ConnectionID conn_id{0};
if (data.skip_until("conn_id")) {
conn_id.id = data.consume_integer<long long>();
if (conn_id.id == -1)
throw std::runtime_error("Invalid error: invalid conn_id value (-1)");
have_conn_id = true;
}
if (data.skip_until("conn_pubkey")) {
if (have_conn_id)
throw std::runtime_error("Internal error: Invalid proxy reply command; conn_id and conn_pubkey are exclusive");
conn_id.pk = data.consume_string();
conn_id.id = ConnectionID::SN_ID;
} else if (!have_conn_id)
throw std::runtime_error("Internal error: Invalid proxy reply command; conn_pubkey or conn_id missing");
if (!data.skip_until("send"))
throw std::runtime_error("Internal error: Invalid proxy reply command; send parts missing");
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
oxenc::bt_list_consumer send = data.consume_list_consumer();
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
auto pr = peers.equal_range(conn_id);
if (pr.first == pr.second) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Unable to send tagged reply: the connection is no longer valid");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return;
}
// We try any connections until one works (for ordinary remotes there will be just one, but for
// SNs there might be one incoming and one outgoing).
for (auto it = pr.first; it != pr.second; ) {
try {
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
send_message_parts(connections[it->second.conn_id], build_send_parts(send, it->second.route));
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
break;
} catch (const zmq::error_t &err) {
if (err.num() == EHOSTUNREACH) {
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
if (it->second.outgoing()) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Unable to send reply to non-SN request on outgoing socket: "
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
"remote is no longer connected; closing connection");
proxy_close_connection(it->second.conn_id, CLOSE_LINGER);
it = peers.erase(it);
++it;
} else {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Unable to send reply to non-SN request on incoming socket: "
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
"remote is no longer connected; removing peer details");
it = peers.erase(it);
}
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
} else {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Unable to send reply to incoming non-SN request: ", err.what());
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
++it;
}
}
}
}
Use fixed array for known-small internal messages Internal messages (control messages, worker messages) are always 3 parts or less, so we can optimize by using a stack allocated std::array for those cases rather than needing to continually clear and expand a heap allocated vector.
2022-05-12 17:42:08 +02:00
void OxenMQ::proxy_control_message(OxenMQ::control_message_array& parts, size_t len) {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
// We throw an uncaught exception here because we only generate control messages internally in
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
// oxenmq code: if one of these condition fail it's a oxenmq bug.
Use fixed array for known-small internal messages Internal messages (control messages, worker messages) are always 3 parts or less, so we can optimize by using a stack allocated std::array for those cases rather than needing to continually clear and expand a heap allocated vector.
2022-05-12 17:42:08 +02:00
if (len < 2)
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
throw std::logic_error("OxenMQ bug: Expected 2-3 message parts for a proxy control message");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
auto route = view(parts[0]), cmd = view(parts[1]);
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("control message: ", cmd);
Use fixed array for known-small internal messages Internal messages (control messages, worker messages) are always 3 parts or less, so we can optimize by using a stack allocated std::array for those cases rather than needing to continually clear and expand a heap allocated vector.
2022-05-12 17:42:08 +02:00
if (len == 3) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("...: ", parts[2]);
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
auto data = view(parts[2]);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (cmd == "SEND") {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("proxying message");
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
return proxy_send(data);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
} else if (cmd == "REPLY") {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("proxying reply to non-SN incoming message");
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
return proxy_reply(data);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
} else if (cmd == "BATCH") {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("proxy batch jobs");
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
auto ptrval = oxenc::bt_deserialize<uintptr_t>(data);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return proxy_batch(reinterpret_cast<detail::Batch*>(ptrval));
Allow injecting tasks into lokimq job queue This allows mixing some outside task into the lokimq job queue for a category (queued up with native LMQ requests for that category) for use when there is some external process that is able to generate messages. For example, the most immediate use for this is to allow an HTTP server to handle incoming RPC requests and, as soon as they arrive, inject them into LokiMQ's queue for the "rpc" category so that native LMQ rpc requests and HTTP rpc requests share the same thread pool and queue. These injected jobs bypass all of LokiMQ's authentication and response mechanisms: that's up to the invoked callback itself to manage. Injected tasks are somewhat similar to batch jobs, but unlike batch jobs the are queued and prioritized as ordinary external LokiMQ requests. (Batch jobs, in contrast, have a higher scheduling priority, no queue limits, and typically a larger available thread pool).
2020-06-08 02:30:46 +02:00
} else if (cmd == "INJECT") {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("proxy inject");
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
return proxy_inject_task(detail::deserialize_object<injected_task>(oxenc::bt_deserialize<uintptr_t>(data)));
Add missing SET_SNS proxy handler
2020-04-13 21:11:30 +02:00
} else if (cmd == "SET_SNS") {
return proxy_set_active_sns(data);
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
} else if (cmd == "UPDATE_SNS") {
return proxy_update_active_sns(data);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
} else if (cmd == "CONNECT_SN") {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
proxy_connect_sn(data);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return;
} else if (cmd == "CONNECT_REMOTE") {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
return proxy_connect_remote(data);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
} else if (cmd == "DISCONNECT") {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
return proxy_disconnect(data);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
} else if (cmd == "TIMER") {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
return proxy_timer(data);
Add timer cancellation & timer tests Updates `add_timer` to return a new opaque TimerID object that can later be passed to `cancel_timer` to cancel an existing timer. Also adds timer tests, which was omitted (except for one in the tagged threads section), along with a new test for timer deletion.
2021-05-21 02:50:25 +02:00
} else if (cmd == "TIMER_DEL") {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
return proxy_timer_del(oxenc::bt_deserialize<int>(data));
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
} else if (cmd == "BIND") {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
auto b = detail::deserialize_object<bind_data>(oxenc::bt_deserialize<uintptr_t>(data));
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
if (proxy_bind(b, bind.size()))
bind.push_back(std::move(b));
return;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Use fixed array for known-small internal messages Internal messages (control messages, worker messages) are always 3 parts or less, so we can optimize by using a stack allocated std::array for those cases rather than needing to continually clear and expand a heap allocated vector.
2022-05-12 17:42:08 +02:00
} else if (len == 2) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (cmd == "START") {
// Command send by the owning thread during startup; we send back a simple READY reply to
// let it know we are running.
return route_control(command, route, "READY");
} else if (cmd == "QUIT") {
// Asked to quit: set max_workers to zero and tell any idle ones to quit. We will
// close workers as they come back to READY status, and then close external
// connections once all workers are done.
max_workers = 0;
Use fixed vector for idle workers Use a count + fixed size vector with a separate variable tracking the size seems to perform slightly better than popping/pushing the vector.
2022-05-12 17:44:54 +02:00
for (size_t i = 0; i < idle_worker_count; i++)
route_control(workers_socket, workers[idle_workers[i]].worker_routing_id, "QUIT");
idle_worker_count = 0;
Tagged threads for jobs, batches, and timers This adds to ability to have lokimq manage specific threads to which jobs (individual, batch jobs, batch completions, or timers) can be directed to. This allows dedicating a thread to some slow or thread-unsafe action where you can dump jobs to the tagged thread as a method of lockless job queuing.
2020-06-05 04:32:28 +02:00
for (auto& [run, busy, queue] : tagged_workers)
if (!busy)
route_control(workers_socket, run.worker_routing_id, "QUIT");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return;
}
}
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
throw std::runtime_error("OxenMQ bug: Proxy received invalid control command: " +
Use fixed array for known-small internal messages Internal messages (control messages, worker messages) are always 3 parts or less, so we can optimize by using a stack allocated std::array for those cases rather than needing to continually clear and expand a heap allocated vector.
2022-05-12 17:42:08 +02:00
std::string{cmd} + " (" + std::to_string(len) + ")");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
bool OxenMQ::proxy_bind(bind_data& b, size_t bind_index) {
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
zmq::socket_t listener{context, zmq::socket_type::router};
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
setup_incoming_socket(listener, b.curve, pubkey, privkey, bind_index);
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
bool good = true;
try {
listener.bind(b.address);
} catch (const zmq::error_t&) {
good = false;
}
if (b.on_bind) {
b.on_bind(good);
b.on_bind = nullptr;
}
if (!good) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "OxenMQ failed to listen on ", b.address);
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
return false;
}
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(info, "OxenMQ listening on ", b.address);
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
b.conn_id = next_conn_id++;
connections.emplace_hint(connections.end(), b.conn_id, std::move(listener));
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
connections_updated = true;
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
return true;
}
Propagate proxy thread startup exceptions Currently if the proxy thread fails to start (typically because a bind fails) the exception happens in the proxy thread which is uncatchable by the caller (and aborts the program). This makes it nicer by transporting startup exceptions back to the start() call.
2021-11-30 18:57:53 +01:00
void OxenMQ::proxy_loop_init() {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Set thread names on proxy/workers Makes debugging which threads are using CPU easier.
2020-04-21 16:57:38 +02:00
#if defined(__linux__) || defined(__sun) || defined(__MINGW32__)
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
pthread_setname_np(pthread_self(), "omq-proxy");
Set thread names on proxy/workers Makes debugging which threads are using CPU easier.
2020-04-21 16:57:38 +02:00
#elif defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__)
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
pthread_set_name_np(pthread_self(), "omq-proxy");
Set thread names on proxy/workers Makes debugging which threads are using CPU easier.
2020-04-21 16:57:38 +02:00
#elif defined(__MACH__)
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
pthread_setname_np("omq-proxy");
Set thread names on proxy/workers Makes debugging which threads are using CPU easier.
2020-04-21 16:57:38 +02:00
#endif
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
zap_auth = zmq::socket_t{context, zmq::socket_type::rep};
cppzmq 4.7+ compatibility Updates bundled cppzmq to 4.7.1, and replaces deprecated functions with new API.
2020-11-10 22:47:31 +01:00
zap_auth.set(zmq::sockopt::linger, 0);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
zap_auth.bind(ZMQ_ADDR_ZAP);
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
workers_socket = zmq::socket_t{context, zmq::socket_type::router};
cppzmq 4.7+ compatibility Updates bundled cppzmq to 4.7.1, and replaces deprecated functions with new API.
2020-11-10 22:47:31 +01:00
workers_socket.set(zmq::sockopt::router_mandatory, true);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
workers_socket.bind(SN_ADDR_WORKERS);
workers.reserve(max_workers);
Use fixed vector for idle workers Use a count + fixed size vector with a separate variable tracking the size seems to perform slightly better than popping/pushing the vector.
2022-05-12 17:44:54 +02:00
idle_workers.resize(max_workers);
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
if (!workers.empty() || !worker_sockets.empty())
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
throw std::logic_error("Internal error: proxy thread started with active worker threads");
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
worker_sockets.reserve(max_workers);
// Pre-initialize these worker sockets rather than creating during thread initialization so that
// we can't hit the zmq socket limit during worker thread startup.
for (int i = 0; i < max_workers; i++)
worker_sockets.emplace_back(context, zmq::socket_type::dealer);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Add support for setting umask when binding This is needed to be able to control the permissions of any created ipc sockets.
2020-05-06 19:52:41 +02:00
#ifndef _WIN32
int saved_umask = -1;
if (STARTUP_UMASK >= 0)
saved_umask = umask(STARTUP_UMASK);
#endif
Add support for inproc: requests inproc support is special in zmq: in particular it completely bypasses the auth layer, which causes problems in OxenMQ because we assume that a message will always have auth information (set during initial connection handshake). This adds an "always-on" inproc listener and adds a new `connect_inproc` method for a caller to establish a connection to it. It also throws exceptions if you try to `listen_plain` or `listen_curve` on an inproc address, because that won't work for the reasons detailed above.
2021-08-04 21:16:36 +02:00
{
zmq::socket_t inproc_listener{context, zmq::socket_type::router};
inproc_listener.bind(SN_ADDR_SELF);
inproc_listener_connid = next_conn_id++;
connections.emplace_hint(connections.end(), inproc_listener_connid, std::move(inproc_listener));
}
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
for (size_t i = 0; i < bind.size(); i++) {
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
if (!proxy_bind(bind[i], i)) {
Propagate proxy thread startup exceptions Currently if the proxy thread fails to start (typically because a bind fails) the exception happens in the proxy thread which is uncatchable by the caller (and aborts the program). This makes it nicer by transporting startup exceptions back to the start() call.
2021-11-30 18:57:53 +01:00
OMQ_LOG(fatal, "OxenMQ failed to listen on ", bind[i].address);
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
throw zmq::error_t{};
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
}
Add support for setting umask when binding This is needed to be able to control the permissions of any created ipc sockets.
2020-05-06 19:52:41 +02:00
#ifndef _WIN32
if (saved_umask != -1)
umask(saved_umask);
allow an IPC socket to set which group it is owned by using SOCKET_GID similar to how STARTUP_UMASK is done.
2020-09-08 19:15:11 +02:00
allow changing uid as root
2020-09-08 19:22:22 +02:00
// set socket gid / uid if it is provided
if (SOCKET_GID != -1 or SOCKET_UID != -1) {
Add support for listening after startup This commit adds support for listening on new ports after startup. This will make things easier in storage server, in particular, where we want to delay listening on public ports until we have an established connection and initial block status update from oxend.
2021-06-23 15:51:08 +02:00
for (auto& b : bind) {
const address addr(b.address);
if (addr.ipc())
if (chown(addr.socket.c_str(), SOCKET_UID, SOCKET_GID) == -1)
allow an IPC socket to set which group it is owned by using SOCKET_GID similar to how STARTUP_UMASK is done.
2020-09-08 19:15:11 +02:00
throw std::runtime_error("cannot set group on " + addr.socket + ": " + strerror(errno));
}
}
Add support for setting umask when binding This is needed to be able to control the permissions of any created ipc sockets.
2020-05-06 19:52:41 +02:00
#endif
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
connections_updated = true;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
// Also add an internal connection to self so that calling code can avoid needing to
// special-case rare situations where we are supposed to talk to a quorum member that happens to
// be ourselves (which can happen, for example, with cross-quoum Blink communication)
// FIXME: not working
//listener.bind(SN_ADDR_SELF);
if (!timers)
timers.reset(zmq_timers_new());
if (-1 == zmq_timers_add(timers.get(),
std::chrono::milliseconds{CONN_CHECK_INTERVAL}.count(),
Propagate proxy thread startup exceptions Currently if the proxy thread fails to start (typically because a bind fails) the exception happens in the proxy thread which is uncatchable by the caller (and aborts the program). This makes it nicer by transporting startup exceptions back to the start() call.
2021-11-30 18:57:53 +01:00
[](int /*timer_id*/, void* self) { static_cast<OxenMQ*>(self)->proxy_conn_cleanup(); },
this)) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
throw zmq::error_t{};
}
Drop tagged thread init function; add synchronization dance The init function doesn't seem all that useful and makes the interface a bit more complicated, so drop it. Also addresses a race condition that can happen with tagged thread startup when the proxy tries to talk to a tagged thread but the tagged thread hasn't connected yet (which then aborts the proxy because it assumes workers are always routable).
2020-06-05 19:06:34 +02:00
// Wait for tagged worker threads to get ready and connect to us (we get a "STARTING" message)
// and send them back a "START" to let them know to go ahead with startup. We need this
// synchronization dance to guarantee that the workers are routable before we can proceed.
if (!tagged_workers.empty()) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Waiting for tagged workers");
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
{
std::unique_lock lock{tagged_startup_mutex};
Fix a race condition with tagged thread startup There's a very rare race condition where a tagged thread doesn't seem to exist when the proxy tries syncing startup with them, and so the proxy thread hangs in startup. This addresses it by avoiding looking at the `proxy_thread` variable (which probably isn't thread safe) in the worker's startup, and signalling the you-need-to-shutdown condition via a third option for the (formerly boolean) `tagged_go`.
2022-10-05 23:01:40 +02:00
tagged_go = tagged_go_mode::GO;
Fix zmq socket limit setting MAX_SOCKETS wasn't working properly because ZMQ uses it when the context is initialized, which happens when the first socket is constructed on that context. For OxenMQ, we had several sockets constructed on the context during OxenMQ construction, which meant the context_t was being initialized during OxenMQ construction, rather than during start(), and so setting MAX_SOCKETS would have no effect and you'd always get the default. This fixes it by making all the member variable zmq::socket_t's default-constructed, then replacing them with proper zmq::socket_t's during startup() so that we also defer zmq::context_t initialization to the right place. A second issue found during testing (also fixed here) is that the socket worker threads use to communicate to the proxy could fail if the worker socket creation would violate the zmq max sockets limit, which wound up throwing an uncaught exception and aborting. This pre-initializes (but doesn't connect) all potential worker threads sockets during start() so that the lazily-initialized worker thread will have one already set up rather than having to create a new one (which could fail).
2022-08-05 04:47:54 +02:00
}
tagged_cv.notify_all();
Drop tagged thread init function; add synchronization dance The init function doesn't seem all that useful and makes the interface a bit more complicated, so drop it. Also addresses a race condition that can happen with tagged thread startup when the proxy tries to talk to a tagged thread but the tagged thread hasn't connected yet (which then aborts the proxy because it assumes workers are always routable).
2020-06-05 19:06:34 +02:00
std::unordered_set<std::string_view> waiting_on;
for (auto& w : tagged_workers)
waiting_on.emplace(std::get<run_info>(w).worker_routing_id);
Propagate proxy thread startup exceptions Currently if the proxy thread fails to start (typically because a bind fails) the exception happens in the proxy thread which is uncatchable by the caller (and aborts the program). This makes it nicer by transporting startup exceptions back to the start() call.
2021-11-30 18:57:53 +01:00
for (std::vector<zmq::message_t> parts; !waiting_on.empty(); parts.clear()) {
Drop tagged thread init function; add synchronization dance The init function doesn't seem all that useful and makes the interface a bit more complicated, so drop it. Also addresses a race condition that can happen with tagged thread startup when the proxy tries to talk to a tagged thread but the tagged thread hasn't connected yet (which then aborts the proxy because it assumes workers are always routable).
2020-06-05 19:06:34 +02:00
recv_message_parts(workers_socket, parts);
if (parts.size() != 2 || view(parts[1]) != "STARTING"sv) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(error, "Received invalid message on worker socket while waiting for tagged thread startup");
Drop tagged thread init function; add synchronization dance The init function doesn't seem all that useful and makes the interface a bit more complicated, so drop it. Also addresses a race condition that can happen with tagged thread startup when the proxy tries to talk to a tagged thread but the tagged thread hasn't connected yet (which then aborts the proxy because it assumes workers are always routable).
2020-06-05 19:06:34 +02:00
continue;
}
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Received STARTING message from ", view(parts[0]));
Drop tagged thread init function; add synchronization dance The init function doesn't seem all that useful and makes the interface a bit more complicated, so drop it. Also addresses a race condition that can happen with tagged thread startup when the proxy tries to talk to a tagged thread but the tagged thread hasn't connected yet (which then aborts the proxy because it assumes workers are always routable).
2020-06-05 19:06:34 +02:00
if (auto it = waiting_on.find(view(parts[0])); it != waiting_on.end())
waiting_on.erase(it);
else
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(error, "Received STARTING message from unknown worker ", view(parts[0]));
Drop tagged thread init function; add synchronization dance The init function doesn't seem all that useful and makes the interface a bit more complicated, so drop it. Also addresses a race condition that can happen with tagged thread startup when the proxy tries to talk to a tagged thread but the tagged thread hasn't connected yet (which then aborts the proxy because it assumes workers are always routable).
2020-06-05 19:06:34 +02:00
}
for (auto&w : tagged_workers) {
Use raw index bytes in worker router Change the internal worker routing id to be "w" followed by the raw integer bytes, so that we can just memcpy them into a uint32_t rather than needing to do str -> integer conversion on each received worker message. (This also eliminates a vestigal call into oxenc internals).
2022-05-12 17:36:33 +02:00
OMQ_LOG(debug, "Telling tagged thread worker ", std::get<run_info>(w).worker_routing_name, " to finish startup");
Drop tagged thread init function; add synchronization dance The init function doesn't seem all that useful and makes the interface a bit more complicated, so drop it. Also addresses a race condition that can happen with tagged thread startup when the proxy tries to talk to a tagged thread but the tagged thread hasn't connected yet (which then aborts the proxy because it assumes workers are always routable).
2020-06-05 19:06:34 +02:00
route_control(workers_socket, std::get<run_info>(w).worker_routing_id, "START");
}
}
Propagate proxy thread startup exceptions Currently if the proxy thread fails to start (typically because a bind fails) the exception happens in the proxy thread which is uncatchable by the caller (and aborts the program). This makes it nicer by transporting startup exceptions back to the start() call.
2021-11-30 18:57:53 +01:00
}
void OxenMQ::proxy_loop(std::promise<void> startup) {
try {
proxy_loop_init();
} catch (...) {
startup.set_exception(std::current_exception());
return;
}
startup.set_value();
Use fixed array for known-small internal messages Internal messages (control messages, worker messages) are always 3 parts or less, so we can optimize by using a stack allocated std::array for those cases rather than needing to continually clear and expand a heap allocated vector.
2022-05-12 17:42:08 +02:00
// Fixed array used for worker and control messages: these are never longer than 3 parts:
std::array<zmq::message_t, 3> control_parts;
// General vector for handling incoming messages:
Propagate proxy thread startup exceptions Currently if the proxy thread fails to start (typically because a bind fails) the exception happens in the proxy thread which is uncatchable by the caller (and aborts the program). This makes it nicer by transporting startup exceptions back to the start() call.
2021-11-30 18:57:53 +01:00
std::vector<zmq::message_t> parts;
Drop tagged thread init function; add synchronization dance The init function doesn't seem all that useful and makes the interface a bit more complicated, so drop it. Also addresses a race condition that can happen with tagged thread startup when the proxy tries to talk to a tagged thread but the tagged thread hasn't connected yet (which then aborts the proxy because it assumes workers are always routable).
2020-06-05 19:06:34 +02:00
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
std::vector<std::pair<const int64_t, zmq::socket_t>*> queue; // Used as a circular buffer
#ifdef OXENMQ_USE_EPOLL
std::vector<struct epoll_event> evs;
#endif
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
while (true) {
std::chrono::milliseconds poll_timeout;
if (max_workers == 0) { // Will be 0 only if we are quitting
Tagged threads for jobs, batches, and timers This adds to ability to have lokimq manage specific threads to which jobs (individual, batch jobs, batch completions, or timers) can be directed to. This allows dedicating a thread to some slow or thread-unsafe action where you can dump jobs to the tagged thread as a method of lockless job queuing.
2020-06-05 04:32:28 +02:00
if (std::none_of(workers.begin(), workers.end(), [](auto &w) { return w.worker_thread.joinable(); }) &&
std::none_of(tagged_workers.begin(), tagged_workers.end(), [](auto &w) { return std::get<0>(w).worker_thread.joinable(); })) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
// All the workers have finished, so we can finish shutting down
return proxy_quit();
}
poll_timeout = 1s; // We don't keep running timers when we're quitting, so don't have a timer to check
} else {
poll_timeout = std::chrono::milliseconds{zmq_timers_timeout(timers.get())};
}
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
if (connections_updated) {
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
rebuild_pollitems();
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
// If we just rebuilt the queue then do a full check of everything, because we might
// have sockets that already edge-triggered that we need to fully drain before we start
// polling.
proxy_skip_one_poll = true;
}
// We round-robin connections when pulling off pending messages one-by-one rather than
// pulling off all messages from one connection before moving to the next; thus in cases of
// contention we end up fairly distributing.
queue.reserve(connections.size() + 1);
#ifdef OXENMQ_USE_EPOLL
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
bool process_command = false, process_worker = false, process_zap = false, process_all = false;
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
if (proxy_skip_one_poll) {
proxy_skip_one_poll = false;
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
process_command = command.get(zmq::sockopt::events) & ZMQ_POLLIN;
process_worker = workers_socket.get(zmq::sockopt::events) & ZMQ_POLLIN;
process_zap = zap_auth.get(zmq::sockopt::events) & ZMQ_POLLIN;
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
process_all = true;
}
else {
OMQ_TRACE("polling for new messages via epoll");
evs.resize(3 + connections.size());
const int max = epoll_wait(epoll_fd, evs.data(), evs.size(), poll_timeout.count());
queue.clear();
for (int i = 0; i < max; i++) {
const auto conn_id = evs[i].data.u64;
if (conn_id == EPOLL_COMMAND_ID)
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
process_command = true;
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
else if (conn_id == EPOLL_WORKER_ID)
process_worker = true;
else if (conn_id == EPOLL_ZAP_ID)
process_zap = true;
else if (auto it = connections.find(conn_id); it != connections.end())
queue.push_back(&*it);
}
queue.push_back(nullptr);
}
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
#else
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (proxy_skip_one_poll)
proxy_skip_one_poll = false;
else {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("polling for new messages");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
// We poll the control socket and worker socket for any incoming messages. If we have
// available worker room then also poll incoming connections and outgoing connections
// for messages to forward to a worker. Otherwise, we just look for a control message
// or a worker coming back with a ready message.
zmq::poll(pollitems.data(), pollitems.size(), poll_timeout);
}
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
constexpr bool process_command = true, process_worker = true, process_zap = true, process_all = true;
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
#endif
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
if (process_command) {
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
OMQ_TRACE("processing control messages");
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
while (size_t len = recv_message_parts(command, control_parts, zmq::recv_flags::dontwait))
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
proxy_control_message(control_parts, len);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
if (process_worker) {
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
OMQ_TRACE("processing worker messages");
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
while (size_t len = recv_message_parts(workers_socket, control_parts, zmq::recv_flags::dontwait))
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
proxy_worker_message(control_parts, len);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("processing timers");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
zmq_timers_execute(timers.get());
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
if (process_zap) {
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
// Handle any zap authentication
OMQ_TRACE("processing zap requests");
process_zap_requests();
}
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
// See if we can drain anything from the current queue before we potentially add to it
// below.
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("processing queued jobs and messages");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
proxy_process_queue();
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("processing new incoming messages");
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
if (process_all) {
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
queue.clear();
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
for (auto& id_sock : connections)
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
if (id_sock.second.get(zmq::sockopt::events) & ZMQ_POLLIN)
queue.push_back(&id_sock);
queue.push_back(nullptr);
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
}
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
size_t end = queue.size() - 1;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
for (size_t pos = 0; pos != end; ++pos %= queue.size()) {
parts.clear();
auto& [id, sock] = *queue[pos];
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (!recv_message_parts(sock, parts, zmq::recv_flags::dontwait))
continue;
// We only pull this one message now but then requeue the socket so that after we check
// all other sockets we come back to this one to check again.
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
queue[end] = queue[pos];
++end %= queue.size();
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (parts.empty()) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Ignoring empty (0-part) incoming message");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
continue;
}
Remove debug
2021-07-01 06:08:32 +02:00
if (!proxy_handle_builtin(id, sock, parts))
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
proxy_to_worker(id, sock, parts);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
if (connections_updated) {
Add epoll support for Linux Each call to zmq::poll is painfully slow when we have many open zmq sockets, such as when we have 1800 outbound connections (i.e. connected to every other service node, as services nodes might have sometimes and the Session push notification server *always* has). In testing on my local Ryzen 5950 system each time we go back to zmq::poll incurs about 1.5ms of (mostly system) CPU time with 2000 open outbound sockets, and so if we're being pelted with a nearly constant stream of requests (such as happens with the Session push notification server) we incur massive CPU costs every time we finish processing messages and go back to wait (via zmq::poll) for more. In testing a simple ZMQ (no OxenMQ) client/server that establishes 2000 connections to a server, and then has the server send a message back on a random connection every 1ms, we get atrocious CPU usage: the proxy thread spends a constant 100% CPU time. Virtually all of this is in the poll call itself, though, so we aren't really bottlenecked by how much can go through the proxy thread: in such a scenario the poll call uses its CPU then returns right away, we process the queue of messages, and return to another poll call. If we have lots of messages received in that time, though (because messages are coming fast and the poll was slow) then we process a lot all at once before going back to the poll, so the main consequences here are that: 1) We use a huge amount of CPU 2) We introduce latency in a busy situation because the CPU has to make the poll call (e.g. 1.5ms) before the next message can be processed. 3) If traffic is very bursty then the latency can manifest another problem: in the time it takes to poll we could accumulate enough incoming messages to overfill our internal per-category job queue, which was happening in the SPNS. (I also tested with 20k connections, and the poll time scaling was linear: we still processed everything, but in larger chunks because every poll call took about 15ms, and so we'd have about 15 messages at a time to process with added latency of up to 15ms). Switching to epoll *drastically* reduces the CPU usage in two ways: 1) It's massively faster by design: there's a single setup and communication of all the polling details to the kernel which we only have to do when our set of zmq sockets changes (which is relatively rare). 2) We can further reduce CPU time because epoll tells us *which* sockets need attention, and so if only 1 connection out of the 2000 sent us something we can only bother checking that single socket for messages. (In theory we can do the same with zmq::poll by querying for events available on the socket, but in practice it doesn't improve anything over just trying to read from them all). In my straight zmq test script, using epoll instead reduced CPU usage in the sends-every-1ms scenario from a constant pegged 100% of a core to an average of 2-3% of a single core. (Moreover this CPU usage level didn't noticeably change when using 20k connections instead of 2k).
2023-09-14 19:38:39 +02:00
// If connections got updated then our points are stale, so restart the proxy loop;
// we'll immediately end up right back here at least once before we resume polling.
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("connections became stale; short-circuiting incoming message loop");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
break;
}
}
epoll: fix hang on heavily loaded sockets This fixes a hang in the epoll code that triggers on heavy, bursty connections (such as the live SPNS APNs notifier). It turns out that side-effects of processing our sockets could leave other sockets (that we processed earlier in the loop) in a needs-attention state which we might not notice if we go back to epoll_wait right away. zmq::poll apparently takes care of this (and so is safe to re-poll even in this state), but when we are using epoll we need to worry about it by always checking for zmq events (which itself has side effects) and, if we get any, re-enter the loop body immediately *without* polling to deal with them.
2023-09-15 20:51:04 +02:00
#ifdef OXENMQ_USE_EPOLL
// If any socket still has ZMQ_POLLIN (which is possible if something we did above changed
// state on another socket, perhaps by writing to it) then we need to repeat the loop
// *without* going back to epoll again, until we get through everything without any
// ZMQ_POLLIN sockets. If we didn't, we could miss it and might end up deadlocked because
// of ZMQ's edge-triggered notifications on zmq fd's.
//
// More info on the complexities here at https://github.com/zeromq/libzmq/issues/3641 and
// https://funcptr.net/2012/09/10/zeromq---edge-triggered-notification/
if (!connections_updated && !proxy_skip_one_poll) {
for (auto* s : {&command, &workers_socket, &zap_auth}) {
if (s->get(zmq::sockopt::events) & ZMQ_POLLIN) {
proxy_skip_one_poll = true;
break;
}
}
if (!proxy_skip_one_poll) {
for (auto& [id, sock] : connections) {
if (sock.get(zmq::sockopt::events) & ZMQ_POLLIN) {
proxy_skip_one_poll = true;
break;
}
}
}
}
#endif
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("done proxy loop");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
}
Require C++17 Removes lokimq::string_view (the type alias is still provided for backwards compat, but now is always std::string_view). Bump version (on dev branch) to 1.2.0
2020-05-12 20:33:59 +02:00
static bool is_error_response(std::string_view cmd) {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
return cmd == "FORBIDDEN" || cmd == "FORBIDDEN_SN" || cmd == "NOT_A_SERVICE_NODE" || cmd == "UNKNOWNCOMMAND" || cmd == "NO_REPLY_TAG";
}
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
// Return true if we recognized/handled the builtin command (even if we reject it for whatever
// reason)
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
bool OxenMQ::proxy_handle_builtin(int64_t conn_id, zmq::socket_t& sock, std::vector<zmq::message_t>& parts) {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
// Doubling as a bool and an offset:
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
size_t incoming = sock.get(zmq::sockopt::type) == ZMQ_ROUTER;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
Require C++17 Removes lokimq::string_view (the type alias is still provided for backwards compat, but now is always std::string_view). Bump version (on dev branch) to 1.2.0
2020-05-12 20:33:59 +02:00
std::string_view route, cmd;
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
if (parts.size() < 1 + incoming) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Received empty message; ignoring");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return true;
}
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
if (incoming) {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
route = view(parts[0]);
cmd = view(parts[1]);
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
} else {
cmd = view(parts[0]);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_TRACE("Checking for builtins: '", cmd, "' from ", peer_address(parts.back()));
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (cmd == "REPLY") {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
size_t tag_pos = 1 + incoming;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (parts.size() <= tag_pos) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Received REPLY without a reply tag; ignoring");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return true;
}
std::string reply_tag{view(parts[tag_pos])};
auto it = pending_requests.find(reply_tag);
if (it != pending_requests.end()) {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(debug, "Received REPLY for pending command ", oxenc::to_hex(reply_tag), "; scheduling callback");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
std::vector<std::string> data;
data.reserve(parts.size() - (tag_pos + 1));
for (auto it = parts.begin() + (tag_pos + 1); it != parts.end(); ++it)
data.emplace_back(view(*it));
proxy_schedule_reply_job([callback=std::move(it->second.second), data=std::move(data)] {
callback(true, std::move(data));
});
pending_requests.erase(it);
} else {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(warn, "Received REPLY with unknown or already handled reply tag (", oxenc::to_hex(reply_tag), "); ignoring");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
return true;
} else if (cmd == "HI") {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
if (!incoming) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Got invalid 'HI' message on an outgoing connection; ignoring");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return true;
}
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Incoming client from ", peer_address(parts.back()), " sent HI, replying with HELLO");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
try {
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
send_routed_message(sock, std::string{route}, "HELLO");
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
} catch (const std::exception &e) { OMQ_LOG(warn, "Couldn't reply with HELLO: ", e.what()); }
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return true;
} else if (cmd == "HELLO") {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
if (incoming) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Got invalid 'HELLO' message on an incoming connection; ignoring");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return true;
}
auto it = std::find_if(pending_connects.begin(), pending_connects.end(),
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
[&](auto& pc) { return std::get<int64_t>(pc) == conn_id; });
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (it == pending_connects.end()) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Got invalid 'HELLO' message on an already handshaked incoming connection; ignoring");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return true;
}
auto& pc = *it;
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
auto pit = peers.find(std::get<int64_t>(pc));
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
if (pit == peers.end()) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Got invalid 'HELLO' message with invalid conn_id; ignoring");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
return true;
}
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Got initial HELLO server response from ", peer_address(parts.back()));
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
proxy_schedule_reply_job([on_success=std::move(std::get<ConnectSuccess>(pc)),
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
conn=pit->first] {
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
on_success(conn);
});
pending_connects.erase(it);
return true;
} else if (cmd == "BYE") {
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
if (!incoming) {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "BYE command received; disconnecting from ", peer_address(parts.back()));
Simplify conn index handling (#41) The existing code was overly complicated by trying to track indices in the `connections` vector, which complication happening because things get removed from `connections` requiring all the internal index values to be updated. So we ended up with a connection ID inside the ConnectionID object, plus a map of those connection IDs to the `connections` index, and need a map back from indices to ConnectionIDs. Though this seems to work usually, I recently noticed an oxen-storage-server sending oxend requests on the wrong connection and so I suspect there is some rare edge cases here where a failed connection index might not be updated properly. This PR simplifies the whole thing by making getting rid of connection ids entirely and keeping the connections in a map (with connection ids that never change). This might end up being a little less efficient than the vector, but it's unlikely to matter and the added complexity isn't worth it.
2021-06-23 22:51:25 +02:00
proxy_close_connection(conn_id, 0s);
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
} else {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Got invalid 'BYE' command on an incoming socket; ignoring");
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
return true;
}
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
else if (is_error_response(cmd)) {
// These messages (FORBIDDEN, UNKNOWNCOMMAND, etc.) are sent in response to us trying to
// invoke something that doesn't exist or we don't have permission to access. These have
Fix pre-1.1.0 UNKNOWNCOMMAND detection 1.0.5 sends just ["UNKNOWNCOMMAND"], so the detection here was broken, which resulted in a warning rather than just a debug log message.
2020-04-15 04:50:54 +02:00
// two forms (the latter is only sent by remotes running 1.1.0+).
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
// - ["XXX", "whatever.command"]
// - ["XXX", "REPLY", replytag]
// (ignoring the routing prefix on incoming commands).
// For the former, we log; for the latter we trigger the reply callback with a failure
Fix pre-1.1.0 UNKNOWNCOMMAND detection 1.0.5 sends just ["UNKNOWNCOMMAND"], so the detection here was broken, which resulted in a warning rather than just a debug log message.
2020-04-15 04:50:54 +02:00
if (parts.size() == (1 + incoming) && cmd == "UNKNOWNCOMMAND") {
// pre-1.1.0 sent just a plain UNKNOWNCOMMAND (without the actual command); this was not
// useful, but also this response is *expected* for things 1.0.5 didn't understand, like
// FORBIDDEN_SN: so log it only at debug level and move on.
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(debug, "Received plain UNKNOWNCOMMAND; remote is probably an older oxenmq. Ignoring.");
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
return true;
}
if (parts.size() == (3 + incoming) && view(parts[1 + incoming]) == "REPLY") {
std::string reply_tag{view(parts[2 + incoming])};
auto it = pending_requests.find(reply_tag);
if (it != pending_requests.end()) {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(debug, "Received ", cmd, " REPLY for pending command ", oxenc::to_hex(reply_tag), "; scheduling failure callback");
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
proxy_schedule_reply_job([callback=std::move(it->second.second), cmd=std::string{cmd}] {
callback(false, {{std::move(cmd)}});
});
pending_requests.erase(it);
} else {
Use oxen-encoding and add compatibility shim headers bt_*, hex, base32z, base64 all moved to oxen-encoding a while ago; this finishes the move by removing them from oxenmq and instead making oxenmq depend on oxen-encoding.
2022-01-18 15:30:23 +01:00
OMQ_LOG(warn, "Received REPLY with unknown or already handled reply tag (", oxenc::to_hex(reply_tag), "); ignoring");
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
}
} else {
Rebrand variables LMQ -> OMQ Various things still were using the lmq (or LMQ) names; change them all to omq/OMQ.
2021-11-30 19:10:47 +01:00
OMQ_LOG(warn, "Received ", cmd, ':', (parts.size() > 1 + incoming ? view(parts[1 + incoming]) : "(unknown command)"sv),
1.1.0: invocation-time SN auth; failure responses This replaces the recognition of SN status to be checked per-command invocation rather than on connection. As this breaks the API quite substantially, though doesn't really affect the functionality, it seems suitable to bump the minor version. This requires a fundamental shift in how the calling application tells LokiMQ about service nodes: rather than using a callback invoked on connection, the application now has to call set_active_sns() (or the more efficient update_active_sns(), if changes are readily available) to update the list whenever it changes. LokiMQ then keeps this list internally and uses it when determining whether to invoke. This release also brings better request responses on errors: when a request fails, the data argument will now be set to the failure reason, one of: - TIMEOUT - UNKNOWNCOMMAND - NOT_A_SERVICE_NODE (the remote isn't running in SN mode) - FORBIDDEN (auth level denies the request) - FORBIDDEN_SN (SN required and the remote doesn't see us as a SN) Some of these (UNKNOWNCOMMAND, NOT_A_SERVICE_NODE, FORBIDDEN) were already sent by remotes, but there was no connection to a request and so they would log a warning, but the request would have to time out. These errors (minus TIMEOUT, plus NO_REPLY_TAG signalling that a command is a request but didn't include a reply tag) are also sent in response to regular commands, but they simply result in a log warning showing the error type and the command that caused the failure when received.
2020-04-13 00:57:19 +02:00
" from ", peer_address(parts.back()));
}
return true;
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
}
return false;
}
Rename LokiMQ to OxenMQ
2021-01-14 19:37:14 +01:00
void OxenMQ::proxy_process_queue() {
Tagged threads for jobs, batches, and timers This adds to ability to have lokimq manage specific threads to which jobs (individual, batch jobs, batch completions, or timers) can be directed to. This allows dedicating a thread to some slow or thread-unsafe action where you can dump jobs to the tagged thread as a method of lockless job queuing.
2020-06-05 04:32:28 +02:00
if (max_workers == 0) // shutting down
return;
// First: send any tagged thread tasks to the tagged threads, if idle
for (auto& [run, busy, queue] : tagged_workers) {
if (!busy && !queue.empty()) {
busy = true;
proxy_run_worker(run.load(std::move(queue.front()), false, run.worker_id));
Change std::queue to std::deque typedef This shouldn't make any difference with an optimizing compiler, but makes it easier a bit easier to experiment with different data structures.
2022-05-12 17:32:17 +02:00
queue.pop_front();
Tagged threads for jobs, batches, and timers This adds to ability to have lokimq manage specific threads to which jobs (individual, batch jobs, batch completions, or timers) can be directed to. This allows dedicating a thread to some slow or thread-unsafe action where you can dump jobs to the tagged thread as a method of lockless job queuing.
2020-06-05 04:32:28 +02:00
}
}
// Second: process any batch jobs; since these are internal they are given higher priority.
Split up into logical headers and compilation units lokimq.cpp and lokimq.h were getting monolithic; this splits lokimq.cpp into multiple smaller cpp files by logical purpose for better parallel compilation ability. It also splits up the lokimq.h header slightly by moving the ConnectionID and Message types into their own headers.
2020-03-13 18:02:54 +01:00
proxy_run_batch_jobs(batch_jobs, batch_jobs_reserved, batch_jobs_active, false);
// Next any reply batch jobs (which are a bit different from the above, since they are
// externally triggered but for things we initiated locally).
proxy_run_batch_jobs(reply_jobs, reply_jobs_reserved, reply_jobs_active, true);
// Finally general incoming commands
for (auto it = pending_commands.begin(); it != pending_commands.end() && active_workers() < max_workers; ) {
auto& pending = *it;
if (pending.cat.active_threads < pending.cat.reserved_threads
|| active_workers() < general_workers) {
proxy_run_worker(get_idle_worker().load(std::move(pending)));
pending.cat.queued--;
pending.cat.active_threads++;
assert(pending.cat.queued >= 0);
it = pending_commands.erase(it);
} else {
++it; // no available general or reserved worker spots for this job right now
}
}
}
}