- Add opt-in tx-extra parsing to `get_transactions`; this now lets you
get most tx details (including service node tx info) via the RPC.
- Add ability for RPC block header requests to include a list of tx
hashes contained in the block.
- remove long-deprecated txs_as_hex from `get_transactions`: this
essentially doubled the length of requests since txs[i].as_hex has the
exact same data. Despite this being the "new" format for several
years, wallet code was still relying on the "old" format. Since 8.x
already breaks the RPC, this seems a good time to remove it.
- Significantly cleaned up and documented how pruned transactions
results are returned in GET_TRANSACTIONS. Previously it was fairly
uncertain when you'd get a pruned or unpruned transaction. (See
comments added in core_rpc_server_commands_defs.h)
- Stop returning the transaction data hex values when you ask for
`decode_as_json`. The documentation already implied you don't get
it, but you do, and this is virtually always just wasted extra data
when you're asking for the json-decoded value.
- fix miner txes being treated in the RPC as pruned transactions (they
aren't: they just have no prunable data).
- Made a bunch of `get_transactions` fields (including virtually all of
the new parsed tx-extras fields) into std::optional's so that the keys
aren't included when the values are empty.
- Fix bug in `get_transactions` that raised an exception if *some* (but
not all) of the requested transactions were missed. Previously you
could only get all-missed, all-found, or an exception.
In short: epee's http client is garbage, standard violating, and
unreliable.
This completely removes the epee http client support and replaces it
with cpr, a curl-based C++ wrapper. rpc/http_client.h wraps cpr for RPC
requests specifically, but it is also usable directly.
This replacement has a number of advantages:
- requests are considerably more reliable. The epee http client code
assumes that a connection will be kept alive forever, and returns a
failure if a connection is ever closed. This results in some very
annoying things: for example, preparing a transaction and then waiting
a long tim before confirming it will usually result in an error
communication with the daemon. This is just terribly behaviour: the
right thing to do on a connection failure is to resubmit the request.
- epee's http client is broken in lots of other ways: for example, it
tries throwing SSL at the port to see if it is HTTPS, but this is
protocol violating and just breaks (with a several second timeout) on
anything that *isn't* epee http server (for example, when lokid is
behind a proxying server).
- even when it isn't doing the above, the client breaks in other ways:
for example, there is a comment (replaced in this PR) in the Trezor PR
code that forces a connection close after every request because epee's
http client doesn't do proper keep-alive request handling.
- it seems noticeably faster to me in practical use in this PR; both
simple requests (for example, when running `lokid status`) and
wallet<->daemon connections are faster, probably because of crappy
code in epee. (I think this is also related to the throw-ssl-at-it
junk above: the epee client always generates an ssl certificate during
static initialization because it might need one at some point).
- significantly reduces the amount of code we have to maintain.
- removes all the epee ssl option code: curl can handle all of that just
fine.
- removes the epee socks proxy code; curl can handle that just fine.
(And can do more: it also supports using HTTP/HTTPS proxies).
- When a cli wallet connection fails we know show why it failed (which
now is an error message from curl), which could have all sorts of
reasons like hostname resolution failure, bad ssl certificate, etc.
Previously you just got a useless generic error that tells you
nothing.
Other related changes in this PR:
- Drops the check-for-update and download-update code. To the best of
my knowledge these have never been supported in loki-core and so it
didn't seem worth the trouble to convert them to use cpr for the
requests.
- Cleaned up node_rpc_proxy return values: there was an inconsistent mix
of ways to return errors and how the returned strings were handled.
Instead this cleans it up to return a pair<bool, val>, which (with
C++17) can be transparently captured as:
auto [success, val] = node.whatever(req);
This drops the failure message string, but it was almost always set to
something fairly useless (if we want to resurrect it we could easily
change the first element to be a custom type with a bool operator for
success, and a `.error` attribute containing some error string, but
for the most part the current code wasn't doing much useful with the
failure string).
- changed local detection (for automatic trusted daemon determination)
to just look for localhost, and to not try to resolve anything.
Trusting non-public IPs does not work well (e.g. with lokinet where
all .loki addresses resolve to a local IP).
- ssl fingerprint option is removed; this isn't supported by curl
(because it is essentially just duplicating what a custom cainfo
bundle does)
- --daemon-ssl-allow-chained is removed; it wasn't a useful option (if
you don't want chaining, don't specify a cainfo chain).
- --daemon-address is now a URL instead of just host:port. (If you omit
the protocol, http:// is prepended).
- --daemon-host and --daemon-port are now deprecated and produce a
warning (in simplewallet) if used; the replacement is to use
--daemon-address.
- --daemon-ssl is deprecated; specify --daemon-address=https://whatever
instead.
- the above three are now hidden from --help
- reordered the wallet connection options to make more logical sense.
This replaces the NIH epee http server which does not work all that well
with an external C++ library called uWebSockets. Fundamentally this
gives the following advantages:
- Much less code to maintain
- Just one thread for handling HTTP connections versus epee's pool of
threads
- Uses existing LokiMQ job server and existing thread pool for handling
the actual tasks; they are processed/scheduled in the same "rpc" or
"admin" queues as lokimq rpc calls. One notable benefit is that "admin"
rpc commands get their own queue (and thus cannot be delayed by long rpc
commands). Currently the lokimq threads and the http rpc thread pool
and the p2p thread pool and the job queue thread pool and the dns lookup
thread pool and... are *all* different thread pools; this is a step
towards consolidating them.
- Very little mutex contention (which has been a major problem with epee
RPC in the past): there is one mutex (inside uWebSockets) for putting
responses back into the thread managing the connection; everything
internally gets handled through (lock-free) lokimq inproc sockets.
- Faster RPC performance on average, and much better worst case
performance. Epee's http interface seems to have some race condition
that ocassionally stalls a request (even a very simple one) for a dozen
or more seconds for no good reason.
- Long polling gets redone here to no longer need threads; instead we
just store the request and respond when the thread pool, or else in a
timer (that runs once/second) for timing out long polls.
---
The basic idea of how this works from a high level:
We launch a single thread to handle HTTP RPC requests and response data.
This uWebSockets thread is essentially running an event loop: it never
actually handles any logic; it only serves to shuttle data that arrives
in a request to some other thread, and then, at some later point, to
send some reply back to that waiting connection. Everything is
asynchronous and non-blocking here: the basic uWebSockets event loop
just operates as things arrive, passes it off immediately, and goes back
to waiting for the next thing to arrive.
The basic flow is like this:
0. uWS thread -- listens on localhost:22023
1. uWS thread -- incoming request on localhost:22023
2. uWS thread -- fires callback, which injects the task into the LokiMQ job queue
3. LMQ main loop -- schedules it as an RPC job
4. LMQ rpc thread -- Some LokiMQ thread runs it, gets the result
5. LMQ rpc thread -- Result gets queued up for the uWS thread
6. uWS thread -- takes the request and starts sending it
(asynchronously) back to the requestor.
In more detail:
uWebSockets has registered has registered handlers for non-jsonrpc
requests (legacy JSON or binary). If the port is restricted then admin
commands get mapped to a "Access denied" response handler, otherwise
public commands (and admin commands on an unrestricted port) go to the
rpc command handler.
POST requests to /json_rpc have their own handler; this is a little
different than the above because it has to parse the request before it
can determine whether it is allowed or not, but once this is done it
continues roughly the same as legacy/binary requests.
uWebSockets then listens on the given IP/port for new incoming requests,
and starts listening for requests in a thread (we own this thread).
When a request arrives, it fires the event handler for that request.
(This may happen multiple times, if the client is sending a bunch of
data in a POST request). Once we have the full request, we then queue
the job in LokiMQ, putting it in the "rpc" or "admin" command
categories. (The one practical different here is that "admin" is
configured to be allowed to start up its own thread if all other threads
are busy, while "rpc" commands are prioritized along with everything
else.) LokiMQ then schedules this, along with native LokiMQ "rpc." or
"admin." requests.
When a LMQ worker thread becomes available, the RPC command gets called
in it and runs. Whatever output it produces (or error message, if it
throws) then gets wrapped up in jsonrpc boilerplate (if necessary), and
delivered to the uWebSockets thread to be sent in reply to that request.
uWebSockets picks up the data and sends whatever it can without
blocking, then buffers whatever it couldn't send to be sent again in a
later event loop iteration once the requestor can accept more data.
(This part is outside lokid; we only have to give uWS the data and let
it worry about delivery).
---
PR specifics:
Things removed from this PR:
1. ssl settings; with this PR the HTTP RPC interface is plain-text. The
previous default generated a self-signed certificate for the server on
startup and then the client accepted any certificate. This is actually
*worse* than unencrypted because it is entirely MITM-readable and yet
might make people think that their RPC communication is encrypted, and
setting up actual certificates is difficult enough that I think most
people don't bother.
uWebSockets *does* support HTTPS, and we could glue the existing options
into it, but I'm not convinced it's worthwhile: it works much better to
put HTTPS in a front-end proxy holding the certificate that proxies
requests to the backend (which can then listen in restricted mode on
some localhost port). One reason this is better is that it is much
easier to reload and/or restart such a front-end server, while
certificate updates with lokid require a full restart. Another reason
is that you get an error page instead of a timeout if something is wrong
with the backend. Finally we also save having to generate a temporary
certificate on *every* lokid invocation.
2. HTTP Digest authentication. Digest authentication is obsolete (and
was already obsolete when it got added to Monero). HTTP-Digest was
originally an attempt to provide a password authentication mechanism
that does not leak the password in transit, but still required that the
server know the password. It only has marginal value against replay
attacks, and is made entirely obsolete by sending traffic over HTTPS
instead. No client out there supports Digest but *not* Basic auth, and
so given the limited usefulness it seems pointless to support more than
Basic auth for HTTP RPC login.
What's worse is that epee's HTTP Digest authentication is a terrible
implementation: it uses boost::spirit -- a recursive descent parser
meant for building complex language grammars -- just to parse a single
HTTP header for Digest auth. This is a big load of crap that should
never have been accepted upstream, and that we should get rid of (even
if we wanted to support Digest auth it takes less than 100 lines of code
to do it when *not* using a recursive descent parser).
The interactive mode handler for `prepare_registration` didn't properly
detect FAKECHAIN mode and so would show mainnet values in the prompts;
this fixes it to notice FAKECHAIN mode properly.
I did this by just checking the `.nettype` string rather than adding a
new field since fakechain is a dev tool that didn't seem to warrant a
new field in the RPC struct.
This removes some boost thread interruption code, but that code is
highly unlikely to have done anything: you cannot preemptively interrupt
a thread, and pretty much anywhere a thread would get stuck is in I/O,
where boost's (voluntary) thread interruption cannot follow.
It also removes thread attributes because they have really limited
usefulness. There *was* a reason to introduce them many years ago (when
musl libc ran into a stack limit problem with libunbound) but musl
increased the default stack size long since.
common/util.h has become something of a dumping ground of random
functions. This splits them up a little by moving the filesystem bits
to common/file.h, the sha256sum functions to common/sha256sum.h, and the
(singleton) signal handler to common/signal_handler.h.
This moves all the conditional HAVE_READLINE into once place rather than
scattering it everywhere we want to suspend readline. (Since the class
does nothing the compiler can trivially optimize it away when we don't
have readline).
Link readline directly into epee; having a separate epee_readline
library is not saving anything since we have it widely linked anyway.
Conditionally linking it to epee simplifies a bit of CMake code.
Also simplify how epee detects cmake to just look for a `readline`
target, which we now only set up if we find readline in the top-level
CMakeLists.txt
- In simplewallet, we don't report invalid/missing commands to the
CLI because we throw and catch errors that are logged silently. Now we
always log the to the user unless it's thrown by something other than
the wallet.
- Add invalid_command exception to distinguish between an error thrown
by the console_handler (i.e. missing/empty command) versus an actual
std::out_of_range thrown by wallet code (previously would have
incorrectly reported "Unknown command ... ", when it was a known command).
- Catch exceptions thrown via commands sent by RPC over the terminal and
shut down the wallet properly.
- In console_handler don't react to cmd_handler failing. Previously, if
a command failed it'd assume invalid command and log or try the next
branch which was detecting application exit.
High-level details:
This redesigns the RPC layer to make it much easier to work with,
decouples it from an embedded HTTP server, and gets the vast majority of
the RPC serialization and dispatch code out of a very commonly included
header.
There is unfortunately rather a lot of interconnected code here that
cannot be easily separated out into separate commits. The full details
of what happens here are as follows:
Major details:
- All of the RPC code is now in a `cryptonote::rpc` namespace; this
renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK
becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME`
becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already
working in the `rpc` namespace).
- `core_rpc_server` is now completely decoupled from providing any
request protocol: it is now *just* the core RPC call handler.
- The HTTP RPC interface now lives in a new rpc/http_server.h; this code
handles listening for HTTP requests and dispatching them to
core_rpc_server, then sending the results back to the caller.
- There is similarly a rpc/lmq_server.h for LMQ RPC code; more details
on this (and other LMQ specifics) below.
- RPC implementing code now returns the response object and throws when
things go wrong which simplifies much of the rpc error handling. They
can throw anything; generic exceptions get logged and a generic
"internal error" message gets returned to the caller, but there is
also an `rpc_error` class to return an error code and message used by
some json-rpc commands.
- RPC implementing functions now overload `core_rpc_server::invoke`
following the pattern:
RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context);
This overloading makes the code vastly simpler: all instantiations are
now done with a small amount of generic instantiation code in a single
.cpp rather than needing to go to hell and back with a nest of epee
macros in a core header.
- each RPC endpoint is now defined by the RPC types themselves,
including its accessible names and permissions, in
core_rpc_server_commands_defs.h:
- every RPC structure now has a static `names()` function that returns
the names by which the end point is accessible. (The first one is
the primary, the others are for deprecated aliases).
- RPC command wrappers define their permissions and type by inheriting
from special tag classes:
- rpc::RPC_COMMAND is a basic, admin-only, JSON command, available
via JSON RPC. *All* JSON commands are now available via JSON RPC,
instead of the previous mix of some being at /foo and others at
/json_rpc. (Ones that were previously at /foo are still there for
backwards compatibility; see `rpc::LEGACY` below).
- rpc::PUBLIC specifies that the command should be available via a
restricted RPC connection.
- rpc::BINARY specifies that the command is not JSON, but rather is
accessible as /name and takes and returns values in the magic epee
binary "portable storage" (lol) data format.
- rpc::LEGACY specifies that the command should be available via the
non-json-rpc interface at `/name` for backwards compatibility (in
addition to the JSON-RPC interface).
- some epee serialization got unwrapped and de-templatized so that it
can be moved into a .cpp file with just declarations in the .h. (This
makes a *huge* difference for core_rpc_server_commands_defs.h and for
every compilation unit that includes it which previously had to
compile all the serialization code and then throw all by one copy away
at link time). This required some new macros so as to not break a ton
of places that will use the old way putting everything in the headers;
The RPC code uses this as does a few other places; there are comments
in contrib/epee/include/serialization/keyvalue_serialization.h as to
how to use it.
- Detemplatized a bunch of epee/storages code. Most of it should have
have been using templates at all (because it can only ever be called
with one type!), and now it isn't. This broke some things that didn't
properly compile because of missing headers or (in one case) a messed
up circular dependency.
- Significantly simplified a bunch of over-templatized serialization
code.
- All RPC serialization definitions is now out of
core_rpc_server_commands_defs.h and into a single .cpp file
(core_rpc_server_commands_defs.cpp).
- core RPC no longer uses the disgusting
BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design
that forced slamming tons of code into a common header that didn't
need to be there.
- epee::struct_init is gone. It was a horrible hack that instiated
multiple templates just so the coder could be so lazy and write
`some_type var;` instead of properly value initializing with
`some_type var{};`.
- Removed a bunch of useless crap from epee. In particular, forcing
extra template instantiations all over the place in order to nest
return objects inside JSON RPC values is no longer needed, as are a
bunch of stuff related to the above de-macroization of the code.
- get_all_service_nodes, get_service_nodes, and get_n_service_nodes are
now combined into a single `get_service_nodes` (with deprecated
aliases for the others), which eliminates a fair amount of
duplication. The biggest obstacle here was getting the requested
fields reference passed through: this is now done by a new ability to
stash a context in the serialization object that can be retrieved by a
sub-serialized type.
LMQ-specifics:
- The LokiMQ instance moves into `cryptonote::core` rather than being
inside cryptonote_protocol. Currently the instance is used both for
qnet and rpc calls (and so needs to be in a common place), but I also
intend future PRs to use the batching code for job processing
(replacing the current threaded job queue).
- rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue.
Unlike http_server it isn't technically running the whole LMQ stack
from here, but the parallel name with http_server seemed appropriate.
- All RPC endpoints are supported by LMQ under the same names as defined
generically, but prefixed with `rpc.` for public commands and `admin.`
for restricted ones.
- service node keys are now always available, even when not running in
`--service-node` mode: this is because we want the x25519 key for
being able to offer CURVE encryption for lmq RPC end-points, and
because it doesn't hurt to have them available all the time. In the
RPC layer this is now called "get_service_keys" (with
"get_service_node_key" as an alias) since they aren't strictly only
for service nodes. This also means code needs to check
m_service_node, and not m_service_node_keys, to tell if it is running
as a service node. (This is also easier to notice because
m_service_node_keys got renamed to `m_service_keys`).
- Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and
`sub.mempool` subscribes the connection for new block and new mempool
TX notifications. The latter can notify on just blink txes, or all
new mempool txes (but only new ones -- txes dumped from a block don't
trigger it). The client gets pushed a [`notify.block`, `height`,
`hash`] or [`notify.tx`, `txhash`, `blob`] message when something
arrives.
Minor details:
- rpc::version_t is now a {major,minor} pair. Forcing everyone to pack
and unpack a uint32_t was gross.
- Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...).
(This immediately revealed a couple of bugs in the RPC code that was
assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because
the macro allows implicit conversion to a char).
- De-templatizing useless templates in epee (i.e. a bunch of templated
types that were never invoked with different types) revealed a painful
circular dependency between epee and non-epee code for tor_address and
i2p_address. This crap is now handled in a suitably named
`net/epee_network_address_hack.cpp` hack because it really isn't
trivial to extricate this mess.
- Removed `epee/include/serialization/serialize_base.h`. Amazingly the
code somehow still all works perfectly with this previously vital
header removed.
- Removed bitrotted, unused epee "crypted_storage" and
"gzipped_inmemstorage" code.
- Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with
LOKI_DEFERs. The epee version involves quite a bit more instantiation
and is ugly as sin. Also made the `loki::defer` class invokable for
some edge cases that need calling before destruction in particular
conditions.
- Moved the systemd code around; it makes much more sense to do the
systemd started notification as in daemon.cpp as late as possible
rather than in core (when we can still have startup failures, e.g. if
the RPC layer can't start).
- Made the systemd short status string available in the get_info RPC
(and no longer require building with systemd).
- during startup, print (only) the x25519 when not in SN mode, and
continue to print all three when in SN mode.
- DRYed out some RPC implementation code (such as set_limit)
- Made wallet_rpc stop using a raw m_wallet pointer
Removes all "using namespace epee;" and "using namespace std;" from the
code and fixes up the various crappy places where unnamespaced types
were being used.
Also removes the ENDL macro (which was defined to be `std::endl`)
because it is retarded, and because even using std::endl instead of a
plain "\n" is usually a mistake (`<< std::endl` is equivalent to `<<
"\n" << std::flush`, and that explicit flush is rarely desirable).
This commit continues the complete replacement of the spaghetti code
mess that was inside daemon/ and daemonize/ which started in #1138, and
looked like a entry level Java programmer threw up inside the code base.
This greatly simplifies it, removing a whole pile of useless abstraction
layers that don't actually abstract anything, and results in
considerably simpler code. (Many of the changes here were also carried
out in #1138; this commit updates them with the merged result which
amends some things from that PR and goes further in some places).
In detail:
- the `--detach` (and related `--pidfile`) options are gone. (--detach
is still handled, but now just prints a fatal error). Detaching a
process is an archaic unix mechanism that has no place on a modern
system. If you *really* want to do it anyway, `nohup lokid &` will do
the job. (The Windows service control code, which is probably seldom
used, is kept because it seems potentially useful for Windows users).
- Many of the `t_whatever` classes in daemon/* are just deleted (mostly
done in #1138); each one was a bunch of junk code that wraps 3-4 lines
but forces an extra layer (not even a generic abstraction, just a
useless wrapper) for no good reason and made the daemon code painfully
hard to understand and work with.
- All of the remaining `t_whatever` classes in daemon/* are either
renamed to `whatever` (because prefixing every class with `t_` is
moronic).
- Some stupid related code (e.g. epee's command handler returning an
unsuitable "usage" string that has to be string modified into what we
want) was replaced with more generic, useful code.
- Replaced boost mutexes/cvs with std ones in epee command handler, and
deleted some commented out code.
- The `--public-node` option handling was terrible: it was being handled
in main, but main doesn't know anything about options, so then it
forced it through the spaghetti objects *beside* the pack of all
options that got passed along. Moved it to a more sane location
(core_rpc_server) and parse it out with some sanity.
- Changed a bunch of std::bind's to lambdas because, at least for small
lambdas (i.e. with only one-or-two pointers for captures) they will
generally be more efficient as the values can be stored in
std::function's without any memory allocations.
This uses templates to simplify the is_yes (etc.) functions, also
removing a boost dependency and making them more flexible where callers
sometimes want "is_yes or one of these other truthy strings".
This was a useless feature to begin with. According to a Monero
insider, this was introduced at the time with an intention of making it
on-by-default on every monerod instance everywhere, but because that was
such an overwhelmingly stupid idea, it never happened yet all this code
(which is probably used by no one anywhere ever) remains in the code
base.
Even if the idea wasn't dumb to start with, this will also become even
more pointless with pulse, so just drop it (it is over 1000 lines of
code, not even counting the extra headers pulled in to do things like
querying CPU usage and battery status).
- updating to latest loki-mq (1.0.0 + various linking fixes)
- BUILD_SHARED_LIBS was being handled very strangely; make it a full
option instead (defaulting to off) that a cmake invoker can specify, as
per cmake recommendations.
- travis ci tweaks/changes:
- Add a static bionic build
- Simplify cmake argument code
- Add `--version` invocation for lokid and loki-wallet-cli to test
that the binaries were linked properly.
- always build an embedded sodium statically; if we do it dynamically
and an older system one exists we are going to have trouble.
- don't force epee and blocks to be static; rather they get controlled
by the above BUILD_SHARED_LIBS, just like all the other internal
libraries.
- use some PkgConfig:: imported targets rather than bunch-of-variables.
The archaic (i.e. decade old) cmake usage here really got in the way of
trying to properly use newer libraries (like lokimq), so this undertakes
overhauling it considerably to make it much more sane (and significantly
reduce the size).
I left more of the architecture-specific bits in the top-level
CMakeLists.txt intact; most of the efforts here are about properly
loading dependencies, specifying dependencies and avoiding a whole pile
of cmake antipatterns.
This bumps the required cmake version to 3.5, which is what xenial comes
with.
- extensive use of interface libraries to include libraries,
definitions, and include paths
- use Boost::whatever instead of ${Boost_WHATEVER_LIBRARY}. The
interface targets are (again) much better as they also give you any
needed include or linking flags without needing to worry about them.
- don't list header files when building things. This has *never* been
correct cmake usage (cmake has always known how to wallet_rpc_headers
the headers that .cpp files include to know about build changes).
- remove the loki_add_library monstrosity; it breaks target names and
makes compiling less efficient because the author couldn't figure out
how to link things together.
- make loki_add_executable take the output filename, and set the output
path to bin/ and install to bin because *every single usage* of
loki_add_executable was immediately followed by setting the output
filename and setting the output path to bin/ and installing to bin.
- move a bunch of crap that is only used in one particular
src/whatever/CMakeLists.txt into that particular CMakeLists.txt instead
of the top level CMakeLists.txt (or src/CMakeLists.txt).
- Remove a bunch of redundant dependencies; most of them look like they
were just copy-and-pasted in, and many more aren't needed (since they
are implied by the PUBLIC linking of other dependencies).
- Removed `die` since it just does a FATAL_ERROR, but adds color (which
is useless since CMake already makes FATAL_ERRORs perfectly visible).
- Change the way LOKI_DAEMON_AND_WALLET_ONLY works to just change the
make targets to daemon and simplewallet rather than changing the build
process (this should make it faster, too, since there are various other
things that will be excluded).
Checkpoint votes internally use a circular buffer, but that's rather
difficult to read for the `print_sn` output; this changes print_sn to
de-circularize the listed votes.
This adds the last lokinet ping time to the second `status` line. It
also compresses this (and storage server's) ping time format to (for
example) `4.7min` instead of `4.7 minutes ago`.
I also add a few tweaks here to the first line of the status message to
save some space for the most common cases:
- just don't show " on mainnet" at all since that's the common case, and
instead capitalize "ON TESTNET"/"ON STAGENET" to make it more obvious
that something is non-default.
- just don't show ", not mining" when not mining
- show the daemon version returned by RPC in addition to the hf version,
as "vX.Y.Z(net vAA)". You usually get a version display from an info
log message when running `lokid version`, but that can be quite
misleading as it's the local lokid binary version which is not
necessarily the same version as the running daemon.
Sample output:
SN:
Height: 174791/174791 (100.0%) ON TESTNET, net hash 455 H/s, v6.0.0(net v14), up to date, 8(out)+12(in) connections, uptime 0d 0h 4m 32s
SN: ff00062fa69397f5580ac2b3d060064d89edac625c77a58fdcd520f94ea54727 active, proof: (never), last pings: 1.6min (storage), 1.6min (lokinet)
Regular node (while mining):
Height: 174791/174791 (100.0%) ON TESTNET, mining at 94 H/s, net hash 455 H/s, v6.0.0(net v14), up to date, 8(out)+32(in) connections, uptime 0d 0h 44m 2s
Fixes:
print_sn SOMEPUBKEY +detail
to suppress the `+detail` (or `+json`) from the error message:
No service node is currently known on the network: 7984cd23f950a52a13199f1ed22c3a396d6eea330b375bd94f6323d8db196953, +detail,
Also fixes a bug in the comma separator logic (see the unwanted
separator above).
- Reformat the Registration details line; I found the formatting really
confusing. Changes from:
Registration Hardfork Version | Register/Expiry Height: 14 | 174767 / Staking Infinitely (stake unlock not requested yet)
to:
Registration: Hardfork Version: 14; Height: 174770; Expiry: Staking Infinitely (stake unlock not requested)
(perhaps not obvious - I also dropped "yet" from the infinite expiry
message to slightly shorten the line as it seemed superfluous).
- add `(Or Penalty)` into last reward value since this value gets reset
without a reward on IP change penalty and recommissions:
Last Reward (Or Penalty) At (Height/TX Index): 177529/4294967295
- add Ed25519 and X25519 pubkeys to +detail mode:
Auxiliary Public Keys:
729916aa41aaa3f2e729e2f9a5516207c39da30debe8e6d0ae7711a3dcfb0d9d (Ed25519)
7984cd23f950a52a13199f1ed22c3a396d6eea330b375bd94f6323d8db196953 (X25519)
The existing code just assumes it is already in the path, which fails
when it's in some prefixed place with a proper .pc file installed (such
as for a homebrew installation).
print_sn was only sorting by last block reward and last tx index, but it
needs to also fallback to sorting by pubkey if both are equal (which
can happen with recomms and ip penalties).
(This is cosmetic only for the `print_sn` command).
- Adds blink signature synchronization and storage through the regular
p2p network
- Adds wallet support (though this is still currently buggy and needs
additional fixes - it sees the tx when it arrives in the mempool but
isn't properly updating when the blink tx gets mined.)
The only difference between long and short is one string (the json
info).
Also remove std::endl because using it like this indicates that the
author doesn't understand what std::endl is for.
Also removed a completely ridiculous inline lambda.
This is the bulk of the work for blink. There is two pieces yet to come
which will follow shortly, which are: the p2p communication of blink
transactions (which needs to be fully synchronized, not just shared,
unlike regular mempool txes); and an implementation of fee burning.
Blink approval, multi-quorum signing, cli wallet and node support for
submission denial are all implemented here.
This overhauls and fixes various parts of the SNNetwork interface to fix
some issues (particularly around non-SN communication with SNs, which
wasn't working).
There are also a few sundry FIXME's and TODO's of other minor details
that will follow shortly under cleanup/testing/etc.
This adds vote relaying via quorumnet.
- The main glue between existing code and quorumnet code is in
cryptonote_protocol/quorumnet.{h,cpp}
- Uses function pointers assigned at initialization to call into the
glue without making cn_core depend on p2p
- Adds ed25519 and quorumnet port to uptime proofs and serialization.
- Added a second uptime proof signature using the ed25519 key to that
SNs are proving they actually have it (otherwise they could spoof
someone else's pubkey).
- Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet
listens on the p2p IPv4 address (but broadcasts the `public_ip` to the
network).
- Derives x25519 when received or deserialized.
- Converted service_info_info::version_t into a scoped enum (with the
same underlying type).
- Added contribution_t::version_t scoped enum. It was using
service_info_info::version for a 0 value which seemed rather wrong.
Random small details:
- Renamed internal `arg_sn_bind_port` for consistency
- Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet
quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port
would collide with the testnet p2p port).
This allows flushing internal caches (for now, the bad tx cache,
which will allow debugging a stuck monerod after it has failed to
verify a transaction in a block, since it would otherwise not try
again, making subsequent log changes pointless)
b3a9a4d add a quick early out to get_blocks.bin when up to date (moneromooo-monero)
2899379 daemon, wallet: new pay for RPC use system (moneromooo-monero)
ffa4602 simplewallet: add public_nodes command (moneromooo-monero)
Daemons intended for public use can be set up to require payment
in the form of hashes in exchange for RPC service. This enables
public daemons to receive payment for their work over a large
number of calls. This system behaves similarly to a pool, so
payment takes the form of valid blocks every so often, yielding
a large one off payment, rather than constant micropayments.
This system can also be used by third parties as a "paywall"
layer, where users of a service can pay for use by mining Monero
to the service provider's address. An example of this for web
site access is Primo, a Monero mining based website "paywall":
https://github.com/selene-kovri/primo
This has some advantages:
- incentive to run a node providing RPC services, thereby promoting the availability of third party nodes for those who can't run their own
- incentive to run your own node instead of using a third party's, thereby promoting decentralization
- decentralized: payment is done between a client and server, with no third party needed
- private: since the system is "pay as you go", you don't need to identify yourself to claim a long lived balance
- no payment occurs on the blockchain, so there is no extra transactional load
- one may mine with a beefy server, and use those credits from a phone, by reusing the client ID (at the cost of some privacy)
- no barrier to entry: anyone may run a RPC node, and your expected revenue depends on how much work you do
- Sybil resistant: if you run 1000 idle RPC nodes, you don't magically get more revenue
- no large credit balance maintained on servers, so they have no incentive to exit scam
- you can use any/many node(s), since there's little cost in switching servers
- market based prices: competition between servers to lower costs
- incentive for a distributed third party node system: if some public nodes are overused/slow, traffic can move to others
- increases network security
- helps counteract mining pools' share of the network hash rate
- zero incentive for a payer to "double spend" since a reorg does not give any money back to the miner
And some disadvantages:
- low power clients will have difficulty mining (but one can optionally mine in advance and/or with a faster machine)
- payment is "random", so a server might go a long time without a block before getting one
- a public node's overall expected payment may be small
Public nodes are expected to compete to find a suitable level for
cost of service.
The daemon can be set up this way to require payment for RPC services:
monerod --rpc-payment-address 4xxxxxx \
--rpc-payment-credits 250 --rpc-payment-difficulty 1000
These values are an example only.
The --rpc-payment-difficulty switch selects how hard each "share" should
be, similar to a mining pool. The higher the difficulty, the fewer
shares a client will find.
The --rpc-payment-credits switch selects how many credits are awarded
for each share a client finds.
Considering both options, clients will be awarded credits/difficulty
credits for every hash they calculate. For example, in the command line
above, 0.25 credits per hash. A client mining at 100 H/s will therefore
get an average of 25 credits per second.
For reference, in the current implementation, a credit is enough to
sync 20 blocks, so a 100 H/s client that's just starting to use Monero
and uses this daemon will be able to sync 500 blocks per second.
The wallet can be set to automatically mine if connected to a daemon
which requires payment for RPC usage. It will try to keep a balance
of 50000 credits, stopping mining when it's at this level, and starting
again as credits are spent. With the example above, a new client will
mine this much credits in about half an hour, and this target is enough
to sync 500000 blocks (currently about a third of the monero blockchain).
There are three new settings in the wallet:
- credits-target: this is the amount of credits a wallet will try to
reach before stopping mining. The default of 0 means 50000 credits.
- auto-mine-for-rpc-payment-threshold: this controls the minimum
credit rate which the wallet considers worth mining for. If the
daemon credits less than this ratio, the wallet will consider mining
to be not worth it. In the example above, the rate is 0.25
- persistent-rpc-client-id: if set, this allows the wallet to reuse
a client id across runs. This means a public node can tell a wallet
that's connecting is the same as one that connected previously, but
allows a wallet to keep their credit balance from one run to the
other. Since the wallet only mines to keep a small credit balance,
this is not normally worth doing. However, someone may want to mine
on a fast server, and use that credit balance on a low power device
such as a phone. If left unset, a new client ID is generated at
each wallet start, for privacy reasons.
To mine and use a credit balance on two different devices, you can
use the --rpc-client-secret-key switch. A wallet's client secret key
can be found using the new rpc_payments command in the wallet.
Note: anyone knowing your RPC client secret key is able to use your
credit balance.
The wallet has a few new commands too:
- start_mining_for_rpc: start mining to acquire more credits,
regardless of the auto mining settings
- stop_mining_for_rpc: stop mining to acquire more credits
- rpc_payments: display information about current credits with
the currently selected daemon
The node has an extra command:
- rpc_payments: display information about clients and their
balances
The node will forget about any balance for clients which have
been inactive for 6 months. Balances carry over on node restart.
* Fix elapsed time in storage server warning message
This was passing the time value rather than the number of seconds so
basically always printed "a long time" instead of the elapsed time.
* Removed pre-HF12 code
* Wait for storage server before sending proof
On startup we send a proof immediately, but this is misleading to the
operator - they may see a proof broadcast over the network suggesting
everything is good even when the storage server is still down. This
makes lokid wait for a first ping before the first proof so that the
user doesn't get mislead.
It also adds the storage server last ping into the `lokid status`
message, such as:
Height: 375634/375634 (100.0%) on mainnet, not mining, net hash 63.00 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 4s
SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: NO PING RECEIVED
or:
Height: 375663/375663 (100.0%) on mainnet, not mining, net hash 76.17 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 32s
SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: last ping 11 seconds ago
You don't want to report your node as failing locally if you can't be
voted on in that quorum (i.e. you re-registered quickly after
deregistration and still exist in the old quorums)
This adds the pubkey, current status, and last uptime proof on a second
line of output in the `status` command (if a service node and we can
query the SN info successfully).
When processing a quorum for a block, if you are not in the quorum to
validate other nodes, check if you're a node that is going to be tested.
If you are, check based on your current data if you're potentially
a candidate to be decommissioned/deregistered and if so report it to the
console log.
Note that this is only a heuristic and ultimately the decision lies on
what the the other Service Nodes perceive the current state of your node
is (i.e. if they're acting malicious then you will be deregistered
irrespectively).
This switches loki 5.x to use a fee formula of
SIZE * PER_BYTE + OUTPUTS * PER_OUTPUT
where we reduce the PER_BYTE fee back to what it was in 3.x; and with
the PER_OUTPUT fee set to 0.02 LOKI. This compares to the 4.x fee of:
SIZE * PER_BYTE * 80
(the *80 multiple was introduced in 4.x).
It also reduces the multiplier for the maximum priority level to 125
instead of 1000 because 1000 produced uselessly high tx fees. The new
multipliers go up 5x at each level: {1, 5, 25, 125} while previously
they went {1, 5, 25, 1000}.
As for the base change: we added the *80 multiplier in 4.x because we
wanted to make a theoretical de-anonymizing tx spam attack more costly.
The unanticipated consequence was that we also made *large* transactions
(such as sweeps) considerably more costly despite the fact that these
transactions typically only create 2 outputs.
This better captures what we meant to do in 4.x (making output creation
relatively more expensive) without making large txes (e.g. sweeps
required for staking) highly expensive.
The end effect is that the fee for a minimum-sized, 1-input/2-output
transaction should stay roughly the same (slightly over 0.04 LOKI),
while a 100-input/2-output transction (a typical spend or sweep from a
wallet with lots of smaller rewards) will drop in fee by somewhere
around 95%.
The most efficient theoretical deanonymizing tx spamming of this sort
was a 1-input/16-output transaction which will become about 2.5x as
expensive as currently the case in v4.x.
new cli options (RPC ones also apply to wallet):
--p2p-bind-ipv6-address (default = "::")
--p2p-bind-port-ipv6 (default same as ipv4 port for given nettype)
--rpc-bind-ipv6-address (default = "::1")
--p2p-use-ipv6 (default false)
--rpc-use-ipv6 (default false)
--p2p-require-ipv4 (default true, if ipv4 bind fails and this is
true, will not continue even if ipv6 bind
successful)
--rpc-require-ipv4 (default true, description as above)
ipv6 addresses are to be specified as "[xx:xx:xx::xx:xx]:port" except
in the cases of the cli args for bind address. For those the square
braces can be omitted.
0605406 daemon: sort alt chains by height (moneromooo-monero)
4228ee0 daemon: add optional arguments to alt_chain_info (moneromooo-monero)
880ebfd daemon: add more chain specific info in alt_chain_info (moneromooo-monero)
new cli options (RPC ones also apply to wallet):
--p2p-bind-ipv6-address (default = "::")
--p2p-bind-port-ipv6 (default same as ipv4 port for given nettype)
--rpc-bind-ipv6-address (default = "::1")
--p2p-use-ipv6 (default false)
--rpc-use-ipv6 (default false)
--p2p-require-ipv4 (default true, if ipv4 bind fails and this is
true, will not continue even if ipv6 bind
successful)
--rpc-require-ipv4 (default true, description as above)
ipv6 addresses are to be specified as "[xx:xx:xx::xx:xx]:port" except
in the cases of the cli args for bind address. For those the square
braces can be omitted.
The doc generator doesn't know how to parse using statements, so upgrade
contributor/contributions to global scope in the RPC definitions file so
it's able to parse it for both service node RPC calls.
The best option will be to eventually deprecate the various ways to
query the network down to 1 call that doesn't duplicate functionality.
KV_SERIALIZE doesn't handle non-primitive types well and requires adding
in extra deserialize/reserialize code. It's more straightforward to
manually serialize the output, rather than add serialization overloads.
Furthermore, KV_SERIALIZE doesn't support serializing enums in a sane
way.
* print_quorum_state displays checkpointing quorums, remove batched call
* Update the help text for quorum commands
* <= for the max_quorum_type not <
* Handle heights greater than the latest
* Don't repeatedly add partially filled quorums
* Revert get_quorum_state to take a range (common case)
* Update the help text from print_quorum_state
* Improved print_checkpoints
* Flesh out print checkpoints and associated RPC call
* Remove debug print checkpoints
* Update help text for print_checkpoints
* Rewrite to fix num_checkpoints != heights as a unit of measurement
* Use GET_ALL_CHECKPOINTS defined value in get_checkpoints_range
* Let T be deduced in parse_if_present, json_rpc_request not rpc_request
* Add deregistration of checkpoints by checking how many votes are missed
Move uptime proofs and add checkpoint counts in the service_node_list
because we typically prune uptime proofs by time, but it seems we want
to switch to a model where we persist proof data until the node expires
otherwise currently we would prune uptime entries and potentially our
checkpoint vote counts which would cause premature deregistration as the
expected vote counts start mismatching with the number of received
votes.
* Revise deregistration
* Fix test breakages
* uint16_t for port, remove debug false, min votes to 2 in integration mode
* Fix integration build
* core: do not commit half constructed batch db txn
* Add defer macro
* Revert dumb extra copy/move change
* Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes
* Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes
* Update tests to compile with relaxed-registration changes
* Get back to feature parity pre-relaxed registration changes
* Remove debug changes noticed in code review and some small bugs
The replaces the deregistration mechanism with a new state change
mechanism (beginning at the v12 fork) which can change a service node's
network status via three potential values (and is extensible in the
future to handle more):
- deregistered -- this is the same as the existing deregistration; the
SN is instantly removed from the SN list.
- decommissioned -- this is a sort of temporary deregistration: your SN
remains in the service node list, but is removed from the rewards list
and from any network duties.
- recommissioned -- this tx is sent by a quorum if they observe a
decommissioned SN sending uptime proofs again. Upon reception, the SN
is reactivated and put on the end of the reward list.
Since this is broadening the quorum use, this also renames the relevant
quorum to a "obligations" quorum (since it validates SN obligations),
while the transactions are "state_change" transactions (since they
change the state of a registered SN).
The new parameters added to service_node_rules.h control how this works:
// Service node decommissioning: as service nodes stay up they earn "credits" (measured in blocks)
// towards a future outage. A new service node starts out with INITIAL_CREDIT, and then builds up
// CREDIT_PER_DAY for each day the service node remains active up to a maximum of
// DECOMMISSION_MAX_CREDIT.
//
// If a service node stops sending uptime proofs, a quorum will consider whether the service node
// has built up enough credits (at least MINIMUM): if so, instead of submitting a deregistration,
// it instead submits a decommission. This removes the service node from the list of active
// service nodes both for rewards and for any active network duties. If the service node comes
// back online (i.e. starts sending the required performance proofs again) before the credits run
// out then a quorum will reinstate the service node using a recommission transaction, which adds
// the service node back to the bottom of the service node reward list, and resets its accumulated
// credits to 0. If it does not come back online within the required number of blocks (i.e. the
// accumulated credit at the point of decommissioning) then a quorum will send a permanent
// deregistration transaction to the network, starting a 30-day deregistration count down.
This commit currently includes values (which are not necessarily
finalized):
- 8 hours (240 blocks) of credit required for activation of a
decommission (rather than a deregister)
- 0 initial credits at registration
- a maximum of 24 hours (720 blocks) of credits
- credits accumulate at a rate that you hit 24 hours of credits after 30
days of operation.
Miscellaneous other details of this PR:
- a new TX extra tag is used for the state change (including
deregistrations). The old extra tag has no version or type tag, so
couldn't be reused. The data in the new tag is slightly more
efficiently packed than the old deregistration transaction, so it gets
used for deregistrations (starting at the v12 fork) as well.
- Correct validator/worker selection required generalizing the shuffle
function to be able to shuffle just part of a vector. This lets us
stick any down service nodes at the end of the potential list, then
select validators by only shuffling the part of the index vector that
contains active service indices. Once the validators are selected, the
remainder of the list (this time including decommissioned SN indices) is
shuffled to select quorum workers to check, thus allowing decommisioned
nodes to be randomly included in the nodes to check without being
selected as a validator.
- Swarm recalculation was not quite right: swarms were recalculated on
SN registrations, even if those registrations were include shared node
registrations, but *not* recalculated on stakes. Starting with the
upgrade this behaviour is fixed (swarms aren't actually used currently
and aren't consensus-relevant so recalculating early won't hurt
anything).
- Details on decomm/dereg are added to RPC info and print_sn/print_sn_status
- Slightly improves the % of reward output in the print_sn output by
rounding it to two digits, and reserves space in the output string to
avoid excessive reallocations.
- Adds various debugging at higher debug levels to quorum voting (into
all of voting itself, vote transmission, and vote reception).
- Reset service node list internal data structure version to 0. The SN
list has to be rescanned anyway at upgrade (its size has changed), so we
might as well reset the version and remove the version-dependent
serialization code. (Note that the affected code here is for SN states
in lmdb storage, not for SN-to-SN communication serialization).
- Don't try participate in quorums at startup if you are not a service node
- Add lock for when potentially writing to the DB
- Pre-existing batch can be open whilst updating checkpoint so can't use
DB guards
- Temporarily emit the ban message using msgwriter instead of logs
* Add debug_mine_singular_block to mine one block at a time for integration tests
* Add workaround for mining thread modifying main thread state causing lock
* Change debug_mine_singular blocks to debug_mine_n_blocks
b3648232 daemon: fix ratio not being floating point (moneromooo-monero)
e1b097b9 core_rpc_server: remove dummy assigning int to bool (moneromooo-monero)
The setup-background-mining option can be used to select
background mining when a wallet loads. The user will be asked
the first time the wallet is created.
* Add the minimal fix for prepare registration failing by dust amounts
* Add dust redistribution at final step
* Comment and clean up useless lines
* Add fudge factor for dust amounts
* Avoid possible overflow with portions if contributor 1 is insufficient
