This purges epee::critical_region/epee::critical_section and the awful
CRITICAL_REGION_LOCAL and CRITICAL_REGION_LOCAL1 and
CRITICAL_REGION_BEGIN1 and all that crap from epee code.
This wrapper class around a mutex is just painful, macro-infested
indirection that accomplishes nothing (and, worse, forces all using code
to use a std::recursive_mutex even when a different mutex type is more
appropriate).
This commit purges it, replacing the "critical_section" mutex wrappers
with either std::mutex, std::recursive_mutex, or std::shared_mutex as
appropriate. I kept anything that looked uncertain as a
recursive_mutex, simple cases that obviously don't recurse as
std::mutex, and simple cases with reader/writing mechanics as a
shared_mutex.
Ideally all the recursive_mutexes should be eliminated because a
recursive_mutex is almost always a design flaw where someone has let the
locking code get impossibly tangled, but that requires a lot more time
to properly trace down all the ways the mutexes are used.
Other notable changes:
- There was one NIH promise/future-like class here that was used in
example one place in p2p/net_node; I replaced it with a
std::promise/future.
- moved the mutex for LMDB resizing into LMDB itself; having it in the
abstract base class is bad design, and also made it impossible to make a
moveable base class (which gets used for the fake db classes in the test
code).
- pessimizing move in wallet2 prevents copy ellision
- various for loops were creating copies (clang-10 now warns about
this). Mostly this is because they had the type wrong when looping
through a map: the iterator type of a `map<K, V>` is `pair<const K,
V>` not `pair<K, V>`. Replaced them with C++17:
for (const auto& [key, val] : var)
which is so much nicer.
- cryptonote::core did not have a virtual destructor, but had virtual
methods (causing both a warning, and likely a crash if we ever have
something inheriting from it held in a unique_ptr<core>).
- core() constructor still had explicit even though it lost the single
argument.
- test code class had a `final` destructor but wasn't marked final. (It
also has a billion superfluous `virtual` declarations but I left them
in place because it's just test code).
common/util.h has become something of a dumping ground of random
functions. This splits them up a little by moving the filesystem bits
to common/file.h, the sha256sum functions to common/sha256sum.h, and the
(singleton) signal handler to common/signal_handler.h.
A huge amount of this is repetitive:
- `boost::get<T>(variant)` becomes `std::get<T>(variant)`
- `boost::get<T>(variant_ptr)` becomes `std::get_if<T>(variant_ptr)`
- `variant.type() == typeid(T)` becomes `std::holds_alternative<T>(variant)`
There are also some simplifications to visitors using simpler stl
visitors, or (simpler still) generic lambdas as visitors.
Also adds boost serialization serializers for std::variant and
std::optional.
This overhauls the (unpleasant) epee "portable storage" interface to be
a bit easier to work with. The diff is fairly large because of the
amount of type indirection that was being used, but at its core does
this simplifications:
- `hsection` was just a typedef for a `section*`, so just use `section*`
instead to not pretend it the value is something it isn't.
- use std::variant instead of boost::variant to store the portable
storage items.
- don't make the variant list recursive with itself since the
serialization doesn't support that anyway (i.e. it doesn't support a
list of lists).
- greatly simplified the variant visiting by using generic lambdas.
- the array traversal interface was a horrible mess. Replaced it with
a simpler custom iterator approach.
- replaced the `selector<bool>` templated class with templated function.
So for example,
epee::serialization::selector<is_store>::serialize_t_val_as_blob(...)
becomes:
epee::serialization::perform_serialize_blob<is_store>(bytes, stg, parent_section, "addr");
and similar for the other types of serializing.
Changes all boost mutexes, locks, and condition_variables to their stl
equivalents.
Changes all lock_guard/unique_lock/shared_lock to not specify the mutex
type (C++17), e.g.
std::lock_guard foo{mutex};
instead of
std::lock_guard<oh::um::what::mutex> foo{mutex};
Also changes some related boost::thread calls to std::thread, and some
related boost chrono calls to stl chrono.
boost::thread isn't changed here to std::thread because some of the
instances rely on some boost thread extensions.
- replace the gross cmake shell script tools being used to create the data file with
cmake code and a single configure_file
- change checkpoint data to string_view
- tools::sha256sum() had two different overloads that did very different
things, yet presents two overloads: one that takes a pointer+size, the
other that takes a string lvalue ref, and yet the string lvalue
reference is *entirely* different (it reads a file). Append _str and
_file suffixes to make it less dangerous.
Switch loki dev branch to C++17 compilation, and update the code with
various C++17 niceties.
- stop including the (deprecated) lokimq/string_view.h header and
instead switch everything to use std::string_view and `""sv` instead of
`""_sv`.
- std::string_view is much nicer than epee::span, so updated various
loki-specific code to use it instead.
- made epee "portable storage" serialization accept a std::string_view
instead of const lvalue std::string so that we can avoid copying.
- switched from mapbox::variant to std::variant
- use `auto [a, b] = whatever()` instead of `T1 a; T2 b; std::tie(a, b)
= whatever()` in a couple places (in the wallet code).
- switch to std::lock(...) instead of boost::lock(...) for simultaneous
lock acquisition. boost::lock() won't compile in C++17 mode when given
locks of different types.
- removed various pre-C++17 workarounds, e.g. for fold expressions,
unused argument attributes, and byte-spannable object detection.
- class template deduction means lock types no longer have to specify
the mutex, so `std::unique_lock<std::mutex> lock{mutex}` can become
`std::unique_lock lock{mutex}`. This will make switching any mutex
types (e.g. from boost to std mutexes) far easier as you just have to
update the type in the header and everything should work. This also
makes the tools::unique_lock and tools::shared_lock methods redundant
(which were a sort of poor-mans-pre-C++17 way to eliminate the
redundancy) so they are now gone and replaced with direct unique_lock or
shared_lock constructions.
- Redid the LNS validation using a string_view; instead of using raw
char pointers the code now uses a string view and chops off parts of the
view as it validates. So, for instance, it starts with "abcd.loki",
validates the ".loki" and chops the view to "abcd", then validates the
first character and chops to "bcd", validates the last and chops to
"bc", then can just check everything remaining for is-valid-middle-char.
- LNS validation gained a couple minor validation checks in the process:
- slightly tightened the requirement on lokinet addresses to require
that the last character of the mapped address is 'y' or 'o' (the
last base32z char holds only one significant bit).
- In parse_owner_to_generic_owner made sure that the owner value has
the correct size (otherwise we could up end not filling or
overfilling the pubkey buffer).
- Replaced base32z/base64/hex conversions with lokimq's versions which
have a nicer interface, are better optimized, and don't depend on epee.
Monero decided to implement this by rebuilding the blockchain class in
tests with another FakeDB. You can trivially avoid this by restructuring
get_block_longhash to not have a high level dependency on the blockchain
itself.
Having the type list defined in the core_rpc_server.cpp file but the
actual list of types in core_rpc_server_command_defs.h is sort of
awkward: you have to change two separate files to add a type. This
moves the type list to the bottom of core_rpc_server_command_defs.h,
instead, which allows cleaning up some bits of core_rpc_server.cpp that
were there to deal with it.
* fix segfault when exiting cli wallet due to running poll thread
* fix whitespace
* style
* support for amount of blacklisted stakes in wallets
* bump version number for blacklist entries with amounts
* default initialize ints in key_image_blacklist_entry
* whitespace
High-level details:
This redesigns the RPC layer to make it much easier to work with,
decouples it from an embedded HTTP server, and gets the vast majority of
the RPC serialization and dispatch code out of a very commonly included
header.
There is unfortunately rather a lot of interconnected code here that
cannot be easily separated out into separate commits. The full details
of what happens here are as follows:
Major details:
- All of the RPC code is now in a `cryptonote::rpc` namespace; this
renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK
becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME`
becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already
working in the `rpc` namespace).
- `core_rpc_server` is now completely decoupled from providing any
request protocol: it is now *just* the core RPC call handler.
- The HTTP RPC interface now lives in a new rpc/http_server.h; this code
handles listening for HTTP requests and dispatching them to
core_rpc_server, then sending the results back to the caller.
- There is similarly a rpc/lmq_server.h for LMQ RPC code; more details
on this (and other LMQ specifics) below.
- RPC implementing code now returns the response object and throws when
things go wrong which simplifies much of the rpc error handling. They
can throw anything; generic exceptions get logged and a generic
"internal error" message gets returned to the caller, but there is
also an `rpc_error` class to return an error code and message used by
some json-rpc commands.
- RPC implementing functions now overload `core_rpc_server::invoke`
following the pattern:
RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context);
This overloading makes the code vastly simpler: all instantiations are
now done with a small amount of generic instantiation code in a single
.cpp rather than needing to go to hell and back with a nest of epee
macros in a core header.
- each RPC endpoint is now defined by the RPC types themselves,
including its accessible names and permissions, in
core_rpc_server_commands_defs.h:
- every RPC structure now has a static `names()` function that returns
the names by which the end point is accessible. (The first one is
the primary, the others are for deprecated aliases).
- RPC command wrappers define their permissions and type by inheriting
from special tag classes:
- rpc::RPC_COMMAND is a basic, admin-only, JSON command, available
via JSON RPC. *All* JSON commands are now available via JSON RPC,
instead of the previous mix of some being at /foo and others at
/json_rpc. (Ones that were previously at /foo are still there for
backwards compatibility; see `rpc::LEGACY` below).
- rpc::PUBLIC specifies that the command should be available via a
restricted RPC connection.
- rpc::BINARY specifies that the command is not JSON, but rather is
accessible as /name and takes and returns values in the magic epee
binary "portable storage" (lol) data format.
- rpc::LEGACY specifies that the command should be available via the
non-json-rpc interface at `/name` for backwards compatibility (in
addition to the JSON-RPC interface).
- some epee serialization got unwrapped and de-templatized so that it
can be moved into a .cpp file with just declarations in the .h. (This
makes a *huge* difference for core_rpc_server_commands_defs.h and for
every compilation unit that includes it which previously had to
compile all the serialization code and then throw all by one copy away
at link time). This required some new macros so as to not break a ton
of places that will use the old way putting everything in the headers;
The RPC code uses this as does a few other places; there are comments
in contrib/epee/include/serialization/keyvalue_serialization.h as to
how to use it.
- Detemplatized a bunch of epee/storages code. Most of it should have
have been using templates at all (because it can only ever be called
with one type!), and now it isn't. This broke some things that didn't
properly compile because of missing headers or (in one case) a messed
up circular dependency.
- Significantly simplified a bunch of over-templatized serialization
code.
- All RPC serialization definitions is now out of
core_rpc_server_commands_defs.h and into a single .cpp file
(core_rpc_server_commands_defs.cpp).
- core RPC no longer uses the disgusting
BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design
that forced slamming tons of code into a common header that didn't
need to be there.
- epee::struct_init is gone. It was a horrible hack that instiated
multiple templates just so the coder could be so lazy and write
`some_type var;` instead of properly value initializing with
`some_type var{};`.
- Removed a bunch of useless crap from epee. In particular, forcing
extra template instantiations all over the place in order to nest
return objects inside JSON RPC values is no longer needed, as are a
bunch of stuff related to the above de-macroization of the code.
- get_all_service_nodes, get_service_nodes, and get_n_service_nodes are
now combined into a single `get_service_nodes` (with deprecated
aliases for the others), which eliminates a fair amount of
duplication. The biggest obstacle here was getting the requested
fields reference passed through: this is now done by a new ability to
stash a context in the serialization object that can be retrieved by a
sub-serialized type.
LMQ-specifics:
- The LokiMQ instance moves into `cryptonote::core` rather than being
inside cryptonote_protocol. Currently the instance is used both for
qnet and rpc calls (and so needs to be in a common place), but I also
intend future PRs to use the batching code for job processing
(replacing the current threaded job queue).
- rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue.
Unlike http_server it isn't technically running the whole LMQ stack
from here, but the parallel name with http_server seemed appropriate.
- All RPC endpoints are supported by LMQ under the same names as defined
generically, but prefixed with `rpc.` for public commands and `admin.`
for restricted ones.
- service node keys are now always available, even when not running in
`--service-node` mode: this is because we want the x25519 key for
being able to offer CURVE encryption for lmq RPC end-points, and
because it doesn't hurt to have them available all the time. In the
RPC layer this is now called "get_service_keys" (with
"get_service_node_key" as an alias) since they aren't strictly only
for service nodes. This also means code needs to check
m_service_node, and not m_service_node_keys, to tell if it is running
as a service node. (This is also easier to notice because
m_service_node_keys got renamed to `m_service_keys`).
- Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and
`sub.mempool` subscribes the connection for new block and new mempool
TX notifications. The latter can notify on just blink txes, or all
new mempool txes (but only new ones -- txes dumped from a block don't
trigger it). The client gets pushed a [`notify.block`, `height`,
`hash`] or [`notify.tx`, `txhash`, `blob`] message when something
arrives.
Minor details:
- rpc::version_t is now a {major,minor} pair. Forcing everyone to pack
and unpack a uint32_t was gross.
- Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...).
(This immediately revealed a couple of bugs in the RPC code that was
assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because
the macro allows implicit conversion to a char).
- De-templatizing useless templates in epee (i.e. a bunch of templated
types that were never invoked with different types) revealed a painful
circular dependency between epee and non-epee code for tor_address and
i2p_address. This crap is now handled in a suitably named
`net/epee_network_address_hack.cpp` hack because it really isn't
trivial to extricate this mess.
- Removed `epee/include/serialization/serialize_base.h`. Amazingly the
code somehow still all works perfectly with this previously vital
header removed.
- Removed bitrotted, unused epee "crypted_storage" and
"gzipped_inmemstorage" code.
- Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with
LOKI_DEFERs. The epee version involves quite a bit more instantiation
and is ugly as sin. Also made the `loki::defer` class invokable for
some edge cases that need calling before destruction in particular
conditions.
- Moved the systemd code around; it makes much more sense to do the
systemd started notification as in daemon.cpp as late as possible
rather than in core (when we can still have startup failures, e.g. if
the RPC layer can't start).
- Made the systemd short status string available in the get_info RPC
(and no longer require building with systemd).
- during startup, print (only) the x25519 when not in SN mode, and
continue to print all three when in SN mode.
- DRYed out some RPC implementation code (such as set_limit)
- Made wallet_rpc stop using a raw m_wallet pointer
Removes all "using namespace epee;" and "using namespace std;" from the
code and fixes up the various crappy places where unnamespaced types
were being used.
Also removes the ENDL macro (which was defined to be `std::endl`)
because it is retarded, and because even using std::endl instead of a
plain "\n" is usually a mistake (`<< std::endl` is equivalent to `<<
"\n" << std::flush`, and that explicit flush is rarely desirable).
This commit continues the complete replacement of the spaghetti code
mess that was inside daemon/ and daemonize/ which started in #1138, and
looked like a entry level Java programmer threw up inside the code base.
This greatly simplifies it, removing a whole pile of useless abstraction
layers that don't actually abstract anything, and results in
considerably simpler code. (Many of the changes here were also carried
out in #1138; this commit updates them with the merged result which
amends some things from that PR and goes further in some places).
In detail:
- the `--detach` (and related `--pidfile`) options are gone. (--detach
is still handled, but now just prints a fatal error). Detaching a
process is an archaic unix mechanism that has no place on a modern
system. If you *really* want to do it anyway, `nohup lokid &` will do
the job. (The Windows service control code, which is probably seldom
used, is kept because it seems potentially useful for Windows users).
- Many of the `t_whatever` classes in daemon/* are just deleted (mostly
done in #1138); each one was a bunch of junk code that wraps 3-4 lines
but forces an extra layer (not even a generic abstraction, just a
useless wrapper) for no good reason and made the daemon code painfully
hard to understand and work with.
- All of the remaining `t_whatever` classes in daemon/* are either
renamed to `whatever` (because prefixing every class with `t_` is
moronic).
- Some stupid related code (e.g. epee's command handler returning an
unsuitable "usage" string that has to be string modified into what we
want) was replaced with more generic, useful code.
- Replaced boost mutexes/cvs with std ones in epee command handler, and
deleted some commented out code.
- The `--public-node` option handling was terrible: it was being handled
in main, but main doesn't know anything about options, so then it
forced it through the spaghetti objects *beside* the pack of all
options that got passed along. Moved it to a more sane location
(core_rpc_server) and parse it out with some sanity.
- Changed a bunch of std::bind's to lambdas because, at least for small
lambdas (i.e. with only one-or-two pointers for captures) they will
generally be more efficient as the values can be stored in
std::function's without any memory allocations.
This was a useless feature to begin with. According to a Monero
insider, this was introduced at the time with an intention of making it
on-by-default on every monerod instance everywhere, but because that was
such an overwhelmingly stupid idea, it never happened yet all this code
(which is probably used by no one anywhere ever) remains in the code
base.
Even if the idea wasn't dumb to start with, this will also become even
more pointless with pulse, so just drop it (it is over 1000 lines of
code, not even counting the extra headers pulled in to do things like
querying CPU usage and battery status).
I started this PR to just fix the upgrades, but when investigating I
found some problematic memory leaks as well and ended up adding a bunch
of nice abstraction layers to make the code nicer.
I'd normally separate these into multiple commits, but they ended up
quite interdependent to the point where it isn't really feasible to do
so; but here's the details on the changes:
- add a `DEFAULT "register_height"` for the new "update_height" field so
that the alter table works
- reorganized the statement preparation and database migration so that
all statement preparation happens *after* migration; otherwise
statement preparation fails because some of the statements reference
columns that aren't created until the migration.
- created a `sql_compiled_statement` class that manages sqlite statement
pointers; there were several places that we were leaking them (the
main ones were okay, but we also have some dynamically constructed
ones for variable size queries).
- added a new abstraction layer around sqlite3 binding and value
extraction that makes the actual sql interaction code much less
verbose.
- removed the `nettype` argument from `sql_run_statement` as it wasn't
being used. (If something in the future needs it, the compiled
statement class has a reference to the db object, so it can be
obtained via `statement.nsdb.network_type()`)
- `bind(statement, 1, someval)` now does what
`sqlite3_bind_whatever(statement, 1, someval, ...)` did before, except
that it infers the "whatever" from the type (for everything except
blobs), and does whatever "..." needs to happen.
- `bind(statement, 1, blob_view{someval.data(), size})` can do the same
for a blob. (There is also `bind_blob(statement, 1, someval.data(),
size)` which does the same thing but is nicer in some contexts).
- `auto x = get<T>(statement, 1)` is the bind counterpart for extracting
a known type (and similarly calls the right sqlite function based on
`T`). There is also a `get(statement, 1, x);` that does the same
thing, but infers the type from whatever `x` is. `get_blob` similarly
gets a blob value, though currently this is only used in the existing
`sql_copy_blob()` wrapper.
- All the `bind` and `get` forms accept an integer enum class as the
position argument, so the code can now write just:
bind(statement, lns_db_setting_column::top_height, value);
instead of:
bind(statement, static_cast<int>(lns_db_setting_column::top_height), value);
- bind_all lets you bind all the parameters in one go (and also clears
existing bindings), letting you write things like:
bind_all(statement, 123, "my string", my_u64_int);
which previously would have needed:
sqlite3_clear_bindings(statement);
sqlite3_bind_int(statement, 1, 123);
sqlite3_bind_text(statement, 2, "my string", 10, nullptr, nullptr);
sqlite3_bind_int64(statement, 3, my_u64_int);
- Also there's a `bind_and_run` which combines this binding with a
sql_run_statement.
- put all of the new binding code in an anonymous namespace, along with
several existing `static` functions (this is equivalent, just easier
than having to write static dozens of times).
- removed a superfluous extra "OR" clause in get_mappings_by_owners. It
was producing queries such as:
... WHERE o1.address IN (?, ?, ?) OR o2.address IN (?, ?, ?) OR o2.address IN (?, ?, ?)
also simplified that part of the code down quite a bit with a
pre-allocated string for the "?, ?, ?" part.
- Simplified some some pointer/size pairs with lokimq::string_view
(which, internally, is just a pointer size pair already).
- Various methods got de-const'ed in this change. They probably
shouldn't have been const before because even though they only access
data, they still mutate internal state of the stored statements, and
making them all mutable would be a bit gross. (They did this before,
too, but since everything was pointers calling into C code I guess
they compiler just allowed it).
Only the initial purchase updated LNS record's register_height. If the
transaction was updated inbetween and we reorganized an update LNS transaction
the details in the LNS wouldn't be reverted correctly and can cause the
replayed TX's to fail.
It fails to revert correctly because when querying the LNS DB for
transactions to revert, we only have the initial registration height,
not the subsequent updates.
This PR adds that update_height field and migrates existing DB's.
Quoting from Jagerman's reason: With the std::hash specialization
already here, an unordered_map here seems an obviously better choice:
hash collisions won't be destructive (they are with std::map + hash as
keys), the code is slightly simpler (you can eliminate the hash call),
and performance is better (map is an insertion sorted container, thus
O(NlogN) on N insertions).
This isn't just an optimal data structure thing: using a map here has
a potential (if rare) data omission on collision.
- Renames generic_key->generic_owner
- Move generic_owner and generic_signature out of crypto.h because they
aren't really crypto items, rather composition of crypto primitives.
generic_owner also needs access to account_public_address, while that is
just 2 public keys, I've decided to include cryptonote_basic.h into
tx_extra.h instead of crypto.h.
- Some generic_owner helper functions were moved into
cryptonote_basic/format_utils as they need to avoid circular
dependencies between cryptonote_core/cryptonote_basic had I included
generic_owner/generic_signature into loki_name_system.h
- Utilise the normal serialize macros since tx_extra.h already includes
the serializing headers.
This allows RPC coming from the loopback interface to not have
to pay for service. This makes it possible to run an externally
accessible RPC server for payment while also having a local RPC
server that can be run unrestricted and payment free.
We still pull out the spend public key, in a future coming PR we will
take the entire wallet address as to improve the usability of LNS.
Said PR will also accept subaddresses which should work out of the box
as long as we use the correct secret key of said subaddress to generate
the required signature.
I opted for just a enum instead of pulling in mapbox::variant for
simplicity. mapbox::variant saves a few lines of code for easily
capturing the type at assignment, whereas here we have helper functions
for assigning type.
Everywhere still uses crypto::hash for the name_hash due to being the
most compact storage form except for the LNS DB which uses base64 so
that it can be indexes in SQL for speed.
But at the boundary between requesting and accessing the DB entry, the
hash gets converted into a base64 representation of the hash.
We opt for base64 as it is more compact than storing the hex
representation.
* Adds GET_SERVICE_NODE_STATUS RPC endpoint to lokid
Previously the getting the service node status was only available in the
lokid console. This commit adds the endpoint to the RPC server
* removed unnecessary variable, allow for json in request
This allows it to be used to fetch the current staking requirement by
just asking for a height of 0, while currently you'd first have to fetch
the height and then pass that in.
- constexpr functions in common/loki.h for inlining
- move hex functions out from common/loki.h to common/hex.h
- use and apply prev_txid on LNS TX's to all LNS types (for updating in the future)
- add lns burn type, for custom burn amounts
- accept and validate lokinet addresses via base32z
- return lokinet addresses in RPC LNS calls via base32z
- updated Messenger references to Session
- update documentation to note that only Session LNS entries are allowed currently
- remove raw c-string interface from LNS db
- update multi-SQL queries into single SQL queries
- remove tx estimation backlog in anticipation for 2 priorities only, blink + unimportant
b90c4bc3 rpc: error out from get_info if the proxied call errors out (moneromooo-monero)
fa16df99 make_test_signature: exit nicely on top level exception (moneromooo-monero)
054b2621 node_rpc_proxy: init some new rpc payment fields in invalidate (moneromooo-monero)
d0faae2a rpc: init a few missing client_info members (moneromooo-monero)
d56a483a rpc: do not propagate exceptions out of a dtor (moneromooo-monero)
3c849188 rpc: always set the update field in update on sucess (moneromooo-monero)
8231c7cd rpc: fix bootstrap RPC payment RPC being made in raw JSON, not JSON RPC (moneromooo-monero)
81c26589 rpc: don't auto fail RPC needing payment in bootstrap mode (moneromooo-monero)
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
Handle errors better when long polling is disabled instead of endlessly
spamming logs.
Avoid lock contention when set_daemon is called. Instead of immediately
affecting the long polling thread (which could be engaging the mutex
until RPC timeout, meaning the program stalls for that duration), update
the address on the next iteration of the long polling thread.
Wallets handle daemons that disable long polling better by sleeping.
`tools::wallet2::rpc_long_poll_timeout` was a static member declaration
without a definition, which isn't allowed before C++17 (although can
work depending on compiler optimizations). Adding the definition in
wallet2.cpp isn't really an option (it would make core depend on the
wallet), so just move it to a constexpr static global (which is allowed
without a definition, even before C++17) in `rpc/` instead.
This is really useful for the blink test suite as it lets us trigger a
resync (which normally only runs every 60s). In particular where we
have a (test) situation like this:
A - B - C
where we want to take down B and bring it up again but want to be sure
that new things learned from A get seen right away by C: if B does a
resync with C *before* it does a resync with A then C wouldn't get the
sync updates for a full minute, while if we force B to sync then force C
to sync we can ensure quick propagation for the test suite.
`--regtest` didn't work in some edge cases, this fixes various things:
- the genesis block wasn't accepted because it needed to be v7, not
vMax
- reduce initial uptime proof delay to 5s in regtest mode
- add --regtest flag to the wallet so that it can talk to a daemon in
--regtest mode.
This also adds two new mining options, available via rpc:
- slow_mining - this avoids the RandomX initialization. It is much
slower, but for regtest with fixed difficulty of 1 that is perfectly
fine.
- `num_blocks` - instruct the miner to mine for the given number of
blocks, then stop. (This can overmine if mining with multiple
threads at a low difficulty, but that's fine).
Blink txes were not being properly passed in/out of the RPC wallet.
This adds the necessary bits both to submit a blink and to get a blink
submission status back from the daemon.
This replaces the horrible, horrible, badly misused templated
once_a_time_seconds and once_a_time_milliseconds with a `periodic_task`
that works the same way but takes parameters as constructor arguments
instead of template parameters.
It also makes various small improvements:
- uses std::chrono::steady_clock instead of ifdef'ing platform dependent
timer code.
- takes a std::chrono duration rather than a template integer and
scaling parameter.
- timers can be reset to trigger on the next invocation, and this is
thread-safe.
- timer intervals can be changed at run-time.
This all then gets used to reset the proof timer immediately upon
receiving a ping (initially or after expiring) from storage server and
lokinet so that we send proofs out faster.
If we've previously printed some alerts about not submitting proofs
because of lokinet/storage server it's nice to print a message as soon
as we get the needed ping. Also nice to see on startup for the first
one to know things are working.
This extracts uptime proof data entirely from service node states,
instead storing (some) proof data as its own first-class object in the
code and backed by the database. We now persistently store:
- timestamp
- version
- ip & ports
- ed25519 pubkey
and update it every time a proof is received. Upon restart, we load all
the proofs from the database, which means we no longer lose last proof
received times on restart, and always have the most recently received ip
and port information (rather than only having whatever it was in the
most recently stored state, which could be ~12 blocks ago). It also
means we don't need to spam the network with proofs for up to half an
hour after a restart because we now know when we last sent (and
received) our own proof before restarting.
This separation required some restructuring: most notably changing
`state_t` to be constructed with a `service_node_list` pointer, which it
needs both directly and for BlockchainDB access. Related to this is
also eliminating the separate BlockchainDB pointer stored in
service_node_list in favour of just accessing it via m_blockchain (which
now has a new `has_db()` method to detect when it has been initialized).
This adds the last lokinet ping time to the second `status` line. It
also compresses this (and storage server's) ping time format to (for
example) `4.7min` instead of `4.7 minutes ago`.
I also add a few tweaks here to the first line of the status message to
save some space for the most common cases:
- just don't show " on mainnet" at all since that's the common case, and
instead capitalize "ON TESTNET"/"ON STAGENET" to make it more obvious
that something is non-default.
- just don't show ", not mining" when not mining
- show the daemon version returned by RPC in addition to the hf version,
as "vX.Y.Z(net vAA)". You usually get a version display from an info
log message when running `lokid version`, but that can be quite
misleading as it's the local lokid binary version which is not
necessarily the same version as the running daemon.
Sample output:
SN:
Height: 174791/174791 (100.0%) ON TESTNET, net hash 455 H/s, v6.0.0(net v14), up to date, 8(out)+12(in) connections, uptime 0d 0h 4m 32s
SN: ff00062fa69397f5580ac2b3d060064d89edac625c77a58fdcd520f94ea54727 active, proof: (never), last pings: 1.6min (storage), 1.6min (lokinet)
Regular node (while mining):
Height: 174791/174791 (100.0%) ON TESTNET, mining at 94 H/s, net hash 455 H/s, v6.0.0(net v14), up to date, 8(out)+32(in) connections, uptime 0d 0h 44m 2s
- actually include the blink hashes in the core sync data
- fix cleanup to delete heights in (0, immutable] instead of [0,
immutable); we want to keep 0 because it signifies the mempool, and we
only need blocks after immutable, not immutable itself.
- fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was
only searching the chain and ignored missed_txs, but missed_txs are ones
we need to look up in the mempool)
- Add a method to tx_pool (needed for the above) to grab multiple txes
by hash (essentially a multi-tx version of `get_transaction()`), and
change get_transaction() to use it with a single value.
- Added various debugging statements
- Added a bunch of comments to each condition of the preliminary blink
data check condition.
- Don't abort blink addition on a single signature failure: if there are
enough valid signatures we should still accept it.
- Check for blink signature approval when receiving blink signatures;
it's not enough to know that all were added successfully, we also have
to ask the blink tx if it is approved (which does additional checks on
subquorum counts) once we add them all.
This adds the ability for check_fee() to also check the burn amount.
This requires passing extra info through `add_tx()` (and the various
things that call it), so I took the:
bool keeped_by_block, bool relayed, bool do_not_relay
argument triplet, moved it into a struct in tx_pool.h, then added the other fee
options there (along with some static factory functions for generating the
typical sets of option).
The majority of this commit is chasing that change through the codebase and
test suite.
This is used by blink but should also help LNS and other future burn
transactions to verify a burn amount simply when adding the transation to the
mempool. It supports a fixed burn amount, a burn amount as a multiple of the
minimum tx fee, and also allows you to increase the minimum tx fee (so that,
for example, we could require blink txes to pay miners 250% of the usual
minimum (unimportant) priority tx fee.
- Removed a useless core::add_new_tx() overload that wasn't used anywhere.
Blink-specific changes:
(I'd normally separate these into a separate commit, but they got interwoven
fairly heavily with the above change).
- changed the way blink burning is specified so that we have three knobs for
fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee).
The fixed amount is currently 0, base fee is 400%, and require miner tx fee is
simply 100% (i.e. no different than a normal transaction). This is the same as
before this commit, but is changing how they are being specified in
cryptonote_config.h.
- blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked
before signing a blink tx. (These fee checks don't apply to anyone else --
when propagating over the network only the miner tx fee is checked).
- Added a couple of checks for blink quorums: 1) make sure they have reached
the blink hf; 2) make sure the submitted tx version conforms to the current hf
min/max tx version.
- print blink fee information in simplewallet's `fee` output
- add "typical" fee calculations in the `fee` output:
[wallet T6SCwL (has locked stakes)]: fee
Current fee is 0.000000850 loki per byte + 0.020000000 loki per output
No backlog at priority 1
No backlog at priority 2
No backlog at priority 3
No backlog at priority 4
Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output
Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink)
where "small" here is the same tx size (2500 bytes + 2 outputs) used to
estimate backlogs.
This catches any exception thrown in the inner quorumnet blink code and
sets it in the promise if it occurs, which propagates it out to
core_rpc_server to catch and deal with.
- Adds blink signature synchronization and storage through the regular
p2p network
- Adds wallet support (though this is still currently buggy and needs
additional fixes - it sees the tx when it arrives in the mempool but
isn't properly updating when the blink tx gets mined.)
There are a bunch trivial forwarding wrappers in cryptonote_core that
simply call the same method in the pool, and blink would require adding
several more. Instead of all of these existing (and new) wrappers, just
directly expose the tx_pool reference so that anything with a `core`
reference can access and call to the mempool directly.
This code was too convoluted not to fix.
- There are 5 linear searches done over 2 vectors (without any changes
done to the vectors in between).
- One of the linear searches calls out to epee to convert the same value
to a hex string for every element.
- Both a set and a map are used here with identical key contents. The
old code looks like it only conditionally adds to the map if it finds
a match in the loop, but the for loop is doing exactly the same test
as the `find_if` that guards the entire `else if` containing it, so it
will *always* match.
- Removed some error code that can never actually run (because of the
above set/map equivalence).
This adds a thread-local, pre-seeded rng at `tools::rng` (to avoid the
multiple places we are creating + seeding such an RNG currently).
This also moves the portable uniform value and generic shuffle code
there as well as neither function is specific to service nodes and this
seems a logical place for them.
This is the bulk of the work for blink. There is two pieces yet to come
which will follow shortly, which are: the p2p communication of blink
transactions (which needs to be fully synchronized, not just shared,
unlike regular mempool txes); and an implementation of fee burning.
Blink approval, multi-quorum signing, cli wallet and node support for
submission denial are all implemented here.
This overhauls and fixes various parts of the SNNetwork interface to fix
some issues (particularly around non-SN communication with SNs, which
wasn't working).
There are also a few sundry FIXME's and TODO's of other minor details
that will follow shortly under cleanup/testing/etc.
Currently we store it as various different things: 3 separate ints, 2
u16s, 3 separate u16s, and a vector of u16s. This unifies all version
values to a `std::array<uint16_t,3>`.
- LOKI_VERSION_{MAJOR,MINOR,PATCH} are now just LOKI_VERSION
- The previous LOKI_VERSION (C-string of the version) is now renamed
LOKI_VERSION_STR
A related change included here is that the restricted RPC now returns
the major version in the get_info rpc call instead of an empty string
(e.g. "5" instead of ""). There is almost certainly enough difference
in the RPC results to distinguish major versions already so this doesn't
seem like it actually leaks anything significant.
This adds vote relaying via quorumnet.
- The main glue between existing code and quorumnet code is in
cryptonote_protocol/quorumnet.{h,cpp}
- Uses function pointers assigned at initialization to call into the
glue without making cn_core depend on p2p
- Adds ed25519 and quorumnet port to uptime proofs and serialization.
- Added a second uptime proof signature using the ed25519 key to that
SNs are proving they actually have it (otherwise they could spoof
someone else's pubkey).
- Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet
listens on the p2p IPv4 address (but broadcasts the `public_ip` to the
network).
- Derives x25519 when received or deserialized.
- Converted service_info_info::version_t into a scoped enum (with the
same underlying type).
- Added contribution_t::version_t scoped enum. It was using
service_info_info::version for a 0 value which seemed rather wrong.
Random small details:
- Renamed internal `arg_sn_bind_port` for consistency
- Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet
quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port
would collide with the testnet p2p port).
Renamed:
- `if_block_not_equal` -> `poll_block_hash`
- `gave_if_not_equal` -> `polling_mode`
Added lokidocs documentation for the new and existing get_n_service_node
fields, and adds magic codes to disable the internal variables.
This simplifies lokinet as it doesn't need to worry about finding the
private key files; sometime after the next fork we do the same
simplification to loki-storage-server to use this RPC call as well.
Aside from simplifying the external components, I also have an idea here
of an advanced SN setup that hosts multiple lokid's on an internal,
beefy dedicated server with good redundant storage and lots of memory,
combined with multiple VPSes with lots of bandwidth for
lokinet/loki-storage but not enough storage/memory capacity to host the
blockchain, and not having to coordinate storage across machines makes
this setup substantially easier.
This allows a `"if_block_not_equal": "hash"` parameter to be given to
`get_n_service_nodes` which, if the given value matches the current
top block hash, skips building and returning a reply.
This is primarily aimed at lokinet which polls fairly frequenty for an
update but where the vast majority of those polls contain no new
information.
It also removes the get_all_service_nodes_public_keys RPC request
completely as lokinet was the only consumer of it (particularly unlikely
that anyone else was using this because it was returning the keys
base32z-encoded).
Neither of these have a place in modern C++11; boost::value_initialized
is entirely superseded by `Type var{};` which does value initialization
(or default construction if a default constructor is defined). More
problematically, each `boost::value_initialized<T>` requires
instantiation of another wrapping templated type which is a pointless
price to pay the compiler in C++11 or newer.
Also removed is the AUTO_VAL_INIT macro (which is just a simple macro
around constructing a boost::value_initialized<T>).
BOOST_FOREACH is a similarly massive pile of code to implement
C++11-style for-each loops. (And bizarrely it *doesn't* appear to fall
back to C++ for-each loops even when under a C++11 compiler!)
This removes both entirely from the codebase.
This allows flushing internal caches (for now, the bad tx cache,
which will allow debugging a stuck monerod after it has failed to
verify a transaction in a block, since it would otherwise not try
again, making subsequent log changes pointless)
Daemons intended for public use can be set up to require payment
in the form of hashes in exchange for RPC service. This enables
public daemons to receive payment for their work over a large
number of calls. This system behaves similarly to a pool, so
payment takes the form of valid blocks every so often, yielding
a large one off payment, rather than constant micropayments.
This system can also be used by third parties as a "paywall"
layer, where users of a service can pay for use by mining Monero
to the service provider's address. An example of this for web
site access is Primo, a Monero mining based website "paywall":
https://github.com/selene-kovri/primo
This has some advantages:
- incentive to run a node providing RPC services, thereby promoting the availability of third party nodes for those who can't run their own
- incentive to run your own node instead of using a third party's, thereby promoting decentralization
- decentralized: payment is done between a client and server, with no third party needed
- private: since the system is "pay as you go", you don't need to identify yourself to claim a long lived balance
- no payment occurs on the blockchain, so there is no extra transactional load
- one may mine with a beefy server, and use those credits from a phone, by reusing the client ID (at the cost of some privacy)
- no barrier to entry: anyone may run a RPC node, and your expected revenue depends on how much work you do
- Sybil resistant: if you run 1000 idle RPC nodes, you don't magically get more revenue
- no large credit balance maintained on servers, so they have no incentive to exit scam
- you can use any/many node(s), since there's little cost in switching servers
- market based prices: competition between servers to lower costs
- incentive for a distributed third party node system: if some public nodes are overused/slow, traffic can move to others
- increases network security
- helps counteract mining pools' share of the network hash rate
- zero incentive for a payer to "double spend" since a reorg does not give any money back to the miner
And some disadvantages:
- low power clients will have difficulty mining (but one can optionally mine in advance and/or with a faster machine)
- payment is "random", so a server might go a long time without a block before getting one
- a public node's overall expected payment may be small
Public nodes are expected to compete to find a suitable level for
cost of service.
The daemon can be set up this way to require payment for RPC services:
monerod --rpc-payment-address 4xxxxxx \
--rpc-payment-credits 250 --rpc-payment-difficulty 1000
These values are an example only.
The --rpc-payment-difficulty switch selects how hard each "share" should
be, similar to a mining pool. The higher the difficulty, the fewer
shares a client will find.
The --rpc-payment-credits switch selects how many credits are awarded
for each share a client finds.
Considering both options, clients will be awarded credits/difficulty
credits for every hash they calculate. For example, in the command line
above, 0.25 credits per hash. A client mining at 100 H/s will therefore
get an average of 25 credits per second.
For reference, in the current implementation, a credit is enough to
sync 20 blocks, so a 100 H/s client that's just starting to use Monero
and uses this daemon will be able to sync 500 blocks per second.
The wallet can be set to automatically mine if connected to a daemon
which requires payment for RPC usage. It will try to keep a balance
of 50000 credits, stopping mining when it's at this level, and starting
again as credits are spent. With the example above, a new client will
mine this much credits in about half an hour, and this target is enough
to sync 500000 blocks (currently about a third of the monero blockchain).
There are three new settings in the wallet:
- credits-target: this is the amount of credits a wallet will try to
reach before stopping mining. The default of 0 means 50000 credits.
- auto-mine-for-rpc-payment-threshold: this controls the minimum
credit rate which the wallet considers worth mining for. If the
daemon credits less than this ratio, the wallet will consider mining
to be not worth it. In the example above, the rate is 0.25
- persistent-rpc-client-id: if set, this allows the wallet to reuse
a client id across runs. This means a public node can tell a wallet
that's connecting is the same as one that connected previously, but
allows a wallet to keep their credit balance from one run to the
other. Since the wallet only mines to keep a small credit balance,
this is not normally worth doing. However, someone may want to mine
on a fast server, and use that credit balance on a low power device
such as a phone. If left unset, a new client ID is generated at
each wallet start, for privacy reasons.
To mine and use a credit balance on two different devices, you can
use the --rpc-client-secret-key switch. A wallet's client secret key
can be found using the new rpc_payments command in the wallet.
Note: anyone knowing your RPC client secret key is able to use your
credit balance.
The wallet has a few new commands too:
- start_mining_for_rpc: start mining to acquire more credits,
regardless of the auto mining settings
- stop_mining_for_rpc: stop mining to acquire more credits
- rpc_payments: display information about current credits with
the currently selected daemon
The node has an extra command:
- rpc_payments: display information about clients and their
balances
The node will forget about any balance for clients which have
been inactive for 6 months. Balances carry over on node restart.
* Fix elapsed time in storage server warning message
This was passing the time value rather than the number of seconds so
basically always printed "a long time" instead of the elapsed time.
* Removed pre-HF12 code
* Wait for storage server before sending proof
On startup we send a proof immediately, but this is misleading to the
operator - they may see a proof broadcast over the network suggesting
everything is good even when the storage server is still down. This
makes lokid wait for a first ping before the first proof so that the
user doesn't get mislead.
It also adds the storage server last ping into the `lokid status`
message, such as:
Height: 375634/375634 (100.0%) on mainnet, not mining, net hash 63.00 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 4s
SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: NO PING RECEIVED
or:
Height: 375663/375663 (100.0%) on mainnet, not mining, net hash 76.17 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 32s
SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: last ping 11 seconds ago
This generates a ed25519 keypair (and from it derives a x25519 keypair)
and broadcasts the ed25519 pubkey in HF13 uptime proofs.
This auxiliary key will be used both inside lokid (starting in HF14) in
places like the upcoming quorumnet code where we need a standard
pub/priv keypair that is usable in external tools (e.g. sodium) without
having to reimplement the incompatible (though still 25519-based) Monero
pubkey format.
This pulls it back into HF13 from the quorumnet code because the
generation code is ready now, and because there may be opportunities to
use this outside of lokid (e.g. in the storage server and in lokinet)
before HF14. Broadcasting it earlier also allows us to be ready to go
as soon as HF14 hits rather than having to wait for every node to have
sent a post-HF14 block uptime proof.
For a similar reason this adds a placeholder for the quorumnet port in
the uptime proof: currently the value is just set to 0 and ignored, but
allowing it to be passed will allow upgraded loki 6.x nodes to start
sending it to each other without having to wait for the fork height so
that they can start using it immediately when HF14 begins.
This puts the SN pubkey/privkey into a single struct (which have
ed25519/x25519 keys added to it in the future), which simplifies various
places to just pass the struct rather than storing and passing the
pubkey and privkey separately.
If the peer (whether pruned or not itself) supports sending pruned blocks
to syncing nodes, the pruned version will be sent along with the hash
of the pruned data and the block weight. The original tx hashes can be
reconstructed from the pruned txes and theur prunable data hash. Those
hashes and the block weights are hashes and checked against the set of
precompiled hashes, ensuring the data we received is the original data.
It is currently not possible to use this system when not using the set
of precompiled hashes, since block weights can not otherwise be checked
for validity.
This is off by default for now, and is enabled by --sync-pruned-blocks
2cd4fd8 Changed the use of boost:value_initialized for C++ list initializer (JesusRami)
4ad191f Removed unused boost/value_init header (whyamiroot)
928f4be Make null hash constants constexpr (whyamiroot)
This switches loki 5.x to use a fee formula of
SIZE * PER_BYTE + OUTPUTS * PER_OUTPUT
where we reduce the PER_BYTE fee back to what it was in 3.x; and with
the PER_OUTPUT fee set to 0.02 LOKI. This compares to the 4.x fee of:
SIZE * PER_BYTE * 80
(the *80 multiple was introduced in 4.x).
It also reduces the multiplier for the maximum priority level to 125
instead of 1000 because 1000 produced uselessly high tx fees. The new
multipliers go up 5x at each level: {1, 5, 25, 125} while previously
they went {1, 5, 25, 1000}.
As for the base change: we added the *80 multiplier in 4.x because we
wanted to make a theoretical de-anonymizing tx spam attack more costly.
The unanticipated consequence was that we also made *large* transactions
(such as sweeps) considerably more costly despite the fact that these
transactions typically only create 2 outputs.
This better captures what we meant to do in 4.x (making output creation
relatively more expensive) without making large txes (e.g. sweeps
required for staking) highly expensive.
The end effect is that the fee for a minimum-sized, 1-input/2-output
transaction should stay roughly the same (slightly over 0.04 LOKI),
while a 100-input/2-output transction (a typical spend or sweep from a
wallet with lots of smaller rewards) will drop in fee by somewhere
around 95%.
The most efficient theoretical deanonymizing tx spamming of this sort
was a 1-input/16-output transaction which will become about 2.5x as
expensive as currently the case in v4.x.
It does not leak much since you can make a fair guess by RPC
version already, and some people want to avoid non release
clients when using third parties' nodes (because they'd never
lie about it)
* Enforce minimum storage server version since hardfork 13
* Send Storage Server version as a triplet
* Send Storage Server version as a triplet of numbers, not strings
* Enforce Storage Server version immediately
new cli options (RPC ones also apply to wallet):
--p2p-bind-ipv6-address (default = "::")
--p2p-bind-port-ipv6 (default same as ipv4 port for given nettype)
--rpc-bind-ipv6-address (default = "::1")
--p2p-use-ipv6 (default false)
--rpc-use-ipv6 (default false)
--p2p-require-ipv4 (default true, if ipv4 bind fails and this is
true, will not continue even if ipv6 bind
successful)
--rpc-require-ipv4 (default true, description as above)
ipv6 addresses are to be specified as "[xx:xx:xx::xx:xx]:port" except
in the cases of the cli args for bind address. For those the square
braces can be omitted.
0605406 daemon: sort alt chains by height (moneromooo-monero)
4228ee0 daemon: add optional arguments to alt_chain_info (moneromooo-monero)
880ebfd daemon: add more chain specific info in alt_chain_info (moneromooo-monero)
This converts the stored service_node_info value into a
`shared_ptr<const service_node_info>` rather than a plain
`service_node_info`. This yields a huge performance benefit by
significantly eliminating the vast majority of service_node_info
construction, destruction, and copying.
Most of the time when we copy a service_node_info nothing in it has
changed, which means we're storing exactly the same thing; this means an
extra construction for every SN info on every block *and* an extra
destruction when we cull old stored history. By using a
shared_ptr, the vast majority of those constructions and destructions
are eliminated.
The immediately previous commit (upon which this one builds) already
reduced a full rescan from 180s to 171s; this commit further reduces
that time to 104s, or about 42% reduced from the rescan time required
before this pair of commits. (All timings are from the dev.lokinet.org
box, tested over multiple runs with the entire lmdb cached in memory).
With the shared_ptr approach, we only make a copy when a change is
actually needed: because of infrequent (at the per-SN level) events like
a state_change, received reward, contribution, etc. The contained
reference is deliberately `const` so that values are not changeable;
there's a new function that does an explicit copying duplication,
returning the new non-const and storing the const ref in the shared
pointer.
Related to this is a small change (and fix) to how proof info and
public_ip/storage_port are stored: rather than store the values in the
service_node_info struct itself, they now gets stored in a shared_ptr
inside the service_node_info that intentionally gets shared among all
copies of the service_node_info (that is, a SN info copy deliberately
copies the pointer rather than the values). This also moves the ip/port
values into the proof struct, since that seemed much easier than
maintaining a separate shared_ptr for each value.
Previously, because these were stored as values in the service_node_info
they would actually get rolled back in the event of a reorg, but that
seems highly undesirable: you would end up rolling back to the old
values of the uptime proof and ip address (for example), but that should
not happen: those values are not dependent on the blockchain and so
should not be affected by a reorg/rollback. With this change they
aren't since there is only one actual proof stored.
Note that the shared storage here only applies to in-memory states;
states loaded from the db will still be duplicated.
This commit makes various simplifications and optimizations, mainly in
the service node list code.
All in all, this shaves about 5% of the CPU time used for re-processing
the entire service node list.
In particular:
- changed m_state_history from a std::vector to a std::set that sorts on
height. This is responsible for the bulk of the CPU reduction by
significant reducing the amount of work required for checkpoint
culling, which has to shuffle a lot of `state_t`s around when removing
from the midde of a vector.
- the above also allows replacing the binary-search `std::lower_bound`
complexity with a much simpler `find()`.
- since the history is now always sorted, removed the error messages
related to sorting that either can't happen (on store) or don't matter
(on load).
- Added some converting constructors to simplify code (for example, a
`state_t` can now be very usefully constructed from an r-value
`state_serialized`).
- Many construct + moves (and a couple construct + copy) were replaced
with in-place constructions.
- removed some unused variables
new cli options (RPC ones also apply to wallet):
--p2p-bind-ipv6-address (default = "::")
--p2p-bind-port-ipv6 (default same as ipv4 port for given nettype)
--rpc-bind-ipv6-address (default = "::1")
--p2p-use-ipv6 (default false)
--rpc-use-ipv6 (default false)
--p2p-require-ipv4 (default true, if ipv4 bind fails and this is
true, will not continue even if ipv6 bind
successful)
--rpc-require-ipv4 (default true, description as above)
ipv6 addresses are to be specified as "[xx:xx:xx::xx:xx]:port" except
in the cases of the cli args for bind address. For those the square
braces can be omitted.
* Add disabled-by-default quorum storage support
This adds support for storing N blocks of recent expired quorum state
history in lokid (and dumping to/from the lmdb).
This isn't useful for regular nodes (and that's why we don't store it)
but is incredibly useful for the block explorer to be able to report
*which* node got deregistered/decommissioned/etc. by a given
state_change tx.
* Fix quorum states copy
KV_SERIALIZE doesn't handle non-primitive types well and requires adding
in extra deserialize/reserialize code. It's more straightforward to
manually serialize the output, rather than add serialization overloads.
Furthermore, KV_SERIALIZE doesn't support serializing enums in a sane
way.
* print_quorum_state displays checkpointing quorums, remove batched call
* Update the help text for quorum commands
* <= for the max_quorum_type not <
* Handle heights greater than the latest
* Don't repeatedly add partially filled quorums
* Revert get_quorum_state to take a range (common case)
* Update the help text from print_quorum_state
* Improved print_checkpoints
* Flesh out print checkpoints and associated RPC call
* Remove debug print checkpoints
* Update help text for print_checkpoints
* Rewrite to fix num_checkpoints != heights as a unit of measurement
* Use GET_ALL_CHECKPOINTS defined value in get_checkpoints_range
* Let T be deduced in parse_if_present, json_rpc_request not rpc_request
Circumvents the need to create a new blockhashing blob when you already
know the data you want to set in the extra_nonce (so use this instead of
reserve_size).
* Add a `state_height` to RPC
The field contains the latest height at which the SN status changed:
that is, the height at which the last contribution was received (if
awaiting); the height of activation (if active and never
decommissioned); the height of decommissioning (if currently
decommissioned); or the height of last recommissioning if applicable.
This is need by the block explorer to be able to report useful
information on the current status.
* Add `funded` boolean to sn status response
* Add deregistration of checkpoints by checking how many votes are missed
Move uptime proofs and add checkpoint counts in the service_node_list
because we typically prune uptime proofs by time, but it seems we want
to switch to a model where we persist proof data until the node expires
otherwise currently we would prune uptime entries and potentially our
checkpoint vote counts which would cause premature deregistration as the
expected vote counts start mismatching with the number of received
votes.
* Revise deregistration
* Fix test breakages
* uint16_t for port, remove debug false, min votes to 2 in integration mode
* Fix integration build
The vast majority of the time `is_active()` is what we actually want.
Most of these are just a name change, but there is also one important
fix here to the next-winner list which needs to use active, not
fully-funded.
Jason Rhinelander
Doyle
cryptonote.social
Doyle
luigi1111
Alexander Blair
Sean Darcy
moneromooo-monero
Maxim Shishmarev
xiphon
naughtyfox
Lee Clagett
Riccardo Spagni
Nathan Dorfman
Howard Chu
Jesus Ramirez
Thomas Winget
sachaaaaa
Jethro Grassie