Manual merge because Loki already applied some of the changes regarding
- Locking in the mixin (we enforced this at the beginning of the chain)
- Enforcing miner TX version (enforced in HF12, we're now at HF15).
Lastly we manually merge restricting miner transactions from having
a RCT Signature, to be effective as of HF16.
High-level details:
This redesigns the RPC layer to make it much easier to work with,
decouples it from an embedded HTTP server, and gets the vast majority of
the RPC serialization and dispatch code out of a very commonly included
header.
There is unfortunately rather a lot of interconnected code here that
cannot be easily separated out into separate commits. The full details
of what happens here are as follows:
Major details:
- All of the RPC code is now in a `cryptonote::rpc` namespace; this
renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK
becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME`
becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already
working in the `rpc` namespace).
- `core_rpc_server` is now completely decoupled from providing any
request protocol: it is now *just* the core RPC call handler.
- The HTTP RPC interface now lives in a new rpc/http_server.h; this code
handles listening for HTTP requests and dispatching them to
core_rpc_server, then sending the results back to the caller.
- There is similarly a rpc/lmq_server.h for LMQ RPC code; more details
on this (and other LMQ specifics) below.
- RPC implementing code now returns the response object and throws when
things go wrong which simplifies much of the rpc error handling. They
can throw anything; generic exceptions get logged and a generic
"internal error" message gets returned to the caller, but there is
also an `rpc_error` class to return an error code and message used by
some json-rpc commands.
- RPC implementing functions now overload `core_rpc_server::invoke`
following the pattern:
RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context);
This overloading makes the code vastly simpler: all instantiations are
now done with a small amount of generic instantiation code in a single
.cpp rather than needing to go to hell and back with a nest of epee
macros in a core header.
- each RPC endpoint is now defined by the RPC types themselves,
including its accessible names and permissions, in
core_rpc_server_commands_defs.h:
- every RPC structure now has a static `names()` function that returns
the names by which the end point is accessible. (The first one is
the primary, the others are for deprecated aliases).
- RPC command wrappers define their permissions and type by inheriting
from special tag classes:
- rpc::RPC_COMMAND is a basic, admin-only, JSON command, available
via JSON RPC. *All* JSON commands are now available via JSON RPC,
instead of the previous mix of some being at /foo and others at
/json_rpc. (Ones that were previously at /foo are still there for
backwards compatibility; see `rpc::LEGACY` below).
- rpc::PUBLIC specifies that the command should be available via a
restricted RPC connection.
- rpc::BINARY specifies that the command is not JSON, but rather is
accessible as /name and takes and returns values in the magic epee
binary "portable storage" (lol) data format.
- rpc::LEGACY specifies that the command should be available via the
non-json-rpc interface at `/name` for backwards compatibility (in
addition to the JSON-RPC interface).
- some epee serialization got unwrapped and de-templatized so that it
can be moved into a .cpp file with just declarations in the .h. (This
makes a *huge* difference for core_rpc_server_commands_defs.h and for
every compilation unit that includes it which previously had to
compile all the serialization code and then throw all by one copy away
at link time). This required some new macros so as to not break a ton
of places that will use the old way putting everything in the headers;
The RPC code uses this as does a few other places; there are comments
in contrib/epee/include/serialization/keyvalue_serialization.h as to
how to use it.
- Detemplatized a bunch of epee/storages code. Most of it should have
have been using templates at all (because it can only ever be called
with one type!), and now it isn't. This broke some things that didn't
properly compile because of missing headers or (in one case) a messed
up circular dependency.
- Significantly simplified a bunch of over-templatized serialization
code.
- All RPC serialization definitions is now out of
core_rpc_server_commands_defs.h and into a single .cpp file
(core_rpc_server_commands_defs.cpp).
- core RPC no longer uses the disgusting
BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design
that forced slamming tons of code into a common header that didn't
need to be there.
- epee::struct_init is gone. It was a horrible hack that instiated
multiple templates just so the coder could be so lazy and write
`some_type var;` instead of properly value initializing with
`some_type var{};`.
- Removed a bunch of useless crap from epee. In particular, forcing
extra template instantiations all over the place in order to nest
return objects inside JSON RPC values is no longer needed, as are a
bunch of stuff related to the above de-macroization of the code.
- get_all_service_nodes, get_service_nodes, and get_n_service_nodes are
now combined into a single `get_service_nodes` (with deprecated
aliases for the others), which eliminates a fair amount of
duplication. The biggest obstacle here was getting the requested
fields reference passed through: this is now done by a new ability to
stash a context in the serialization object that can be retrieved by a
sub-serialized type.
LMQ-specifics:
- The LokiMQ instance moves into `cryptonote::core` rather than being
inside cryptonote_protocol. Currently the instance is used both for
qnet and rpc calls (and so needs to be in a common place), but I also
intend future PRs to use the batching code for job processing
(replacing the current threaded job queue).
- rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue.
Unlike http_server it isn't technically running the whole LMQ stack
from here, but the parallel name with http_server seemed appropriate.
- All RPC endpoints are supported by LMQ under the same names as defined
generically, but prefixed with `rpc.` for public commands and `admin.`
for restricted ones.
- service node keys are now always available, even when not running in
`--service-node` mode: this is because we want the x25519 key for
being able to offer CURVE encryption for lmq RPC end-points, and
because it doesn't hurt to have them available all the time. In the
RPC layer this is now called "get_service_keys" (with
"get_service_node_key" as an alias) since they aren't strictly only
for service nodes. This also means code needs to check
m_service_node, and not m_service_node_keys, to tell if it is running
as a service node. (This is also easier to notice because
m_service_node_keys got renamed to `m_service_keys`).
- Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and
`sub.mempool` subscribes the connection for new block and new mempool
TX notifications. The latter can notify on just blink txes, or all
new mempool txes (but only new ones -- txes dumped from a block don't
trigger it). The client gets pushed a [`notify.block`, `height`,
`hash`] or [`notify.tx`, `txhash`, `blob`] message when something
arrives.
Minor details:
- rpc::version_t is now a {major,minor} pair. Forcing everyone to pack
and unpack a uint32_t was gross.
- Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...).
(This immediately revealed a couple of bugs in the RPC code that was
assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because
the macro allows implicit conversion to a char).
- De-templatizing useless templates in epee (i.e. a bunch of templated
types that were never invoked with different types) revealed a painful
circular dependency between epee and non-epee code for tor_address and
i2p_address. This crap is now handled in a suitably named
`net/epee_network_address_hack.cpp` hack because it really isn't
trivial to extricate this mess.
- Removed `epee/include/serialization/serialize_base.h`. Amazingly the
code somehow still all works perfectly with this previously vital
header removed.
- Removed bitrotted, unused epee "crypted_storage" and
"gzipped_inmemstorage" code.
- Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with
LOKI_DEFERs. The epee version involves quite a bit more instantiation
and is ugly as sin. Also made the `loki::defer` class invokable for
some edge cases that need calling before destruction in particular
conditions.
- Moved the systemd code around; it makes much more sense to do the
systemd started notification as in daemon.cpp as late as possible
rather than in core (when we can still have startup failures, e.g. if
the RPC layer can't start).
- Made the systemd short status string available in the get_info RPC
(and no longer require building with systemd).
- during startup, print (only) the x25519 when not in SN mode, and
continue to print all three when in SN mode.
- DRYed out some RPC implementation code (such as set_limit)
- Made wallet_rpc stop using a raw m_wallet pointer
I started this PR to just fix the upgrades, but when investigating I
found some problematic memory leaks as well and ended up adding a bunch
of nice abstraction layers to make the code nicer.
I'd normally separate these into multiple commits, but they ended up
quite interdependent to the point where it isn't really feasible to do
so; but here's the details on the changes:
- add a `DEFAULT "register_height"` for the new "update_height" field so
that the alter table works
- reorganized the statement preparation and database migration so that
all statement preparation happens *after* migration; otherwise
statement preparation fails because some of the statements reference
columns that aren't created until the migration.
- created a `sql_compiled_statement` class that manages sqlite statement
pointers; there were several places that we were leaking them (the
main ones were okay, but we also have some dynamically constructed
ones for variable size queries).
- added a new abstraction layer around sqlite3 binding and value
extraction that makes the actual sql interaction code much less
verbose.
- removed the `nettype` argument from `sql_run_statement` as it wasn't
being used. (If something in the future needs it, the compiled
statement class has a reference to the db object, so it can be
obtained via `statement.nsdb.network_type()`)
- `bind(statement, 1, someval)` now does what
`sqlite3_bind_whatever(statement, 1, someval, ...)` did before, except
that it infers the "whatever" from the type (for everything except
blobs), and does whatever "..." needs to happen.
- `bind(statement, 1, blob_view{someval.data(), size})` can do the same
for a blob. (There is also `bind_blob(statement, 1, someval.data(),
size)` which does the same thing but is nicer in some contexts).
- `auto x = get<T>(statement, 1)` is the bind counterpart for extracting
a known type (and similarly calls the right sqlite function based on
`T`). There is also a `get(statement, 1, x);` that does the same
thing, but infers the type from whatever `x` is. `get_blob` similarly
gets a blob value, though currently this is only used in the existing
`sql_copy_blob()` wrapper.
- All the `bind` and `get` forms accept an integer enum class as the
position argument, so the code can now write just:
bind(statement, lns_db_setting_column::top_height, value);
instead of:
bind(statement, static_cast<int>(lns_db_setting_column::top_height), value);
- bind_all lets you bind all the parameters in one go (and also clears
existing bindings), letting you write things like:
bind_all(statement, 123, "my string", my_u64_int);
which previously would have needed:
sqlite3_clear_bindings(statement);
sqlite3_bind_int(statement, 1, 123);
sqlite3_bind_text(statement, 2, "my string", 10, nullptr, nullptr);
sqlite3_bind_int64(statement, 3, my_u64_int);
- Also there's a `bind_and_run` which combines this binding with a
sql_run_statement.
- put all of the new binding code in an anonymous namespace, along with
several existing `static` functions (this is equivalent, just easier
than having to write static dozens of times).
- removed a superfluous extra "OR" clause in get_mappings_by_owners. It
was producing queries such as:
... WHERE o1.address IN (?, ?, ?) OR o2.address IN (?, ?, ?) OR o2.address IN (?, ?, ?)
also simplified that part of the code down quite a bit with a
pre-allocated string for the "?, ?, ?" part.
- Simplified some some pointer/size pairs with lokimq::string_view
(which, internally, is just a pointer size pair already).
- Various methods got de-const'ed in this change. They probably
shouldn't have been const before because even though they only access
data, they still mutate internal state of the stored statements, and
making them all mutable would be a bit gross. (They did this before,
too, but since everything was pointers calling into C code I guess
they compiler just allowed it).
We allow arbitrary reorganizations for the LNS DB by storing the
previous TXID that purchased the LNS record. When reorganizing, we
iterate back through the blockchain looking for the first LNS entry that
is before the reorganize height and revert the LNS entry to that record.
Call service_node_lists's block added hook function manually instead of
hooking the hook. There's a common operation between the 2 subsystems,
Loki Name Service and Service Node List, in which they generate state
based off incoming blocks from the blockchain.
Upon profiling, the slowest factor of this is retrieving the blocks from
the DB in batches of a 1000 rather than the processing of the blocks in
the subsystems. So flatten out the hierarchy so that we can retrieve the
batch and simultaneously process the blocks in both subsystems on the
same fetch request.
A complete removal of the hook system so we have more flexibility at the
callsite can be done in a later PR, to avoid too many unneccesary
changes in the LNS PR.
Blockchain::prepare_handle_incoming_blocks locks m_tx_pool, but uses a
local RAII lock on the blockchain object itself, then also starts a
batch. Blockchain::cleanup_handle_incoming_blocks then also takes out a
local RAII blockchain lock, then cleans up the batch.
But the lmdb layer is retarded in that it throws an exception if any
thread tries to write to the database while a batch is active in another
thread, and so the blockchain lock is *also* used as a guard writes.
Holding an open batch but *not* holding the blockchain lock then hits
this exception if a write arrives in another thread at just the right
time.
This is, of course, terrible design at multiple layers, but this close
to release I am reluctant to make more drastic fixes.
Other small changes here:
- All the locks in `blockchain.cpp` now use tools::unique_lock or
tools::unique_locks rather than the nasty epee macro. This also
reduces the likelihood of accidental deadlock because this means the
dual txpool-blockchain locks are not taken out simultaneously via
std::lock rather than sequentially.
- Removed a completely useless "if (1)". Git blame shows that there was
previously a condition here, but apparently the upstream monero author
who changed it was afraid of removing the `if` keyword.
- Reduced the sleep in the loop that waits for a batch to 100ms from
1000ms because sleeping for a full second for a fairly light test is
insane.
- boost isn't happy calling boost::lock() on the tx pool or blockchain
object because the lock/unlock/try_lock methods are const, and so the
workaround of using boost::lock because std::lock and
std::shared_time_mutex are broken on the macOS SDK 10.11 that we use
for mac builds now requires extra workarounds. Joy.
This extracts uptime proof data entirely from service node states,
instead storing (some) proof data as its own first-class object in the
code and backed by the database. We now persistently store:
- timestamp
- version
- ip & ports
- ed25519 pubkey
and update it every time a proof is received. Upon restart, we load all
the proofs from the database, which means we no longer lose last proof
received times on restart, and always have the most recently received ip
and port information (rather than only having whatever it was in the
most recently stored state, which could be ~12 blocks ago). It also
means we don't need to spam the network with proofs for up to half an
hour after a restart because we now know when we last sent (and
received) our own proof before restarting.
This separation required some restructuring: most notably changing
`state_t` to be constructed with a `service_node_list` pointer, which it
needs both directly and for BlockchainDB access. Related to this is
also eliminating the separate BlockchainDB pointer stored in
service_node_list in favour of just accessing it via m_blockchain (which
now has a new `has_db()` method to detect when it has been initialized).
Extracts the tx removal lambda into a new method.
DRYs the nearly identical prune loop bodies (the only difference between
them is the iteration order).
Fix undefined behaviour in mempool pruning: on removal of a standard tx
the iterator was updated to the result of `.erase()`, but that's wrong
because the loop is going in backwards order, and so the first prune
would leave the iterator pointing at `.end()` which would be deferenced
in the next iteration. The (new, DRY) lambda now iterates in the
desired direction.
This also makes (and uses) blockchain and tx_pool implement the standard
library Lockable concept by adding a `try_lock()` so that they can be
locked simultaneously. It also updates them to hold a
boost::recursive_mutex directly rather than going through the pointless
and horribly designed epee `critical_section` wrapper (seriously who
thinks that making a mutex class copyable where a copy holds a new,
entirely unrelated mutex is a good idea?)
This adds blink tx synchronization: when doing a periodic sync with
other nodes each node includes a map of {HEIGHT, HASH} pairs where
HEIGHT is a mined, post-immutable block height and HASH is an xor of all
the tx hashes mined into that block.
If upon receipt the node disagrees about what it thinks the HASH should
be, it can request a list of txes for one or more disagreeing heights,
for which it gets a list of tx hashes of all blink txes mined into that
block.
If it is then missing any of those TXes, this adds an ability to request
that the remote send TXes via the existing NOTIFY_NEW_TRANSACTIONS
mechanism, but with an added flag (so that these don't get relayed).
This originally was going to request the TXes via the existing
NOTIFY_REQUEST_GET_OBJECTS, which has a `txs` field, but unfortunately
any txs passed to it are completely ignored; it is *only* usable for
block synchronization. As such I renamed it and removed the `txs` field
to make the responsibility/capability clearer.
- Simplify the algorithm (while keeping the results the same)
- Don't use a return value for a function that doesn't have a meaningful
return value
- Clarify and correct documentation of what it does
This adds the ability for check_fee() to also check the burn amount.
This requires passing extra info through `add_tx()` (and the various
things that call it), so I took the:
bool keeped_by_block, bool relayed, bool do_not_relay
argument triplet, moved it into a struct in tx_pool.h, then added the other fee
options there (along with some static factory functions for generating the
typical sets of option).
The majority of this commit is chasing that change through the codebase and
test suite.
This is used by blink but should also help LNS and other future burn
transactions to verify a burn amount simply when adding the transation to the
mempool. It supports a fixed burn amount, a burn amount as a multiple of the
minimum tx fee, and also allows you to increase the minimum tx fee (so that,
for example, we could require blink txes to pay miners 250% of the usual
minimum (unimportant) priority tx fee.
- Removed a useless core::add_new_tx() overload that wasn't used anywhere.
Blink-specific changes:
(I'd normally separate these into a separate commit, but they got interwoven
fairly heavily with the above change).
- changed the way blink burning is specified so that we have three knobs for
fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee).
The fixed amount is currently 0, base fee is 400%, and require miner tx fee is
simply 100% (i.e. no different than a normal transaction). This is the same as
before this commit, but is changing how they are being specified in
cryptonote_config.h.
- blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked
before signing a blink tx. (These fee checks don't apply to anyone else --
when propagating over the network only the miner tx fee is checked).
- Added a couple of checks for blink quorums: 1) make sure they have reached
the blink hf; 2) make sure the submitted tx version conforms to the current hf
min/max tx version.
- print blink fee information in simplewallet's `fee` output
- add "typical" fee calculations in the `fee` output:
[wallet T6SCwL (has locked stakes)]: fee
Current fee is 0.000000850 loki per byte + 0.020000000 loki per output
No backlog at priority 1
No backlog at priority 2
No backlog at priority 3
No backlog at priority 4
Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output
Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink)
where "small" here is the same tx size (2500 bytes + 2 outputs) used to
estimate backlogs.
* Add a 1 atomic unit fudge factor on reward calculations
If the thread's rounding mode is different (because RandomX changed it
in the current thread or in the thread that generated the block) then
the reward calculation can be up to 1 atomic unit off. This lets those
pass rather than rejecting the block.
(I tested LOKI's reward calculations up to height 10M to verify that the
maximum error in reward values is never off by more than 1).
* Remove partial_block_reward
This variable is not actually used anywhere.
* Add 1ULP tolerance for SN reward checks
* Rewrite confusing logic and error
- fix the broken error message which was printing "base(base+fees)" instead
of "total(base+fees)".
- improve the error message wording to actually indicate what the values
signify.
- don't use base_reward as a temporary value (the intermediate value
assigned to it is completely replaced later anyway).
- use some better variable names to describe what is happening here
* Add check for overflow
If the peer (whether pruned or not itself) supports sending pruned blocks
to syncing nodes, the pruned version will be sent along with the hash
of the pruned data and the block weight. The original tx hashes can be
reconstructed from the pruned txes and theur prunable data hash. Those
hashes and the block weights are hashes and checked against the set of
precompiled hashes, ensuring the data we received is the original data.
It is currently not possible to use this system when not using the set
of precompiled hashes, since block weights can not otherwise be checked
for validity.
This is off by default for now, and is enabled by --sync-pruned-blocks
Add migratory code for new alt_block_data_t.checkpointed field
Code review
Redo switch to alt chain logic to handle PoW properly
We always switch to the chain with prevailing checkpoints. If an old
chain reappears with more checkpoints it will be rebroadcasted and at
that point we will switch over then. So here we restore the old
behaviour regarding keeping alt chains around or not depending on the
scenario that caused the chain switch
Move service node list methods to state_t methods
Add querying state from alt blocks and put key image parsing into function
Incorporate hash into state_t to find alt states
Add a way to query alternate testing quorums
Rebase cleanup
