- Replace all cryptonote_config macros with constexpr variables. Some
become integer types, some become chrono types.
- generally this involved removing a "CRYPTONOTE_" prefix since the
values are now in the `cryptonote` namespace
- some constants are grouped into sub-namespaces (e.g.
cryptonote::p2p)
- deprecated constants (i.e. for old HFs) are in the `cryptonote::old`
namespace.
- all the magic hash key domain separating strings are now in
cryptonote::hashkey::WHATEVER.
- Move some economy-related constants to oxen_economy.h instead
- Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more
straightforward `BLOCKS_PER_DAY` value (i.e. old
`BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`.
- Replaced `network_version` unscoped enum with a scoped enum
`cryptonote::hf`, replacing all the raw uint8_t values where it was
currently accepted with the new `hf` type.
- Made `network_type` a scoped enum so that it now has to be qualified
(network_type::TESTNET) and can't be arbitrarily/unintentionally
converted to/from an int.
- HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER
constexpr hf values.
- Add `revision` to rpc hard_fork_info response
- Don't build trezor code at all (previously we were pointlessly
building an empty dummy lib).
This updates the coinbase transactions to reward service nodes
periodically rather than every block. If you recieve a service node
reward this reward will be delayed x blocks, if you receive another
reward to the same wallet before those blocks have been completed it
will be added to your total and all will be paid out after those x
blocks has passed.
For example if our batching interval is 2 blocks:
Block 1 - Address A receives reward of 10 oxen - added to batch
Block 2 - Address A receives reward of 10 oxen - added to batch
Block 3 - Address A is paid out 20 oxen.
Batching accumulates a small reward for all nodes every block
The batching of service node rewards allows us to drip feed rewards
to service nodes. Rather than accruing each service node 16.5 oxen every
time they are pulse block leader we now reward every node the 16.5 /
num_service_nodes every block and pay each wallet the full amount that
has been accrued after a period of time (Likely 3.5 days).
To spread each payment evenly we now pay the rewards based on the
address of the recipient. This modulus of their address determines which
block the address should be paid and by setting the interval to our
service_node_batching interval we can guarantee they will be paid out
regularly and evenly distribute the payments for all wallets over this
Remove misc_language.h: Half of it is unused, half of it is crap doesn't
need to be used, and the two useful things (median calculator and a
scope exit caller) were poorly written.
Rewrote median from scratch and moved it out of epee.
Simplified the scope exit handler and moved it to its own small header
in epee.
All the encoding parts move to oxen-encoding recently; this updates to
the latest version of oxen-mq, adds oxen-encoding, and converts
everything to use oxenc headers rather than the oxenmq compatibility
shims.
Snode revisions are a secondary version that let us put out a mandatory
update for snodes that isn't a hardfork (and so isn't mandatory for
wallets/exchanges/etc.).
The main point of this is to let us make a 9.2.0 release that includes
new mandatory minimums of future versions of storage server (2.2.0) and
lokinet (0.9.4) to bring upgrades to the network.
This slightly changes the HF7 blocks to 0 (instead of 1) because,
apparently, we weren't properly checking the HF value of the
pre-first-hf genesis block at all before. (In practice this changes
nothing because genesis blocks are v7 anyway).
This also changes (slightly) how we check for hard forks: now if we skip
some hard forks then we still want to know the height when a hard fork
triggers. For example, if the hf tables contains {7,14} then we still
need to know that the HF14 block height also is the height that
activates HF9, 10, etc.
A couple people have messaged me now because they tried transferring
keys and used the `restore` command on their legacy `key` file, but it
restored an ed25519 key.
This adds a red warning if attempting to restore an ed key to a filename
ending in `/key` with a note about probably wanting the restore-legacy
command instead.
We shouldn't actually use it in `main.cpp` because it is called before
the log system is initialized, and it is a wrapper that saves basically
nothing, so just replace it everywhere with direct calls to
fs::create_directories and delete it.
- Remove some useless epee functions, and add deprecated markers to ones
that have good replacements already.
- Don't use boost::lexical_cast when std::to_string or direct stream
output can be used just as well.
- Get rid of dumb epee "include_base_utils.h" header
Converts all use of boost::filesystem to std::filesystem.
For macos and potentially other exotic systems where std::filesystem
isn't available, we use ghc::filesystem instead (which is a drop-in
replacement for std::filesystem, unlike boost::filesystem).
This also greatly changes how we handle filenames internally by holding
them in filesystem::path objects as soon as possible (using
fs::u8path()), rather than strings, which avoids a ton of issues around
unicode filenames. As a result this lets us drop the boost::locale
dependency on Windows along with a bunch of messy Windows ifdef code,
and avoids the need for doing gross boost locale codecvt calls.
When targetting macos <10.14 macos won't allow use of anything from
C++17 that throws, such as:
- std::get on a variant
- std::visit
- std::optional::value()
- std::any_cast
This avoids all of these.
For std::get, we either replace with std::get_if (where appropriate), or
else use a `var::get` implementation of std::get added to lokimq (also
updated here). (This `var` namespace is just an `std` alias everywhere
*except* old target macos).
For std::visit, likewise lokimq adds an var::visit implementation for
old macos that we use.
std::optional::value() uses weren't useful anyway as everywhere it calls
them we've already checked that the option has a value, in which case we
can use `*opt` (which doesn't check for contents and throw).
std::any just has to be avoided as far as I can tell, but the one place
we used it is only ever a block, so I just replaced it with a `const
block*`.
We always rescan from 0 because of a difficulty bug, so instead just
move it to blockchain_import where you might occasionally want
fine-tuned access to it.
This allows inspecting, generating, and restoring both Ed25519 and
legacy (naming in line with PR #1240 now avoiding the non-Ed25519 keys
by default) secret key files as needed by lokid:
$ ./loki-sn-keys generate key_ed25519
Generated SN Ed25519 secret key in key_ed25519
Public key: 8f94f6bbc1d5876484309fc7fd41396c6bd66c4631a34f04895887547cb374dd
X25519 pubkey: f3e42d1aff8db8232433ec86e3f56cdda44d4c510144f9b6786520175b08c97f
Lokinet address: t6kxpq6b4sdsjbbou9d94oj3pti7c5ngggtw6brjmndie9fuquqo.snode
$ ./loki-sn-keys legacy key
Generated SN legacy private key in key
Public key: f3a513ddfbe946c79c62c27cca2260b80606a2da1b4f75ba16c793a5750a8510
$ ./loki-sn-keys show key
key (legacy SN keypair)
==========
Private key: c8c17cc296c3e8bb603098b5d2535ad86662cdb24cd4c785c0485d0a490a2356
Public key: f3a513ddfbe946c79c62c27cca2260b80606a2da1b4f75ba16c793a5750a8510
$ ./loki-sn-keys show key_ed25519
key_ed25519 (Ed25519 SN keypair)
==========
Secret key: 3f078900bac5f3e97d5fe451a6e332826d29aae2e13fec895c1b527af88093c3
Public key: 8f94f6bbc1d5876484309fc7fd41396c6bd66c4631a34f04895887547cb374dd
X25519 pubkey: f3e42d1aff8db8232433ec86e3f56cdda44d4c510144f9b6786520175b08c97f
Lokinet address: t6kxpq6b4sdsjbbou9d94oj3pti7c5ngggtw6brjmndie9fuquqo.snode
$ ./loki-sn-keys restore key_ed25519-2
Enter the Ed25519 secret key:
3f078900bac5f3e97d5fe451a6e332826d29aae2e13fec895c1b527af88093c3
Public key: 8f94f6bbc1d5876484309fc7fd41396c6bd66c4631a34f04895887547cb374dd
X25519 pubkey: f3e42d1aff8db8232433ec86e3f56cdda44d4c510144f9b6786520175b08c97f
Lokinet address: t6kxpq6b4sdsjbbou9d94oj3pti7c5ngggtw6brjmndie9fuquqo.snode
Is this correct? Press Enter to continue, Ctrl-C to cancel.
Saved secret key to key_ed25519-2
$ ./loki-sn-keys restore-legacy key-2
Enter the legacy SN private key:
c8c17cc296c3e8bb603098b5d2535ad86662cdb24cd4c785c0485d0a490a2356
Public key: f3a513ddfbe946c79c62c27cca2260b80606a2da1b4f75ba16c793a5750a8510
Is this correct? Press Enter to continue, Ctrl-C to cancel.
Saved secret key to key-2
Because:
boost::join(v | boost::adaptors::transformed([](uint64_t out){return std::to_string(out);}), " ")
is ugly as sin, while:
tools::join(" ", v)
is nice and simple.
Also removes a few unnecessary boost iterator adaptor includes and uses.
- pessimizing move in wallet2 prevents copy ellision
- various for loops were creating copies (clang-10 now warns about
this). Mostly this is because they had the type wrong when looping
through a map: the iterator type of a `map<K, V>` is `pair<const K,
V>` not `pair<K, V>`. Replaced them with C++17:
for (const auto& [key, val] : var)
which is so much nicer.
- cryptonote::core did not have a virtual destructor, but had virtual
methods (causing both a warning, and likely a crash if we ever have
something inheriting from it held in a unique_ptr<core>).
- core() constructor still had explicit even though it lost the single
argument.
- test code class had a `final` destructor but wasn't marked final. (It
also has a billion superfluous `virtual` declarations but I left them
in place because it's just test code).
common/util.h has become something of a dumping ground of random
functions. This splits them up a little by moving the filesystem bits
to common/file.h, the sha256sum functions to common/sha256sum.h, and the
(singleton) signal handler to common/signal_handler.h.
Add number parsing and basic string splitting to common/string_util.h
and use it (replacing some regexes and boost string utilities).
Includes unit tests.
A huge amount of this is repetitive:
- `boost::get<T>(variant)` becomes `std::get<T>(variant)`
- `boost::get<T>(variant_ptr)` becomes `std::get_if<T>(variant_ptr)`
- `variant.type() == typeid(T)` becomes `std::holds_alternative<T>(variant)`
There are also some simplifications to visitors using simpler stl
visitors, or (simpler still) generic lambdas as visitors.
Also adds boost serialization serializers for std::variant and
std::optional.
High-level details:
This redesigns the RPC layer to make it much easier to work with,
decouples it from an embedded HTTP server, and gets the vast majority of
the RPC serialization and dispatch code out of a very commonly included
header.
There is unfortunately rather a lot of interconnected code here that
cannot be easily separated out into separate commits. The full details
of what happens here are as follows:
Major details:
- All of the RPC code is now in a `cryptonote::rpc` namespace; this
renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK
becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME`
becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already
working in the `rpc` namespace).
- `core_rpc_server` is now completely decoupled from providing any
request protocol: it is now *just* the core RPC call handler.
- The HTTP RPC interface now lives in a new rpc/http_server.h; this code
handles listening for HTTP requests and dispatching them to
core_rpc_server, then sending the results back to the caller.
- There is similarly a rpc/lmq_server.h for LMQ RPC code; more details
on this (and other LMQ specifics) below.
- RPC implementing code now returns the response object and throws when
things go wrong which simplifies much of the rpc error handling. They
can throw anything; generic exceptions get logged and a generic
"internal error" message gets returned to the caller, but there is
also an `rpc_error` class to return an error code and message used by
some json-rpc commands.
- RPC implementing functions now overload `core_rpc_server::invoke`
following the pattern:
RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context);
This overloading makes the code vastly simpler: all instantiations are
now done with a small amount of generic instantiation code in a single
.cpp rather than needing to go to hell and back with a nest of epee
macros in a core header.
- each RPC endpoint is now defined by the RPC types themselves,
including its accessible names and permissions, in
core_rpc_server_commands_defs.h:
- every RPC structure now has a static `names()` function that returns
the names by which the end point is accessible. (The first one is
the primary, the others are for deprecated aliases).
- RPC command wrappers define their permissions and type by inheriting
from special tag classes:
- rpc::RPC_COMMAND is a basic, admin-only, JSON command, available
via JSON RPC. *All* JSON commands are now available via JSON RPC,
instead of the previous mix of some being at /foo and others at
/json_rpc. (Ones that were previously at /foo are still there for
backwards compatibility; see `rpc::LEGACY` below).
- rpc::PUBLIC specifies that the command should be available via a
restricted RPC connection.
- rpc::BINARY specifies that the command is not JSON, but rather is
accessible as /name and takes and returns values in the magic epee
binary "portable storage" (lol) data format.
- rpc::LEGACY specifies that the command should be available via the
non-json-rpc interface at `/name` for backwards compatibility (in
addition to the JSON-RPC interface).
- some epee serialization got unwrapped and de-templatized so that it
can be moved into a .cpp file with just declarations in the .h. (This
makes a *huge* difference for core_rpc_server_commands_defs.h and for
every compilation unit that includes it which previously had to
compile all the serialization code and then throw all by one copy away
at link time). This required some new macros so as to not break a ton
of places that will use the old way putting everything in the headers;
The RPC code uses this as does a few other places; there are comments
in contrib/epee/include/serialization/keyvalue_serialization.h as to
how to use it.
- Detemplatized a bunch of epee/storages code. Most of it should have
have been using templates at all (because it can only ever be called
with one type!), and now it isn't. This broke some things that didn't
properly compile because of missing headers or (in one case) a messed
up circular dependency.
- Significantly simplified a bunch of over-templatized serialization
code.
- All RPC serialization definitions is now out of
core_rpc_server_commands_defs.h and into a single .cpp file
(core_rpc_server_commands_defs.cpp).
- core RPC no longer uses the disgusting
BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design
that forced slamming tons of code into a common header that didn't
need to be there.
- epee::struct_init is gone. It was a horrible hack that instiated
multiple templates just so the coder could be so lazy and write
`some_type var;` instead of properly value initializing with
`some_type var{};`.
- Removed a bunch of useless crap from epee. In particular, forcing
extra template instantiations all over the place in order to nest
return objects inside JSON RPC values is no longer needed, as are a
bunch of stuff related to the above de-macroization of the code.
- get_all_service_nodes, get_service_nodes, and get_n_service_nodes are
now combined into a single `get_service_nodes` (with deprecated
aliases for the others), which eliminates a fair amount of
duplication. The biggest obstacle here was getting the requested
fields reference passed through: this is now done by a new ability to
stash a context in the serialization object that can be retrieved by a
sub-serialized type.
LMQ-specifics:
- The LokiMQ instance moves into `cryptonote::core` rather than being
inside cryptonote_protocol. Currently the instance is used both for
qnet and rpc calls (and so needs to be in a common place), but I also
intend future PRs to use the batching code for job processing
(replacing the current threaded job queue).
- rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue.
Unlike http_server it isn't technically running the whole LMQ stack
from here, but the parallel name with http_server seemed appropriate.
- All RPC endpoints are supported by LMQ under the same names as defined
generically, but prefixed with `rpc.` for public commands and `admin.`
for restricted ones.
- service node keys are now always available, even when not running in
`--service-node` mode: this is because we want the x25519 key for
being able to offer CURVE encryption for lmq RPC end-points, and
because it doesn't hurt to have them available all the time. In the
RPC layer this is now called "get_service_keys" (with
"get_service_node_key" as an alias) since they aren't strictly only
for service nodes. This also means code needs to check
m_service_node, and not m_service_node_keys, to tell if it is running
as a service node. (This is also easier to notice because
m_service_node_keys got renamed to `m_service_keys`).
- Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and
`sub.mempool` subscribes the connection for new block and new mempool
TX notifications. The latter can notify on just blink txes, or all
new mempool txes (but only new ones -- txes dumped from a block don't
trigger it). The client gets pushed a [`notify.block`, `height`,
`hash`] or [`notify.tx`, `txhash`, `blob`] message when something
arrives.
Minor details:
- rpc::version_t is now a {major,minor} pair. Forcing everyone to pack
and unpack a uint32_t was gross.
- Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...).
(This immediately revealed a couple of bugs in the RPC code that was
assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because
the macro allows implicit conversion to a char).
- De-templatizing useless templates in epee (i.e. a bunch of templated
types that were never invoked with different types) revealed a painful
circular dependency between epee and non-epee code for tor_address and
i2p_address. This crap is now handled in a suitably named
`net/epee_network_address_hack.cpp` hack because it really isn't
trivial to extricate this mess.
- Removed `epee/include/serialization/serialize_base.h`. Amazingly the
code somehow still all works perfectly with this previously vital
header removed.
- Removed bitrotted, unused epee "crypted_storage" and
"gzipped_inmemstorage" code.
- Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with
LOKI_DEFERs. The epee version involves quite a bit more instantiation
and is ugly as sin. Also made the `loki::defer` class invokable for
some edge cases that need calling before destruction in particular
conditions.
- Moved the systemd code around; it makes much more sense to do the
systemd started notification as in daemon.cpp as late as possible
rather than in core (when we can still have startup failures, e.g. if
the RPC layer can't start).
- Made the systemd short status string available in the get_info RPC
(and no longer require building with systemd).
- during startup, print (only) the x25519 when not in SN mode, and
continue to print all three when in SN mode.
- DRYed out some RPC implementation code (such as set_limit)
- Made wallet_rpc stop using a raw m_wallet pointer
Removes all "using namespace epee;" and "using namespace std;" from the
code and fixes up the various crappy places where unnamespaced types
were being used.
Also removes the ENDL macro (which was defined to be `std::endl`)
because it is retarded, and because even using std::endl instead of a
plain "\n" is usually a mistake (`<< std::endl` is equivalent to `<<
"\n" << std::flush`, and that explicit flush is rarely desirable).
We have never had decomposed amounts. (And I think the stuck v1
transactions are stuck precisely *because* they are v1 transactions
which are not valid decomposed amounts).
