This replaces the current epee logging system with our oxen::log
library. It replaces the easylogging library with spdlog, removes the
macros and replaces with functions and standardises how we call the
logs.
Fixes a few merge conflicts, several compilation errors, and
some behavioral incorrectness. Still a few bugs with wallet3
but as far as I can tell wallet2 and daemon etc. should be working
correctly.
This was nasty.
Remove --block-rate-notify entirely; it's nearly useless on Oxen.
block and reorg notify remain, but now use a post-block hook rather than
shoving toxic Notify crap into cryptonote_core headers.
Add hooks that are called *after* a block is successfully added to the
blockchain.
This also fixes a race condition with OMQ notify.block subscriptions:
because the notification was firing during (instead of after) block
addition, the lokinet/storage server receiving the notify would race to
fetch the block info, but that request could happen *before* the block
addition is finished, ending up with lokinet/SS sometimes getting a
stale block.
This hook *isn't* called after a block is added, but rather it is part
of the block addition process and can abort the whole thing by raising
an exception (or returning false, prior to this PR).
This is unintuitive and causes bugs if using it as a "block has been
added" hook.
bool returns suck in general, but in most cases here they are also a
pain in the ass because *each* place that returns false is also issuing
a log statement. If only there were a way to return error information
to the common caller to have the common caller handle it... oh wait,
there is!
- Instead of inheriting from a pure virtual base class, we know just use
lambdas for hook callbacks.
- The signature of hooks also changes to take an info struct rather than
a list of parameters, so that you don't have to change everything to
ignore unwanted parameters if someone adds something to a hook.
With batching, individual blocks can have a negative coinbase emission
because the tx fee gets added to the batch rewards database and not paid
out immediately, which then results in an negative overflow to a value
close to 2^64. Thus a block with no payout and a tx fee will have an
erroneous huge positive coinbase emission when queried via
`get_coinbase_tx_sum`. For example block 1094068 queried with:
{"jsonrpc":"2.0","id":"0","method":"get_coinbase_tx_sum","params":{"height": 1094068, "count": 1}}
returns:
{
"jsonrpc": "2.0",
"id": "0",
"result": {
"burn_amount": 0,
"emission_amount": 18446744073699378616,
"fee_amount": 10173000,
"status": "OK"
}
}
This commit fixes it by making the values signed (and also serves as an
example of why unsigned integers are usually the wrong choice):
{
"jsonrpc": "2.0",
"id": "0",
"result": {
"burn_amount": 0,
"emission_amount": -10173000,
"fee_amount": 10173000,
"status": "OK"
}
}
If we pass the blob into parse_and_validate_tx_from_blob we get
`deserialization or varint failed` error logs, which are harmless bug
annoying. This catches the empty tx blob case earlier and logs at a
lesser severity.
On devnet we don't have a storage server, and so storage ports are left
uninitialized and effectively become random ports on the network. This
initializes them to 0, and avoids comparing SS ports for devnet for the
duplicate ip/ports warning.
- Replace all cryptonote_config macros with constexpr variables. Some
become integer types, some become chrono types.
- generally this involved removing a "CRYPTONOTE_" prefix since the
values are now in the `cryptonote` namespace
- some constants are grouped into sub-namespaces (e.g.
cryptonote::p2p)
- deprecated constants (i.e. for old HFs) are in the `cryptonote::old`
namespace.
- all the magic hash key domain separating strings are now in
cryptonote::hashkey::WHATEVER.
- Move some economy-related constants to oxen_economy.h instead
- Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more
straightforward `BLOCKS_PER_DAY` value (i.e. old
`BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`.
- Replaced `network_version` unscoped enum with a scoped enum
`cryptonote::hf`, replacing all the raw uint8_t values where it was
currently accepted with the new `hf` type.
- Made `network_type` a scoped enum so that it now has to be qualified
(network_type::TESTNET) and can't be arbitrarily/unintentionally
converted to/from an int.
- HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER
constexpr hf values.
- Add `revision` to rpc hard_fork_info response
- Don't build trezor code at all (previously we were pointlessly
building an empty dummy lib).
This updates the coinbase transactions to reward service nodes
periodically rather than every block. If you recieve a service node
reward this reward will be delayed x blocks, if you receive another
reward to the same wallet before those blocks have been completed it
will be added to your total and all will be paid out after those x
blocks has passed.
For example if our batching interval is 2 blocks:
Block 1 - Address A receives reward of 10 oxen - added to batch
Block 2 - Address A receives reward of 10 oxen - added to batch
Block 3 - Address A is paid out 20 oxen.
Batching accumulates a small reward for all nodes every block
The batching of service node rewards allows us to drip feed rewards
to service nodes. Rather than accruing each service node 16.5 oxen every
time they are pulse block leader we now reward every node the 16.5 /
num_service_nodes every block and pay each wallet the full amount that
has been accrued after a period of time (Likely 3.5 days).
To spread each payment evenly we now pay the rewards based on the
address of the recipient. This modulus of their address determines which
block the address should be paid and by setting the interval to our
service_node_batching interval we can guarantee they will be paid out
regularly and evenly distribute the payments for all wallets over this
Remove misc_language.h: Half of it is unused, half of it is crap doesn't
need to be used, and the two useful things (median calculator and a
scope exit caller) were poorly written.
Rewrote median from scratch and moved it out of epee.
Simplified the scope exit handler and moved it to its own small header
in epee.
openssl is a miserable dependency to fight with, especially for
iOS/Android, and we use it for very little:
- it gets (mis)used to write base64 data to a file in wallet2 for things
like multisig data. However this mode is *not* enabled by default,
basically completely unknown, completely unused, only exists in the
cli wallet, and is just dumb. (Honestly the justification given in
the PR is that "Windows users might want it", presupposing that there
exists Windows users who are capable of generating a multisig wallet
in a CLI-only application and yet are incapable of dealing with binary
files).
- it's a dependency of unbound (in order to do dnssec, I believe).
Unbound itself is fairly useless for Oxen, so I've removed it too:
- it does OpenAlias lookups, which are a Monero thing that has never
been used outside Monero, doesn't work reliably (because it fails
if the result isn't DNSSEC validated) and is pointless when we
have ONS.
- it does DNS resolution on seed nodes, but we have never set seed
nodes by name and just use seed node IPs instead (which seems a
bit better anyway since the DNS lookup leaks some metadata).
- it *was* being used for sha256, but an earlier commit in this PR
already replaced that with libsodium (entirely coincidentally).
- for static deps, it enables HTTPS support for the wallet. However
only the CLI wallet actually supports this (the GUI and mobile wallets
don't), and since oxend hasn't support this for a while I have strong
doubts it is being used anywhere. (Plus wallet3 will do everything
encrypted using zmq/libsodium, so doesn't need this to be secure).
Note here that it is *only* removed by this commit for static builds:
if doing a system build that links to libcurl supporting HTTPS then
HTTPS support will still work.
Libexpat is also gone because it was only there for libunbound.
All the encoding parts move to oxen-encoding recently; this updates to
the latest version of oxen-mq, adds oxen-encoding, and converts
everything to use oxenc headers rather than the oxenmq compatibility
shims.
This drops the dedicated GET_TRANSACTION_POOL endpoint entirely and
folds the txpool fetching into GET_TRANSACTIONS via a new `memory_pool`
parameter. (This is backwards incompatible, of course, but the results
may not be too harmful: the wallet doesn't actually use this endpoint,
so it mainly affects internal oxend commands (fixed here) and the block
explorer (which is fairly easily fixed).
This also removes some of the useless GET_TRANSACTIONS bits such as
decode_as_json.
`fee` and `burned` are now always provided in the return (for both
specific requested transactions and mempool transactions).
As a side effect, this code ventured deep into core/blockchain in terms
of dealing with missed transactions: previously they were always
returned (via lvalue ref) as a vector, and always had to be specified.
This changes to take via pointer to an unordered_set, which can be null
if you don't care about the missed ones. This reduces several calls
that indeed didn't care about collecting the missed values, and improved
virtually all the places that *did* care which need to query which ones
were missed rather than needing a specific order.
At MERROR they don't print at the default --log-level=0, so you get
output such as:
2021-08-06 18:04:08.073 I Starting up oxend services...
2021-08-06 18:04:08.073 I Starting core
2021-08-06 18:04:08.073 F Failed to start core
2021-08-06 18:04:08.173 I Deinitializing daemon objects...
with no actual error displayed as to why it failed to start. With this
change you get:
2021-08-06 18:04:14.004 F Please specify an IPv4 public address which the service node & storage server is accessible from with: '--service-node-public-ip <ip address>'
2021-08-06 18:04:14.004 F IMPORTANT: One or more required service node-related configuration settings/options were omitted or invalid; please fix them and restart oxend.
2021-08-06 18:04:14.004 F Failed to start core
- make checkpoint and pulse participation distinct types
- make the container act a little more like an stl container
- replace check_participation() with a failures() method that just
returns the number of failures (so the caller can decide whether that
is bad or not).
- As a consequence of the above, failures now trigger on N+1 failures,
while previously they required N+1 failures *and* 8 total responses.
There is no need to wait for all 8 as far as I can see since we will
be failing no matter what the next 3 responses are.
- Removed the serialization code (hopefully this doesn't break
everything outside the RPC code).
- Simplify/DRY the code that records participation
This code is bitrotting, doesn't compile, and isn't being maintained
anymore.
The integration test suite was an interesting idea, in early Loki days,
but is no longer being maintained and is quite cumbersome to run (for
instance, it is not possible to run it via CI because it depends on
xterm to actually run). The code to actually run it (in doy-lee's
loki-integration-testing repository) is also a large burden of "janky"
code that isn't worth maintaining.
Remove this from the code; if someone wants to pick it back up in the
future reverting this commit shouldn't be too difficult (though I'd
suggest that a much better approach to integration testing would be to
run different daemons/wallets via rpc commands, as the network-tests do,
rather than trying to feed stdin and parse stdout from running
individual oxends/wallets).
Snode revisions are a secondary version that let us put out a mandatory
update for snodes that isn't a hardfork (and so isn't mandatory for
wallets/exchanges/etc.).
The main point of this is to let us make a 9.2.0 release that includes
new mandatory minimums of future versions of storage server (2.2.0) and
lokinet (0.9.4) to bring upgrades to the network.
This slightly changes the HF7 blocks to 0 (instead of 1) because,
apparently, we weren't properly checking the HF value of the
pre-first-hf genesis block at all before. (In practice this changes
nothing because genesis blocks are v7 anyway).
This also changes (slightly) how we check for hard forks: now if we skip
some hard forks then we still want to know the height when a hard fork
triggers. For example, if the hf tables contains {7,14} then we still
need to know that the HF14 block height also is the height that
activates HF9, 10, etc.
It works just like storage server testing.
Renames the report_peer_storage_server_status to report_peer_status, and
repurposes the code to handle both SS and lokinet.
This *doesn't* need a HF by design because the reason bit field was
deliberately designed so that we can add reason fields (older clients
will just ignore unknown bits).
bt-encoded proofs have a bug for nodes with different legacy/ed25519
pubkeys that isn't easily solvable (it needs a fix on *both* sender and
receiver ends). That fix is in here (in uptime_proof.cpp) but that
isn't enough to solve it immediately.
This works around the issue by submitting old-style proofs if we are a
node with different legacy/ed25519 pubkeys.
Improves the oxend<->storage server communications protocol:
- pass storage server HTTPS port as part of the storage server ping
(which already carries the also-required OMQ port) rather than needing
to provide it when starting up oxend. --storage-server-port is now
obsolete (and ignored, if specified).
- Fix up the internal API to use `storage_https_port` and
`storage_omq_port` rather than `storage_port` and `storage_lmq_port`.
- Redo and the SS ping RPC endpoint so that it is less verbose and more
closely matches the lokinet endpoint; instead of:
{ "version_major": 2, "version_minor": 0, "version_patch": 9, "storage_lmq_port": 22222 }
we now expect:
{ "version": [2,0,9], "https_port": 11111, "omq_port": 22222 }
- Tweaks the (not-yet-released) SS proof key names: "s"->"shp" and "slp"->"sop"
m_nettype won't be set properly during construction so we can't call
get_net_config() in the constructor arguments here; set a reasonable
value and then update during actual initialization instead.
This makes uptime proof times network-dependent, and tweaks them a bit.
Also converts the times to type-safe std::chrono types rather than
macros.
Mainnet/testnet/devnet:
- Send the first proof 30s after startup rather than waiting 2 minutes.
- Check for whether we need to send a proof every 30s rather than every
5mins.
Mainnet:
Other times unchanged.
Testnet/devnet:
- Send proofs every 10min instead of 1h, and consider nodes to be down
after 21m instead of 2h5m.
Fakechain:
- Send 5s after startup, check every 5s, and send every 1min.
- Expiry after 2min5s
Also remove the cmake debug option for short proofs since the fakechain
changes above basically incorporate what it did.
The coinbase tx sum rpc call had an off-by-one error that made it
double-count a block the first time it was reloaded at any height. This
caused the deviation of oxen.observer and lokiblocks.com emissions
counts (because one is being called more frequently than the other), and
caused *both* to grow too quickly over time.
Sean Darcy
Jason Rhinelander
Thomas Winget
Sean Darcy