This adds a new table to the batching schema to copy the accrued
balances every so often. This means that we can jump back to a
previous point without popping blocks.
The archiving is triggered in sql every 100 blocks and added to the
archive table, then pruned from the archive table at a later time to
ensure the size is kept small. Rebuilding 100 blocks is pretty
reasonable and should be less than 10s.
For longer distance pop_blocks and blockchain detached every 10k blocks
is kept in the archiving table. This takes longer to rebuild but is
better than rebuilding from scratch.
The blockchain detached function is also added to our regular blockchain
detached hooks so that it gets called every time the blockchain
detaches. Which appears to have caused some issues previously when some
of the modules would detach but batching would be stuck in an advanced
state.
This replaces the current epee logging system with our oxen::log
library. It replaces the easylogging library with spdlog, removes the
macros and replaces with functions and standardises how we call the
logs.
Profiling shows noticeable CPU spend in allocating memory for this
vector (which makes sense, since we are looping through ~1700 nodes and
building a reward vector for each one). Avoid it by reusing a single
vector that gets cleared (but not reallocated more than a handful of
times).
This reduces batching CPU time in a debug build by about 12%; curiously
I didn't find a noticeable reduction in a release build.
Converting between internal addresses and string wallet addresses is
surprisingly expensive (partly because a keccak hash is involved);
normally on the blockchain that doesn't matter because string
conversions are rare, but batching exposed some code paths that do the
conversion a lot.
This makes two optimizations to considerably speed up batching
processing time:
1. Add an address string hash for batch addresses. We don't see that
many, but we tend to see the same addresses for *every* block. By
caching it we can considerably reduce the time needed to do the
actual conversions.
2. In many places we don't really need strings at all: we can work just
with the address_infos (which are cheap). This eliminates some of
the places we use addresses and pushes address_infos further through
the code.
In total, this plus the previous optimization drop batch processing time
by this cuts batch processing CPU time by about 85%, roughly half from
this optimization and roughly half from the store-offsets optimization.
On a debug build, timing mainnet resyncing from just before HF19 to
current:
- neither optimization: 223.60s spent processing batch jobs (of 282s
total recalculating subsystems).
- just offsets: 131s spent in batch processing
- both optimizations: 33s spent in batch processing (down to 60s total
recalculating).
On a release build:
- before: 107.35s (4.73s snl; 95.55s batch)
- after: 28.27s (4.66s snl; 0.00s ons; 21.72s batch)
We don't do anything differently for a nullopt versus an empty vector,
so just return an empty vector for both not-at-the-hf and no-payments
cases.
(This also fixes a segfault and/or chainsync bug in the previous commit
because the optional is dereferenced without checking when on hf19).
Using SQLite::Statement directly requires sqlite to re-prepare the
statement every time it is executed; going via our `prepared_st` wrapper
caches (per thread) and reuses previously prepared statements for better
performance.
(It also, in some cases, simplifies the code a bit).
This allows a user to query the daemon and ask for the accrued balance
of oxen rewards that have been batched but not yet paid out.
The endpoint takes in an array of addresses (Strings) and returns an
array of atomic oxen amounts (integers) that reflect the current
database balances for those requested addresses.
For example an address has accrued 2x batch payments of 16.5 oxen but
has not yet been paid out will show 33000000000 being the atomic amount
in the database.
- Replace all cryptonote_config macros with constexpr variables. Some
become integer types, some become chrono types.
- generally this involved removing a "CRYPTONOTE_" prefix since the
values are now in the `cryptonote` namespace
- some constants are grouped into sub-namespaces (e.g.
cryptonote::p2p)
- deprecated constants (i.e. for old HFs) are in the `cryptonote::old`
namespace.
- all the magic hash key domain separating strings are now in
cryptonote::hashkey::WHATEVER.
- Move some economy-related constants to oxen_economy.h instead
- Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more
straightforward `BLOCKS_PER_DAY` value (i.e. old
`BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`.
- Replaced `network_version` unscoped enum with a scoped enum
`cryptonote::hf`, replacing all the raw uint8_t values where it was
currently accepted with the new `hf` type.
- Made `network_type` a scoped enum so that it now has to be qualified
(network_type::TESTNET) and can't be arbitrarily/unintentionally
converted to/from an int.
- HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER
constexpr hf values.
- Add `revision` to rpc hard_fork_info response
- Don't build trezor code at all (previously we were pointlessly
building an empty dummy lib).
This updates the coinbase transactions to reward service nodes
periodically rather than every block. If you recieve a service node
reward this reward will be delayed x blocks, if you receive another
reward to the same wallet before those blocks have been completed it
will be added to your total and all will be paid out after those x
blocks has passed.
For example if our batching interval is 2 blocks:
Block 1 - Address A receives reward of 10 oxen - added to batch
Block 2 - Address A receives reward of 10 oxen - added to batch
Block 3 - Address A is paid out 20 oxen.
Batching accumulates a small reward for all nodes every block
The batching of service node rewards allows us to drip feed rewards
to service nodes. Rather than accruing each service node 16.5 oxen every
time they are pulse block leader we now reward every node the 16.5 /
num_service_nodes every block and pay each wallet the full amount that
has been accrued after a period of time (Likely 3.5 days).
To spread each payment evenly we now pay the rewards based on the
address of the recipient. This modulus of their address determines which
block the address should be paid and by setting the interval to our
service_node_batching interval we can guarantee they will be paid out
regularly and evenly distribute the payments for all wallets over this
