This makes reachability testing activate at HF19. We probably want to
come back and update this before HF19, but for now we just check but
don't enforce lokinet reachability.
It works just like storage server testing.
Renames the report_peer_storage_server_status to report_peer_status, and
repurposes the code to handle both SS and lokinet.
This *doesn't* need a HF by design because the reason bit field was
deliberately designed so that we can add reason fields (older clients
will just ignore unknown bits).
This moves all the responsibility of ping testing (deciding when it's
unreachable, etc.) into oxend, allowing for better reporting on SS ping
results and eliminating some edge cases that can lead to oxend and
storage server getting "stuck" thinking each is in a different state.
This makes uptime proof times network-dependent, and tweaks them a bit.
Also converts the times to type-safe std::chrono types rather than
macros.
Mainnet/testnet/devnet:
- Send the first proof 30s after startup rather than waiting 2 minutes.
- Check for whether we need to send a proof every 30s rather than every
5mins.
Mainnet:
Other times unchanged.
Testnet/devnet:
- Send proofs every 10min instead of 1h, and consider nodes to be down
after 21m instead of 2h5m.
Fakechain:
- Send 5s after startup, check every 5s, and send every 1min.
- Expiry after 2min5s
Also remove the cmake debug option for short proofs since the fakechain
changes above basically incorporate what it did.
- A complete fix would involve setting the DIFFICULTY_WINDOW to
1 unified constant for both pre/post HF16 code. My attempts to get this
to work have not succeeded and so I'm putting this aside to allocate
more time on getting Pulse ready for testnet.
For completeness sake, I'll be copying verbatim the changes requested below.
Of the changes requested only adjusting the DIFFICULTY_WINDOW to 59 was
done (with the appropriate offset changes to the other constants that
interact with it).
The changes requested by jagerman
(from https://github.com/loki-project/loki-core/pull/1235)
Two points about this analysis:
1. Most of the comments here belong in the commit message rather than
code comments.
2. Well before I joined LOKI I worked with zawy for a good bit of time
on LWMA implementations, and so I found this description very confusing
because adding 1 is the correct thing to do for an LWMA. You need 61
observations to do a LWMA 60 because you need solve times where are
timestamp differences of adjacent blocks.
However what you describe above is also correct. So then I investigated
the disparity, and figured it out:
We don't do a proper LWMA(60) because of this: N = (DIFFICULTY_WINDOW - 1)
That is just wrong: it means we are doing a LWMA(59). But even that
isn't quite right because the resize(N+1) done in next_difficulty_v2
also throws away the top value. So what we end up calculating is
a LWMA(59) with a lag of one (that is, we throw away the most recent
usable observation).
The fix in this commit is making the math line up, but in not quite the
right away because there is still a bunch of stuff that is off by one.
Instead what we need is:
DIFFICULTY_WINDOW_V2 should be changed to 59, since that's what it is
(and has always) actually done.
DIFFICULTY_BLOCKS_COUNT_V2 should remain as DIFFICULTY_WINDOW_V2 + 1 because that's the proper number of
required blocks needed for a LWMA(DIFFICULTY_WINDOW_V2).
- the -1 should be removed from next_difficulty_v2 (so that it remains at 59, with the
above change).
- if we're on HF15 or earlier then we still need
DIFFICULTY_BLOCKS_COUNT_V2 blocks, but because of this bug they need
to be the values from blocks (H-61) through (H-2) rather than (H-60)
through (H-1).
- The // TODO: put asserts here, so that the difficulty algorithm is never
called with an oversized window (removed in this commit) should be
actually done as it should work fine with these fixes.
- Pulse blocks will forcibly get the difficulty set to
1'000'000 * TARGET_BLOCK_TIME throughout time
- When PoW is required again, the past window of blocks will use these
difficulties, i.e. setup the chain for mining at 1'000'000 difficulty
which is easily mineable to continue the network and continue to pull
difficulties from the new-er mined blocks until the network is ready
for Pulse again.
- Difficulty is still necessary for falling back to mining when Pulse
fails. Switching between the two systems seamlessly can be done by
continuing to set the difficulty for Pulse blocks.
- Keep none for blocks
- Remove HF_VERSION_PULSE and prefer network_version_16 for big features
- Keep HF_VERSION_... for smaller feature flags
- Add operator< for pulse_sort_key
- Avoid erase() in validators gen loop for pulse
Replace stagenet parameters with new devnet parameters:
- Wallet prefix is now dV[123] for regular wallets, dV[ABC] for
integrated addresses, and dV[abc] for subaddresses. Wallet length is
the same as testnet (i.e. 2 longer than mainnet).
- devnet has a single v7 genesis block (because change this is too
intrusive) and then immediately jumps to v15 at height 2.
- Moved default ports from 3805x to 3885x (to avoid conflicting with
potential existing stagenet daemons)
- DEVNET staking requirement is 1/10 of the mainnet requirement
beginning at height 600000. i.e. devnet staking requirement at height
1234 == 1/10 of mainnet requirement at height 601234. (This makes
lets devnet test out the decreasing stake, which testnet's fixed
requirement has never allowed).
This prevents staking transactions from being accepted if they overstake
the available contribution room by more than 1%. This is to prevent a
case that has happened a few times where there are competing partial
stakes submitted for the same SN at the same time (i.e. before a block
gets mined with the stakes). For example:
- Operator registers service node with 30% contribution
- Staker 1 submits stake with 40% contribution
- Staker 2 submits stake with 60% contribution
The wallet avoids stake 2 if the 40% has been accepted into a block, but
doesn't if it is still in the mempool. Later, when the contributions
get mined, both stakes are admitted because whichever one goes first
doesn't complete the stake, and the second one is still valid (since
there is a spot, and since it contributes >= the required amount).
Whichever stake gets added to a block second, however, will only be
counted as a contribution of the available amount. So, for example, if
stake 1 gets added first and then stake 2 gets added you'll end up with
an active service node of:
- operator has 30% contributed and locked
- staker 1 has 40% contributed and locked
- staker 2 has 30% contributed but 60% locked.
This commit adds an upper bound for an acceptable stake that is 101% of
the available contribution room so that, in the above situation,
whichever stake gets added first will be a contribution and the second
one will fall through as an ordinary transaction back to the staker's
wallet so that the staker the re-contribute the proper amount.
The parameters for estimating the mainnet height have drifted slightly
over the past two years -- there is now only about 2 days grace time (of
the 7 days added).
Testnet height estimates were already invalid (producing estimates a
little more than 3 weeks into the future because of some testnet stalls
and rollbacks), which means a newly generated testnet wallet created
without a daemon connection wouldn't work for ~3 weeks.
Updated them to values from the current mainnet and testnet chains.
This also moves the height and timestamp estimate parameters to
constexpr's in cryptonote_config as that seems a much better place for
them than line 13120 of wallet2.cpp.
Currently all the variables in here need to be added into every
compilation unit; C++17 inline constexpr avoids that by throwing away
the duplicates at link time, plus lets the compiler optimize things
away.
Also eliminates a side effect in `get_config()` that mutated the
variable (so that one `get_config()` would end up changing the data of a
previous `get_config()`). Instead `get_config()` is now always constant
and doesn't care about the hard fork version; instead there is a
convenience method to get the governance wallet_address which takes the
hf version.
Add a multi-address lookup mode to the unbound lookup code and remove
the gross code in net_node.inl that creates one thread per address.
(And worse, tries to use boost thread interruption which cannot work at
all since any such interruption is voluntary and won't do anything to a
thread that is stalled waiting for a synchronous libunbound lookup).
The replacement has unbound create and manage a single thread for async
lookups then waits up to the given timeout for results.
Currently this code isn't actually used because our seed node DNS list
is empty, but this lets us eliminate a boost::thread dependency while
not removing the feature (also includes a test case).
Manual merge because Loki already applied some of the changes regarding
- Locking in the mixin (we enforced this at the beginning of the chain)
- Enforcing miner TX version (enforced in HF12, we're now at HF15).
Lastly we manually merge restricting miner transactions from having
a RCT Signature, to be effective as of HF16.
High-level details:
This redesigns the RPC layer to make it much easier to work with,
decouples it from an embedded HTTP server, and gets the vast majority of
the RPC serialization and dispatch code out of a very commonly included
header.
There is unfortunately rather a lot of interconnected code here that
cannot be easily separated out into separate commits. The full details
of what happens here are as follows:
Major details:
- All of the RPC code is now in a `cryptonote::rpc` namespace; this
renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK
becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME`
becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already
working in the `rpc` namespace).
- `core_rpc_server` is now completely decoupled from providing any
request protocol: it is now *just* the core RPC call handler.
- The HTTP RPC interface now lives in a new rpc/http_server.h; this code
handles listening for HTTP requests and dispatching them to
core_rpc_server, then sending the results back to the caller.
- There is similarly a rpc/lmq_server.h for LMQ RPC code; more details
on this (and other LMQ specifics) below.
- RPC implementing code now returns the response object and throws when
things go wrong which simplifies much of the rpc error handling. They
can throw anything; generic exceptions get logged and a generic
"internal error" message gets returned to the caller, but there is
also an `rpc_error` class to return an error code and message used by
some json-rpc commands.
- RPC implementing functions now overload `core_rpc_server::invoke`
following the pattern:
RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context);
This overloading makes the code vastly simpler: all instantiations are
now done with a small amount of generic instantiation code in a single
.cpp rather than needing to go to hell and back with a nest of epee
macros in a core header.
- each RPC endpoint is now defined by the RPC types themselves,
including its accessible names and permissions, in
core_rpc_server_commands_defs.h:
- every RPC structure now has a static `names()` function that returns
the names by which the end point is accessible. (The first one is
the primary, the others are for deprecated aliases).
- RPC command wrappers define their permissions and type by inheriting
from special tag classes:
- rpc::RPC_COMMAND is a basic, admin-only, JSON command, available
via JSON RPC. *All* JSON commands are now available via JSON RPC,
instead of the previous mix of some being at /foo and others at
/json_rpc. (Ones that were previously at /foo are still there for
backwards compatibility; see `rpc::LEGACY` below).
- rpc::PUBLIC specifies that the command should be available via a
restricted RPC connection.
- rpc::BINARY specifies that the command is not JSON, but rather is
accessible as /name and takes and returns values in the magic epee
binary "portable storage" (lol) data format.
- rpc::LEGACY specifies that the command should be available via the
non-json-rpc interface at `/name` for backwards compatibility (in
addition to the JSON-RPC interface).
- some epee serialization got unwrapped and de-templatized so that it
can be moved into a .cpp file with just declarations in the .h. (This
makes a *huge* difference for core_rpc_server_commands_defs.h and for
every compilation unit that includes it which previously had to
compile all the serialization code and then throw all by one copy away
at link time). This required some new macros so as to not break a ton
of places that will use the old way putting everything in the headers;
The RPC code uses this as does a few other places; there are comments
in contrib/epee/include/serialization/keyvalue_serialization.h as to
how to use it.
- Detemplatized a bunch of epee/storages code. Most of it should have
have been using templates at all (because it can only ever be called
with one type!), and now it isn't. This broke some things that didn't
properly compile because of missing headers or (in one case) a messed
up circular dependency.
- Significantly simplified a bunch of over-templatized serialization
code.
- All RPC serialization definitions is now out of
core_rpc_server_commands_defs.h and into a single .cpp file
(core_rpc_server_commands_defs.cpp).
- core RPC no longer uses the disgusting
BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design
that forced slamming tons of code into a common header that didn't
need to be there.
- epee::struct_init is gone. It was a horrible hack that instiated
multiple templates just so the coder could be so lazy and write
`some_type var;` instead of properly value initializing with
`some_type var{};`.
- Removed a bunch of useless crap from epee. In particular, forcing
extra template instantiations all over the place in order to nest
return objects inside JSON RPC values is no longer needed, as are a
bunch of stuff related to the above de-macroization of the code.
- get_all_service_nodes, get_service_nodes, and get_n_service_nodes are
now combined into a single `get_service_nodes` (with deprecated
aliases for the others), which eliminates a fair amount of
duplication. The biggest obstacle here was getting the requested
fields reference passed through: this is now done by a new ability to
stash a context in the serialization object that can be retrieved by a
sub-serialized type.
LMQ-specifics:
- The LokiMQ instance moves into `cryptonote::core` rather than being
inside cryptonote_protocol. Currently the instance is used both for
qnet and rpc calls (and so needs to be in a common place), but I also
intend future PRs to use the batching code for job processing
(replacing the current threaded job queue).
- rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue.
Unlike http_server it isn't technically running the whole LMQ stack
from here, but the parallel name with http_server seemed appropriate.
- All RPC endpoints are supported by LMQ under the same names as defined
generically, but prefixed with `rpc.` for public commands and `admin.`
for restricted ones.
- service node keys are now always available, even when not running in
`--service-node` mode: this is because we want the x25519 key for
being able to offer CURVE encryption for lmq RPC end-points, and
because it doesn't hurt to have them available all the time. In the
RPC layer this is now called "get_service_keys" (with
"get_service_node_key" as an alias) since they aren't strictly only
for service nodes. This also means code needs to check
m_service_node, and not m_service_node_keys, to tell if it is running
as a service node. (This is also easier to notice because
m_service_node_keys got renamed to `m_service_keys`).
- Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and
`sub.mempool` subscribes the connection for new block and new mempool
TX notifications. The latter can notify on just blink txes, or all
new mempool txes (but only new ones -- txes dumped from a block don't
trigger it). The client gets pushed a [`notify.block`, `height`,
`hash`] or [`notify.tx`, `txhash`, `blob`] message when something
arrives.
Minor details:
- rpc::version_t is now a {major,minor} pair. Forcing everyone to pack
and unpack a uint32_t was gross.
- Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...).
(This immediately revealed a couple of bugs in the RPC code that was
assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because
the macro allows implicit conversion to a char).
- De-templatizing useless templates in epee (i.e. a bunch of templated
types that were never invoked with different types) revealed a painful
circular dependency between epee and non-epee code for tor_address and
i2p_address. This crap is now handled in a suitably named
`net/epee_network_address_hack.cpp` hack because it really isn't
trivial to extricate this mess.
- Removed `epee/include/serialization/serialize_base.h`. Amazingly the
code somehow still all works perfectly with this previously vital
header removed.
- Removed bitrotted, unused epee "crypted_storage" and
"gzipped_inmemstorage" code.
- Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with
LOKI_DEFERs. The epee version involves quite a bit more instantiation
and is ugly as sin. Also made the `loki::defer` class invokable for
some edge cases that need calling before destruction in particular
conditions.
- Moved the systemd code around; it makes much more sense to do the
systemd started notification as in daemon.cpp as late as possible
rather than in core (when we can still have startup failures, e.g. if
the RPC layer can't start).
- Made the systemd short status string available in the get_info RPC
(and no longer require building with systemd).
- during startup, print (only) the x25519 when not in SN mode, and
continue to print all three when in SN mode.
- DRYed out some RPC implementation code (such as set_limit)
- Made wallet_rpc stop using a raw m_wallet pointer
This changes the formula for HF15 to 66% service nodes, 24% miner, 10%
foundation, and changes the block reward from the current 28 and a bit
to a fixed 25 LOKI per block. SN reward and LF reward both increase,
miner reward gets cut substantially. This change is reflecting that
with checkpoints and blink the security provided by miners is
substantially reduced.
In HF16 (with pulse) we plan to become mining-free, with final rewards
also included here for that fork of 21 LOKI per block with 90% to SNs
and 10% to foundation. (This is tentative assuming pulse is implemented
in HF16).
This reduces blink fees by half (from 5x to 2.5x base fee) at HF15, and
makes anything other than "unimportant" priority map to blink for
ordinary transactions.
For non-blink txes priorities are still accepted (so that, if the
mempool is clogged, you can still get a registration or stake through by
upping the priority).
It also updates the wallets default priority for transactions to blink
(that is, "default" now becomes blink).
With the priorities gone and default set to blink, the backlog-checking
code and automatic priority bumping code don't serve any useful purpose,
so this rips them out, along with a few other related code
simplifications.
