The archaic (i.e. decade old) cmake usage here really got in the way of
trying to properly use newer libraries (like lokimq), so this undertakes
overhauling it considerably to make it much more sane (and significantly
reduce the size).
I left more of the architecture-specific bits in the top-level
CMakeLists.txt intact; most of the efforts here are about properly
loading dependencies, specifying dependencies and avoiding a whole pile
of cmake antipatterns.
This bumps the required cmake version to 3.5, which is what xenial comes
with.
- extensive use of interface libraries to include libraries,
definitions, and include paths
- use Boost::whatever instead of ${Boost_WHATEVER_LIBRARY}. The
interface targets are (again) much better as they also give you any
needed include or linking flags without needing to worry about them.
- don't list header files when building things. This has *never* been
correct cmake usage (cmake has always known how to wallet_rpc_headers
the headers that .cpp files include to know about build changes).
- remove the loki_add_library monstrosity; it breaks target names and
makes compiling less efficient because the author couldn't figure out
how to link things together.
- make loki_add_executable take the output filename, and set the output
path to bin/ and install to bin because *every single usage* of
loki_add_executable was immediately followed by setting the output
filename and setting the output path to bin/ and installing to bin.
- move a bunch of crap that is only used in one particular
src/whatever/CMakeLists.txt into that particular CMakeLists.txt instead
of the top level CMakeLists.txt (or src/CMakeLists.txt).
- Remove a bunch of redundant dependencies; most of them look like they
were just copy-and-pasted in, and many more aren't needed (since they
are implied by the PUBLIC linking of other dependencies).
- Removed `die` since it just does a FATAL_ERROR, but adds color (which
is useless since CMake already makes FATAL_ERRORs perfectly visible).
- Change the way LOKI_DAEMON_AND_WALLET_ONLY works to just change the
make targets to daemon and simplewallet rather than changing the build
process (this should make it faster, too, since there are various other
things that will be excluded).
Checkpoint votes internally use a circular buffer, but that's rather
difficult to read for the `print_sn` output; this changes print_sn to
de-circularize the listed votes.
This adds the last lokinet ping time to the second `status` line. It
also compresses this (and storage server's) ping time format to (for
example) `4.7min` instead of `4.7 minutes ago`.
I also add a few tweaks here to the first line of the status message to
save some space for the most common cases:
- just don't show " on mainnet" at all since that's the common case, and
instead capitalize "ON TESTNET"/"ON STAGENET" to make it more obvious
that something is non-default.
- just don't show ", not mining" when not mining
- show the daemon version returned by RPC in addition to the hf version,
as "vX.Y.Z(net vAA)". You usually get a version display from an info
log message when running `lokid version`, but that can be quite
misleading as it's the local lokid binary version which is not
necessarily the same version as the running daemon.
Sample output:
SN:
Height: 174791/174791 (100.0%) ON TESTNET, net hash 455 H/s, v6.0.0(net v14), up to date, 8(out)+12(in) connections, uptime 0d 0h 4m 32s
SN: ff00062fa69397f5580ac2b3d060064d89edac625c77a58fdcd520f94ea54727 active, proof: (never), last pings: 1.6min (storage), 1.6min (lokinet)
Regular node (while mining):
Height: 174791/174791 (100.0%) ON TESTNET, mining at 94 H/s, net hash 455 H/s, v6.0.0(net v14), up to date, 8(out)+32(in) connections, uptime 0d 0h 44m 2s
Fixes:
print_sn SOMEPUBKEY +detail
to suppress the `+detail` (or `+json`) from the error message:
No service node is currently known on the network: 7984cd23f950a52a13199f1ed22c3a396d6eea330b375bd94f6323d8db196953, +detail,
Also fixes a bug in the comma separator logic (see the unwanted
separator above).
- Reformat the Registration details line; I found the formatting really
confusing. Changes from:
Registration Hardfork Version | Register/Expiry Height: 14 | 174767 / Staking Infinitely (stake unlock not requested yet)
to:
Registration: Hardfork Version: 14; Height: 174770; Expiry: Staking Infinitely (stake unlock not requested)
(perhaps not obvious - I also dropped "yet" from the infinite expiry
message to slightly shorten the line as it seemed superfluous).
- add `(Or Penalty)` into last reward value since this value gets reset
without a reward on IP change penalty and recommissions:
Last Reward (Or Penalty) At (Height/TX Index): 177529/4294967295
- add Ed25519 and X25519 pubkeys to +detail mode:
Auxiliary Public Keys:
729916aa41aaa3f2e729e2f9a5516207c39da30debe8e6d0ae7711a3dcfb0d9d (Ed25519)
7984cd23f950a52a13199f1ed22c3a396d6eea330b375bd94f6323d8db196953 (X25519)
The existing code just assumes it is already in the path, which fails
when it's in some prefixed place with a proper .pc file installed (such
as for a homebrew installation).
print_sn was only sorting by last block reward and last tx index, but it
needs to also fallback to sorting by pubkey if both are equal (which
can happen with recomms and ip penalties).
(This is cosmetic only for the `print_sn` command).
- Adds blink signature synchronization and storage through the regular
p2p network
- Adds wallet support (though this is still currently buggy and needs
additional fixes - it sees the tx when it arrives in the mempool but
isn't properly updating when the blink tx gets mined.)
The only difference between long and short is one string (the json
info).
Also remove std::endl because using it like this indicates that the
author doesn't understand what std::endl is for.
Also removed a completely ridiculous inline lambda.
This is the bulk of the work for blink. There is two pieces yet to come
which will follow shortly, which are: the p2p communication of blink
transactions (which needs to be fully synchronized, not just shared,
unlike regular mempool txes); and an implementation of fee burning.
Blink approval, multi-quorum signing, cli wallet and node support for
submission denial are all implemented here.
This overhauls and fixes various parts of the SNNetwork interface to fix
some issues (particularly around non-SN communication with SNs, which
wasn't working).
There are also a few sundry FIXME's and TODO's of other minor details
that will follow shortly under cleanup/testing/etc.
This adds vote relaying via quorumnet.
- The main glue between existing code and quorumnet code is in
cryptonote_protocol/quorumnet.{h,cpp}
- Uses function pointers assigned at initialization to call into the
glue without making cn_core depend on p2p
- Adds ed25519 and quorumnet port to uptime proofs and serialization.
- Added a second uptime proof signature using the ed25519 key to that
SNs are proving they actually have it (otherwise they could spoof
someone else's pubkey).
- Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet
listens on the p2p IPv4 address (but broadcasts the `public_ip` to the
network).
- Derives x25519 when received or deserialized.
- Converted service_info_info::version_t into a scoped enum (with the
same underlying type).
- Added contribution_t::version_t scoped enum. It was using
service_info_info::version for a 0 value which seemed rather wrong.
Random small details:
- Renamed internal `arg_sn_bind_port` for consistency
- Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet
quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port
would collide with the testnet p2p port).
* Fix elapsed time in storage server warning message
This was passing the time value rather than the number of seconds so
basically always printed "a long time" instead of the elapsed time.
* Removed pre-HF12 code
* Wait for storage server before sending proof
On startup we send a proof immediately, but this is misleading to the
operator - they may see a proof broadcast over the network suggesting
everything is good even when the storage server is still down. This
makes lokid wait for a first ping before the first proof so that the
user doesn't get mislead.
It also adds the storage server last ping into the `lokid status`
message, such as:
Height: 375634/375634 (100.0%) on mainnet, not mining, net hash 63.00 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 4s
SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: NO PING RECEIVED
or:
Height: 375663/375663 (100.0%) on mainnet, not mining, net hash 76.17 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 32s
SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: last ping 11 seconds ago
You don't want to report your node as failing locally if you can't be
voted on in that quorum (i.e. you re-registered quickly after
deregistration and still exist in the old quorums)
This adds the pubkey, current status, and last uptime proof on a second
line of output in the `status` command (if a service node and we can
query the SN info successfully).
When processing a quorum for a block, if you are not in the quorum to
validate other nodes, check if you're a node that is going to be tested.
If you are, check based on your current data if you're potentially
a candidate to be decommissioned/deregistered and if so report it to the
console log.
Note that this is only a heuristic and ultimately the decision lies on
what the the other Service Nodes perceive the current state of your node
is (i.e. if they're acting malicious then you will be deregistered
irrespectively).
This switches loki 5.x to use a fee formula of
SIZE * PER_BYTE + OUTPUTS * PER_OUTPUT
where we reduce the PER_BYTE fee back to what it was in 3.x; and with
the PER_OUTPUT fee set to 0.02 LOKI. This compares to the 4.x fee of:
SIZE * PER_BYTE * 80
(the *80 multiple was introduced in 4.x).
It also reduces the multiplier for the maximum priority level to 125
instead of 1000 because 1000 produced uselessly high tx fees. The new
multipliers go up 5x at each level: {1, 5, 25, 125} while previously
they went {1, 5, 25, 1000}.
As for the base change: we added the *80 multiplier in 4.x because we
wanted to make a theoretical de-anonymizing tx spam attack more costly.
The unanticipated consequence was that we also made *large* transactions
(such as sweeps) considerably more costly despite the fact that these
transactions typically only create 2 outputs.
This better captures what we meant to do in 4.x (making output creation
relatively more expensive) without making large txes (e.g. sweeps
required for staking) highly expensive.
The end effect is that the fee for a minimum-sized, 1-input/2-output
transaction should stay roughly the same (slightly over 0.04 LOKI),
while a 100-input/2-output transction (a typical spend or sweep from a
wallet with lots of smaller rewards) will drop in fee by somewhere
around 95%.
The most efficient theoretical deanonymizing tx spamming of this sort
was a 1-input/16-output transaction which will become about 2.5x as
expensive as currently the case in v4.x.
new cli options (RPC ones also apply to wallet):
--p2p-bind-ipv6-address (default = "::")
--p2p-bind-port-ipv6 (default same as ipv4 port for given nettype)
--rpc-bind-ipv6-address (default = "::1")
--p2p-use-ipv6 (default false)
--rpc-use-ipv6 (default false)
--p2p-require-ipv4 (default true, if ipv4 bind fails and this is
true, will not continue even if ipv6 bind
successful)
--rpc-require-ipv4 (default true, description as above)
ipv6 addresses are to be specified as "[xx:xx:xx::xx:xx]:port" except
in the cases of the cli args for bind address. For those the square
braces can be omitted.
The doc generator doesn't know how to parse using statements, so upgrade
contributor/contributions to global scope in the RPC definitions file so
it's able to parse it for both service node RPC calls.
The best option will be to eventually deprecate the various ways to
query the network down to 1 call that doesn't duplicate functionality.
KV_SERIALIZE doesn't handle non-primitive types well and requires adding
in extra deserialize/reserialize code. It's more straightforward to
manually serialize the output, rather than add serialization overloads.
Furthermore, KV_SERIALIZE doesn't support serializing enums in a sane
way.
* print_quorum_state displays checkpointing quorums, remove batched call
* Update the help text for quorum commands
* <= for the max_quorum_type not <
* Handle heights greater than the latest
* Don't repeatedly add partially filled quorums
* Revert get_quorum_state to take a range (common case)
* Update the help text from print_quorum_state
* Improved print_checkpoints
* Flesh out print checkpoints and associated RPC call
* Remove debug print checkpoints
* Update help text for print_checkpoints
* Rewrite to fix num_checkpoints != heights as a unit of measurement
* Use GET_ALL_CHECKPOINTS defined value in get_checkpoints_range
* Let T be deduced in parse_if_present, json_rpc_request not rpc_request
* Add deregistration of checkpoints by checking how many votes are missed
Move uptime proofs and add checkpoint counts in the service_node_list
because we typically prune uptime proofs by time, but it seems we want
to switch to a model where we persist proof data until the node expires
otherwise currently we would prune uptime entries and potentially our
checkpoint vote counts which would cause premature deregistration as the
expected vote counts start mismatching with the number of received
votes.
* Revise deregistration
* Fix test breakages
* uint16_t for port, remove debug false, min votes to 2 in integration mode
* Fix integration build
* core: do not commit half constructed batch db txn
* Add defer macro
* Revert dumb extra copy/move change
* Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes
* Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes
* Update tests to compile with relaxed-registration changes
* Get back to feature parity pre-relaxed registration changes
* Remove debug changes noticed in code review and some small bugs
The replaces the deregistration mechanism with a new state change
mechanism (beginning at the v12 fork) which can change a service node's
network status via three potential values (and is extensible in the
future to handle more):
- deregistered -- this is the same as the existing deregistration; the
SN is instantly removed from the SN list.
- decommissioned -- this is a sort of temporary deregistration: your SN
remains in the service node list, but is removed from the rewards list
and from any network duties.
- recommissioned -- this tx is sent by a quorum if they observe a
decommissioned SN sending uptime proofs again. Upon reception, the SN
is reactivated and put on the end of the reward list.
Since this is broadening the quorum use, this also renames the relevant
quorum to a "obligations" quorum (since it validates SN obligations),
while the transactions are "state_change" transactions (since they
change the state of a registered SN).
The new parameters added to service_node_rules.h control how this works:
// Service node decommissioning: as service nodes stay up they earn "credits" (measured in blocks)
// towards a future outage. A new service node starts out with INITIAL_CREDIT, and then builds up
// CREDIT_PER_DAY for each day the service node remains active up to a maximum of
// DECOMMISSION_MAX_CREDIT.
//
// If a service node stops sending uptime proofs, a quorum will consider whether the service node
// has built up enough credits (at least MINIMUM): if so, instead of submitting a deregistration,
// it instead submits a decommission. This removes the service node from the list of active
// service nodes both for rewards and for any active network duties. If the service node comes
// back online (i.e. starts sending the required performance proofs again) before the credits run
// out then a quorum will reinstate the service node using a recommission transaction, which adds
// the service node back to the bottom of the service node reward list, and resets its accumulated
// credits to 0. If it does not come back online within the required number of blocks (i.e. the
// accumulated credit at the point of decommissioning) then a quorum will send a permanent
// deregistration transaction to the network, starting a 30-day deregistration count down.
This commit currently includes values (which are not necessarily
finalized):
- 8 hours (240 blocks) of credit required for activation of a
decommission (rather than a deregister)
- 0 initial credits at registration
- a maximum of 24 hours (720 blocks) of credits
- credits accumulate at a rate that you hit 24 hours of credits after 30
days of operation.
Miscellaneous other details of this PR:
- a new TX extra tag is used for the state change (including
deregistrations). The old extra tag has no version or type tag, so
couldn't be reused. The data in the new tag is slightly more
efficiently packed than the old deregistration transaction, so it gets
used for deregistrations (starting at the v12 fork) as well.
- Correct validator/worker selection required generalizing the shuffle
function to be able to shuffle just part of a vector. This lets us
stick any down service nodes at the end of the potential list, then
select validators by only shuffling the part of the index vector that
contains active service indices. Once the validators are selected, the
remainder of the list (this time including decommissioned SN indices) is
shuffled to select quorum workers to check, thus allowing decommisioned
nodes to be randomly included in the nodes to check without being
selected as a validator.
- Swarm recalculation was not quite right: swarms were recalculated on
SN registrations, even if those registrations were include shared node
registrations, but *not* recalculated on stakes. Starting with the
upgrade this behaviour is fixed (swarms aren't actually used currently
and aren't consensus-relevant so recalculating early won't hurt
anything).
- Details on decomm/dereg are added to RPC info and print_sn/print_sn_status
- Slightly improves the % of reward output in the print_sn output by
rounding it to two digits, and reserves space in the output string to
avoid excessive reallocations.
- Adds various debugging at higher debug levels to quorum voting (into
all of voting itself, vote transmission, and vote reception).
- Reset service node list internal data structure version to 0. The SN
list has to be rescanned anyway at upgrade (its size has changed), so we
might as well reset the version and remove the version-dependent
serialization code. (Note that the affected code here is for SN states
in lmdb storage, not for SN-to-SN communication serialization).
