Increase the range to mitigate frequent rescanning when reorganising,
technically with checkpointing we only need up to the 2nd oldest
checkpoint, but I'm sure there's some case we're forgetting that would
benefit from a larger rollback range.
Also considering there were only 160 events in the month with a mass
deregistration- keeping 2 days worth of rollbacks around is not going to
be too expensive.
Start storing service node list state for past blocks
Separate quorums and reduce storage to 1 days worth
Don't use pointer, declare m_state by value
Fix rebase to dev
Fix double serialise of service node state and expensive blob copy
std::move quorums into storage, reserve vector size
Don't +1 checkpoint finality height
Code review
Code review 2
* Recalculate difficulty on reorganisation workaround
* Gate difficulty workaround to only be on mainnet
* Make difficulty changes emit logs on diff delta only
* Improved print_checkpoints
* Flesh out print checkpoints and associated RPC call
* Remove debug print checkpoints
* Update help text for print_checkpoints
* Rewrite to fix num_checkpoints != heights as a unit of measurement
* Use GET_ALL_CHECKPOINTS defined value in get_checkpoints_range
* Let T be deduced in parse_if_present, json_rpc_request not rpc_request
We're going to be coming down off a ~300MH/s high, so add a difficulty
cap of 30MH/s for the first 60 blocks so that we don't end with
extremely long blocks.
Fixes some potential architecture issues with the checkpoint lmdb:
- size_t is not a fixed size type (so 32-bit architectures and 64-bit
architectures would store a different amount of data here)
- add endian conversions to the integer fields before/after loading
(these are no-ops on little-endian architectures, i.e. in the vast
majority of the time).
This converts the transaction type and version to scoped enum, giving
type safety and making the tx type assignment less error prone because
there is no implicit conversion or comparison with raw integers that has
to be worried about.
This ends up converting any use of `cryptonote::transaction::type_xyz`
to `cryptonote::transaction::txtype::xyz`. For version, names like
`transaction::version_v4` become `cryptonote::txversion::v4_tx_types`.
This also allows/includes various other simplifications related to or
enabled by this change:
- handle `is_deregister` dynamically in serialization code (setting
`type::standard` or `type::deregister` rather than using a
version-determined union)
- `get_type()` is no longer needed with the above change: it is now
much simpler to directly access `type` which will always have the
correct value (even for v2 or v3 transaction types). And though there
was an assertion on the enum value, `get_type()` was being used only
sporadically: many places accessed `.type` directly.
- the old unscoped enum didn't have a type but was assumed castable
to/from `uint16_t`, which technically meant there was potential
undefined behaviour when deserializing any type values >= 8.
- tx type range checks weren't being done in all serialization paths;
they are now. Because `get_type()` was not used everywhere (lots of
places simply accessed `.type` directory) these might not have been
caught.
- `set_type()` is not needed; it was only being used in a single place
(wallet2.cpp) and only for v4 txes, so the version protection code was
never doing anything.
- added a std::ostream << operator for the enum types so that they can be
output with `<< tx_type <<` rather than needing to wrap it in
`type_to_string(tx_type)` everywhere. For the versions, you get the
annotated version string (e.g. 4_tx_types) rather than just the number
4.
* Initial updates to allow syncing of checkpoints in protocol_handler
* Handle checkpoints in prepare_handle_incoming_blocks
* Debug changes for testing, cancel changes later
* Add checkpoint pruning code
* Reduce DB checkpoint accesses, sync votes for up to 60 blocks
* Remove duplicate lifetime variable
* Move parsing checkpoints to above early return for consistency
* Various integration test fixes
- Don't try participate in quorums at startup if you are not a service node
- Add lock for when potentially writing to the DB
- Pre-existing batch can be open whilst updating checkpoint so can't use
DB guards
- Temporarily emit the ban message using msgwriter instead of logs
* Integration mode bug fixes
- Always deserialize empty quorums so get_testing_quorum only fails when
an actual critical error has occurred
- Cache the last culled height so we don't needlessly query the DB over
the same heights trying to delete already-deleted checkpoints
- Submit votes when new blocks arrive for more reliable vote
transportation
* Undo debug changes for testing
* Update incorrect DB message and stale comment
* Unify checkpointing and uptime quorums
* Begin making checkpoints cull old votes/checkpoints
* Begin rehaul of service node code out of core, to assist checkpoints
* Begin overhaul of votes to move resposibility into quorum_cop
* Update testing suite to work with the new system
* Remove vote culling from checkpoints and into voting_pool
* Fix bugs making integration deregistration fail
* Votes don't always specify an index in the validators
* Update tests for validator index member change
* Rename deregister to voting, fix subtle hashing bug
Update the deregister hash derivation to use uint32_t as originally set
not uint64_t otherwise this affects the result and produces different
results.
* Remove un-needed nettype from vote pool
* PR review, use <algorithms>
* Rename uptime_deregister/uptime quorums to just deregister quorums
* Remove unused add_deregister_vote, move side effect out of macro
* Beging adding functions to recalculate difficulty
* Add command line args to utility for recalculating difficulty
* Exception safety for recalculating difficulty
* Update help text for recalc difficulty
* Add recalc flag on the daemon
* Make context be const, signify intent for var++ to 1
* Add functions for storing checkpoints to the DB
* Allocate the DB entry on the stack instead of heap
* Add virtual overrides for new checkpoint functions
* Clean up for pull request, simplify some logic
* Revise API to include height in checkpoint header
* Move log to top of function even if early exit
* Begin moving checkpoints to db
* Allow storing of checkpoints to DB
* Cleanup for code reviewer, fix unit tests
* Fix tests, fix casting issues
* Don't use DUPSORT, use height->checkpoint mapping in DB
* Remove if 0 disabling checkpoint vote, we already check HF12
* Fix unit test infinite loop
* Update db schemas to match blk_checkpoint_header
* Code review
The db txn in add_block ending caused the entire overarching
batch txn to stop.
Also add a new guard class so a db txn can be stopped in the
face of exceptions.
Also use a read only db txn in init when the db itself is
read only, and do not save the max tx size in that case.
This curbs runaway growth while still allowing substantial
spikes in block weight
Original specification from ArticMine:
here is the scaling proposal
Define: LongTermBlockWeight
Before fork:
LongTermBlockWeight = BlockWeight
At or after fork:
LongTermBlockWeight = min(BlockWeight, 1.4*LongTermEffectiveMedianBlockWeight)
Note: To avoid possible consensus issues over rounding the LongTermBlockWeight for a given block should be calculated to the nearest byte, and stored as a integer in the block itself. The stored LongTermBlockWeight is then used for future calculations of the LongTermEffectiveMedianBlockWeight and not recalculated each time.
Define: LongTermEffectiveMedianBlockWeight
LongTermEffectiveMedianBlockWeight = max(300000, MedianOverPrevious100000Blocks(LongTermBlockWeight))
Change Definition of EffectiveMedianBlockWeight
From (current definition)
EffectiveMedianBlockWeight = max(300000, MedianOverPrevious100Blocks(BlockWeight))
To (proposed definition)
EffectiveMedianBlockWeight = min(max(300000, MedianOverPrevious100Blocks(BlockWeight)), 50*LongTermEffectiveMedianBlockWeight)
Notes:
1) There are no other changes to the existing penalty formula, median calculation, fees etc.
2) There is the requirement to store the LongTermBlockWeight of a block unencrypted in the block itself. This is to avoid possible consensus issues over rounding and also to prevent the calculations from becoming unwieldy as we move away from the fork.
3) When the EffectiveMedianBlockWeight cap is reached it is still possible to mine blocks up to 2x the EffectiveMedianBlockWeight by paying the corresponding penalty.
Note: the long term block weight is stored in the database, but not in the actual block itself,
since it requires recalculating anyway for verification.
This curbs runaway growth while still allowing substantial
spikes in block weight
Original specification from ArticMine:
here is the scaling proposal
Define: LongTermBlockWeight
Before fork:
LongTermBlockWeight = BlockWeight
At or after fork:
LongTermBlockWeight = min(BlockWeight, 1.4*LongTermEffectiveMedianBlockWeight)
Note: To avoid possible consensus issues over rounding the LongTermBlockWeight for a given block should be calculated to the nearest byte, and stored as a integer in the block itself. The stored LongTermBlockWeight is then used for future calculations of the LongTermEffectiveMedianBlockWeight and not recalculated each time.
Define: LongTermEffectiveMedianBlockWeight
LongTermEffectiveMedianBlockWeight = max(300000, MedianOverPrevious100000Blocks(LongTermBlockWeight))
Change Definition of EffectiveMedianBlockWeight
From (current definition)
EffectiveMedianBlockWeight = max(300000, MedianOverPrevious100Blocks(BlockWeight))
To (proposed definition)
EffectiveMedianBlockWeight = min(max(300000, MedianOverPrevious100Blocks(BlockWeight)), 50*LongTermEffectiveMedianBlockWeight)
Notes:
1) There are no other changes to the existing penalty formula, median calculation, fees etc.
2) There is the requirement to store the LongTermBlockWeight of a block unencrypted in the block itself. This is to avoid possible consensus issues over rounding and also to prevent the calculations from becoming unwieldy as we move away from the fork.
3) When the EffectiveMedianBlockWeight cap is reached it is still possible to mine blocks up to 2x the EffectiveMedianBlockWeight by paying the corresponding penalty.
* Cleanup and undoing some protocol breakages
* Simplify expiration of nodes
* Request unlock schedules entire node for expiration
* Fix off by one in expiring nodes
* Undo expiring code for pre v10 nodes
* Fix RPC returning register as unlock height and not checking 0
* Rename key image unlock height const
* Undo testnet hardfork debug changes
* Remove is_type for get_type, fix missing var rename
* Move serialisable data into public namespace
* Serialise tx types properly
* Fix typo in no service node known msg
* Code review
* Fix == to >= on serialising tx type
* Code review 2
* Fix tests and key image unlock
* Add command to print locked key images
* Update ui to display lock stakes, query in print cmd blacklist
* Modify print stakes to be less slow
* Remove autostaking code
* Refactor staking into sweep functions
It appears staking was derived off stake_main written separately at
implementation at the beginning. This merges them back into a common
code path, after removing autostake there's only some minor differences.
It also makes sure that any changes to sweeping upstream are going to be
considered in the staking process which we want.
* Display unlock height for stakes
* Begin creating output blacklist
* Make blacklist output a migration step
* Implement get_output_blacklist for lmdb
* In wallet output selection ignore blacklisted outputs
* Apply blacklisted outputs to output selection
* Fix broken tests, switch key image unlock
* Fix broken unit_tests
* Begin change to limit locked key images to 4 globally
* Revamp prepare registration for new min contribution rules
* Fix up old back case in prepare registration
* Remove debug code
* Cleanup debug code and some unecessary changes
* Fix migration step on mainnet db
* Fix blacklist outputs for pre-existing DB's
* Remove irrelevant note
* Tweak scanning addresses for locked stakes
Since we only now allow contributions from the primary address we can
skip checking all subaddress + lookahead to speed up wallet scanning
* Define macro for SCNu64 for Mingw
* Fix failure on empty DB
* Add missing error msg, remove contributor from stake
* Improve staking messages
* Flush prompt to always display
* Return the msg from stake failure and fix stake parsing error
* Tweak fork rules for smaller bulletproofs
* Tweak pooled nodes minimum amounts
* Fix crash on exit, there's no need to store on destructor
Since all information about service nodes is derived from the blockchain
and we store state every time we receive a block, storing in the
destructor is redundant as there is no new information to store.
* Make prompt be consistent with CLI
* Check max number of key images from per user to node
* Implement error message on get_output_blacklist failure
* Remove resolved TODO's/comments
* Handle infinite staking in print_sn
* Atoi->strtol, fix prepare_registration, virtual override, stale msgs
The blockchain prunes seven eighths of prunable tx data.
This saves about two thirds of the blockchain size, while
keeping the node useful as a sync source for an eighth
of the blockchain.
No other data is currently pruned.
There are three ways to prune a blockchain:
- run monerod with --prune-blockchain
- run "prune_blockchain" in the monerod console
- run the monero-blockchain-prune utility
The first two will prune in place. Due to how LMDB works, this
will not reduce the blockchain size on disk. Instead, it will
mark parts of the file as free, so that future data will use
that free space, causing the file to not grow until free space
grows scarce.
The third way will create a second database, a pruned copy of
the original one. Since this is a new file, this one will be
smaller than the original one.
Once the database is pruned, it will stay pruned as it syncs.
That is, there is no need to use --prune-blockchain again, etc.
get_output_key method is commonly used when working with txs and their key images. Because the method is not const, passing blockchain object though const& or pointers to const is not possible in this context. This is especially problematic in external projects (e.g., projects in moneroexamples) that use monero C++ api to operate on the blockchain and txs.
Thus, having get_output_key method will simplify moving blockchain object around through const references and pointers to const objects.
6644b9b blockchain_db: remove a couple unused functions (moneromooo-monero)
ce594f5 blockchain_db: allocate known size vector only once (moneromooo-monero)
8332698 db_lmdb: inline check_open, it's trivial and called everywhere (moneromooo-monero)
5511563 db_lmdb: avoid pointless division (moneromooo-monero)
d1efe3d cryptonote: set tx hash on newly parsed txes when known (moneromooo-monero)
9cc68a2 tx_pool: add a few std::move where it can make a difference (moneromooo-monero)
a48f2dab blockchain_prune_known_spent_data: blackball file is now optional (moneromooo-monero)
17b45725 Outputs where all amounts are known spent can now be pruned (moneromooo-monero)
Only for pre rct for obvious reasons.
Note: DO NOT use a known spent list which includes outputs
which are not known spent. If the list includes any output
that's just strongly thought to be spent, but not provably
so, you risk finding yourself unable to sync past the point
where that output is spent.
I estimate only 200 MB saved on current mainnet though,
unless the new blackballing rule unearths a good amount of
large-amount-set extra spent outs.
5808530f blockchain: remove unused output_scan_worker parameter (moneromooo-monero)
1426209a blockchain: don't run threads if we have just one function to run (moneromooo-monero)
6f7a5fd4 db_lmdb: slight speedup getting array data from the blockchain (moneromooo-monero)
99fbe100 db_lmdb: save some string copies for readonly db keys/values (moneromooo-monero)
bf31447e tx_pool: speed up take_tx for transactions from blocks (moneromooo-monero)
4f005a77 tx_pool: remove unnecessary get_transaction_hash (moneromooo-monero)
593ef598 perf_timer: call reserve on new timer array (moneromooo-monero)
6ecc99ad core: avoid unnecessary tx/blob conversions (moneromooo-monero)
00cc1a16 unit_tests: notify test special case for the usual weirdo (moneromooo-monero)
bcf3f6af fuzz_tests: catch unhandled exceptions (moneromooo-monero)
3ebd05d4 miner: restore stream flags after changing them (moneromooo-monero)
a093092e levin_protocol_handler_async: do not propagate exception through dtor (moneromooo-monero)
1eebb82b net_helper: do not propagate exceptions through dtor (moneromooo-monero)
fb6a3630 miner: do not propagate exceptions through dtor (moneromooo-monero)
2e2139ff epee: do not propagate exception through dtor (moneromooo-monero)
0749a8bd db_lmdb: do not propagate exceptions in dtor (moneromooo-monero)
1b0afeeb wallet_rpc_server: exit cleanly on unhandled exceptions (moneromooo-monero)
418a9936 unit_tests: catch unhandled exceptions (moneromooo-monero)
ea7f9543 threadpool: do not propagate exceptions through the dtor (moneromooo-monero)
6e855422 gen_multisig: nice exit on unhandled exception (moneromooo-monero)
53df2deb db_lmdb: catch error in mdb_stat calls during migration (moneromooo-monero)
e67016dd blockchain_blackball: catch failure to commit db transaction (moneromooo-monero)
661439f4 mlog: don't remove old logs if we failed to rename the current file (moneromooo-monero)
5fdcda50 easylogging++: test for NULL before dereference (moneromooo-monero)
7ece1550 performance_test: fix bad last argument calling add_arg (moneromooo-monero)
a085da32 unit_tests: add check for page size > 0 before dividing (moneromooo-monero)
d8b1ec8b unit_tests: use std::shared_ptr to shut coverity up about leaks (moneromooo-monero)
02563bf4 simplewallet: top level exception catcher to print nicer messages (moneromooo-monero)
c57a65b2 blockchain_blackball: fix shift range for 32 bit archs (moneromooo-monero)
29dea03 epee: resize vectors where possible in serialization (moneromooo-monero)
76affd9 epee: some speedup in parsing (moneromooo-monero)
dc6c069 db_lmdb: speedup the get_output_distribution common case (moneromooo-monero)
76ac5a8 wallet2: ask for a binary output distribution, for speed (moneromooo-monero)
This reverts commit c4988f5a1f.
This un-reverts the changes to store the service node info in the
database. This could easily just be its own commit instead of a
revert, but doing a revert is amusing (and easier in this case).
* add per-output unlock time field to transaction class
* get output unlock time (and relevant const fixes)
* move output unlocked check to separate function
* tx unlocked -> output unlocked in Blockchain
* per output unlock in tx creation; needs fork rules and testing
* per output unlock in tx prefix so it is signed...
also fix a couple typos/goofs
* wallet: check if using per output unlock time
* add rules.h and rules.cpp. git woopsie.
* update test for BlockchainDB changes
* Change tx v3 deregister to be a bool in tx header
* service_node_list: handle per output unlock times in contribution txs
* transaction: added verification of unlock_times.size()
149da42 db_lmdb: enable batch transactions by default (stoffu)
34cb6b4 add --regtest and --fixed-difficulty for regression testing (vicsn)
9e1403e update get_info RPC and bump RPC version (vicsn)
207b66e first new functional tests (vicsn)
This gets rid of the temporary precalc cache.
Also make the RPC able to send data back in binary or JSON,
since there can be a lot of data
This bumps the LMDB database format to v3, with migration.
on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest.
Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain
Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'.
Queries hard fork heights info of other network types
740da1b core: fix automatic safe db sync mode switching (moneromooo-monero)
e942d34 protocol: do not switch to unsafe sync mode for just a few blocks (moneromooo-monero)
e4646379 keccak: fix mdlen bounds sanity checking (moneromooo-monero)
2e3e90ac pass large parameters by const ref, not value (moneromooo-monero)
61defd89 blockchain: sanity check number of precomputed hash of hash blocks (moneromooo-monero)
9af6b2d1 ringct: fix infinite loop in unused h2b function (moneromooo-monero)
8cea8d0c simplewallet: double check a new multisig wallet is multisig (moneromooo-monero)
9b98a6ac threadpool: catch exceptions in dtor, to avoid terminate (moneromooo-monero)
24803ed9 blockchain_export: fix buffer overflow in exporter (moneromooo-monero)
f3f7da62 perf_timer: rewrite to make it clear there is no division by zero (moneromooo-monero)
c6ea3df0 performance_tests: remove add_arg call stray extra param (moneromooo-monero)
fa6b4566 fuzz_tests: fix an uninitialized var in setup (moneromooo-monero)
03887f11 keccak: fix sanity check bounds test (moneromooo-monero)
ad11db91 blockchain_db: initialize m_open in base class ctor (moneromooo-monero)
bece67f9 miner: restore std::cout precision after modification (moneromooo-monero)
1aabd14c db_lmdb: check hard fork info drop succeeded (moneromooo-monero)
