oxen-io/oxen-core
2
1
Fork 0
mirror of https://github.com/oxen-io/oxen-core.git synced 2023-12-14 02:22:56 +01:00
Code Issues Projects Releases Wiki Activity
oxen-core/src/cryptonote_core/cryptonote_core.cpp

2236 lines
90 KiB
C++
Raw Normal View History

Update monero copyright to 2019 pre-emptively to make merge simpler
2019-04-12 06:36:43 +02:00
// Copyright (c) 2014-2019, The Monero Project
First pass Monero to Loki refs
2018-04-10 06:49:20 +02:00
// Copyright (c) 2018, The Loki Project
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// All rights reserved.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// Redistribution and use in source and binary forms, with or without modification, are
// permitted provided that the following conditions are met:
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// 1. Redistributions of source code must retain the above copyright notice, this list of
// conditions and the following disclaimer.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
// of conditions and the following disclaimer in the documentation and/or other
// materials provided with the distribution.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// 3. Neither the name of the copyright holder nor the names of its contributors may be
// used to endorse or promote products derived from this software without specific
// prior written permission.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
moved all stuff to github
2014-03-03 23:07:58 +01:00
move includes around to lessen overall load
2017-11-25 23:25:05 +01:00
#include <boost/algorithm/string.hpp>
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
#include <boost/endian/conversion.hpp>
move includes around to lessen overall load
2017-11-25 23:25:05 +01:00
#include "string_tools.h"
moved all stuff to github
2014-03-03 23:07:58 +01:00
using namespace epee;
#include <unordered_set>
service node registration: add signatures from service node pubkey to registration
2018-07-21 03:27:13 +02:00
#include <iomanip>
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
moved all stuff to github
2014-03-03 23:07:58 +01:00
#include "cryptonote_core.h"
#include "common/util.h"
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
#include "common/updates.h"
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
#include "common/download.h"
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
#include "common/threadpool.h"
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
#include "common/command_line.h"
moved all stuff to github
2014-03-03 23:07:58 +01:00
#include "warnings.h"
#include "crypto/crypto.h"
#include "cryptonote_config.h"
#include "misc_language.h"
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
#include "file_io_utils.h"
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
#include <csignal>
move checkpoints in a separate library
2017-09-10 18:35:59 +02:00
#include "checkpoints/checkpoints.h"
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
#include "ringct/rctTypes.h"
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
#include "blockchain_db/blockchain_db.h"
rct: split rct checks between semantics and other Semantics can be checked early
2017-01-14 14:29:08 +01:00
#include "ringct/rctSigs.h"
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 22:18:08 +02:00
#include "common/notify.h"
cryptonote_core does not depend on p2p anymore As a followon side effect, this makes a lot of inline code included only in particular cpp files (and instanciated when necessary.
2017-11-29 23:53:58 +01:00
#include "version.h"
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
#include "wipeable_string.h"
service node registration: add signatures from service node pubkey to registration
2018-07-21 03:27:13 +02:00
#include "common/i18n.h"
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
#include "net/local_ip.h"
moved all stuff to github
2014-03-03 23:07:58 +01:00
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
#include "common/loki_integration_test_hooks.h"
First pass Monero to Loki refs
2018-04-10 06:49:20 +02:00
#undef LOKI_DEFAULT_LOG_CATEGORY
#define LOKI_DEFAULT_LOG_CATEGORY "cn"
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
moved all stuff to github
2014-03-03 23:07:58 +01:00
DISABLE_VS_WARNINGS(4355)
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
#define MERROR_VER(x) MCERROR("verify", x)
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
#define BAD_SEMANTICS_TXES_MAX_SIZE 100
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
// basically at least how many bytes the block itself serializes to without the miner tx
#define BLOCK_SIZE_SANITY_LEEWAY 100
moved all stuff to github
2014-03-03 23:07:58 +01:00
namespace cryptonote
{
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
const command_line::arg_descriptor<bool, false> arg_testnet_on = {
"testnet"
, "Run on testnet. The wallet must be launched with --testnet flag."
, false
};
Stagenet
2018-02-16 12:04:04 +01:00
const command_line::arg_descriptor<bool, false> arg_stagenet_on = {
"stagenet"
, "Run on stagenet. The wallet must be launched with --stagenet flag."
, false
};
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
const command_line::arg_descriptor<bool> arg_regtest_on = {
"regtest"
, "Run in a regression testing mode."
, false
};
const command_line::arg_descriptor<difficulty_type> arg_fixed_difficulty = {
"fixed-difficulty"
, "Fixed difficulty used for testing."
, 0
};
Stagenet
2018-02-16 12:04:04 +01:00
const command_line::arg_descriptor<std::string, false, true, 2> arg_data_dir = {
options: remove testnet-* options
2018-01-21 16:29:55 +01:00
"data-dir"
, "Specify data directory"
, tools::get_default_data_dir()
Stagenet
2018-02-16 12:04:04 +01:00
, {{ &arg_testnet_on, &arg_stagenet_on }}
fix lambda compile error on openbsd
2018-03-27 15:47:57 +02:00
, [](std::array<bool, 2> testnet_stagenet, bool defaulted, std::string val)->std::string {
Stagenet
2018-02-16 12:04:04 +01:00
if (testnet_stagenet[0])
options: add testnet option dependencies
2018-01-22 02:49:51 +01:00
return (boost::filesystem::path(val) / "testnet").string();
Stagenet
2018-02-16 12:04:04 +01:00
else if (testnet_stagenet[1])
return (boost::filesystem::path(val) / "stagenet").string();
options: add testnet option dependencies
2018-01-22 02:49:51 +01:00
return val;
}
options: remove testnet-* options
2018-01-21 16:29:55 +01:00
};
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
const command_line::arg_descriptor<bool> arg_offline = {
"offline"
, "Do not listen for peers, nor connect to any"
};
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
const command_line::arg_descriptor<size_t> arg_block_download_max_size = {
"block-download-max-size"
, "Set maximum size of block download queue in bytes (0 for default)"
, 0
};
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
static const command_line::arg_descriptor<bool> arg_test_drop_download = {
"test-drop-download"
, "For net tests: in download, discard ALL blocks instead checking/saving them (very fast)"
};
static const command_line::arg_descriptor<uint64_t> arg_test_drop_download_height = {
"test-drop-download-height"
Fix typos in various files
2018-03-01 12:36:19 +01:00
, "Like test-drop-download but discards only after around certain height"
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
, 0
};
static const command_line::arg_descriptor<int> arg_test_dbg_lock_sleep = {
"test-dbg-lock-sleep"
, "Sleep time in ms, defaults to 0 (off), used to debug before/after locking mutex. Values 100 to 1000 are good for tests."
, 0
};
static const command_line::arg_descriptor<uint64_t> arg_fast_block_sync = {
"fast-block-sync"
, "Sync up most of the way by using embedded, known block hashes."
, 1
};
static const command_line::arg_descriptor<uint64_t> arg_prep_blocks_threads = {
"prep-blocks-threads"
, "Max number of threads to use when preparing block hashes in groups."
, 4
};
static const command_line::arg_descriptor<uint64_t> arg_show_time_stats = {
"show-time-stats"
, "Show time-stats when processing blocks/txs and disk synchronization."
, 0
};
static const command_line::arg_descriptor<size_t> arg_block_sync_size = {
"block-sync-size"
, "How many blocks to sync at once during chain synchronization (0 = adaptive)."
, 0
};
static const command_line::arg_descriptor<std::string> arg_check_updates = {
"check-updates"
Update cmakelists and misc refs to Loki
2018-04-25 08:13:38 +02:00
, "Check for new versions of loki: [disabled|notify|download|update]"
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
, "notify"
};
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 23:27:25 +01:00
static const command_line::arg_descriptor<bool> arg_pad_transactions = {
"pad-transactions"
, "Pad relayed transactions to help defend against traffic volume analysis"
, false
};
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
static const command_line::arg_descriptor<size_t> arg_max_txpool_weight = {
"max-txpool-weight"
, "Set maximum txpool weight in bytes."
, DEFAULT_TXPOOL_MAX_WEIGHT
tx_pool: add a max pool size, settable with --max-txpool-size
2018-01-29 22:20:24 +01:00
};
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
static const command_line::arg_descriptor<bool> arg_service_node = {
"service-node"
Removes sn-public-ip and combines into --service-node (#676) * Removes sn-public-ip and combines into --service-node * Only rename the argument, don't combine args for clarity * Update help message to use --service-node-public-ip
2019-06-27 08:03:22 +02:00
, "Run as a service node, options 'service-node-public-ip' and 'storage-server-port' must be set"
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
};
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
static const command_line::arg_descriptor<std::string> arg_public_ip = {
Removes sn-public-ip and combines into --service-node (#676) * Removes sn-public-ip and combines into --service-node * Only rename the argument, don't combine args for clarity * Update help message to use --service-node-public-ip
2019-06-27 08:03:22 +02:00
"service-node-public-ip"
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
, "Public IP address on which this service node's services (such as the Loki "
"storage server) are accessible. This IP address will be advertised to the "
"network via the service node uptime proofs. Required if operating as a "
"service node."
};
static const command_line::arg_descriptor<uint16_t> arg_sn_bind_port = {
"storage-server-port"
, "The port on which this service node's storage server is accessible. A listening "
"storage server is required for service nodes. (This option is specified "
"automatically when using Loki Launcher.)"
, 0};
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 22:18:08 +02:00
static const command_line::arg_descriptor<std::string> arg_block_notify = {
"block-notify"
, "Run a program for each new block, '%s' will be replaced by the block hash"
, ""
};
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
static const command_line::arg_descriptor<bool> arg_prune_blockchain = {
"prune-blockchain"
, "Prune blockchain"
, false
};
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 15:29:28 +01:00
static const command_line::arg_descriptor<std::string> arg_reorg_notify = {
"reorg-notify"
, "Run a program for each reorg, '%s' will be replaced by the split height, "
"'%h' will be replaced by the new blockchain height, and '%n' will be "
"replaced by the number of new blocks in the new chain"
, ""
};
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
static const command_line::arg_descriptor<std::string> arg_block_rate_notify = {
"block-rate-notify"
, "Run a program when the block rate undergoes large fluctuations. This might "
"be a sign of large amounts of hash rate going on and off the Monero network, "
"and thus be of potential interest in predicting attacks. %t will be replaced "
"by the number of minutes for the observation window, %b by the number of "
"blocks observed within that window, and %e by the number of blocks that was "
"expected in that window. It is suggested that this notification is used to "
"automatically increase the number of confirmations required before a payment "
"is acted upon."
, ""
};
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
static const command_line::arg_descriptor<bool> arg_keep_alt_blocks = {
"keep-alt-blocks"
, "Keep alternative blocks on restart"
, false
};
moved all stuff to github
2014-03-03 23:07:58 +01:00
Add flag to recalculate difficulty per block (#598) * Beging adding functions to recalculate difficulty * Add command line args to utility for recalculating difficulty * Exception safety for recalculating difficulty * Update help text for recalc difficulty * Add recalc flag on the daemon * Make context be const, signify intent for var++ to 1
2019-05-12 01:40:57 +02:00
const command_line::arg_descriptor<uint64_t> arg_recalculate_difficulty = {
"recalculate-difficulty",
"Recalculate per-block difficulty starting from the height specified",
0};
(Low priority) Add disabled-by-default quorum storage support (#702) * Add disabled-by-default quorum storage support This adds support for storing N blocks of recent expired quorum state history in lokid (and dumping to/from the lmdb). This isn't useful for regular nodes (and that's why we don't store it) but is incredibly useful for the block explorer to be able to report *which* node got deregistered/decommissioned/etc. by a given state_change tx. * Fix quorum states copy
2019-07-17 08:01:54 +02:00
static const command_line::arg_descriptor<uint64_t> arg_store_quorum_history = {
"store-quorum-history",
"Store the service node quorum history for the last N blocks to allow historic quorum lookups "
"(e.g. by a block explorer). Specify the number of blocks of history to store, or 1 to store "
"the entire history. Requires considerably more memory and block chain storage.",
0};
moved all stuff to github
2014-03-03 23:07:58 +01:00
//-----------------------------------------------------------------------------------------------
core::core(i_cryptonote_protocol* pprotocol):
m_mempool(m_blockchain_storage),
servicenodes: Added service node list
2018-06-29 06:47:00 +02:00
m_service_node_list(m_blockchain_storage),
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
m_blockchain_storage(m_mempool, m_service_node_list),
Making service nodes list and quorum cop thread-safe (#285) * Remove dependency on service node list in quorum cop * making service node list therad-safe * improve thread-safety of quorum cop
2018-10-10 07:24:01 +02:00
m_quorum_cop(*this),
RandomX integration Support RandomX PoW algorithm
2019-04-23 21:32:27 +02:00
m_miner(this, &m_blockchain_storage),
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
m_miner_address(boost::value_initialized<account_public_address>()),
'getinfo' daemon HTTP-RPC returns 'target_height' for progress estimations
2014-06-04 22:50:13 +02:00
m_starter_message_showed(false),
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 07:01:58 +02:00
m_target_blockchain_height(0),
m_checkpoints_path(""),
core: make update download cancellable
2017-02-26 22:00:38 +01:00
m_last_json_checkpoints_update(0),
core: fix use of uninitialised data
2018-03-19 00:36:20 +01:00
m_update_download(0),
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 16:31:34 +02:00
m_nettype(UNDEFINED),
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 23:27:25 +01:00
m_update_available(false),
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
m_last_storage_server_ping(time(nullptr)), // Reset the storage server last ping to make sure the very first uptime proof works
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 23:27:25 +01:00
m_pad_transactions(false)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
cryptonote_core: initialize checkpoint flag
2017-07-02 19:42:45 +02:00
m_checkpoints_updating.clear();
moved all stuff to github
2014-03-03 23:07:58 +01:00
set_cryptonote_protocol(pprotocol);
}
void core::set_cryptonote_protocol(i_cryptonote_protocol* pprotocol)
{
if(pprotocol)
m_pprotocol = pprotocol;
else
m_pprotocol = &m_protocol_stub;
}
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 07:15:28 +02:00
//-----------------------------------------------------------------------------------------------
Rollback chain when receiving conflicting checkpoints (#726)
2019-07-11 06:16:55 +02:00
bool core::update_checkpoints_from_json_file()
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 07:15:28 +02:00
{
Merge dev into upstream
2019-05-01 08:01:17 +02:00
if (m_nettype != MAINNET) return true;
DNS Checkpoint updating-related fixes/changes Only one thread will be doing the updating. Two valid responses must match, and the first two that match will be used.
2015-04-22 10:36:39 +02:00
if (m_checkpoints_updating.test_and_set()) return true;
Service Node Checkpoint Storage (#579) * Add functions for storing checkpoints to the DB * Allocate the DB entry on the stack instead of heap * Add virtual overrides for new checkpoint functions * Clean up for pull request, simplify some logic * Revise API to include height in checkpoint header * Move log to top of function even if early exit * Begin moving checkpoints to db * Allow storing of checkpoints to DB * Cleanup for code reviewer, fix unit tests * Fix tests, fix casting issues * Don't use DUPSORT, use height->checkpoint mapping in DB * Remove if 0 disabling checkpoint vote, we already check HF12 * Fix unit test infinite loop * Update db schemas to match blk_checkpoint_header * Code review
2019-05-01 07:57:41 +02:00
// load json checkpoints every 10min and verify them with respect to what blocks we already have
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 07:01:58 +02:00
bool res = true;
Merge dev into upstream
2019-05-01 08:01:17 +02:00
if (time(NULL) - m_last_json_checkpoints_update >= 600)
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 07:15:28 +02:00
{
Rollback chain when receiving conflicting checkpoints (#726)
2019-07-11 06:16:55 +02:00
res = m_blockchain_storage.update_checkpoints_from_json_file(m_checkpoints_path);
various changes to runtime checkpoint updating json checkpoints will be checked every 10 minutes, dns every 60. json checkpoints always enforced, dns still with flag. conflicting checkpoints is hard fail, but soft if dns enforce flag not set and dns checkpoints are wonky.
2014-09-29 22:30:47 +02:00
m_last_json_checkpoints_update = time(NULL);
}
DNS Checkpoint updating-related fixes/changes Only one thread will be doing the updating. Two valid responses must match, and the first two that match will be used.
2015-04-22 10:36:39 +02:00
m_checkpoints_updating.clear();
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
// if anything fishy happened getting new checkpoints, bring down the house
if (!res)
{
graceful_exit();
}
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 07:01:58 +02:00
return res;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------
RPC calls for background daemon added in The RPC calls the daemon executable uses to talk to the running daemon instance have mostly been added back in. Rate limiting has not been added in upstream, but is on its way in a separate effort, so those calls are still NOPed out.
2015-02-05 10:11:20 +01:00
void core::stop()
{
daemon: stop miner before we bring the whole thing down This avoids the miner erroring out trying to submit blocks to a core that's already shut down (and avoids pegging the CPU while we're busy shutting down).
2018-11-30 15:53:38 +01:00
m_miner.stop();
Fix delayed exit when syncing
2016-12-04 13:27:45 +01:00
m_blockchain_storage.cancel();
core: make update download cancellable
2017-02-26 22:00:38 +01:00
tools::download_async_handle handle;
{
boost::lock_guard<boost::mutex> lock(m_update_mutex);
handle = m_update_download;
m_update_download = 0;
}
if (handle)
tools::download_cancel(handle);
RPC calls for background daemon added in The RPC calls the daemon executable uses to talk to the running daemon instance have mostly been added back in. Rate limiting has not been added in upstream, but is on its way in a separate effort, so those calls are still NOPed out.
2015-02-05 10:11:20 +01:00
}
//-----------------------------------------------------------------------------------
Register daemon command line arguments to core if they're used in core This fixes coretests, which does not register daemon specific arguments, but uses core, which uses those arguments. Also gets rid of an unwanted dependency on daemon code from core.
2015-12-09 00:06:29 +01:00
void core::init_options(boost::program_options::options_description& desc)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
options: remove testnet-* options
2018-01-21 16:29:55 +01:00
command_line::add_arg(desc, arg_data_dir);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
command_line::add_arg(desc, arg_test_drop_download);
command_line::add_arg(desc, arg_test_drop_download_height);
command_line::add_arg(desc, arg_testnet_on);
Stagenet
2018-02-16 12:04:04 +01:00
command_line::add_arg(desc, arg_stagenet_on);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
command_line::add_arg(desc, arg_regtest_on);
command_line::add_arg(desc, arg_fixed_difficulty);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
command_line::add_arg(desc, arg_prep_blocks_threads);
command_line::add_arg(desc, arg_fast_block_sync);
command_line::add_arg(desc, arg_show_time_stats);
command_line::add_arg(desc, arg_block_sync_size);
command_line::add_arg(desc, arg_check_updates);
command_line::add_arg(desc, arg_test_dbg_lock_sleep);
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
command_line::add_arg(desc, arg_offline);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
command_line::add_arg(desc, arg_block_download_max_size);
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
command_line::add_arg(desc, arg_max_txpool_weight);
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
command_line::add_arg(desc, arg_service_node);
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
command_line::add_arg(desc, arg_public_ip);
command_line::add_arg(desc, arg_sn_bind_port);
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 23:27:25 +01:00
command_line::add_arg(desc, arg_pad_transactions);
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 22:18:08 +02:00
command_line::add_arg(desc, arg_block_notify);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
command_line::add_arg(desc, arg_prune_blockchain);
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 15:29:28 +01:00
command_line::add_arg(desc, arg_reorg_notify);
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
command_line::add_arg(desc, arg_block_rate_notify);
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
command_line::add_arg(desc, arg_keep_alt_blocks);
core: add p2p bind port options from net_node They're now used by core to determine the data directory to use for the txpool directory. This fixes an assert in the core tests, which don't use the RPC server, which normally initializes the P2P code.
2017-02-09 22:29:35 +01:00
Add flag to recalculate difficulty per block (#598) * Beging adding functions to recalculate difficulty * Add command line args to utility for recalculating difficulty * Exception safety for recalculating difficulty * Update help text for recalc difficulty * Add recalc flag on the daemon * Make context be const, signify intent for var++ to 1
2019-05-12 01:40:57 +02:00
command_line::add_arg(desc, arg_recalculate_difficulty);
(Low priority) Add disabled-by-default quorum storage support (#702) * Add disabled-by-default quorum storage support This adds support for storing N blocks of recent expired quorum state history in lokid (and dumping to/from the lmdb). This isn't useful for regular nodes (and that's why we don't store it) but is incredibly useful for the block explorer to be able to report *which* node got deregistered/decommissioned/etc. by a given state_change tx. * Fix quorum states copy
2019-07-17 08:01:54 +02:00
command_line::add_arg(desc, arg_store_quorum_history);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
#if defined(LOKI_ENABLE_INTEGRATION_TEST_HOOKS)
Add ability to handle multiple daemon/wallets
2018-11-23 05:18:08 +01:00
command_line::add_arg(desc, loki::arg_integration_test_hardforks_override);
Revamp to use semaphores for IPC Delay hooks into stdin/out until core is idle for daemons Disable logging for daemon (super problematic)
2019-01-15 10:09:17 +01:00
command_line::add_arg(desc, loki::arg_integration_test_shared_mem_name);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
#endif
Fix core_tests breaking on startup You're wondering how this fixes core tests, aren't you... It prevents the miner (initialized by cryptonote::core) from breaking trying to access arguments that were not added. Since the tests don't use the miner directly, it makes more sense to have cryptonote_core add those, since it also uses the miner.
2017-02-21 16:06:07 +01:00
miner::init_options(desc);
move db specific options to BlockchainDB Avoids common depending on blockchain_db, which can cause link errors.
2017-08-29 12:14:42 +02:00
BlockchainDB::init_options(desc);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
bool core::handle_command_line(const boost::program_options::variables_map& vm)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Stagenet
2018-02-16 12:04:04 +01:00
if (m_nettype != FAKECHAIN)
{
const bool testnet = command_line::get_arg(vm, arg_testnet_on);
const bool stagenet = command_line::get_arg(vm, arg_stagenet_on);
m_nettype = testnet ? TESTNET : stagenet ? STAGENET : MAINNET;
}
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
options: remove testnet-* options
2018-01-21 16:29:55 +01:00
m_config_folder = command_line::get_arg(vm, arg_data_dir);
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
test_drop_download_height(command_line::get_arg(vm, arg_test_drop_download_height));
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 23:27:25 +01:00
m_pad_transactions = get_arg(vm, arg_pad_transactions);
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
m_offline = get_arg(vm, arg_offline);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
if (command_line::get_arg(vm, arg_test_drop_download) == true)
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
test_drop_download();
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
m_service_node = command_line::get_arg(vm, arg_service_node);
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
if (m_service_node) {
/// TODO: parse these options early, before we start p2p server etc?
m_storage_port = command_line::get_arg(vm, arg_sn_bind_port);
bool storage_ok = true;
if (m_storage_port == 0) {
Make --service-node give more verbose failure flags (#630)
2019-06-06 09:39:54 +02:00
MERROR("Please specify the port on which the storage server is listening with: '--" << arg_sn_bind_port.name << " <port>'");
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
storage_ok = false;
}
const std::string pub_ip = command_line::get_arg(vm, arg_public_ip);
Make --service-node give more verbose failure flags (#630)
2019-06-06 09:39:54 +02:00
if (pub_ip.size())
{
if (!epee::string_tools::get_ip_int32_from_string(m_sn_public_ip, pub_ip)) {
MERROR("Unable to parse IPv4 public address from: " << pub_ip);
storage_ok = false;
}
Tighten allowed IPv4 ranges epee's is_ip_local is missing two ranges that are commonly found: link-local auto-config addresses (169.254.0.0/16) that are sometimes used as a fallback when DHCP isn't present; and the carrier-grade NAT range (100.64.0.0/10) reserved for carriers who impose NAT on their customers. There are also other ranges that aren't exactly "local" but aren't public either: 0.0.0.0/8 isn't a valid destination address; and 224.0.0.0/3 (includes includes both the 224/4 multicast range, and the reserved (but will most likely never be used) 240.0.0.0/4 range). These are now added to a new `is_ip_public` function that returns true if it's not local or loopback, and not one of these special ranges. This also simplifies some convoluted netmask logic. (The simplification would look better except that epee took the extremely bizarre and wrong decision to store IPv4 addresses in little-endian order).
2019-06-25 18:31:38 +02:00
if (!epee::net_utils::is_ip_public(m_sn_public_ip)) {
Format sn-public-ip option in error message Prints a human-readable IP instead of a 32-bit integer
2019-06-19 07:26:48 +02:00
MERROR("Address given for public-ip is not public: " << epee::string_tools::get_ip_string_from_int32(m_sn_public_ip));
Make --service-node give more verbose failure flags (#630)
2019-06-06 09:39:54 +02:00
storage_ok = false;
}
}
else
{
MERROR("Please specify an IPv4 public address which the service node & storage server is accessible from with: '--" << arg_public_ip.name << " <ip address>'");
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
storage_ok = false;
}
if (!storage_ok) {
Make --service-node give more verbose failure flags (#630)
2019-06-06 09:39:54 +02:00
MERROR("IMPORTANT: All service node operators are now required to run the loki storage "
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
<< "server and provide the public ip and port on which it can be accessed on the internet.");
return false;
}
MGINFO("Storage server endpoint is set to: "
<< (epee::net_utils::ipv4_network_address{ m_sn_public_ip, m_storage_port }).str());
}
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
epee::debug::g_test_dbg_lock_sleep() = command_line::get_arg(vm, arg_test_dbg_lock_sleep);
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
uint64_t core::get_current_blockchain_height() const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_current_blockchain_height();
}
//-----------------------------------------------------------------------------------------------
get_blockchain_top now returns void It was always returning true, and could not be foreseen to usefully return errors in the future. This silences CID 162652 as well as saves some checking code in a few places.
2017-09-09 13:06:24 +02:00
void core::get_blockchain_top(uint64_t& height, crypto::hash& top_id) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
top_id = m_blockchain_storage.get_tail_id(height);
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_blocks(uint64_t start_offset, size_t count, std::vector<std::pair<cryptonote::blobdata,block>>& blocks, std::vector<cryptonote::blobdata>& txs) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_blocks(start_offset, count, blocks, txs);
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_blocks(uint64_t start_offset, size_t count, std::vector<std::pair<cryptonote::blobdata,block>>& blocks) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_blocks(start_offset, count, blocks);
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_blocks(uint64_t start_offset, size_t count, std::vector<block>& blocks) const
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<std::pair<cryptonote::blobdata, cryptonote::block>> bs;
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
if (!m_blockchain_storage.get_blocks(start_offset, count, bs))
return false;
for (const auto &b: bs)
blocks.push_back(b.second);
return true;
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_transactions(const std::vector<crypto::hash>& txs_ids, std::vector<cryptonote::blobdata>& txs, std::vector<crypto::hash>& missed_txs) const
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
{
return m_blockchain_storage.get_transactions_blobs(txs_ids, txs, missed_txs);
}
//-----------------------------------------------------------------------------------------------
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
bool core::get_split_transactions_blobs(const std::vector<crypto::hash>& txs_ids, std::vector<std::tuple<crypto::hash, cryptonote::blobdata, crypto::hash, cryptonote::blobdata>>& txs, std::vector<crypto::hash>& missed_txs) const
{
return m_blockchain_storage.get_split_transactions_blobs(txs_ids, txs, missed_txs);
}
//-----------------------------------------------------------------------------------------------
rpc: add a new RPC to get current txpool backlog (sizes and fees)
2017-08-26 17:23:31 +02:00
bool core::get_txpool_backlog(std::vector<tx_backlog_entry>& backlog) const
{
m_mempool.get_transaction_backlog(backlog);
return true;
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_transactions(const std::vector<crypto::hash>& txs_ids, std::vector<transaction>& txs, std::vector<crypto::hash>& missed_txs) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_transactions(txs_ids, txs, missed_txs);
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_alternative_blocks(std::vector<block>& blocks) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_alternative_blocks(blocks);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
size_t core::get_alternative_blocks_count() const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_alternative_blocks_count();
}
//-----------------------------------------------------------------------------------------------
remove some unused code Found by codacy.com
2018-11-23 14:47:51 +01:00
bool core::init(const boost::program_options::variables_map& vm, const cryptonote::test_options *test_options, const GetCheckpointsCallback& get_checkpoints/* = nullptr */)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Add start_time to get_info methods and show uptime
2017-01-09 00:50:29 +01:00
start_time = std::time(nullptr);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
#if defined(LOKI_ENABLE_INTEGRATION_TEST_HOOKS)
Add ability to handle multiple daemon/wallets
2018-11-23 05:18:08 +01:00
const std::string arg_integration_test_override_hardforks = command_line::get_arg(vm, loki::arg_integration_test_hardforks_override);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
std::vector<std::pair<uint8_t, uint64_t>> integration_test_hardforks;
if (!arg_integration_test_override_hardforks.empty())
{
// Expected format: <fork_version>:<fork_height>, ...
// Example: 7:0, 8:10, 9:20, 10:100
char const *ptr = arg_integration_test_override_hardforks.c_str();
while (ptr[0])
{
int hf_version = atoi(ptr);
while(ptr[0] != ':') ptr++;
++ptr;
int hf_height = atoi(ptr);
while(ptr[0] && ptr[0] != ',') ptr++;
integration_test_hardforks.push_back(std::make_pair(static_cast<uint8_t>(hf_version), static_cast<uint64_t>(hf_height)));
if (!ptr[0]) break;
ptr++;
}
}
cryptonote::test_options integration_hardfork_override = {integration_test_hardforks};
if (!arg_integration_test_override_hardforks.empty())
test_options = &integration_hardfork_override;
Add ability to handle multiple daemon/wallets
2018-11-23 05:18:08 +01:00
{
Revamp to use semaphores for IPC Delay hooks into stdin/out until core is idle for daemons Disable logging for daemon (super problematic)
2019-01-15 10:09:17 +01:00
const std::string arg_shared_mem_name = command_line::get_arg(vm, loki::arg_integration_test_shared_mem_name);
loki::init_integration_test_context(arg_shared_mem_name);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
}
#endif
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
const bool regtest = command_line::get_arg(vm, arg_regtest_on);
if (test_options != NULL || regtest)
Stagenet
2018-02-16 12:04:04 +01:00
{
m_nettype = FAKECHAIN;
}
Add support for fakechain in wallet
2018-11-21 00:43:09 +01:00
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
bool r = handle_command_line(vm);
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
/// Currently terminating before blockchain is initialized results in a crash
/// during deinitialization... TODO: fix that
CHECK_AND_ASSERT_MES(r, false, "Failed to apply command line options.");
Separate data per P2P port poolstate.bin and p2pstate.bin are stored in .bitmonero/ if the default P2P port is being used. If another port is used both files are stored in .bitmonero/PORTNUMBER/.
2017-01-28 01:36:39 +01:00
move db specific options to BlockchainDB Avoids common depending on blockchain_db, which can cause link errors.
2017-08-29 12:14:42 +02:00
std::string db_type = command_line::get_arg(vm, cryptonote::arg_db_type);
std::string db_sync_mode = command_line::get_arg(vm, cryptonote::arg_db_sync_mode);
bool db_salvage = command_line::get_arg(vm, cryptonote::arg_db_salvage) != 0;
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
bool fast_sync = command_line::get_arg(vm, arg_fast_block_sync) != 0;
uint64_t blocks_threads = command_line::get_arg(vm, arg_prep_blocks_threads);
std::string check_updates_string = command_line::get_arg(vm, arg_check_updates);
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
size_t max_txpool_weight = command_line::get_arg(vm, arg_max_txpool_weight);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
bool prune_blockchain = command_line::get_arg(vm, arg_prune_blockchain);
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
bool keep_alt_blocks = command_line::get_arg(vm, arg_keep_alt_blocks);
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
if (m_service_node)
{
r = init_service_node_key();
CHECK_AND_ASSERT_MES(r, false, "Failed to create or load service node key");
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
m_service_node_list.set_my_service_node_keys(&m_service_node_pubkey);
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
}
Nicer looking exit when blockchain.bin is found Do not print the exception message, and write the important bit in red, since people will only read the last line otherwise.
2015-12-31 18:09:00 +01:00
boost::filesystem::path folder(m_config_folder);
Stagenet
2018-02-16 12:04:04 +01:00
if (m_nettype == FAKECHAIN)
Nicer looking exit when blockchain.bin is found Do not print the exception message, and write the important bit in red, since people will only read the last line otherwise.
2015-12-31 18:09:00 +01:00
folder /= "fake";
core: move the db lock to the data directory Locking just one db turns out to not have been a good idea, since the pool and p2p state fdles have to be used anyway. Also ensure the directory exists before tring to lock.
2016-02-04 19:09:45 +01:00
// make sure the data directory exists, and try to lock it
CHECK_AND_ASSERT_MES (boost::filesystem::exists(folder) || boost::filesystem::create_directories(folder), false,
std::string("Failed to create directory ").append(folder.string()).c_str());
Nicer looking exit when blockchain.bin is found Do not print the exception message, and write the important bit in red, since people will only read the last line otherwise.
2015-12-31 18:09:00 +01:00
// check for blockchain.bin
try
{
const boost::filesystem::path old_files = folder;
if (boost::filesystem::exists(old_files / "blockchain.bin"))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MWARNING("Found old-style blockchain.bin in " << old_files.string());
Update cmakelists and misc refs to Loki
2018-04-25 08:13:38 +02:00
MWARNING("Loki now uses a new format. You can either remove blockchain.bin to start syncing");
Update output binaries to loki
2018-04-10 06:51:12 +02:00
MWARNING("the blockchain anew, or use loki-blockchain-export and loki-blockchain-import to");
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MWARNING("convert your existing blockchain.bin to the new format. See README.md for instructions.");
Nicer looking exit when blockchain.bin is found Do not print the exception message, and write the important bit in red, since people will only read the last line otherwise.
2015-12-31 18:09:00 +01:00
return false;
}
}
// folder might not be a directory, etc, etc
catch (...) { }
cryptonote_core: fix db leak on error
2017-12-13 15:04:19 +01:00
std::unique_ptr<BlockchainDB> db(new_db(db_type));
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
if (db == NULL)
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
{
Fix a few minor typos
2016-12-04 14:13:54 +01:00
LOG_ERROR("Attempted to use non-existent database type");
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
return false;
}
folder /= db->get_db_name();
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MGINFO("Loading blockchain from folder " << folder.string() << " ...");
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
const std::string filename = folder.string();
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
// default to fast:async:1 if overridden
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 20:36:51 +02:00
blockchain_db_sync_mode sync_mode = db_defaultsync;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
bool sync_on_blocks = true;
uint64_t sync_threshold = 1;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
if (m_nettype == FAKECHAIN)
{
Disable automatic DB delete on fakechain for integration test
2018-12-12 02:12:15 +01:00
#if !defined(LOKI_ENABLE_INTEGRATION_TEST_HOOKS) // In integration mode, don't delete the DB. This should be explicitly done in the tests. Otherwise the more likely behaviour is persisting the DB across multiple daemons in the same test.
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
// reset the db by removing the database file before opening it
if (!db->remove_data_file(filename))
{
MERROR("Failed to remove data file in " << filename);
return false;
}
Disable automatic DB delete on fakechain for integration test
2018-12-12 02:12:15 +01:00
#endif
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
}
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
try
{
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
uint64_t db_flags = 0;
std::vector<std::string> options;
boost::trim(db_sync_mode);
boost::split(options, db_sync_mode, boost::is_any_of(" :"));
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
const bool db_sync_mode_is_default = command_line::is_arg_defaulted(vm, cryptonote::arg_db_sync_mode);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
for(const auto &option : options)
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MDEBUG("option: " << option);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 04:38:55 +01:00
// default to fast:async:1
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
uint64_t DEFAULT_FLAGS = DBF_FAST;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
if(options.size() == 0)
{
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 04:38:55 +01:00
// default to fast:async:1
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
db_flags = DEFAULT_FLAGS;
}
bool safemode = false;
if(options.size() >= 1)
{
if(options[0] == "safe")
{
safemode = true;
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
db_flags = DBF_SAFE;
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_nosync;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
}
else if(options[0] == "fast")
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 20:36:51 +02:00
{
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
db_flags = DBF_FAST;
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 20:36:51 +02:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
else if(options[0] == "fastest")
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 04:38:55 +01:00
{
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
db_flags = DBF_FASTEST;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
sync_threshold = 1000; // default to fastest:async:1000
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 04:38:55 +01:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
else
db_flags = DEFAULT_FLAGS;
}
if(options.size() >= 2 && !safemode)
{
if(options[1] == "sync")
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_sync;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
else if(options[1] == "async")
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
}
if(options.size() >= 3 && !safemode)
{
Remove blocks_per_sync limits The code used to cap at 5000 blocks per sync. It also treated 0 as 1. Remove these checks; if specified as 0 do no periodic syncs at all. Then the user is responsible for syncing in some external process.
2016-08-29 17:42:52 +02:00
char *endptr;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
uint64_t threshold = strtoull(options[2].c_str(), &endptr, 0);
if (*endptr == '\0' || !strcmp(endptr, "blocks"))
{
sync_on_blocks = true;
sync_threshold = threshold;
}
else if (!strcmp(endptr, "bytes"))
{
sync_on_blocks = false;
sync_threshold = threshold;
}
else
{
LOG_ERROR("Invalid db sync mode: " << options[2]);
return false;
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
}
core: add --db-salvage command line flag Use to load the database when the primary meta page is corrupted
2017-07-31 00:59:52 +02:00
if (db_salvage)
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
db_flags |= DBF_SALVAGE;
core: add --db-salvage command line flag Use to load the database when the primary meta page is corrupted
2017-07-31 00:59:52 +02:00
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
db->open(filename, db_flags);
if(!db->m_open)
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
return false;
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
}
catch (const DB_ERROR& e)
{
Moved logging to target functions rather than caller
2016-10-03 23:11:00 +02:00
LOG_ERROR("Error opening database: " << e.what());
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
return false;
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
m_blockchain_storage.set_user_options(blocks_threads,
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
sync_on_blocks, sync_threshold, sync_mode, fast_sync);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 22:18:08 +02:00
try
{
if (!command_line::is_arg_defaulted(vm, arg_block_notify))
m_blockchain_storage.set_block_notify(std::shared_ptr<tools::Notify>(new tools::Notify(command_line::get_arg(vm, arg_block_notify).c_str())));
}
catch (const std::exception &e)
{
MERROR("Failed to parse block notify spec");
}
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 15:29:28 +01:00
try
{
if (!command_line::is_arg_defaulted(vm, arg_reorg_notify))
m_blockchain_storage.set_reorg_notify(std::shared_ptr<tools::Notify>(new tools::Notify(command_line::get_arg(vm, arg_reorg_notify).c_str())));
}
catch (const std::exception &e)
{
MERROR("Failed to parse reorg notify spec");
}
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
try
{
if (!command_line::is_arg_defaulted(vm, arg_block_rate_notify))
m_block_rate_notify.reset(new tools::Notify(command_line::get_arg(vm, arg_block_rate_notify).c_str()));
}
catch (const std::exception &e)
{
MERROR("Failed to parse block rate notify spec");
}
Merge dev changes
2018-10-10 08:51:16 +02:00
const std::vector<std::pair<uint8_t, uint64_t>> regtest_hard_forks = {std::make_pair(1, 0), std::make_pair(Blockchain::get_hard_fork_heights(MAINNET).back().version, 1), std::make_pair(0, 0)};
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
const cryptonote::test_options regtest_test_options = {
ArticMine's new block weight algorithm This curbs runaway growth while still allowing substantial spikes in block weight Original specification from ArticMine: here is the scaling proposal Define: LongTermBlockWeight Before fork: LongTermBlockWeight = BlockWeight At or after fork: LongTermBlockWeight = min(BlockWeight, 1.4*LongTermEffectiveMedianBlockWeight) Note: To avoid possible consensus issues over rounding the LongTermBlockWeight for a given block should be calculated to the nearest byte, and stored as a integer in the block itself. The stored LongTermBlockWeight is then used for future calculations of the LongTermEffectiveMedianBlockWeight and not recalculated each time. Define: LongTermEffectiveMedianBlockWeight LongTermEffectiveMedianBlockWeight = max(300000, MedianOverPrevious100000Blocks(LongTermBlockWeight)) Change Definition of EffectiveMedianBlockWeight From (current definition) EffectiveMedianBlockWeight = max(300000, MedianOverPrevious100Blocks(BlockWeight)) To (proposed definition) EffectiveMedianBlockWeight = min(max(300000, MedianOverPrevious100Blocks(BlockWeight)), 50*LongTermEffectiveMedianBlockWeight) Notes: 1) There are no other changes to the existing penalty formula, median calculation, fees etc. 2) There is the requirement to store the LongTermBlockWeight of a block unencrypted in the block itself. This is to avoid possible consensus issues over rounding and also to prevent the calculations from becoming unwieldy as we move away from the fork. 3) When the EffectiveMedianBlockWeight cap is reached it is still possible to mine blocks up to 2x the EffectiveMedianBlockWeight by paying the corresponding penalty.
2019-01-21 18:18:50 +01:00
regtest_hard_forks,
0
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
};
const difficulty_type fixed_difficulty = command_line::get_arg(vm, arg_fixed_difficulty);
Merge commit '025187e' into LokiMergeUpstream20180821
2018-09-19 06:05:07 +02:00
Revert "Revert "Store Service Node State to DB as Blob (#125)" (#127)" This reverts commit c4988f5a1fca4aff2c9ca0cd0988e9dd536f6bd0. This un-reverts the changes to store the service node info in the database. This could easily just be its own commit instead of a revert, but doing a revert is amusing (and easier in this case).
2018-08-16 05:08:36 +02:00
BlockchainDB *initialized_db = db.release();
Service Node Checkpoint Storage (#579) * Add functions for storing checkpoints to the DB * Allocate the DB entry on the stack instead of heap * Add virtual overrides for new checkpoint functions * Clean up for pull request, simplify some logic * Revise API to include height in checkpoint header * Move log to top of function even if early exit * Begin moving checkpoints to db * Allow storing of checkpoints to DB * Cleanup for code reviewer, fix unit tests * Fix tests, fix casting issues * Don't use DUPSORT, use height->checkpoint mapping in DB * Remove if 0 disabling checkpoint vote, we already check HF12 * Fix unit test infinite loop * Update db schemas to match blk_checkpoint_header * Code review
2019-05-01 07:57:41 +02:00
// Service Nodes
{
m_service_node_list.set_db_pointer(initialized_db);
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
(Low priority) Add disabled-by-default quorum storage support (#702) * Add disabled-by-default quorum storage support This adds support for storing N blocks of recent expired quorum state history in lokid (and dumping to/from the lmdb). This isn't useful for regular nodes (and that's why we don't store it) but is incredibly useful for the block explorer to be able to report *which* node got deregistered/decommissioned/etc. by a given state_change tx. * Fix quorum states copy
2019-07-17 08:01:54 +02:00
m_service_node_list.set_quorum_history_storage(command_line::get_arg(vm, arg_store_quorum_history));
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
m_blockchain_storage.hook_block_added(m_service_node_list);
m_blockchain_storage.hook_blockchain_detached(m_service_node_list);
m_blockchain_storage.hook_init(m_service_node_list);
m_blockchain_storage.hook_validate_miner_tx(m_service_node_list);
// NOTE: There is an implicit dependency on service node lists being hooked first!
m_blockchain_storage.hook_init(m_quorum_cop);
m_blockchain_storage.hook_block_added(m_quorum_cop);
m_blockchain_storage.hook_blockchain_detached(m_quorum_cop);
Service Node Checkpoint Storage (#579) * Add functions for storing checkpoints to the DB * Allocate the DB entry on the stack instead of heap * Add virtual overrides for new checkpoint functions * Clean up for pull request, simplify some logic * Revise API to include height in checkpoint header * Move log to top of function even if early exit * Begin moving checkpoints to db * Allow storing of checkpoints to DB * Cleanup for code reviewer, fix unit tests * Fix tests, fix casting issues * Don't use DUPSORT, use height->checkpoint mapping in DB * Remove if 0 disabling checkpoint vote, we already check HF12 * Fix unit test infinite loop * Update db schemas to match blk_checkpoint_header * Code review
2019-05-01 07:57:41 +02:00
}
// Checkpoints
{
auto data_dir = boost::filesystem::path(m_config_folder);
boost::filesystem::path json(JSON_HASH_FILE_NAME);
boost::filesystem::path checkpoint_json_hashfile_fullpath = data_dir / json;
m_checkpoints_path = checkpoint_json_hashfile_fullpath.string();
}
Merge commit '8f9c381' into LokiMergeUpstream
2018-10-31 07:26:00 +01:00
r = m_blockchain_storage.init(initialized_db, m_nettype, m_offline, regtest ? &regtest_test_options : test_options, fixed_difficulty, get_checkpoints);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
Add flag to recalculate difficulty per block (#598) * Beging adding functions to recalculate difficulty * Add command line args to utility for recalculating difficulty * Exception safety for recalculating difficulty * Update help text for recalc difficulty * Add recalc flag on the daemon * Make context be const, signify intent for var++ to 1
2019-05-12 01:40:57 +02:00
if (!command_line::is_arg_defaulted(vm, arg_recalculate_difficulty))
{
uint64_t recalc_diff_from_block = command_line::get_arg(vm, arg_recalculate_difficulty);
cryptonote::BlockchainDB::fixup_context context = {};
context.type = cryptonote::BlockchainDB::fixup_type::calculate_difficulty;
context.calculate_difficulty_params.start_height = recalc_diff_from_block;
initialized_db->fixup(context);
}
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
r = m_mempool.init(max_txpool_weight);
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize memory pool");
Fix V1/V2 use of hard fork related parameters Some of it uses hardcoded height, which will need some thinking for next (voted upon) fork.
2016-01-29 16:09:17 +01:00
// now that we have a valid m_blockchain_storage, we can clean out any
// transactions in the pool that do not conform to the current fork
m_mempool.validate(m_blockchain_storage.get_current_hard_fork_version());
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
bool show_time_stats = command_line::get_arg(vm, arg_show_time_stats) != 0;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
m_blockchain_storage.set_show_time_stats(show_time_stats);
moved all stuff to github
2014-03-03 23:07:58 +01:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize blockchain storage");
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
block_sync_size = command_line::get_arg(vm, arg_block_sync_size);
RandomX integration Support RandomX PoW algorithm
2019-04-23 21:32:27 +02:00
if (block_sync_size > BLOCKS_SYNCHRONIZING_MAX_COUNT)
MERROR("Error --dblock-sync-size cannot be greater than " << BLOCKS_SYNCHRONIZING_MAX_COUNT);
core: make the sync chunk block count overridable
2016-09-24 18:00:19 +02:00
core: add a message when loading checkpoints at init time This uses DNS, which can take a while, so it's useful to know this is the culprit when loading pauses
2017-07-31 07:43:13 +02:00
MGINFO("Loading checkpoints");
Rollback chain when receiving conflicting checkpoints (#726)
2019-07-11 06:16:55 +02:00
CHECK_AND_ASSERT_MES(update_checkpoints_from_json_file(), false, "One or more checkpoints loaded from json conflicted with existing checkpoints.");
Fixed segfault with checkpoints loading Bounds checking on blockchain_storage' m_blocks.size() when validating against checkpoints. Also moved initial json & DNS checkpoints load to after blockchain init.
2014-09-27 23:46:12 +02:00
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
// DNS versions checking
if (check_updates_string == "disabled")
check_updates_level = UPDATES_DISABLED;
else if (check_updates_string == "notify")
check_updates_level = UPDATES_NOTIFY;
else if (check_updates_string == "download")
check_updates_level = UPDATES_DOWNLOAD;
else if (check_updates_string == "update")
check_updates_level = UPDATES_UPDATE;
else {
MERROR("Invalid argument to --dns-versions-check: " << check_updates_string);
return false;
}
Stagenet
2018-02-16 12:04:04 +01:00
r = m_miner.init(vm, m_nettype);
Merge pull request #749 bfd4a28 Update BlockchainDB documentation (Thomas Winget) 797357e Change Doxyfile, Blockchain not blockchain_storage (Thomas Winget) c835215 remove defunct code from cryptonote::core (Thomas Winget) 50dba6d cryptonote::core doxygen documentation (Thomas Winget) 8ac329d doxygen documentation for difficulty functions (Thomas Winget) 540a76c Move checkpoint functions into checkpoints class (Thomas Winget) 1b0c98e doxygen documentation for checkpoints.{h,cpp} (Thomas Winget) 89c24ac Remove unnecessary or defunct code (Thomas Winget) ab0ed14 doxygen include private and static members (Thomas Winget) 3a48449 Updated documentation for blockchain.* (Thomas Winget)
2016-03-25 07:22:06 +01:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize miner instance");
moved all stuff to github
2014-03-03 23:07:58 +01:00
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
if (!keep_alt_blocks && !m_blockchain_storage.get_db().is_read_only())
m_blockchain_storage.get_db().drop_alt_blocks();
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
if (prune_blockchain)
{
// display a message if the blockchain is not pruned yet
if (m_blockchain_storage.get_current_blockchain_height() > 1 && !m_blockchain_storage.get_blockchain_pruning_seed())
core: update pruning if using --prune-blockchain on a pruned blockchain Avoids a massive amount of spurious warnings if the last update before the daemon exited was a while ago and the daemon was syncing
2019-05-28 11:09:30 +02:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
MGINFO("Pruning blockchain...");
core: update pruning if using --prune-blockchain on a pruned blockchain Avoids a massive amount of spurious warnings if the last update before the daemon exited was a while ago and the daemon was syncing
2019-05-28 11:09:30 +02:00
CHECK_AND_ASSERT_MES(m_blockchain_storage.prune_blockchain(), false, "Failed to prune blockchain");
}
else
{
CHECK_AND_ASSERT_MES(m_blockchain_storage.update_blockchain_pruning(), false, "Failed to update blockchain pruning");
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
return load_state_data();
}
//-----------------------------------------------------------------------------------------------
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
bool core::init_service_node_key()
{
std::string keypath = m_config_folder + "/key";
if (epee::file_io_utils::is_file_exist(keypath))
{
std::string keystr;
bool r = epee::file_io_utils::load_file_to_string(keypath, keystr);
Fix build warning on Mingw
2018-10-24 03:00:33 +02:00
memcpy(&unwrap(unwrap(m_service_node_key)), keystr.data(), sizeof(m_service_node_key));
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
wipeable_string wipe(keystr);
CHECK_AND_ASSERT_MES(r, false, "failed to load service node key from file");
r = crypto::secret_key_to_public_key(m_service_node_key, m_service_node_pubkey);
CHECK_AND_ASSERT_MES(r, false, "failed to generate pubkey from secret key");
}
else
{
cryptonote::keypair keypair = keypair::generate(hw::get_device("default"));
m_service_node_pubkey = keypair.pub;
m_service_node_key = keypair.sec;
std::string keystr(reinterpret_cast<const char *>(&m_service_node_key), sizeof(m_service_node_key));
bool r = epee::file_io_utils::save_string_to_file(keypath, keystr);
wipeable_string wipe(keystr);
CHECK_AND_ASSERT_MES(r, false, "failed to save service node key to file");
using namespace boost::filesystem;
permissions(keypath, owner_read);
}
MGINFO_YELLOW("Service node pubkey is " << epee::string_tools::pod_to_hex(m_service_node_pubkey));
return true;
}
//-----------------------------------------------------------------------------------------------
moved all stuff to github
2014-03-03 23:07:58 +01:00
bool core::set_genesis_block(const block& b)
{
return m_blockchain_storage.reset_and_set_genesis_block(b);
}
//-----------------------------------------------------------------------------------------------
bool core::load_state_data()
{
// may be some code later
return true;
}
//-----------------------------------------------------------------------------------------------
Fix destruction of service node list locking destroyed mutex
2019-01-18 06:16:23 +01:00
bool core::deinit()
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Fix destruction of service node list locking destroyed mutex
2019-01-18 06:16:23 +01:00
m_service_node_list.store();
m_service_node_list.set_db_pointer(nullptr);
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
m_miner.stop();
m_mempool.deinit();
Removed all code related to fast_exit
2016-10-03 03:06:55 +02:00
m_blockchain_storage.deinit();
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
}
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 20:59:39 +01:00
//-----------------------------------------------------------------------------------------------
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
void core::test_drop_download()
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 20:59:39 +01:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
m_test_drop_download = false;
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 20:59:39 +01:00
}
//-----------------------------------------------------------------------------------------------
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
void core::test_drop_download_height(uint64_t height)
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 20:59:39 +01:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
m_test_drop_download_height = height;
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::get_test_drop_download() const
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
return m_test_drop_download;
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::get_test_drop_download_height() const
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
if (m_test_drop_download_height == 0)
return true;
if (get_blockchain_storage().get_current_blockchain_height() <= m_test_drop_download_height)
return true;
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
return false;
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 20:59:39 +01:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
//-----------------------------------------------------------------------------------------------
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
bool core::handle_incoming_tx_pre(const blobdata& tx_blob, tx_verification_context& tvc, cryptonote::transaction &tx, crypto::hash &tx_hash, bool keeped_by_block, bool relayed, bool do_not_relay)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
tvc = boost::value_initialized<tx_verification_context>();
if(tx_blob.size() > get_max_tx_size())
{
moved non-critical warnings and errors to log level 1
2014-09-09 11:32:00 +02:00
LOG_PRINT_L1("WRONG TRANSACTION BLOB, too big size " << tx_blob.size() << ", rejected");
moved all stuff to github
2014-03-03 23:07:58 +01:00
tvc.m_verifivation_failed = true;
Convey tx verification failure reasons to the RPC client This allows appropriate action to be taken, like displaying the reason to the user. Do just that in simplewallet, which should help a lot in determining why users fail to send. Also make it so a tx which is accepted but not relayed is seen as a success rather than a failure.
2016-03-27 13:35:36 +02:00
tvc.m_too_big = true;
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
remove "using namespace std" from headers It's nasty, and actually breaks on Solaris, where if.h fails to build due to: struct map *if_memmap;
2017-10-10 16:47:08 +02:00
tx_hash = crypto::null_hash;
moved all stuff to github
2014-03-03 23:07:58 +01:00
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
if(!parse_tx_from_blob(tx, tx_hash, tx_blob))
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
moved non-critical warnings and errors to log level 1
2014-09-09 11:32:00 +02:00
LOG_PRINT_L1("WRONG TRANSACTION BLOB, Failed to parse, rejected");
moved all stuff to github
2014-03-03 23:07:58 +01:00
tvc.m_verifivation_failed = true;
return false;
}
//std::cout << "!"<< tx.vin.size() << std::endl;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
bad_semantics_txes_lock.lock();
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
for (int idx = 0; idx < 2; ++idx)
core: cache tx hashes of failing semantics txes
2017-01-22 17:13:31 +01:00
{
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
if (bad_semantics_txes[idx].find(tx_hash) != bad_semantics_txes[idx].end())
{
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
bad_semantics_txes_lock.unlock();
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
LOG_PRINT_L1("Transaction already seen with bad semantics, rejected");
tvc.m_verifivation_failed = true;
return false;
}
core: cache tx hashes of failing semantics txes
2017-01-22 17:13:31 +01:00
}
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
bad_semantics_txes_lock.unlock();
return true;
}
//-----------------------------------------------------------------------------------------------
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
bool core::handle_incoming_tx_post(const blobdata& tx_blob, tx_verification_context& tvc, cryptonote::transaction &tx, crypto::hash &tx_hash, bool keeped_by_block, bool relayed, bool do_not_relay)
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
{
moved all stuff to github
2014-03-03 23:07:58 +01:00
if(!check_tx_syntax(tx))
{
moved non-critical warnings and errors to log level 1
2014-09-09 11:32:00 +02:00
LOG_PRINT_L1("WRONG TRANSACTION BLOB, Failed to check tx " << tx_hash << " syntax, rejected");
moved all stuff to github
2014-03-03 23:07:58 +01:00
tvc.m_verifivation_failed = true;
return false;
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
return true;
}
//-----------------------------------------------------------------------------------------------
void core::set_semantics_failed(const crypto::hash &tx_hash)
{
LOG_PRINT_L1("WRONG TRANSACTION BLOB, Failed to check tx " << tx_hash << " semantic, rejected");
bad_semantics_txes_lock.lock();
bad_semantics_txes[0].insert(tx_hash);
if (bad_semantics_txes[0].size() >= BAD_SEMANTICS_TXES_MAX_SIZE)
{
std::swap(bad_semantics_txes[0], bad_semantics_txes[1]);
bad_semantics_txes[0].clear();
}
bad_semantics_txes_lock.unlock();
}
//-----------------------------------------------------------------------------------------------
require canonical multi output bulletproof layout
2018-03-31 11:49:53 +02:00
static bool is_canonical_bulletproof_layout(const std::vector<rct::Bulletproof> &proofs)
{
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
if (proofs.size() != 1)
return false;
const size_t sz = proofs[0].V.size();
if (sz == 0 || sz > BULLETPROOF_MAX_OUTPUTS)
return false;
require canonical multi output bulletproof layout
2018-03-31 11:49:53 +02:00
return true;
}
//-----------------------------------------------------------------------------------------------
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
bool core::handle_incoming_tx_accumulated_batch(std::vector<tx_verification_batch_info> &tx_info, bool keeped_by_block)
{
bool ret = true;
blockchain: skip checking tx semantics in embedded block hash range If the txes are bad, this'll be picked up by the block hash mismatch since the tx merkle root is part of the block hash.
2017-07-30 10:45:55 +02:00
if (keeped_by_block && get_blockchain_storage().is_within_compiled_block_hash_area())
{
MTRACE("Skipping semantics check for tx kept by block in embedded hash area");
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
return true;
blockchain: skip checking tx semantics in embedded block hash range If the txes are bad, this'll be picked up by the block hash mismatch since the tx merkle root is part of the block hash.
2017-07-30 10:45:55 +02:00
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
std::vector<const rct::rctSig*> rvv;
for (size_t n = 0; n < tx_info.size(); ++n)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
if (!check_tx_semantic(*tx_info[n].tx, keeped_by_block))
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
{
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
continue;
}
Make tx type and version scoped enums This converts the transaction type and version to scoped enum, giving type safety and making the tx type assignment less error prone because there is no implicit conversion or comparison with raw integers that has to be worried about. This ends up converting any use of `cryptonote::transaction::type_xyz` to `cryptonote::transaction::txtype::xyz`. For version, names like `transaction::version_v4` become `cryptonote::txversion::v4_tx_types`. This also allows/includes various other simplifications related to or enabled by this change: - handle `is_deregister` dynamically in serialization code (setting `type::standard` or `type::deregister` rather than using a version-determined union) - `get_type()` is no longer needed with the above change: it is now much simpler to directly access `type` which will always have the correct value (even for v2 or v3 transaction types). And though there was an assertion on the enum value, `get_type()` was being used only sporadically: many places accessed `.type` directly. - the old unscoped enum didn't have a type but was assumed castable to/from `uint16_t`, which technically meant there was potential undefined behaviour when deserializing any type values >= 8. - tx type range checks weren't being done in all serialization paths; they are now. Because `get_type()` was not used everywhere (lots of places simply accessed `.type` directory) these might not have been caught. - `set_type()` is not needed; it was only being used in a single place (wallet2.cpp) and only for v4 txes, so the version protection code was never doing anything. - added a std::ostream << operator for the enum types so that they can be output with `<< tx_type <<` rather than needing to wrap it in `type_to_string(tx_type)` everywhere. For the versions, you get the annotated version string (e.g. 4_tx_types) rather than just the number 4.
2019-06-11 20:53:46 +02:00
if (tx_info[n].tx->type != txtype::standard)
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
continue;
const rct::rctSig &rv = tx_info[n].tx->rct_signatures;
switch (rv.type) {
case rct::RCTTypeNull:
// coinbase should not come here, so we reject for all other types
MERROR_VER("Unexpected Null rctSig type");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
case rct::RCTTypeSimple:
if (!rct::verRctSemanticsSimple(rv))
{
MERROR_VER("rct signature semantics check failed");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
break;
case rct::RCTTypeFull:
if (!rct::verRct(rv, true))
{
MERROR_VER("rct signature semantics check failed");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
break;
case rct::RCTTypeBulletproof:
add a bulletproof version, new bulletproof type, and rct config This makes it easier to modify the bulletproof format
2019-01-06 14:47:16 +01:00
case rct::RCTTypeBulletproof2:
require canonical multi output bulletproof layout
2018-03-31 11:49:53 +02:00
if (!is_canonical_bulletproof_layout(rv.p.bulletproofs))
{
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
MERROR_VER("Bulletproof does not have canonical form");
require canonical multi output bulletproof layout
2018-03-31 11:49:53 +02:00
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
rvv.push_back(&rv); // delayed batch verification
break;
default:
MERROR_VER("Unknown rct type: " << rv.type);
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
}
if (!rvv.empty() && !rct::verRctSemanticsSimple(rvv))
{
LOG_PRINT_L1("One transaction among this group has bad semantics, verifying one at a time");
ret = false;
const bool assumed_bad = rvv.size() == 1; // if there's only one tx, it must be the bad one
for (size_t n = 0; n < tx_info.size(); ++n)
{
if (!tx_info[n].result)
continue;
add a bulletproof version, new bulletproof type, and rct config This makes it easier to modify the bulletproof format
2019-01-06 14:47:16 +01:00
if (tx_info[n].tx->rct_signatures.type != rct::RCTTypeBulletproof && tx_info[n].tx->rct_signatures.type != rct::RCTTypeBulletproof2)
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
continue;
if (assumed_bad || !rct::verRctSemanticsSimple(tx_info[n].tx->rct_signatures))
{
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
}
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
return ret;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::handle_incoming_txs(const std::vector<blobdata>& tx_blobs, std::vector<tx_verification_context>& tvc, bool keeped_by_block, bool relayed, bool do_not_relay)
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
{
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
TRY_ENTRY();
core: lock incoming tx lock when checking the txpool and chain This gets rid of an innocuous race trying to add the same tx twice to the txpool
2018-05-18 10:38:38 +02:00
CRITICAL_REGION_LOCAL(m_incoming_tx_lock);
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
struct result { bool res; cryptonote::transaction tx; crypto::hash hash; };
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
std::vector<result> results(tx_blobs.size());
tvc.resize(tx_blobs.size());
core: remove threadpool dependency from header
2018-04-23 19:55:58 +02:00
tools::threadpool& tpool = tools::threadpool::getInstance();
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
tools::threadpool::waiter waiter;
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<blobdata>::const_iterator it = tx_blobs.begin();
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
for (size_t i = 0; i < tx_blobs.size(); i++, ++it) {
core: remove threadpool dependency from header
2018-04-23 19:55:58 +02:00
tpool.submit(&waiter, [&, i, it] {
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
try
{
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
results[i].res = handle_incoming_tx_pre(*it, tvc[i], results[i].tx, results[i].hash, keeped_by_block, relayed, do_not_relay);
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
}
catch (const std::exception &e)
{
MERROR_VER("Exception in handle_incoming_tx_pre: " << e.what());
cryptonote: fix calculating coinbase tx hash Also set error flag on exception when handling new txes to keep tests working
2019-03-20 13:11:24 +01:00
tvc[i].m_verifivation_failed = true;
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
results[i].res = false;
}
});
}
threadpool: allow leaf functions to run concurrently Decrease the number of worker threads by one to account for the fact the calling thread acts as a worker thread now
2018-04-26 12:44:47 +02:00
waiter.wait(&tpool);
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
it = tx_blobs.begin();
core: don't verify range proofs multiple times
2018-10-17 18:22:27 +02:00
std::vector<bool> already_have(tx_blobs.size(), false);
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
for (size_t i = 0; i < tx_blobs.size(); i++, ++it) {
if (!results[i].res)
continue;
if(m_mempool.have_tx(results[i].hash))
{
LOG_PRINT_L2("tx " << results[i].hash << "already have transaction in tx_pool");
core: don't verify range proofs multiple times
2018-10-17 18:22:27 +02:00
already_have[i] = true;
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
}
else if(m_blockchain_storage.have_tx(results[i].hash))
{
LOG_PRINT_L2("tx " << results[i].hash << " already have transaction in blockchain");
core: don't verify range proofs multiple times
2018-10-17 18:22:27 +02:00
already_have[i] = true;
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
}
else
{
core: remove threadpool dependency from header
2018-04-23 19:55:58 +02:00
tpool.submit(&waiter, [&, i, it] {
core: guard against exceptions in tx verification worker threads
2017-09-10 16:33:13 +02:00
try
{
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
results[i].res = handle_incoming_tx_post(*it, tvc[i], results[i].tx, results[i].hash, keeped_by_block, relayed, do_not_relay);
core: guard against exceptions in tx verification worker threads
2017-09-10 16:33:13 +02:00
}
catch (const std::exception &e)
{
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
MERROR_VER("Exception in handle_incoming_tx_post: " << e.what());
cryptonote: fix calculating coinbase tx hash Also set error flag on exception when handling new txes to keep tests working
2019-03-20 13:11:24 +01:00
tvc[i].m_verifivation_failed = true;
core: guard against exceptions in tx verification worker threads
2017-09-10 16:33:13 +02:00
results[i].res = false;
}
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
});
}
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
}
threadpool: allow leaf functions to run concurrently Decrease the number of worker threads by one to account for the fact the calling thread acts as a worker thread now
2018-04-26 12:44:47 +02:00
waiter.wait(&tpool);
moved all stuff to github
2014-03-03 23:07:58 +01:00
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
std::vector<tx_verification_batch_info> tx_info;
tx_info.reserve(tx_blobs.size());
for (size_t i = 0; i < tx_blobs.size(); i++) {
core: don't verify range proofs multiple times
2018-10-17 18:22:27 +02:00
if (!results[i].res || already_have[i])
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
continue;
tx_info.push_back({&results[i].tx, results[i].hash, tvc[i], results[i].res});
}
if (!tx_info.empty())
handle_incoming_tx_accumulated_batch(tx_info, keeped_by_block);
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
bool ok = true;
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
it = tx_blobs.begin();
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
for (size_t i = 0; i < tx_blobs.size(); i++, ++it) {
if (!results[i].res)
{
ok = false;
continue;
}
core: fix adding new pre-hoh block when a tx is already in the pool
2018-11-27 00:42:14 +01:00
if (keeped_by_block)
get_blockchain_storage().on_new_tx_from_block(results[i].tx);
if (already_have[i])
continue;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
const size_t weight = get_transaction_weight(results[i].tx, it->size());
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
ok &= add_new_tx(results[i].tx, results[i].hash, tx_blobs[i], weight, tvc[i], keeped_by_block, relayed, do_not_relay);
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
if(tvc[i].m_verifivation_failed)
Fix TXBuilder default hf value causing regression in core_tests
2018-11-21 06:30:16 +01:00
{
MERROR_VER("Transaction verification failed: " << results[i].hash);
}
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
else if(tvc[i].m_verifivation_impossible)
Fix TXBuilder default hf value causing regression in core_tests
2018-11-21 06:30:16 +01:00
{
MERROR_VER("Transaction verification impossible: " << results[i].hash);
}
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
if(tvc[i].m_added_to_pool)
MDEBUG("tx added: " << results[i].hash);
}
return ok;
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
CATCH_ENTRY_L0("core::handle_incoming_txs()", false);
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::handle_incoming_tx(const blobdata& tx_blob, tx_verification_context& tvc, bool keeped_by_block, bool relayed, bool do_not_relay)
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<cryptonote::blobdata> tx_blobs;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
tx_blobs.push_back(tx_blob);
std::vector<tx_verification_context> tvcv(1);
bool r = handle_incoming_txs(tx_blobs, tvcv, keeped_by_block, relayed, do_not_relay);
tvc = tvcv[0];
moved all stuff to github
2014-03-03 23:07:58 +01:00
return r;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::get_stat_info(core_stat_info& st_inf) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
st_inf.mining_speed = m_miner.get_speed();
st_inf.alternative_blocks = m_blockchain_storage.get_alternative_blocks_count();
st_inf.blockchain_height = m_blockchain_storage.get_current_blockchain_height();
st_inf.tx_pool_size = m_mempool.get_transactions_count();
st_inf.top_block_id_str = epee::string_tools::pod_to_hex(m_blockchain_storage.get_tail_id());
return true;
}
Add minor whitespace/delete dead code after diffing Monero and Loki Mostly small whitespace changes that make comparing a diff of the entirety of Loki and Monero less noisy.
2018-10-24 02:33:48 +02:00
moved all stuff to github
2014-03-03 23:07:58 +01:00
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::check_tx_semantic(const transaction& tx, bool keeped_by_block) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Make tx type and version scoped enums This converts the transaction type and version to scoped enum, giving type safety and making the tx type assignment less error prone because there is no implicit conversion or comparison with raw integers that has to be worried about. This ends up converting any use of `cryptonote::transaction::type_xyz` to `cryptonote::transaction::txtype::xyz`. For version, names like `transaction::version_v4` become `cryptonote::txversion::v4_tx_types`. This also allows/includes various other simplifications related to or enabled by this change: - handle `is_deregister` dynamically in serialization code (setting `type::standard` or `type::deregister` rather than using a version-determined union) - `get_type()` is no longer needed with the above change: it is now much simpler to directly access `type` which will always have the correct value (even for v2 or v3 transaction types). And though there was an assertion on the enum value, `get_type()` was being used only sporadically: many places accessed `.type` directly. - the old unscoped enum didn't have a type but was assumed castable to/from `uint16_t`, which technically meant there was potential undefined behaviour when deserializing any type values >= 8. - tx type range checks weren't being done in all serialization paths; they are now. Because `get_type()` was not used everywhere (lots of places simply accessed `.type` directory) these might not have been caught. - `set_type()` is not needed; it was only being used in a single place (wallet2.cpp) and only for v4 txes, so the version protection code was never doing anything. - added a std::ostream << operator for the enum types so that they can be output with `<< tx_type <<` rather than needing to wrap it in `type_to_string(tx_type)` everywhere. For the versions, you get the annotated version string (e.g. 4_tx_types) rather than just the number 4.
2019-06-11 20:53:46 +02:00
if (tx.type != txtype::standard)
Check that inputs/outputs size is 0 for deregister
2018-07-25 03:13:38 +02:00
{
if (tx.vin.size() != 0)
{
Make tx type and version scoped enums This converts the transaction type and version to scoped enum, giving type safety and making the tx type assignment less error prone because there is no implicit conversion or comparison with raw integers that has to be worried about. This ends up converting any use of `cryptonote::transaction::type_xyz` to `cryptonote::transaction::txtype::xyz`. For version, names like `transaction::version_v4` become `cryptonote::txversion::v4_tx_types`. This also allows/includes various other simplifications related to or enabled by this change: - handle `is_deregister` dynamically in serialization code (setting `type::standard` or `type::deregister` rather than using a version-determined union) - `get_type()` is no longer needed with the above change: it is now much simpler to directly access `type` which will always have the correct value (even for v2 or v3 transaction types). And though there was an assertion on the enum value, `get_type()` was being used only sporadically: many places accessed `.type` directly. - the old unscoped enum didn't have a type but was assumed castable to/from `uint16_t`, which technically meant there was potential undefined behaviour when deserializing any type values >= 8. - tx type range checks weren't being done in all serialization paths; they are now. Because `get_type()` was not used everywhere (lots of places simply accessed `.type` directory) these might not have been caught. - `set_type()` is not needed; it was only being used in a single place (wallet2.cpp) and only for v4 txes, so the version protection code was never doing anything. - added a std::ostream << operator for the enum types so that they can be output with `<< tx_type <<` rather than needing to wrap it in `type_to_string(tx_type)` everywhere. For the versions, you get the annotated version string (e.g. 4_tx_types) rather than just the number 4.
2019-06-11 20:53:46 +02:00
MERROR_VER("tx type: " << tx.type << " must have 0 inputs, received: " << tx.vin.size() << ", rejected for tx id = " << get_transaction_hash(tx));
Check that inputs/outputs size is 0 for deregister
2018-07-25 03:13:38 +02:00
return false;
}
}
else if (!tx.vin.size())
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx with empty inputs, rejected for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
if(!check_inputs_types_supported(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("unsupported input types for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
if(!check_outs_valid(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx with invalid outputs, rejected for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
Make tx type and version scoped enums This converts the transaction type and version to scoped enum, giving type safety and making the tx type assignment less error prone because there is no implicit conversion or comparison with raw integers that has to be worried about. This ends up converting any use of `cryptonote::transaction::type_xyz` to `cryptonote::transaction::txtype::xyz`. For version, names like `transaction::version_v4` become `cryptonote::txversion::v4_tx_types`. This also allows/includes various other simplifications related to or enabled by this change: - handle `is_deregister` dynamically in serialization code (setting `type::standard` or `type::deregister` rather than using a version-determined union) - `get_type()` is no longer needed with the above change: it is now much simpler to directly access `type` which will always have the correct value (even for v2 or v3 transaction types). And though there was an assertion on the enum value, `get_type()` was being used only sporadically: many places accessed `.type` directly. - the old unscoped enum didn't have a type but was assumed castable to/from `uint16_t`, which technically meant there was potential undefined behaviour when deserializing any type values >= 8. - tx type range checks weren't being done in all serialization paths; they are now. Because `get_type()` was not used everywhere (lots of places simply accessed `.type` directory) these might not have been caught. - `set_type()` is not needed; it was only being used in a single place (wallet2.cpp) and only for v4 txes, so the version protection code was never doing anything. - added a std::ostream << operator for the enum types so that they can be output with `<< tx_type <<` rather than needing to wrap it in `type_to_string(tx_type)` everywhere. For the versions, you get the annotated version string (e.g. 4_tx_types) rather than just the number 4.
2019-06-11 20:53:46 +02:00
if (tx.version >= txversion::v2_ringct)
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
{
if (tx.rct_signatures.outPk.size() != tx.vout.size())
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx with mismatched vout/outPk count, rejected for tx id= " << get_transaction_hash(tx));
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
return false;
}
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
if(!check_money_overflow(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx has money overflow, rejected for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
Make tx type and version scoped enums This converts the transaction type and version to scoped enum, giving type safety and making the tx type assignment less error prone because there is no implicit conversion or comparison with raw integers that has to be worried about. This ends up converting any use of `cryptonote::transaction::type_xyz` to `cryptonote::transaction::txtype::xyz`. For version, names like `transaction::version_v4` become `cryptonote::txversion::v4_tx_types`. This also allows/includes various other simplifications related to or enabled by this change: - handle `is_deregister` dynamically in serialization code (setting `type::standard` or `type::deregister` rather than using a version-determined union) - `get_type()` is no longer needed with the above change: it is now much simpler to directly access `type` which will always have the correct value (even for v2 or v3 transaction types). And though there was an assertion on the enum value, `get_type()` was being used only sporadically: many places accessed `.type` directly. - the old unscoped enum didn't have a type but was assumed castable to/from `uint16_t`, which technically meant there was potential undefined behaviour when deserializing any type values >= 8. - tx type range checks weren't being done in all serialization paths; they are now. Because `get_type()` was not used everywhere (lots of places simply accessed `.type` directory) these might not have been caught. - `set_type()` is not needed; it was only being used in a single place (wallet2.cpp) and only for v4 txes, so the version protection code was never doing anything. - added a std::ostream << operator for the enum types so that they can be output with `<< tx_type <<` rather than needing to wrap it in `type_to_string(tx_type)` everywhere. For the versions, you get the annotated version string (e.g. 4_tx_types) rather than just the number 4.
2019-06-11 20:53:46 +02:00
if (tx.version == txversion::v1)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
uint64_t amount_in = 0;
get_inputs_money_amount(tx, amount_in);
uint64_t amount_out = get_outs_money_amount(tx);
if(amount_in <= amount_out)
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx with wrong amounts: ins " << amount_in << ", outs " << amount_out << ", rejected for tx id= " << get_transaction_hash(tx));
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
return false;
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
if(!keeped_by_block && get_transaction_weight(tx) >= m_blockchain_storage.get_current_cumulative_block_weight_limit() - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
MERROR_VER("tx is too large " << get_transaction_weight(tx) << ", expected not bigger than " << m_blockchain_storage.get_current_cumulative_block_weight_limit() - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE);
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
if(!check_tx_inputs_keyimages_diff(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx uses a single key image more than once");
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
core: forbid duplicate ring members from v6 This avoids someone adding what amounts to mixin 0 rings in practice, as there is no other good reason to allow this.
2017-07-18 23:38:26 +02:00
if (!check_tx_inputs_ring_members_diff(tx))
{
MERROR_VER("tx uses duplicate ring members");
return false;
}
core: test key images against validity domain
2017-02-20 18:41:39 +01:00
if (!check_tx_inputs_keyimages_domain(tx))
{
MERROR_VER("tx uses key image not in the valid domain");
return false;
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::is_key_image_spent(const crypto::key_image &key_image) const
Add a is_key_image_spent daemon command and RPC call
2015-08-11 11:49:15 +02:00
{
return m_blockchain_storage.have_tx_keyimg_as_spent(key_image);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::are_key_images_spent(const std::vector<crypto::key_image>& key_im, std::vector<bool> &spent) const
Add a is_key_image_spent daemon command and RPC call
2015-08-11 11:49:15 +02:00
{
spent.clear();
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 21:38:10 +01:00
for(auto& ki: key_im)
Add a is_key_image_spent daemon command and RPC call
2015-08-11 11:49:15 +02:00
{
spent.push_back(m_blockchain_storage.have_tx_keyimg_as_spent(ki));
}
return true;
}
//-----------------------------------------------------------------------------------------------
cryptonote_protocol: large block sync size before v4
2017-08-16 22:19:53 +02:00
size_t core::get_block_sync_size(uint64_t height) const
{
Stagenet
2018-02-16 12:04:04 +01:00
static const uint64_t quick_height = m_nettype == TESTNET ? 801219 : m_nettype == MAINNET ? 1220516 : 0;
cryptonote_protocol: large block sync size before v4
2017-08-16 22:19:53 +02:00
if (block_sync_size > 0)
return block_sync_size;
if (height >= quick_height)
return BLOCKS_SYNCHRONIZING_DEFAULT_COUNT;
return BLOCKS_SYNCHRONIZING_DEFAULT_COUNT_PRE_V4;
}
//-----------------------------------------------------------------------------------------------
json serialization for rpc-relevant monero types Structured {de-,}serialization methods for (many new) types which are used for requests or responses in the RPC. New types include RPC requests and responses, and structs which compose types within those. # Conflicts: # src/cryptonote_core/blockchain.cpp
2017-09-05 18:20:27 +02:00
bool core::are_key_images_spent_in_pool(const std::vector<crypto::key_image>& key_im, std::vector<bool> &spent) const
{
spent.clear();
return m_mempool.check_for_key_images(key_im, spent);
}
//-----------------------------------------------------------------------------------------------
patched parameter type bug that was causing failure on 32 bit
2016-10-13 18:49:25 +02:00
std::pair<uint64_t, uint64_t> core::get_coinbase_tx_sum(const uint64_t start_offset, const size_t count)
added print_coinbase_tx_sum option
2016-10-10 21:45:51 +02:00
{
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-11 01:55:18 +02:00
uint64_t emission_amount = 0;
uint64_t total_fee_amount = 0;
Don't copy blockchain for coinbase_tx_sum Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:29:51 +02:00
if (count)
added print_coinbase_tx_sum option
2016-10-10 21:45:51 +02:00
{
Don't copy blockchain for coinbase_tx_sum Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:29:51 +02:00
const uint64_t end = start_offset + count - 1;
m_blockchain_storage.for_blocks_range(start_offset, end,
[this, &emission_amount, &total_fee_amount](uint64_t, const crypto::hash& hash, const block& b){
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<transaction> txs;
std::vector<crypto::hash> missed_txs;
core: fix recalculation of emission/fees It was counting things many, many times
2016-12-21 12:18:33 +01:00
uint64_t coinbase_amount = get_outs_money_amount(b.miner_tx);
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-11 01:55:18 +02:00
this->get_transactions(b.tx_hashes, txs, missed_txs);
core: fix recalculation of emission/fees It was counting things many, many times
2016-12-21 12:18:33 +01:00
uint64_t tx_fee_amount = 0;
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 21:38:10 +01:00
for(const auto& tx: txs)
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-11 01:55:18 +02:00
{
tx_fee_amount += get_tx_fee(tx);
}
emission_amount += coinbase_amount - tx_fee_amount;
total_fee_amount += tx_fee_amount;
Don't copy blockchain for coinbase_tx_sum Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:29:51 +02:00
return true;
});
added print_coinbase_tx_sum option
2016-10-10 21:45:51 +02:00
}
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-11 01:55:18 +02:00
return std::pair<uint64_t, uint64_t>(emission_amount, total_fee_amount);
added print_coinbase_tx_sum option
2016-10-10 21:45:51 +02:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::check_tx_inputs_keyimages_diff(const transaction& tx) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
std::unordered_set<crypto::key_image> ki;
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 21:38:10 +01:00
for(const auto& in: tx.vin)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
CHECKED_GET_SPECIFIC_VARIANT(in, const txin_to_key, tokey_in, false);
if(!ki.insert(tokey_in.k_image).second)
return false;
}
return true;
}
//-----------------------------------------------------------------------------------------------
core: forbid duplicate ring members from v6 This avoids someone adding what amounts to mixin 0 rings in practice, as there is no other good reason to allow this.
2017-07-18 23:38:26 +02:00
bool core::check_tx_inputs_ring_members_diff(const transaction& tx) const
{
const uint8_t version = m_blockchain_storage.get_current_hard_fork_version();
if (version >= 6)
{
for(const auto& in: tx.vin)
{
CHECKED_GET_SPECIFIC_VARIANT(in, const txin_to_key, tokey_in, false);
for (size_t n = 1; n < tokey_in.key_offsets.size(); ++n)
if (tokey_in.key_offsets[n] == 0)
return false;
}
}
return true;
}
//-----------------------------------------------------------------------------------------------
core: test key images against validity domain
2017-02-20 18:41:39 +01:00
bool core::check_tx_inputs_keyimages_domain(const transaction& tx) const
{
std::unordered_set<crypto::key_image> ki;
for(const auto& in: tx.vin)
{
CHECKED_GET_SPECIFIC_VARIANT(in, const txin_to_key, tokey_in, false);
if (!(rct::scalarmultKey(rct::ki2rct(tokey_in.k_image), rct::curveOrder()) == rct::identity()))
return false;
}
return true;
}
//-----------------------------------------------------------------------------------------------
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
bool core::add_new_tx(transaction& tx, tx_verification_context& tvc, bool keeped_by_block, bool relayed, bool do_not_relay)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
crypto::hash tx_hash = get_transaction_hash(tx);
blobdata bl;
t_serializable_object_to_blob(tx, bl);
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
size_t tx_weight = get_transaction_weight(tx, bl.size());
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
return add_new_tx(tx, tx_hash, bl, tx_weight, tvc, keeped_by_block, relayed, do_not_relay);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
size_t core::get_blockchain_total_transactions() const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_total_transactions();
}
//-----------------------------------------------------------------------------------------------
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
bool core::add_new_tx(transaction& tx, const crypto::hash& tx_hash, const cryptonote::blobdata &blob, size_t tx_weight, tx_verification_context& tvc, bool keeped_by_block, bool relayed, bool do_not_relay)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
if(m_mempool.have_tx(tx_hash))
{
LOG_PRINT_L2("tx " << tx_hash << "already have transaction in tx_pool");
return true;
}
if(m_blockchain_storage.have_tx(tx_hash))
{
LOG_PRINT_L2("tx " << tx_hash << " already have transaction in blockchain");
return true;
}
Fix V1/V2 use of hard fork related parameters Some of it uses hardcoded height, which will need some thinking for next (voted upon) fork.
2016-01-29 16:09:17 +01:00
uint8_t version = m_blockchain_storage.get_current_hard_fork_version();
Resolve pubkeys from state change in dupe check in the tx_pool (#707) * Resolve public keys from state changes for the TX pool to avoid dupes * Update copy pasted error message for quorum index OOB * Code review
2019-07-04 09:25:02 +02:00
return m_mempool.add_tx(tx, tx_hash, blob, tx_weight, tvc, keeped_by_block, relayed, do_not_relay, version, m_service_node_list);
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
}
//-----------------------------------------------------------------------------------------------
bool core::relay_txpool_transactions()
{
// we attempt to relay txes that should be relayed, but were not
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<std::pair<crypto::hash, cryptonote::blobdata>> txs;
core: don't try to re-relay an empty set of pool transactions
2017-03-12 15:51:32 +01:00
if (m_mempool.get_relayable_transactions(txs) && !txs.empty())
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
{
cryptonote_connection_context fake_context = AUTO_VAL_INIT(fake_context);
tx_verification_context tvc = AUTO_VAL_INIT(tvc);
NOTIFY_NEW_TRANSACTIONS::request r;
for (auto it = txs.begin(); it != txs.end(); ++it)
{
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
r.txs.push_back(it->second);
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
}
get_protocol()->relay_transactions(r, fake_context);
m_mempool.set_relayed(txs);
}
return true;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
quorum_cop: added quorum cop and service node keys to lokid Submit deregistration autonomously, prune uptime_proofs Move pruner to relayer, fix inf. loop, quorum participation bool return find_if, fix reorg logic, make 2hrs check a variable Add iterator assign on erase, adjust detached check qacop: review iteration, changed warnings and std::find call Accept proofs excluding the added time buffer
2018-07-12 03:58:17 +02:00
bool core::submit_uptime_proof()
{
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
if (!m_service_node)
return true;
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
NOTIFY_UPTIME_PROOF::request req = m_service_node_list.generate_uptime_proof(m_service_node_pubkey, m_service_node_key, m_sn_public_ip, m_storage_port);
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
cryptonote_connection_context fake_context = AUTO_VAL_INIT(fake_context);
bool relayed = get_protocol()->relay_uptime_proof(req, fake_context);
if (relayed)
MGINFO("Submitted uptime-proof for service node (yours): " << m_service_node_pubkey);
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
return true;
}
//-----------------------------------------------------------------------------------------------
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
bool core::handle_uptime_proof(const NOTIFY_UPTIME_PROOF::request &proof)
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
{
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
return m_service_node_list.handle_uptime_proof(proof);
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
}
//-----------------------------------------------------------------------------------------------
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
void core::on_transaction_relayed(const cryptonote::blobdata& tx_blob)
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<std::pair<crypto::hash, cryptonote::blobdata>> txs;
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
cryptonote::transaction tx;
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
crypto::hash tx_hash;
if (!parse_and_validate_tx_from_blob(tx_blob, tx, tx_hash))
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
{
Fix a few minor typos
2016-12-04 14:13:54 +01:00
LOG_ERROR("Failed to parse relayed transaction");
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
return;
}
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
txs.push_back(std::make_pair(tx_hash, std::move(tx_blob)));
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
m_mempool.set_relayed(txs);
}
//-----------------------------------------------------------------------------------------------
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
bool core::relay_service_node_votes()
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
{
Remove NOTIFY_NEW_DEREGISTER_VOTE code, deprecated by service node vote
2019-08-07 02:18:53 +02:00
NOTIFY_NEW_SERVICE_NODE_VOTE::request req = {};
req.votes = m_quorum_cop.get_relayable_votes(get_current_blockchain_height());
if (req.votes.size())
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
{
Remove NOTIFY_NEW_DEREGISTER_VOTE code, deprecated by service node vote
2019-08-07 02:18:53 +02:00
cryptonote_connection_context fake_context = AUTO_VAL_INIT(fake_context);
if (get_protocol()->relay_service_node_votes(req, fake_context))
Merge dev into upstream
2019-05-01 08:01:17 +02:00
{
Remove NOTIFY_NEW_DEREGISTER_VOTE code, deprecated by service node vote
2019-08-07 02:18:53 +02:00
m_quorum_cop.set_votes_relayed(req.votes);
Merge dev into upstream
2019-05-01 08:01:17 +02:00
}
}
return true;
}
//-----------------------------------------------------------------------------------------------
Add expected total reward to RPC "getblocktemplate". Only works from V5 fork onward - returns 0 before that block.
2017-03-28 16:55:38 +02:00
bool core::get_block_template(block& b, const account_public_address& adr, difficulty_type& diffic, uint64_t& height, uint64_t& expected_reward, const blobdata& ex_nonce)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Add expected total reward to RPC "getblocktemplate". Only works from V5 fork onward - returns 0 before that block.
2017-03-28 16:55:38 +02:00
return m_blockchain_storage.create_block_template(b, adr, diffic, height, expected_reward, ex_nonce);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
functional_tests: add more blockchain related tests Related to emission, reorgs, getting tx data back, output distribution and histogram
2019-03-23 17:20:08 +01:00
bool core::get_block_template(block& b, const crypto::hash *prev_block, const account_public_address& adr, difficulty_type& diffic, uint64_t& height, uint64_t& expected_reward, const blobdata& ex_nonce)
{
return m_blockchain_storage.create_block_template(b, prev_block, adr, diffic, height, expected_reward, ex_nonce);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, NOTIFY_RESPONSE_CHAIN_ENTRY::request& resp) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.find_blockchain_supplement(qblock_ids, resp);
}
//-----------------------------------------------------------------------------------------------
rpc: rework to avoid repeated calculations in get_blocks.bin
2018-04-16 21:16:00 +02:00
bool core::find_blockchain_supplement(const uint64_t req_start_block, const std::list<crypto::hash>& qblock_ids, std::vector<std::pair<std::pair<cryptonote::blobdata, crypto::hash>, std::vector<std::pair<crypto::hash, cryptonote::blobdata> > > >& blocks, uint64_t& total_height, uint64_t& start_height, bool pruned, bool get_miner_tx_hash, size_t max_count) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
rpc: rework to avoid repeated calculations in get_blocks.bin
2018-04-16 21:16:00 +02:00
return m_blockchain_storage.find_blockchain_supplement(req_start_block, qblock_ids, blocks, total_height, start_height, pruned, get_miner_tx_hash, max_count);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
Add a get_outs (fully text based) version of get_outs.bin
2016-11-22 21:00:40 +01:00
bool core::get_outs(const COMMAND_RPC_GET_OUTPUTS_BIN::request& req, COMMAND_RPC_GET_OUTPUTS_BIN::response& res) const
Fake outs set is now decided by the wallet This plugs a privacy leak from the wallet to the daemon, as the daemon could previously see what input is included as a transaction input, which the daemon hadn't previously supplied. Now, the wallet requests a particular set of outputs, including the real one. This can result in transactions that can't be accepted if the wallet happens to select too many outputs with non standard unlock times. The daemon could know this and select another output, but the wallet is blind to it. It's currently very unlikely since I don't think anything uses non default unlock times. The wallet requests more outputs than necessary so it can use spares if any of the returns outputs are still locked. If there are not enough spares to reach the desired mixin, the transaction will fail.
2016-08-02 22:48:09 +02:00
{
return m_blockchain_storage.get_outs(req, res);
}
//-----------------------------------------------------------------------------------------------
speed up get_output_distribution (and precalc common case)
2018-05-16 23:41:50 +02:00
bool core::get_output_distribution(uint64_t amount, uint64_t from_height, uint64_t to_height, uint64_t &start_height, std::vector<uint64_t> &distribution, uint64_t &base) const
add RPC to get a histogram of outputs of a given amount
2018-02-19 12:15:15 +01:00
{
speed up get_output_distribution (and precalc common case)
2018-05-16 23:41:50 +02:00
return m_blockchain_storage.get_output_distribution(amount, from_height, to_height, start_height, distribution, base);
add RPC to get a histogram of outputs of a given amount
2018-02-19 12:15:15 +01:00
}
//-----------------------------------------------------------------------------------------------
Infinite Staking Part 2 (#406) * Cleanup and undoing some protocol breakages * Simplify expiration of nodes * Request unlock schedules entire node for expiration * Fix off by one in expiring nodes * Undo expiring code for pre v10 nodes * Fix RPC returning register as unlock height and not checking 0 * Rename key image unlock height const * Undo testnet hardfork debug changes * Remove is_type for get_type, fix missing var rename * Move serialisable data into public namespace * Serialise tx types properly * Fix typo in no service node known msg * Code review * Fix == to >= on serialising tx type * Code review 2 * Fix tests and key image unlock * Add command to print locked key images * Update ui to display lock stakes, query in print cmd blacklist * Modify print stakes to be less slow * Remove autostaking code * Refactor staking into sweep functions It appears staking was derived off stake_main written separately at implementation at the beginning. This merges them back into a common code path, after removing autostake there's only some minor differences. It also makes sure that any changes to sweeping upstream are going to be considered in the staking process which we want. * Display unlock height for stakes * Begin creating output blacklist * Make blacklist output a migration step * Implement get_output_blacklist for lmdb * In wallet output selection ignore blacklisted outputs * Apply blacklisted outputs to output selection * Fix broken tests, switch key image unlock * Fix broken unit_tests * Begin change to limit locked key images to 4 globally * Revamp prepare registration for new min contribution rules * Fix up old back case in prepare registration * Remove debug code * Cleanup debug code and some unecessary changes * Fix migration step on mainnet db * Fix blacklist outputs for pre-existing DB's * Remove irrelevant note * Tweak scanning addresses for locked stakes Since we only now allow contributions from the primary address we can skip checking all subaddress + lookahead to speed up wallet scanning * Define macro for SCNu64 for Mingw * Fix failure on empty DB * Add missing error msg, remove contributor from stake * Improve staking messages * Flush prompt to always display * Return the msg from stake failure and fix stake parsing error * Tweak fork rules for smaller bulletproofs * Tweak pooled nodes minimum amounts * Fix crash on exit, there's no need to store on destructor Since all information about service nodes is derived from the blockchain and we store state every time we receive a block, storing in the destructor is redundant as there is no new information to store. * Make prompt be consistent with CLI * Check max number of key images from per user to node * Implement error message on get_output_blacklist failure * Remove resolved TODO's/comments * Handle infinite staking in print_sn * Atoi->strtol, fix prepare_registration, virtual override, stale msgs
2019-02-14 02:12:57 +01:00
bool core::get_output_blacklist(std::vector<uint64_t> &blacklist) const
{
return m_blockchain_storage.get_output_blacklist(blacklist);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_tx_outputs_gindexs(tx_id, indexs);
}
//-----------------------------------------------------------------------------------------------
blockchain_db: speedup tx output gathering We know all the data we'll want for getblocks.bin is contiguous
2018-12-16 14:28:49 +01:00
bool core::get_tx_outputs_gindexs(const crypto::hash& tx_id, size_t n_txes, std::vector<std::vector<uint64_t>>& indexs) const
{
return m_blockchain_storage.get_tx_outputs_gindexs(tx_id, n_txes, indexs);
}
//-----------------------------------------------------------------------------------------------
moved all stuff to github
2014-03-03 23:07:58 +01:00
void core::pause_mine()
{
m_miner.pause();
}
//-----------------------------------------------------------------------------------------------
void core::resume_mine()
{
m_miner.resume();
}
//-----------------------------------------------------------------------------------------------
core: call {prepare|cleanup}_handle_incoming_blocks when adding a mined block This ensures that a batch can't also be started/stopped out of sync by another thread and us getting in the middle
2017-03-19 11:25:12 +01:00
block_complete_entry get_block_complete_entry(block& b, tx_memory_pool &pool)
{
block_complete_entry bce;
bce.block = cryptonote::block_to_blob(b);
for (const auto &tx_hash: b.tx_hashes)
{
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
cryptonote::blobdata txblob;
CHECK_AND_ASSERT_THROW_MES(pool.get_transaction(tx_hash, txblob), "Transaction not found in pool");
bce.txs.push_back(txblob);
core: call {prepare|cleanup}_handle_incoming_blocks when adding a mined block This ensures that a batch can't also be started/stopped out of sync by another thread and us getting in the middle
2017-03-19 11:25:12 +01:00
}
return bce;
}
//-----------------------------------------------------------------------------------------------
functional_tests: add more blockchain related tests Related to emission, reorgs, getting tx data back, output distribution and histogram
2019-03-23 17:20:08 +01:00
bool core::handle_block_found(block& b, block_verification_context &bvc)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
functional_tests: add more blockchain related tests Related to emission, reorgs, getting tx data back, output distribution and histogram
2019-03-23 17:20:08 +01:00
bvc = boost::value_initialized<block_verification_context>();
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<block_complete_entry> blocks;
Handle multi state changes in the pool (#781) * Update state transition check to account for height and universally set timestamp on recommission Reject invalidated state changes by their height after HF13 * Prune invalidated state changes on blockchain increment Simplify check_tx_inputs for state_changes by using service node list Instead of querying the last 60 historical blocks for every transaction, use the service node list and determine the state of the service node and if it can transition to its new state. We also now enforce at hardfork 13, that the network cannot commit transactions to the network if they would have been invalidated by a newer state change that already is already on the blockchain. This is backwards compatible all the way back to hardfork 9. Greatly simplify state change tx pruning on block added Use the new stricter rules for pruning state changes in the txpool We can do so because pruning the TX pool won't cause issues at the protocol level, but the more people we can upgrade the better network behaviour we get in terms of propagating more intrinsically correct ordering of state changes to other peers. * Don't generate state changes if not valid, disallow voting if node is non-votable
2019-08-12 01:48:47 +02:00
m_miner.pause();
cryptonote: avoid double parsing blocks when syncing
2018-11-23 14:05:48 +01:00
{
Handle multi state changes in the pool (#781) * Update state transition check to account for height and universally set timestamp on recommission Reject invalidated state changes by their height after HF13 * Prune invalidated state changes on blockchain increment Simplify check_tx_inputs for state_changes by using service node list Instead of querying the last 60 historical blocks for every transaction, use the service node list and determine the state of the service node and if it can transition to its new state. We also now enforce at hardfork 13, that the network cannot commit transactions to the network if they would have been invalidated by a newer state change that already is already on the blockchain. This is backwards compatible all the way back to hardfork 9. Greatly simplify state change tx pruning on block added Use the new stricter rules for pruning state changes in the txpool We can do so because pruning the TX pool won't cause issues at the protocol level, but the more people we can upgrade the better network behaviour we get in terms of propagating more intrinsically correct ordering of state changes to other peers. * Don't generate state changes if not valid, disallow voting if node is non-votable
2019-08-12 01:48:47 +02:00
LOKI_DEFER { m_miner.resume(); };
try
{
blocks.push_back(get_block_complete_entry(b, m_mempool));
}
catch (const std::exception &e)
{
return false;
}
std::vector<block> pblocks;
if (!prepare_handle_incoming_blocks(blocks, pblocks))
{
MERROR("Block found, but failed to prepare to add");
return false;
}
add_new_block(b, bvc, nullptr /*checkpoint*/);
cleanup_handle_incoming_blocks(true);
m_miner.on_block_chain_update();
cryptonote: avoid double parsing blocks when syncing
2018-11-23 14:05:48 +01:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
CHECK_AND_ASSERT_MES(!bvc.m_verifivation_failed, false, "mined block failed verification");
if(bvc.m_added_to_main_chain)
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<crypto::hash> missed_txs;
std::vector<cryptonote::blobdata> txs;
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
m_blockchain_storage.get_transactions_blobs(b.tx_hashes, txs, missed_txs);
moved all stuff to github
2014-03-03 23:07:58 +01:00
if(missed_txs.size() && m_blockchain_storage.get_block_id_by_height(get_block_height(b)) != get_block_hash(b))
{
moved non-critical warnings and errors to log level 1
2014-09-09 11:32:00 +02:00
LOG_PRINT_L1("Block found but, seems that reorganize just happened after that, do not relay this block");
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
}
Fix typos in various files
2018-03-01 12:36:19 +01:00
CHECK_AND_ASSERT_MES(txs.size() == b.tx_hashes.size() && !missed_txs.size(), false, "can't find some transactions in found block:" << get_block_hash(b) << " txs.size()=" << txs.size()
moved all stuff to github
2014-03-03 23:07:58 +01:00
<< ", b.tx_hashes.size()=" << b.tx_hashes.size() << ", missed_txs.size()" << missed_txs.size());
Remove non-fluffy blockchain syncing path (#618) Obsolete since forever, fluffy blocks is by default- it's more space efficient when syncing the blockchain.
2019-06-04 04:28:56 +02:00
cryptonote_connection_context exclude_context = boost::value_initialized<cryptonote_connection_context>();
NOTIFY_NEW_FLUFFY_BLOCK::request arg = AUTO_VAL_INIT(arg);
arg.current_blockchain_height = m_blockchain_storage.get_current_blockchain_height();
arg.b = blocks[0];
moved all stuff to github
2014-03-03 23:07:58 +01:00
m_pprotocol->relay_block(arg, exclude_context);
}
functional_tests: add more blockchain related tests Related to emission, reorgs, getting tx data back, output distribution and histogram
2019-03-23 17:20:08 +01:00
return true;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
void core::on_synchronized()
{
m_miner.on_synchronized();
}
//-----------------------------------------------------------------------------------------------
Cleanup test impact of adding safesyncmode() method
2017-08-22 15:54:18 +02:00
void core::safesyncmode(const bool onoff)
{
m_blockchain_storage.safesyncmode(onoff);
}
//-----------------------------------------------------------------------------------------------
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
bool core::add_new_block(const block& b, block_verification_context& bvc, checkpoint_t const *checkpoint)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Service node checkpointing soft fork for mainnet (#719) * Add soft forking for checkpointing on mainnet * Clear checkpoints on softfork on mainnet * Move softfork date until we're ready, fix test results and modulo addition * Only round up delete_height if it's not a multiple of CHECKPOINT_INTERVAL * Remove unused variable soft fork in service_node_rules (replaced by cryptonote_config)
2019-07-11 08:16:16 +02:00
// TODO(loki): Temporary soft-fork code can be removed
uint64_t latest_height = std::max(get_current_blockchain_height(), get_target_blockchain_height());
if (get_nettype() == cryptonote::MAINNET &&
latest_height >= HF_VERSION_12_CHECKPOINTING_SOFT_FORK_HEIGHT &&
get_block_height(b) < HF_VERSION_12_CHECKPOINTING_SOFT_FORK_HEIGHT)
{
// NOTE: After the soft fork, ignore all checkpoints from before the fork so we
// only create and enforce checkpoints from after the soft-fork.
checkpoint = nullptr;
}
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
bool result = m_blockchain_storage.add_new_block(b, bvc, checkpoint);
if (result)
{
// TODO(loki): PERF(loki): This causes perf problems in integration mode, so in real-time operation it may not be
// noticeable but could bubble up and cause slowness if the runtime variables align up undesiredly.
relay_service_node_votes(); // NOTE: nop if synchronising due to not accepting votes whilst syncing
}
return result;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
//-----------------------------------------------------------------------------------------------
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
bool core::prepare_handle_incoming_blocks(const std::vector<block_complete_entry> &blocks_entry, std::vector<block> &blocks)
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
{
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
m_incoming_tx_lock.lock();
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
if (!m_blockchain_storage.prepare_handle_incoming_blocks(blocks_entry, blocks))
cryptonote: avoid double parsing blocks when syncing
2018-11-23 14:05:48 +01:00
{
cleanup_handle_incoming_blocks(false);
return false;
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
return true;
}
//-----------------------------------------------------------------------------------------------
bool core::cleanup_handle_incoming_blocks(bool force_sync)
{
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 16:43:32 +02:00
bool success = false;
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
try {
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 16:43:32 +02:00
success = m_blockchain_storage.cleanup_handle_incoming_blocks(force_sync);
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
}
catch (...) {}
m_incoming_tx_lock.unlock();
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 16:43:32 +02:00
return success;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
//-----------------------------------------------------------------------------------------------
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
bool core::handle_incoming_block(const blobdata& block_blob, const block *b, block_verification_context& bvc, checkpoint_t const *checkpoint, bool update_miner_blocktemplate)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
TRY_ENTRY();
moved all stuff to github
2014-03-03 23:07:58 +01:00
bvc = boost::value_initialized<block_verification_context>();
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
if (!check_incoming_block_size(block_blob))
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
bvc.m_verifivation_failed = true;
return false;
}
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
if (((size_t)-1) <= 0xffffffff && block_blob.size() >= 0x3fffffff)
MWARNING("This block's size is " << block_blob.size() << ", closing on the 32 bit limit");
Rollback chain when receiving conflicting checkpoints (#726)
2019-07-11 06:16:55 +02:00
CHECK_AND_ASSERT_MES(update_checkpoints_from_json_file(), false, "One or more checkpoints loaded from json conflicted with existing checkpoints.");
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
core: avoid double parsing blocks after hoh
2018-11-23 12:15:04 +01:00
block lb;
if (!b)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
cryptonote: block parsing + hash calculation speedup This saves a duplicate serialization step
2018-12-13 20:47:47 +01:00
crypto::hash block_hash;
if(!parse_and_validate_block_from_blob(block_blob, lb, block_hash))
core: avoid double parsing blocks after hoh
2018-11-23 12:15:04 +01:00
{
LOG_PRINT_L1("Failed to parse and validate new block");
bvc.m_verifivation_failed = true;
return false;
}
b = &lb;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
add_new_block(*b, bvc, checkpoint);
moved all stuff to github
2014-03-03 23:07:58 +01:00
if(update_miner_blocktemplate && bvc.m_added_to_main_chain)
Inline update_miner_block_template and remove from core header
2019-08-07 02:30:46 +02:00
m_miner.on_block_chain_update();
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
CATCH_ENTRY_L0("core::handle_incoming_block()", false);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
Fix for orphan fix (check blocksize)
2014-06-11 23:32:53 +02:00
// Used by the RPC server to check the size of an incoming
// block_blob
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::check_incoming_block_size(const blobdata& block_blob) const
Fix for orphan fix (check blocksize)
2014-06-11 23:32:53 +02:00
{
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
// note: we assume block weight is always >= block blob size, so we check incoming
// blob size against the block weight limit, which acts as a sanity check without
// having to parse/weigh first; in fact, since the block blob is the block header
// plus the tx hashes, the weight will typically be much larger than the blob size
if(block_blob.size() > m_blockchain_storage.get_current_cumulative_block_weight_limit() + BLOCK_SIZE_SANITY_LEEWAY)
Fix for orphan fix (check blocksize)
2014-06-11 23:32:53 +02:00
{
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
LOG_PRINT_L1("WRONG BLOCK BLOB, sanity check failed on size " << block_blob.size() << ", rejected");
Fix for orphan fix (check blocksize)
2014-06-11 23:32:53 +02:00
return false;
}
return true;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
crypto::hash core::get_tail_id() const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_tail_id();
}
//-----------------------------------------------------------------------------------------------
protocol: pass blockchain cumulative difficulty when syncing Not used yet.
2017-08-12 16:24:39 +02:00
difficulty_type core::get_block_cumulative_difficulty(uint64_t height) const
{
return m_blockchain_storage.get_db().get_block_cumulative_difficulty(height);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
size_t core::get_pool_transactions_count() const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_mempool.get_transactions_count();
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::have_block(const crypto::hash& id) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.have_block(id);
}
//-----------------------------------------------------------------------------------------------
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
bool core::parse_tx_from_blob(transaction& tx, crypto::hash& tx_hash, const blobdata& blob) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
return parse_and_validate_tx_from_blob(blob, tx, tx_hash);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::check_tx_syntax(const transaction& tx) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return true;
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_pool_transactions(std::vector<transaction>& txs, bool include_sensitive_data) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 13:06:41 +01:00
m_mempool.get_transactions(txs, include_sensitive_data);
Use const where appropriate in tx_pool
2014-07-17 16:31:44 +02:00
return true;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
core, wallet: faster tx pool scanning Includes a new RPC to get tx pool hashes fast.
2017-03-22 19:03:23 +01:00
//-----------------------------------------------------------------------------------------------
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 13:06:41 +01:00
bool core::get_pool_transaction_hashes(std::vector<crypto::hash>& txs, bool include_sensitive_data) const
core, wallet: faster tx pool scanning Includes a new RPC to get tx pool hashes fast.
2017-03-22 19:03:23 +01:00
{
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 13:06:41 +01:00
m_mempool.get_transaction_hashes(txs, include_sensitive_data);
core, wallet: faster tx pool scanning Includes a new RPC to get tx pool hashes fast.
2017-03-22 19:03:23 +01:00
return true;
}
//-----------------------------------------------------------------------------------------------
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 13:06:41 +01:00
bool core::get_pool_transaction_stats(struct txpool_stats& stats, bool include_sensitive_data) const
Speedup print_pool_stats Since we're just counting txs, there's no reason to deserialize all the blobs.
2017-05-31 20:11:56 +02:00
{
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 13:06:41 +01:00
m_mempool.get_transaction_stats(stats, include_sensitive_data);
Speedup print_pool_stats Since we're just counting txs, there's no reason to deserialize all the blobs.
2017-05-31 20:11:56 +02:00
return true;
}
//-----------------------------------------------------------------------------------------------
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
bool core::get_pool_transaction(const crypto::hash &id, cryptonote::blobdata& tx) const
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 21:00:08 +02:00
{
return m_mempool.get_transaction(id, tx);
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
//-----------------------------------------------------------------------------------------------
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
bool core::pool_has_tx(const crypto::hash &id) const
{
return m_mempool.have_tx(id);
}
//-----------------------------------------------------------------------------------------------
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 13:06:41 +01:00
bool core::get_pool_transactions_and_spent_keys_info(std::vector<tx_info>& tx_infos, std::vector<spent_key_image_info>& key_image_infos, bool include_sensitive_data) const
Add mempool output to daemon via command and RPC This is for the "print_pool" command and "get_transaction_pool" RPC method. Add mempool's spent key images to the results.
2015-04-23 14:13:07 +02:00
{
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 13:06:41 +01:00
return m_mempool.get_transactions_and_spent_keys_info(tx_infos, key_image_infos, include_sensitive_data);
Add mempool output to daemon via command and RPC This is for the "print_pool" command and "get_transaction_pool" RPC method. Add mempool's spent key images to the results.
2015-04-23 14:13:07 +02:00
}
//-----------------------------------------------------------------------------------------------
json serialization for rpc-relevant monero types Structured {de-,}serialization methods for (many new) types which are used for requests or responses in the RPC. New types include RPC requests and responses, and structs which compose types within those. # Conflicts: # src/cryptonote_core/blockchain.cpp
2017-09-05 18:20:27 +02:00
bool core::get_pool_for_rpc(std::vector<cryptonote::rpc::tx_in_pool>& tx_infos, cryptonote::rpc::key_images_with_tx_hashes& key_image_infos) const
{
return m_mempool.get_pool_for_rpc(tx_infos, key_image_infos);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::get_short_chain_history(std::list<crypto::hash>& ids) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_short_chain_history(ids);
}
//-----------------------------------------------------------------------------------------------
bool core::handle_get_objects(NOTIFY_REQUEST_GET_OBJECTS::request& arg, NOTIFY_RESPONSE_GET_OBJECTS::request& rsp, cryptonote_connection_context& context)
{
return m_blockchain_storage.handle_get_objects(arg, rsp);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
crypto::hash core::get_block_id_by_height(uint64_t height) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_block_id_by_height(height);
}
//-----------------------------------------------------------------------------------------------
rpc: fix orphan_status when getting blocks It was always set to false, even for orphan blocks
2017-01-22 13:20:55 +01:00
bool core::get_block_by_hash(const crypto::hash &h, block &blk, bool *orphan) const
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
{
rpc: fix orphan_status when getting blocks It was always set to false, even for orphan blocks
2017-01-22 13:20:55 +01:00
return m_blockchain_storage.get_block_by_hash(h, blk, orphan);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
std::string core::print_pool(bool short_format) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_mempool.print_pool(short_format);
}
//-----------------------------------------------------------------------------------------------
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
static bool check_storage_server_ping(time_t last_time_storage_server_pinged)
{
const auto elapsed = std::time(nullptr) - last_time_storage_server_pinged;
if (elapsed > STORAGE_SERVER_PING_LIFETIME)
{
MWARNING("Have not heard from the storage server since at least: "
<< tools::get_human_readable_timespan(std::chrono::seconds(last_time_storage_server_pinged)));
return false;
}
return true;
}
//-----------------------------------------------------------------------------------------------
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
void core::do_uptime_proof_call()
{
// wait one block before starting uptime proofs.
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
std::vector<service_nodes::service_node_pubkey_info> const states = get_service_node_list_state({ m_service_node_pubkey });
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
if (!states.empty() && (states[0].info.registration_height + 1) < get_current_blockchain_height())
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
{
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
// Code snippet from Github @Jagerman
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
service_nodes::service_node_info const &info = states[0].info;
m_check_uptime_proof_interval.do_call([&info, this]() {
if (info.proof.timestamp <= static_cast<uint64_t>(time(nullptr) - UPTIME_PROOF_FREQUENCY_IN_SECONDS))
{
uint8_t hf_version = get_blockchain_storage().get_current_hard_fork_version();
if (!check_storage_server_ping(m_last_storage_server_ping))
{
if (hf_version >= cryptonote::network_version_12_checkpointing)
{
MGINFO_RED(
"Failed to submit uptime proof: have not heard from the storage server recently. Make sure that it "
"is running! It is required to run alongside the Loki daemon after hard-fork 12");
return true;
}
else
{
MGINFO_RED(
"We have not heard from the storage server recently. Make sure that it is running! After hard fork "
"12, this Service Node will stop submitting uptime proofs if it does not hear from the Loki Storage "
"Server.");
}
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
}
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
this->submit_uptime_proof();
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
}
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
return true;
});
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
}
else
{
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
// reset the interval so that we're ready when we register, OR if we get deregistered this primes us up for re-registration in the same session
m_check_uptime_proof_interval = epee::math_helper::once_a_time_seconds<UPTIME_PROOF_BUFFER_IN_SECONDS, true /*start_immediately*/>();
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
}
}
//-----------------------------------------------------------------------------------------------
moved all stuff to github
2014-03-03 23:07:58 +01:00
bool core::on_idle()
{
if(!m_starter_message_showed)
{
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
std::string main_message;
if (m_offline)
First pass Monero to Loki refs
2018-04-10 06:49:20 +02:00
main_message = "The daemon is running offline and will not attempt to sync to the Loki network.";
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
else
main_message = "The daemon will start synchronizing with the network. This may take a long time to complete.";
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MGINFO_YELLOW(ENDL << "**********************************************************************" << ENDL
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
<< main_message << ENDL
moved all stuff to github
2014-03-03 23:07:58 +01:00
<< ENDL
Small cleanup of daemon synchronization output Add period to second sentence
2017-11-27 22:27:26 +01:00
<< "You can set the level of process detailization through \"set_log <level|categories>\" command," << ENDL
<< "where <level> is between 0 (no details) and 4 (very verbose), or custom category based levels (eg, *:WARNING)." << ENDL
moved all stuff to github
2014-03-03 23:07:58 +01:00
<< ENDL
Update log messages to reflect fact we no longer need to type `exit` to save database before quitting
2017-01-23 02:03:04 +01:00
<< "Use the \"help\" command to see the list of available commands." << ENDL
Added command descriptions
2017-11-22 13:53:18 +01:00
<< "Use \"help <command>\" to see a command's documentation." << ENDL
Update log messages to reflect fact we no longer need to type `exit` to save database before quitting
2017-01-23 02:03:04 +01:00
<< "**********************************************************************" << ENDL);
moved all stuff to github
2014-03-03 23:07:58 +01:00
m_starter_message_showed = true;
}
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
m_fork_moaner.do_call(boost::bind(&core::check_fork_time, this));
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
m_txpool_auto_relayer.do_call(boost::bind(&core::relay_txpool_transactions, this));
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
m_service_node_vote_relayer.do_call(boost::bind(&core::relay_service_node_votes, this));
Merge commit '4a652d6' into LokiMergeUpstreamUntil_20180911_e6d36c1
2018-10-10 04:48:01 +02:00
// m_check_updates_interval.do_call(boost::bind(&core::check_updates, this));
core: warn when free disk space is low
2017-11-12 00:25:12 +01:00
m_check_disk_space_interval.do_call(boost::bind(&core::check_disk_space, this));
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
m_block_rate_interval.do_call(boost::bind(&core::check_block_rate, this));
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
time_t const lifetime = time(nullptr) - get_start_time();
if (m_service_node && lifetime > DIFFICULTY_TARGET_V2) // Give us some time to connect to peers before sending uptimes
{
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
do_uptime_proof_call();
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
m_blockchain_pruning_interval.do_call(boost::bind(&core::update_blockchain_pruning, this));
moved all stuff to github
2014-03-03 23:07:58 +01:00
m_miner.on_idle();
proper tx_pool handling from CryptoZoidberg / BBR
2014-06-15 09:48:13 +02:00
m_mempool.on_idle();
Revamp to use semaphores for IPC Delay hooks into stdin/out until core is idle for daemons Disable logging for daemon (super problematic)
2019-01-15 10:09:17 +01:00
#if defined(LOKI_ENABLE_INTEGRATION_TEST_HOOKS)
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
loki::integration_test.core_is_idle = true;
Revamp to use semaphores for IPC Delay hooks into stdin/out until core is idle for daemons Disable logging for daemon (super problematic)
2019-01-15 10:09:17 +01:00
#endif
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
bool core::check_fork_time()
{
core: don't check block rate nor fork time in regtest mode
2019-04-07 15:29:21 +02:00
if (m_nettype == FAKECHAIN)
return true;
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
HardFork::State state = m_blockchain_storage.get_hard_fork_state();
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
const el::Level level = el::Level::Warning;
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
switch (state) {
case HardFork::LikelyForked:
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MCLOG_RED(level, "global", "**********************************************************************");
MCLOG_RED(level, "global", "Last scheduled hard fork is too far in the past.");
MCLOG_RED(level, "global", "We are most likely forked from the network. Daemon update needed now.");
MCLOG_RED(level, "global", "**********************************************************************");
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
break;
case HardFork::UpdateNeeded:
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MCLOG_RED(level, "global", "**********************************************************************");
cryptonote_core: change wording of fork warning message An udpate may or may not be available now, but should be soon if not. This will prevent too many people freaking out.
2018-02-19 00:44:59 +01:00
MCLOG_RED(level, "global", "Last scheduled hard fork time shows a daemon update is needed soon.");
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MCLOG_RED(level, "global", "**********************************************************************");
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
break;
default:
break;
}
return true;
}
//-----------------------------------------------------------------------------------------------
Only log an error if fork version is higher AND is not known.
2018-04-16 04:02:26 +02:00
uint8_t core::get_ideal_hard_fork_version() const
{
return get_blockchain_storage().get_ideal_hard_fork_version();
}
//-----------------------------------------------------------------------------------------------
tests: fix tests build Add get_fork_version and add_ideal_fork_version to core so cryptonote_protocol does not have to need the Blockchain class directly, as it's not in its dependencies, and add those to the fake core classes in tests too.
2017-08-10 11:10:04 +02:00
uint8_t core::get_ideal_hard_fork_version(uint64_t height) const
{
return get_blockchain_storage().get_ideal_hard_fork_version(height);
}
//-----------------------------------------------------------------------------------------------
uint8_t core::get_hard_fork_version(uint64_t height) const
{
return get_blockchain_storage().get_hard_fork_version(height);
}
//-----------------------------------------------------------------------------------------------
core: add get_earliest_ideal_height_for_version()
2018-03-20 12:30:09 +01:00
uint64_t core::get_earliest_ideal_height_for_version(uint8_t version) const
{
return get_blockchain_storage().get_earliest_ideal_height_for_version(version);
}
//-----------------------------------------------------------------------------------------------
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
bool core::check_updates()
{
Update cmakelists and misc refs to Loki
2018-04-25 08:13:38 +02:00
static const char software[] = "loki";
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
#ifdef BUILD_TAG
static const char buildtag[] = BOOST_PP_STRINGIZE(BUILD_TAG);
Source updates are in a source subdirectory rather than in the same directory as the prebuilt versions
2017-09-22 22:48:19 +02:00
static const char subdir[] = "cli"; // because it can never be simple
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
#else
static const char buildtag[] = "source";
Source updates are in a source subdirectory rather than in the same directory as the prebuilt versions
2017-09-22 22:48:19 +02:00
static const char subdir[] = "source"; // because it can never be simple
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
#endif
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
if (m_offline)
return true;
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
if (check_updates_level == UPDATES_DISABLED)
return true;
std::string version, hash;
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
MCDEBUG("updates", "Checking for a new " << software << " version for " << buildtag);
updates: remove testnet case It need not be any different
2017-02-22 12:19:20 +01:00
if (!tools::check_updates(software, buildtag, version, hash))
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
return false;
First pass Monero to Loki refs
2018-04-10 06:49:20 +02:00
if (tools::vercmp(version.c_str(), LOKI_VERSION) <= 0)
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 16:31:34 +02:00
{
m_update_available = false;
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
return true;
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 16:31:34 +02:00
}
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
updates: we now have a user URL, and an automatic one So the user can use https, while the automatic does not have to
2017-02-22 12:11:45 +01:00
std::string url = tools::get_update_url(software, subdir, buildtag, version, true);
core: print the "new update found" message in cyan, for visibility
2017-03-04 19:29:18 +01:00
MCLOG_CYAN(el::Level::Info, "global", "Version " << version << " of " << software << " for " << buildtag << " is available: " << url << ", SHA256 hash " << hash);
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 16:31:34 +02:00
m_update_available = true;
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
if (check_updates_level == UPDATES_NOTIFY)
return true;
updates: we now have a user URL, and an automatic one So the user can use https, while the automatic does not have to
2017-02-22 12:11:45 +01:00
url = tools::get_update_url(software, subdir, buildtag, version, false);
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
std::string filename;
const char *slash = strrchr(url.c_str(), '/');
if (slash)
filename = slash + 1;
else
filename = std::string(software) + "-update-" + version;
boost::filesystem::path path(epee::string_tools::get_current_module_folder());
path /= filename;
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
core: make update download cancellable
2017-02-26 22:00:38 +01:00
boost::unique_lock<boost::mutex> lock(m_update_mutex);
if (m_update_download != 0)
{
MCDEBUG("updates", "Already downloading update");
return true;
}
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
crypto::hash file_hash;
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
if (!tools::sha256sum(path.string(), file_hash) || (hash != epee::string_tools::pod_to_hex(file_hash)))
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
{
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
MCDEBUG("updates", "We don't have that file already, downloading");
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
const std::string tmppath = path.string() + ".tmp";
if (epee::file_io_utils::is_file_exist(tmppath))
{
MCDEBUG("updates", "We have part of the file already, resuming download");
}
core: make update download cancellable
2017-02-26 22:00:38 +01:00
m_last_update_length = 0;
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
m_update_download = tools::download_async(tmppath, url, [this, hash, path](const std::string &tmppath, const std::string &uri, bool success) {
bool remove = false, good = true;
core: make update download cancellable
2017-02-26 22:00:38 +01:00
if (success)
{
crypto::hash file_hash;
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
if (!tools::sha256sum(tmppath, file_hash))
core: make update download cancellable
2017-02-26 22:00:38 +01:00
{
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
MCERROR("updates", "Failed to hash " << tmppath);
remove = true;
good = false;
core: make update download cancellable
2017-02-26 22:00:38 +01:00
}
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
else if (hash != epee::string_tools::pod_to_hex(file_hash))
core: make update download cancellable
2017-02-26 22:00:38 +01:00
{
MCERROR("updates", "Download from " << uri << " does not match the expected hash");
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
remove = true;
good = false;
core: make update download cancellable
2017-02-26 22:00:38 +01:00
}
}
else
{
MCERROR("updates", "Failed to download " << uri);
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
good = false;
core: make update download cancellable
2017-02-26 22:00:38 +01:00
}
boost::unique_lock<boost::mutex> lock(m_update_mutex);
m_update_download = 0;
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
if (success && !remove)
{
std::error_code e = tools::replace_file(tmppath, path.string());
if (e)
{
MCERROR("updates", "Failed to rename downloaded file");
good = false;
}
}
else if (remove)
{
if (!boost::filesystem::remove(tmppath))
{
MCERROR("updates", "Failed to remove invalid downloaded file");
good = false;
}
}
if (good)
MCLOG_CYAN(el::Level::Info, "updates", "New version downloaded to " << path.string());
core: make update download cancellable
2017-02-26 22:00:38 +01:00
}, [this](const std::string &path, const std::string &uri, size_t length, ssize_t content_length) {
if (length >= m_last_update_length + 1024 * 1024 * 10)
{
m_last_update_length = length;
MCDEBUG("updates", "Downloaded " << length << "/" << (content_length ? std::to_string(content_length) : "unknown"));
}
return true;
});
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
}
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
else
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
{
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
MCDEBUG("updates", "We already have " << path << " with expected hash");
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
}
core: make update download cancellable
2017-02-26 22:00:38 +01:00
lock.unlock();
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
if (check_updates_level == UPDATES_DOWNLOAD)
return true;
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
MCERROR("updates", "Download/update not implemented yet");
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: warn when free disk space is low
2017-11-12 00:25:12 +01:00
bool core::check_disk_space()
{
uint64_t free_space = get_free_space();
if (free_space < 1ull * 1024 * 1024 * 1024) // 1 GB
{
const el::Level level = el::Level::Warning;
MCLOG_RED(level, "global", "Free space is below 1 GB on " << m_config_folder);
}
return true;
}
//-----------------------------------------------------------------------------------------------
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
double factorial(unsigned int n)
{
if (n <= 1)
return 1.0;
double f = n;
while (n-- > 1)
f *= n;
return f;
}
//-----------------------------------------------------------------------------------------------
core: improve block rate monitor trigger probabilities The original intent of one false positive a week on average was not met, since what we really want is not the probability of having N blocks in T seconds, but either N blocks of fewer in T seconds, or N blocks or more in T seconds. Some of this could be cached since it calculates the same fairly complex floating point values, but it seems pretty fast already.
2019-03-19 17:30:17 +01:00
static double probability1(unsigned int blocks, unsigned int expected)
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
{
// https://www.umass.edu/wsp/resources/poisson/#computing
return pow(expected, blocks) / (factorial(blocks) * exp(expected));
}
//-----------------------------------------------------------------------------------------------
core: improve block rate monitor trigger probabilities The original intent of one false positive a week on average was not met, since what we really want is not the probability of having N blocks in T seconds, but either N blocks of fewer in T seconds, or N blocks or more in T seconds. Some of this could be cached since it calculates the same fairly complex floating point values, but it seems pretty fast already.
2019-03-19 17:30:17 +01:00
static double probability(unsigned int blocks, unsigned int expected)
{
double p = 0.0;
if (blocks <= expected)
{
for (unsigned int b = 0; b <= blocks; ++b)
p += probability1(b, expected);
}
else if (blocks > expected)
{
for (unsigned int b = blocks; b <= expected * 3 /* close enough */; ++b)
p += probability1(b, expected);
}
return p;
}
//-----------------------------------------------------------------------------------------------
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
bool core::check_block_rate()
{
core: don't check block rate nor fork time in regtest mode
2019-04-07 15:29:21 +02:00
if (m_offline || m_nettype == FAKECHAIN || m_target_blockchain_height > get_current_blockchain_height())
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
{
MDEBUG("Not checking block rate, offline or syncing");
return true;
}
Disable block rate and min life time in integration mode
2018-11-29 08:20:31 +01:00
#if defined(LOKI_ENABLE_INTEGRATION_TEST_HOOKS)
MDEBUG("Not checking block rate, integration test mode");
return true;
#endif
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
static constexpr double threshold = 1. / (864000 / DIFFICULTY_TARGET_V2); // one false positive every 10 days
const time_t now = time(NULL);
const std::vector<time_t> timestamps = m_blockchain_storage.get_last_block_timestamps(60);
core: add a few more block rate window sizes The 10 minute one will never trigger for 0 blocks, as it's still fairly likely to happen even without the actual hash rate changing much, so we add a 20 minute window, where it will (for 0 blocks) and a one hour window.
2019-02-06 20:04:47 +01:00
static const unsigned int seconds[] = { 5400, 3600, 1800, 1200, 600 };
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
for (size_t n = 0; n < sizeof(seconds)/sizeof(seconds[0]); ++n)
{
unsigned int b = 0;
cryptonote_core: avoid gratuitous recalculations in check_block_rate()
2018-11-05 01:13:51 +01:00
const time_t time_boundary = now - static_cast<time_t>(seconds[n]);
for (time_t ts: timestamps) b += ts >= time_boundary;
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
const double p = probability(b, seconds[n] / DIFFICULTY_TARGET_V2);
MDEBUG("blocks in the last " << seconds[n] / 60 << " minutes: " << b << " (probability " << p << ")");
if (p < threshold)
{
Fix broken builds from upstream merge
2018-11-05 01:43:15 +01:00
MWARNING("There were " << b << " blocks in the last " << seconds[n] / 60 << " minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack. Or it could be just sheer bad luck.");
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
std::shared_ptr<tools::Notify> block_rate_notify = m_block_rate_notify;
if (block_rate_notify)
{
auto expected = seconds[n] / DIFFICULTY_TARGET_V2;
block_rate_notify->notify("%t", std::to_string(seconds[n] / 60).c_str(), "%b", std::to_string(b).c_str(), "%e", std::to_string(expected).c_str(), NULL);
}
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
break; // no need to look further
}
}
return true;
}
//-----------------------------------------------------------------------------------------------
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
bool core::update_blockchain_pruning()
{
return m_blockchain_storage.update_blockchain_pruning();
}
//-----------------------------------------------------------------------------------------------
bool core::check_blockchain_pruning()
{
return m_blockchain_storage.check_blockchain_pruning();
}
//-----------------------------------------------------------------------------------------------
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
void core::set_target_blockchain_height(uint64_t target_blockchain_height)
{
Revert "Fix fake 'network synchronized, begin using' messages" This reverts commit 78035d2b6c9922f4cd730df0766aa74f4854ccb2. The patch doesn't work, and causes constant SYNCHRONIZED OK spam.
2016-09-25 20:08:31 +02:00
m_target_blockchain_height = target_blockchain_height;
'getinfo' daemon HTTP-RPC returns 'target_height' for progress estimations
2014-06-04 22:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
uint64_t core::get_target_blockchain_height() const
{
'getinfo' daemon HTTP-RPC returns 'target_height' for progress estimations
2014-06-04 22:50:13 +02:00
return m_target_blockchain_height;
}
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
uint64_t core::prevalidate_block_hashes(uint64_t height, const std::vector<crypto::hash> &hashes)
precomputed block hashes are now in blocks of N (currently 256) This shaves a lot of space off binaries
2017-09-13 22:03:04 +02:00
{
return get_blockchain_storage().prevalidate_block_hashes(height, hashes);
}
//-----------------------------------------------------------------------------------------------
core: warn when free disk space is low
2017-11-12 00:25:12 +01:00
uint64_t core::get_free_space() const
{
boost::filesystem::path path(m_config_folder);
boost::filesystem::space_info si = boost::filesystem::space(path);
return si.available;
}
//-----------------------------------------------------------------------------------------------
(Low priority) Add disabled-by-default quorum storage support (#702) * Add disabled-by-default quorum storage support This adds support for storing N blocks of recent expired quorum state history in lokid (and dumping to/from the lmdb). This isn't useful for regular nodes (and that's why we don't store it) but is incredibly useful for the block explorer to be able to report *which* node got deregistered/decommissioned/etc. by a given state_change tx. * Fix quorum states copy
2019-07-17 08:01:54 +02:00
std::shared_ptr<const service_nodes::testing_quorum> core::get_testing_quorum(service_nodes::quorum_type type, uint64_t height, bool include_old) const
Merge dev into upstream
2019-05-01 08:01:17 +02:00
{
(Low priority) Add disabled-by-default quorum storage support (#702) * Add disabled-by-default quorum storage support This adds support for storing N blocks of recent expired quorum state history in lokid (and dumping to/from the lmdb). This isn't useful for regular nodes (and that's why we don't store it) but is incredibly useful for the block explorer to be able to report *which* node got deregistered/decommissioned/etc. by a given state_change tx. * Fix quorum states copy
2019-07-17 08:01:54 +02:00
return m_service_node_list.get_testing_quorum(type, height, include_old);
Making service nodes list and quorum cop thread-safe (#285) * Remove dependency on service node list in quorum cop * making service node list therad-safe * improve thread-safety of quorum cop
2018-10-10 07:24:01 +02:00
}
//-----------------------------------------------------------------------------------------------
Relax deregistration rules The replaces the deregistration mechanism with a new state change mechanism (beginning at the v12 fork) which can change a service node's network status via three potential values (and is extensible in the future to handle more): - deregistered -- this is the same as the existing deregistration; the SN is instantly removed from the SN list. - decommissioned -- this is a sort of temporary deregistration: your SN remains in the service node list, but is removed from the rewards list and from any network duties. - recommissioned -- this tx is sent by a quorum if they observe a decommissioned SN sending uptime proofs again. Upon reception, the SN is reactivated and put on the end of the reward list. Since this is broadening the quorum use, this also renames the relevant quorum to a "obligations" quorum (since it validates SN obligations), while the transactions are "state_change" transactions (since they change the state of a registered SN). The new parameters added to service_node_rules.h control how this works: // Service node decommissioning: as service nodes stay up they earn "credits" (measured in blocks) // towards a future outage. A new service node starts out with INITIAL_CREDIT, and then builds up // CREDIT_PER_DAY for each day the service node remains active up to a maximum of // DECOMMISSION_MAX_CREDIT. // // If a service node stops sending uptime proofs, a quorum will consider whether the service node // has built up enough credits (at least MINIMUM): if so, instead of submitting a deregistration, // it instead submits a decommission. This removes the service node from the list of active // service nodes both for rewards and for any active network duties. If the service node comes // back online (i.e. starts sending the required performance proofs again) before the credits run // out then a quorum will reinstate the service node using a recommission transaction, which adds // the service node back to the bottom of the service node reward list, and resets its accumulated // credits to 0. If it does not come back online within the required number of blocks (i.e. the // accumulated credit at the point of decommissioning) then a quorum will send a permanent // deregistration transaction to the network, starting a 30-day deregistration count down. This commit currently includes values (which are not necessarily finalized): - 8 hours (240 blocks) of credit required for activation of a decommission (rather than a deregister) - 0 initial credits at registration - a maximum of 24 hours (720 blocks) of credits - credits accumulate at a rate that you hit 24 hours of credits after 30 days of operation. Miscellaneous other details of this PR: - a new TX extra tag is used for the state change (including deregistrations). The old extra tag has no version or type tag, so couldn't be reused. The data in the new tag is slightly more efficiently packed than the old deregistration transaction, so it gets used for deregistrations (starting at the v12 fork) as well. - Correct validator/worker selection required generalizing the shuffle function to be able to shuffle just part of a vector. This lets us stick any down service nodes at the end of the potential list, then select validators by only shuffling the part of the index vector that contains active service indices. Once the validators are selected, the remainder of the list (this time including decommissioned SN indices) is shuffled to select quorum workers to check, thus allowing decommisioned nodes to be randomly included in the nodes to check without being selected as a validator. - Swarm recalculation was not quite right: swarms were recalculated on SN registrations, even if those registrations were include shared node registrations, but *not* recalculated on stakes. Starting with the upgrade this behaviour is fixed (swarms aren't actually used currently and aren't consensus-relevant so recalculating early won't hurt anything). - Details on decomm/dereg are added to RPC info and print_sn/print_sn_status - Slightly improves the % of reward output in the print_sn output by rounding it to two digits, and reserves space in the output string to avoid excessive reallocations. - Adds various debugging at higher debug levels to quorum voting (into all of voting itself, vote transmission, and vote reception). - Reset service node list internal data structure version to 0. The SN list has to be rescanned anyway at upgrade (its size has changed), so we might as well reset the version and remove the version-dependent serialization code. (Note that the affected code here is for SN states in lmdb storage, not for SN-to-SN communication serialization).
2019-06-18 23:57:02 +02:00
bool core::is_service_node(const crypto::public_key& pubkey, bool require_active) const
Making service nodes list and quorum cop thread-safe (#285) * Remove dependency on service node list in quorum cop * making service node list therad-safe * improve thread-safety of quorum cop
2018-10-10 07:24:01 +02:00
{
Relax deregistration rules The replaces the deregistration mechanism with a new state change mechanism (beginning at the v12 fork) which can change a service node's network status via three potential values (and is extensible in the future to handle more): - deregistered -- this is the same as the existing deregistration; the SN is instantly removed from the SN list. - decommissioned -- this is a sort of temporary deregistration: your SN remains in the service node list, but is removed from the rewards list and from any network duties. - recommissioned -- this tx is sent by a quorum if they observe a decommissioned SN sending uptime proofs again. Upon reception, the SN is reactivated and put on the end of the reward list. Since this is broadening the quorum use, this also renames the relevant quorum to a "obligations" quorum (since it validates SN obligations), while the transactions are "state_change" transactions (since they change the state of a registered SN). The new parameters added to service_node_rules.h control how this works: // Service node decommissioning: as service nodes stay up they earn "credits" (measured in blocks) // towards a future outage. A new service node starts out with INITIAL_CREDIT, and then builds up // CREDIT_PER_DAY for each day the service node remains active up to a maximum of // DECOMMISSION_MAX_CREDIT. // // If a service node stops sending uptime proofs, a quorum will consider whether the service node // has built up enough credits (at least MINIMUM): if so, instead of submitting a deregistration, // it instead submits a decommission. This removes the service node from the list of active // service nodes both for rewards and for any active network duties. If the service node comes // back online (i.e. starts sending the required performance proofs again) before the credits run // out then a quorum will reinstate the service node using a recommission transaction, which adds // the service node back to the bottom of the service node reward list, and resets its accumulated // credits to 0. If it does not come back online within the required number of blocks (i.e. the // accumulated credit at the point of decommissioning) then a quorum will send a permanent // deregistration transaction to the network, starting a 30-day deregistration count down. This commit currently includes values (which are not necessarily finalized): - 8 hours (240 blocks) of credit required for activation of a decommission (rather than a deregister) - 0 initial credits at registration - a maximum of 24 hours (720 blocks) of credits - credits accumulate at a rate that you hit 24 hours of credits after 30 days of operation. Miscellaneous other details of this PR: - a new TX extra tag is used for the state change (including deregistrations). The old extra tag has no version or type tag, so couldn't be reused. The data in the new tag is slightly more efficiently packed than the old deregistration transaction, so it gets used for deregistrations (starting at the v12 fork) as well. - Correct validator/worker selection required generalizing the shuffle function to be able to shuffle just part of a vector. This lets us stick any down service nodes at the end of the potential list, then select validators by only shuffling the part of the index vector that contains active service indices. Once the validators are selected, the remainder of the list (this time including decommissioned SN indices) is shuffled to select quorum workers to check, thus allowing decommisioned nodes to be randomly included in the nodes to check without being selected as a validator. - Swarm recalculation was not quite right: swarms were recalculated on SN registrations, even if those registrations were include shared node registrations, but *not* recalculated on stakes. Starting with the upgrade this behaviour is fixed (swarms aren't actually used currently and aren't consensus-relevant so recalculating early won't hurt anything). - Details on decomm/dereg are added to RPC info and print_sn/print_sn_status - Slightly improves the % of reward output in the print_sn output by rounding it to two digits, and reserves space in the output string to avoid excessive reallocations. - Adds various debugging at higher debug levels to quorum voting (into all of voting itself, vote transmission, and vote reception). - Reset service node list internal data structure version to 0. The SN list has to be rescanned anyway at upgrade (its size has changed), so we might as well reset the version and remove the version-dependent serialization code. (Note that the affected code here is for SN states in lmdb storage, not for SN-to-SN communication serialization).
2019-06-18 23:57:02 +02:00
return m_service_node_list.is_service_node(pubkey, require_active);
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
}
//-----------------------------------------------------------------------------------------------
Infinite Staking Part 1 (#387) * Remove dead branches in hot-path check_tx_inputs Also renames #define for mixins to better match naming convention * Shuffle around some more code into common branches * Fix min/max tx version rules, since there 1 tx v2 on v9 fork * First draft infinite staking implementation * Actually generate the right key image and expire appropriately * Add framework to lock key images after expiry * Return locked key images for nodes, add request unlock option * Introduce transaction types for key image unlock * Update validation steps to accept tx types, key_image_unlock * Add mapping for lockable key images to amounts * Change inconsistent naming scheme of contributors * Create key image unlock transaction type and process it * Update tx params to allow v4 types and as a result construct_tx* * Fix some serialisation issues not sending all the information * Fix dupe tx extra tag causing incorrect deserialisation * Add warning comments * Fix key image unlocks parsing error * Simplify key image proof checks * Fix rebase errors * Correctly calculate the key image unlock times * Blacklist key image on deregistration * Serialise key image blacklist * Rollback blacklisted key images * Fix expiry logic error * Disallow requesting stake unlock if already unlocked client side * Add double spend checks for key image unlocks * Rename get_staking_requirement_lock_blocks To staking_initial_num_lock_blocks * Begin modifying output selection to not use locked outputs * Modify output selection to avoid locked/blacklisted key images * Cleanup and undoing some protocol breakages * Simplify expiration of nodes * Request unlock schedules entire node for expiration * Fix off by one in expiring nodes * Undo expiring code for pre v10 nodes * Fix RPC returning register as unlock height and not checking 0 * Rename key image unlock height const * Undo testnet hardfork debug changes * Remove is_type for get_type, fix missing var rename * Move serialisable data into public namespace * Serialise tx types properly * Fix typo in no service node known msg * Code review * Fix == to >= on serialising tx type * Code review 2 * Fix tests and key image unlock * Add additional test, fix assert * Remove debug code in wallet * Fix merge dev problem
2019-01-25 04:15:52 +01:00
const std::vector<service_nodes::key_image_blacklist_entry> &core::get_service_node_blacklisted_key_images() const
{
const auto &result = m_service_node_list.get_blacklisted_key_images();
return result;
}
//-----------------------------------------------------------------------------------------------
Service Nodes States Endpoint (#145) * Add get_service_node_list_state command for analytics * Update service_node_list_state to search particular pubkey * Service node list state sorts display results by longest waiting * Fix up leftover todos/unused data structures * Change get_service_node_list_state to print_sn
2018-08-15 04:59:05 +02:00
std::vector<service_nodes::service_node_pubkey_info> core::get_service_node_list_state(const std::vector<crypto::public_key> &service_node_pubkeys) const
{
std::vector<service_nodes::service_node_pubkey_info> result = m_service_node_list.get_service_node_list_state(service_node_pubkeys);
return result;
}
//-----------------------------------------------------------------------------------------------
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
bool core::add_service_node_vote(const service_nodes::quorum_vote_t& vote, vote_verification_context &vvc)
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
{
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
bool result = m_quorum_cop.handle_vote(vote, vvc);
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
return result;
}
First service node test (#190) * core_tests: added service node tests * core_tests: check balance after registration tx
2018-08-29 07:53:13 +02:00
//-----------------------------------------------------------------------------------------------
quorum_cop: added quorum cop and service node keys to lokid Submit deregistration autonomously, prune uptime_proofs Move pruner to relayer, fix inf. loop, quorum participation bool return find_if, fix reorg logic, make 2hrs check a variable Add iterator assign on erase, adjust detached check qacop: review iteration, changed warnings and std::find call Accept proofs excluding the added time buffer
2018-07-12 03:58:17 +02:00
bool core::get_service_node_keys(crypto::public_key &pub_key, crypto::secret_key &sec_key) const
{
if (m_service_node)
{
pub_key = m_service_node_pubkey;
sec_key = m_service_node_key;
}
return m_service_node;
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
uint32_t core::get_blockchain_pruning_seed() const
{
return get_blockchain_storage().get_blockchain_pruning_seed();
}
//-----------------------------------------------------------------------------------------------
bool core::prune_blockchain(uint32_t pruning_seed)
{
return get_blockchain_storage().prune_blockchain(pruning_seed);
}
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
//-----------------------------------------------------------------------------------------------
Replace is_fully_funded() calls with is_active() The vast majority of the time `is_active()` is what we actually want. Most of these are just a name change, but there is also one important fix here to the next-winner list which needs to use active, not fully-funded.
2019-06-26 05:00:05 +02:00
void core::get_all_service_nodes_public_keys(std::vector<crypto::public_key>& keys, bool active_nodes_only) const
All Service Node Keys as Base32z RPC Call (#404) * create get_service_node_list rpc call currently does nothing, just a shell (that compiles) * implement get_all_service_node_keys rpc call * change get_all_service_node_keys rpc to use json rpc also change the result to be vector of hex strings rather than binary keys * Make nodes be plural, add hex to base32z for keys Base32z for Lokinet internal usage * Add option to return fully funded service nodes only * Add nullptr check for conversion * Add assert for incorrect usage of to base32z
2019-01-25 00:50:42 +01:00
{
Replace is_fully_funded() calls with is_active() The vast majority of the time `is_active()` is what we actually want. Most of these are just a name change, but there is also one important fix here to the next-winner list which needs to use active, not fully-funded.
2019-06-26 05:00:05 +02:00
m_service_node_list.get_all_service_nodes_public_keys(keys, active_nodes_only);
All Service Node Keys as Base32z RPC Call (#404) * create get_service_node_list rpc call currently does nothing, just a shell (that compiles) * implement get_all_service_node_keys rpc call * change get_all_service_node_keys rpc to use json rpc also change the result to be vector of hex strings rather than binary keys * Make nodes be plural, add hex to base32z for keys Base32z for Lokinet internal usage * Add option to return fully funded service nodes only * Add nullptr check for conversion * Add assert for incorrect usage of to base32z
2019-01-25 00:50:42 +01:00
}
//-----------------------------------------------------------------------------------------------
Add start_time to get_info methods and show uptime
2017-01-09 00:50:29 +01:00
std::time_t core::get_start_time() const
{
return start_time;
}
//-----------------------------------------------------------------------------------------------
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
void core::graceful_exit()
{
raise(SIGTERM);
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Reference in a new issue Copy permalink