oxen-io/oxen-core
2
1
Fork 0
mirror of https://github.com/oxen-io/oxen-core.git synced 2023-12-14 02:22:56 +01:00
Code Issues Projects Releases Wiki Activity
oxen-core/src/cryptonote_core/cryptonote_core.cpp

2605 lines
106 KiB
C++
Raw Normal View History

Update monero copyright to 2019 pre-emptively to make merge simpler
2019-04-12 06:36:43 +02:00
// Copyright (c) 2014-2019, The Monero Project
First pass Monero to Loki refs
2018-04-10 06:49:20 +02:00
// Copyright (c) 2018, The Loki Project
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// All rights reserved.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// Redistribution and use in source and binary forms, with or without modification, are
// permitted provided that the following conditions are met:
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// 1. Redistributions of source code must retain the above copyright notice, this list of
// conditions and the following disclaimer.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
// of conditions and the following disclaimer in the documentation and/or other
// materials provided with the distribution.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// 3. Neither the name of the copyright holder nor the names of its contributors may be
// used to endorse or promote products derived from this software without specific
// prior written permission.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
moved all stuff to github
2014-03-03 23:07:58 +01:00
move includes around to lessen overall load
2017-11-25 23:25:05 +01:00
#include <boost/algorithm/string.hpp>
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
#include <boost/endian/conversion.hpp>
move includes around to lessen overall load
2017-11-25 23:25:05 +01:00
#include "string_tools.h"
moved all stuff to github
2014-03-03 23:07:58 +01:00
#include <unordered_set>
service node registration: add signatures from service node pubkey to registration
2018-07-21 03:27:13 +02:00
#include <iomanip>
"Remove namespace pollution" << ENDL Removes all "using namespace epee;" and "using namespace std;" from the code and fixes up the various crappy places where unnamespaced types were being used. Also removes the ENDL macro (which was defined to be `std::endl`) because it is retarded, and because even using std::endl instead of a plain "\n" is usually a mistake (`<< std::endl` is equivalent to `<< "\n" << std::flush`, and that explicit flush is rarely desirable).
2020-04-03 18:30:11 +02:00
#include <lokimq/hex.h>
C++17 Switch loki dev branch to C++17 compilation, and update the code with various C++17 niceties. - stop including the (deprecated) lokimq/string_view.h header and instead switch everything to use std::string_view and `""sv` instead of `""_sv`. - std::string_view is much nicer than epee::span, so updated various loki-specific code to use it instead. - made epee "portable storage" serialization accept a std::string_view instead of const lvalue std::string so that we can avoid copying. - switched from mapbox::variant to std::variant - use `auto [a, b] = whatever()` instead of `T1 a; T2 b; std::tie(a, b) = whatever()` in a couple places (in the wallet code). - switch to std::lock(...) instead of boost::lock(...) for simultaneous lock acquisition. boost::lock() won't compile in C++17 mode when given locks of different types. - removed various pre-C++17 workarounds, e.g. for fold expressions, unused argument attributes, and byte-spannable object detection. - class template deduction means lock types no longer have to specify the mutex, so `std::unique_lock<std::mutex> lock{mutex}` can become `std::unique_lock lock{mutex}`. This will make switching any mutex types (e.g. from boost to std mutexes) far easier as you just have to update the type in the header and everything should work. This also makes the tools::unique_lock and tools::shared_lock methods redundant (which were a sort of poor-mans-pre-C++17 way to eliminate the redundancy) so they are now gone and replaced with direct unique_lock or shared_lock constructions. - Redid the LNS validation using a string_view; instead of using raw char pointers the code now uses a string view and chops off parts of the view as it validates. So, for instance, it starts with "abcd.loki", validates the ".loki" and chops the view to "abcd", then validates the first character and chops to "bcd", validates the last and chops to "bc", then can just check everything remaining for is-valid-middle-char. - LNS validation gained a couple minor validation checks in the process: - slightly tightened the requirement on lokinet addresses to require that the last character of the mapped address is 'y' or 'o' (the last base32z char holds only one significant bit). - In parse_owner_to_generic_owner made sure that the owner value has the correct size (otherwise we could up end not filling or overfilling the pubkey buffer). - Replaced base32z/base64/hex conversions with lokimq's versions which have a nicer interface, are better optimized, and don't depend on epee.
2020-05-13 20:12:49 +02:00
#include <lokimq/base32z.h>
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
extern "C" {
#include <sodium.h>
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
#ifdef ENABLE_SYSTEMD
# include <systemd/sd-daemon.h>
#endif
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
}
Remove vendored SQLite, rely on system installed lib
2020-02-03 03:03:32 +01:00
#include <sqlite3.h>
moved all stuff to github
2014-03-03 23:07:58 +01:00
#include "cryptonote_core.h"
#include "common/util.h"
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
#include "common/updates.h"
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
#include "common/download.h"
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
#include "common/threadpool.h"
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
#include "common/command_line.h"
moved all stuff to github
2014-03-03 23:07:58 +01:00
#include "warnings.h"
#include "crypto/crypto.h"
#include "cryptonote_config.h"
#include "misc_language.h"
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
#include "file_io_utils.h"
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
#include <csignal>
move checkpoints in a separate library
2017-09-10 18:35:59 +02:00
#include "checkpoints/checkpoints.h"
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
#include "ringct/rctTypes.h"
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
#include "blockchain_db/blockchain_db.h"
rct: split rct checks between semantics and other Semantics can be checked early
2017-01-14 14:29:08 +01:00
#include "ringct/rctSigs.h"
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 22:18:08 +02:00
#include "common/notify.h"
cryptonote_core does not depend on p2p anymore As a followon side effect, this makes a lot of inline code included only in particular cpp files (and instanciated when necessary.
2017-11-29 23:53:58 +01:00
#include "version.h"
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
#include "memwipe.h"
service node registration: add signatures from service node pubkey to registration
2018-07-21 03:27:13 +02:00
#include "common/i18n.h"
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
#include "net/local_ip.h"
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
#include "cryptonote_protocol/quorumnet.h"
moved all stuff to github
2014-03-03 23:07:58 +01:00
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
#include "common/loki_integration_test_hooks.h"
First pass Monero to Loki refs
2018-04-10 06:49:20 +02:00
#undef LOKI_DEFAULT_LOG_CATEGORY
#define LOKI_DEFAULT_LOG_CATEGORY "cn"
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
moved all stuff to github
2014-03-03 23:07:58 +01:00
DISABLE_VS_WARNINGS(4355)
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
#define MERROR_VER(x) MCERROR("verify", x)
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
#define BAD_SEMANTICS_TXES_MAX_SIZE 100
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
// basically at least how many bytes the block itself serializes to without the miner tx
#define BLOCK_SIZE_SANITY_LEEWAY 100
moved all stuff to github
2014-03-03 23:07:58 +01:00
namespace cryptonote
{
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
const command_line::arg_descriptor<bool, false> arg_testnet_on = {
"testnet"
, "Run on testnet. The wallet must be launched with --testnet flag."
, false
};
Stagenet
2018-02-16 12:04:04 +01:00
const command_line::arg_descriptor<bool, false> arg_stagenet_on = {
"stagenet"
, "Run on stagenet. The wallet must be launched with --stagenet flag."
, false
};
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
const command_line::arg_descriptor<bool> arg_regtest_on = {
"regtest"
, "Run in a regression testing mode."
, false
};
Add a --keep-fakechain option to keep fakechain databases This is handy when doing tests that generate a lot of transactions, since that takes time it's preferable to re-use the database for future runs.
2019-11-01 16:16:37 +01:00
const command_line::arg_descriptor<bool> arg_keep_fakechain = {
"keep-fakechain"
, "Don't delete any existing database when in fakechain mode."
, false
};
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
const command_line::arg_descriptor<difficulty_type> arg_fixed_difficulty = {
"fixed-difficulty"
, "Fixed difficulty used for testing."
, 0
};
Add --dev-allow-local-ip flag for allowing local network public-ip This is very useful for testing. Also adds a suitable scary warning at startup when used.
2019-09-17 03:20:15 +02:00
const command_line::arg_descriptor<bool> arg_dev_allow_local = {
"dev-allow-local-ips"
, "Allow a local IPs for local and received service node public IP (for local testing only)"
, false
};
Stagenet
2018-02-16 12:04:04 +01:00
const command_line::arg_descriptor<std::string, false, true, 2> arg_data_dir = {
options: remove testnet-* options
2018-01-21 16:29:55 +01:00
"data-dir"
, "Specify data directory"
, tools::get_default_data_dir()
Stagenet
2018-02-16 12:04:04 +01:00
, {{ &arg_testnet_on, &arg_stagenet_on }}
fix lambda compile error on openbsd
2018-03-27 15:47:57 +02:00
, [](std::array<bool, 2> testnet_stagenet, bool defaulted, std::string val)->std::string {
Stagenet
2018-02-16 12:04:04 +01:00
if (testnet_stagenet[0])
options: add testnet option dependencies
2018-01-22 02:49:51 +01:00
return (boost::filesystem::path(val) / "testnet").string();
Stagenet
2018-02-16 12:04:04 +01:00
else if (testnet_stagenet[1])
return (boost::filesystem::path(val) / "stagenet").string();
options: add testnet option dependencies
2018-01-22 02:49:51 +01:00
return val;
}
options: remove testnet-* options
2018-01-21 16:29:55 +01:00
};
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
const command_line::arg_descriptor<bool> arg_offline = {
"offline"
, "Do not listen for peers, nor connect to any"
};
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
const command_line::arg_descriptor<size_t> arg_block_download_max_size = {
"block-download-max-size"
, "Set maximum size of block download queue in bytes (0 for default)"
, 0
};
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
static const command_line::arg_descriptor<bool> arg_test_drop_download = {
"test-drop-download"
, "For net tests: in download, discard ALL blocks instead checking/saving them (very fast)"
};
static const command_line::arg_descriptor<uint64_t> arg_test_drop_download_height = {
"test-drop-download-height"
Fix typos in various files
2018-03-01 12:36:19 +01:00
, "Like test-drop-download but discards only after around certain height"
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
, 0
};
static const command_line::arg_descriptor<int> arg_test_dbg_lock_sleep = {
"test-dbg-lock-sleep"
, "Sleep time in ms, defaults to 0 (off), used to debug before/after locking mutex. Values 100 to 1000 are good for tests."
, 0
};
static const command_line::arg_descriptor<uint64_t> arg_fast_block_sync = {
"fast-block-sync"
, "Sync up most of the way by using embedded, known block hashes."
, 1
};
static const command_line::arg_descriptor<uint64_t> arg_prep_blocks_threads = {
"prep-blocks-threads"
, "Max number of threads to use when preparing block hashes in groups."
, 4
};
static const command_line::arg_descriptor<uint64_t> arg_show_time_stats = {
"show-time-stats"
, "Show time-stats when processing blocks/txs and disk synchronization."
, 0
};
static const command_line::arg_descriptor<size_t> arg_block_sync_size = {
"block-sync-size"
, "How many blocks to sync at once during chain synchronization (0 = adaptive)."
, 0
};
static const command_line::arg_descriptor<std::string> arg_check_updates = {
"check-updates"
Update cmakelists and misc refs to Loki
2018-04-25 08:13:38 +02:00
, "Check for new versions of loki: [disabled|notify|download|update]"
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
, "notify"
};
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 23:27:25 +01:00
static const command_line::arg_descriptor<bool> arg_pad_transactions = {
"pad-transactions"
, "Pad relayed transactions to help defend against traffic volume analysis"
, false
};
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
static const command_line::arg_descriptor<size_t> arg_max_txpool_weight = {
"max-txpool-weight"
, "Set maximum txpool weight in bytes."
, DEFAULT_TXPOOL_MAX_WEIGHT
tx_pool: add a max pool size, settable with --max-txpool-size
2018-01-29 22:20:24 +01:00
};
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
static const command_line::arg_descriptor<bool> arg_service_node = {
"service-node"
Removes sn-public-ip and combines into --service-node (#676) * Removes sn-public-ip and combines into --service-node * Only rename the argument, don't combine args for clarity * Update help message to use --service-node-public-ip
2019-06-27 08:03:22 +02:00
, "Run as a service node, options 'service-node-public-ip' and 'storage-server-port' must be set"
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
};
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
static const command_line::arg_descriptor<std::string> arg_public_ip = {
Removes sn-public-ip and combines into --service-node (#676) * Removes sn-public-ip and combines into --service-node * Only rename the argument, don't combine args for clarity * Update help message to use --service-node-public-ip
2019-06-27 08:03:22 +02:00
"service-node-public-ip"
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
, "Public IP address on which this service node's services (such as the Loki "
"storage server) are accessible. This IP address will be advertised to the "
"network via the service node uptime proofs. Required if operating as a "
"service node."
};
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
static const command_line::arg_descriptor<uint16_t> arg_storage_server_port = {
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
"storage-server-port"
, "The port on which this service node's storage server is accessible. A listening "
"storage server is required for service nodes. (This option is specified "
"automatically when using Loki Launcher.)"
, 0};
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
static const command_line::arg_descriptor<uint16_t, false, true, 2> arg_quorumnet_port = {
"quorumnet-port"
, "The port on which this service node listen for direct connections from other "
"service nodes for quorum messages. The port must be publicly reachable "
"on the `--service-node-public-ip' address and binds to the p2p IP address."
" Only applies when running as a service node."
, config::QNET_DEFAULT_PORT
, {{ &cryptonote::arg_testnet_on, &cryptonote::arg_stagenet_on }}
, [](std::array<bool, 2> testnet_stagenet, bool defaulted, uint16_t val) -> uint16_t {
return defaulted && testnet_stagenet[0] ? config::testnet::QNET_DEFAULT_PORT :
defaulted && testnet_stagenet[1] ? config::stagenet::QNET_DEFAULT_PORT :
val;
}
};
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
static const command_line::arg_descriptor<bool> arg_lmq_quorumnet_public{
"lmq-public-quorumnet",
"Allow the curve-enabled quorumnet address (for a Service Node) to be used for public RPC commands as if passed to --lmq-curve-public. "
"Note that even without this option the quorumnet port can be used for RPC commands by --lmq-admin and --lmq-user pubkeys.",
false};
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 22:18:08 +02:00
static const command_line::arg_descriptor<std::string> arg_block_notify = {
"block-notify"
, "Run a program for each new block, '%s' will be replaced by the block hash"
, ""
};
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
static const command_line::arg_descriptor<bool> arg_prune_blockchain = {
"prune-blockchain"
, "Prune blockchain"
, false
};
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 15:29:28 +01:00
static const command_line::arg_descriptor<std::string> arg_reorg_notify = {
"reorg-notify"
, "Run a program for each reorg, '%s' will be replaced by the split height, "
"'%h' will be replaced by the new blockchain height, and '%n' will be "
"replaced by the number of new blocks in the new chain"
, ""
};
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
static const command_line::arg_descriptor<std::string> arg_block_rate_notify = {
"block-rate-notify"
, "Run a program when the block rate undergoes large fluctuations. This might "
Loki-ify --block-rate-notify - Monero -> Loki - Remove the part about attacks as checkpointing alleviates that concern. - Remove the recommendation about using it for confirmations, since again checkpointing and blink make it unnecessary.
2020-04-01 19:43:33 +02:00
"be a sign of large amounts of hash rate going on and off the Loki network, "
"or could be a sign that lokid is not properly synchronizing with the network. %t will be replaced "
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
"by the number of minutes for the observation window, %b by the number of "
"blocks observed within that window, and %e by the number of blocks that was "
Loki-ify --block-rate-notify - Monero -> Loki - Remove the part about attacks as checkpointing alleviates that concern. - Remove the recommendation about using it for confirmations, since again checkpointing and blink make it unnecessary.
2020-04-01 19:43:33 +02:00
"expected in that window."
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
, ""
};
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
static const command_line::arg_descriptor<bool> arg_keep_alt_blocks = {
"keep-alt-blocks"
, "Keep alternative blocks on restart"
, false
};
moved all stuff to github
2014-03-03 23:07:58 +01:00
Add flag to recalculate difficulty per block (#598) * Beging adding functions to recalculate difficulty * Add command line args to utility for recalculating difficulty * Exception safety for recalculating difficulty * Update help text for recalc difficulty * Add recalc flag on the daemon * Make context be const, signify intent for var++ to 1
2019-05-12 01:40:57 +02:00
const command_line::arg_descriptor<uint64_t> arg_recalculate_difficulty = {
"recalculate-difficulty",
"Recalculate per-block difficulty starting from the height specified",
Add grace to required difficulty - fix 526484 At height 526483 a mined block arrived that had *slightly* too little proof of work, yet difficulties on the network had apparently diverged enough that half the network accepted it and the other half didn't -- nodes with the correct difficulty refuse it, so a recalculate difficulty cannot help. This hacks around the problem from 526483 until the next hard fork by adding a grace to the accepted block difficulty - we'll accept a block from the network if it has 99.8% of the calculated difficulty value. It also defaults --recalculate-difficulty to 1 so that lokid does a full difficulty rescan at startup. (This seems to only take a second or two so seems like a good thing to enable by default).
2020-05-05 17:52:35 +02:00
// This is now enabled by default because the network broke at 526483 because of divergent
// difficulty values (and the chain that kept going violated the correct difficulty, and got
// checkpointed multiple times because enough of the network followed it).
//
// TODO: We can disable this post-pulse (since diff won't matter anymore), but until then there
// is a subtle bug somewhere in difficulty calculations that can cause divergence; this seems
// important enough to just rescan at every startup (and only takes a few seconds).
1};
Add flag to recalculate difficulty per block (#598) * Beging adding functions to recalculate difficulty * Add command line args to utility for recalculating difficulty * Exception safety for recalculating difficulty * Update help text for recalc difficulty * Add recalc flag on the daemon * Make context be const, signify intent for var++ to 1
2019-05-12 01:40:57 +02:00
(Low priority) Add disabled-by-default quorum storage support (#702) * Add disabled-by-default quorum storage support This adds support for storing N blocks of recent expired quorum state history in lokid (and dumping to/from the lmdb). This isn't useful for regular nodes (and that's why we don't store it) but is incredibly useful for the block explorer to be able to report *which* node got deregistered/decommissioned/etc. by a given state_change tx. * Fix quorum states copy
2019-07-17 08:01:54 +02:00
static const command_line::arg_descriptor<uint64_t> arg_store_quorum_history = {
"store-quorum-history",
"Store the service node quorum history for the last N blocks to allow historic quorum lookups "
"(e.g. by a block explorer). Specify the number of blocks of history to store, or 1 to store "
"the entire history. Requires considerably more memory and block chain storage.",
0};
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
// Loads stubs that fail if invoked. The stubs are replaced in the cryptonote_protocol/quorumnet.cpp glue code.
[[noreturn]] static void need_core_init() {
throw std::logic_error("Internal error: quorumnet::init_core_callbacks() should have been called");
}
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
void *(*quorumnet_new)(core&);
void (*quorumnet_delete)(void*&self);
void (*quorumnet_relay_obligation_votes)(void* self, const std::vector<service_nodes::quorum_vote_t>&);
std::future<std::pair<blink_result, std::string>> (*quorumnet_send_blink)(core& core, const std::string& tx_blob);
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
static bool init_core_callback_stubs() {
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
quorumnet_new = [](core&) -> void* { need_core_init(); };
quorumnet_delete = [](void*&) { need_core_init(); };
quorumnet_relay_obligation_votes = [](void*, const std::vector<service_nodes::quorum_vote_t>&) { need_core_init(); };
quorumnet_send_blink = [](core&, const std::string&) -> std::future<std::pair<blink_result, std::string>> { need_core_init(); };
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
return false;
}
bool init_core_callback_complete = init_core_callback_stubs();
(Low priority) Add disabled-by-default quorum storage support (#702) * Add disabled-by-default quorum storage support This adds support for storing N blocks of recent expired quorum state history in lokid (and dumping to/from the lmdb). This isn't useful for regular nodes (and that's why we don't store it) but is incredibly useful for the block explorer to be able to report *which* node got deregistered/decommissioned/etc. by a given state_change tx. * Fix quorum states copy
2019-07-17 08:01:54 +02:00
moved all stuff to github
2014-03-03 23:07:58 +01:00
//-----------------------------------------------------------------------------------------------
Remove optional dependencies from constructor, no code actually uses it
2020-04-23 07:52:19 +02:00
core::core()
: m_mempool(m_blockchain_storage)
, m_service_node_list(m_blockchain_storage)
, m_blockchain_storage(m_mempool, m_service_node_list)
, m_quorum_cop(*this)
Merge commit '8cc7d5b6c0c611dd03eba21c9cc7a9a043fa7b04' into MergeUpstream3
2020-05-25 08:08:05 +02:00
, m_miner(this, [this](const cryptonote::block &b, uint64_t height, unsigned int threads, crypto::hash &hash) {
Provide randomx data when filling nonce Monero decided to implement this by rebuilding the blockchain class in tests with another FakeDB. You can trivially avoid this by restructuring get_block_longhash to not have a high level dependency on the blockchain itself.
2020-05-29 05:37:35 +02:00
hash = cryptonote::get_block_longhash_w_blockchain(&m_blockchain_storage, b, height, threads);
return true;
Merge commit '8cc7d5b6c0c611dd03eba21c9cc7a9a043fa7b04' into MergeUpstream3
2020-05-25 08:08:05 +02:00
})
Remove optional dependencies from constructor, no code actually uses it
2020-04-23 07:52:19 +02:00
, m_pprotocol(&m_protocol_stub)
, m_starter_message_showed(false)
, m_target_blockchain_height(0)
, m_checkpoints_path("")
, m_last_json_checkpoints_update(0)
, m_update_download(0)
, m_nettype(UNDEFINED)
, m_update_available(false)
, m_last_storage_server_ping(0)
, m_last_lokinet_ping(0)
, m_pad_transactions(false)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
cryptonote_core: initialize checkpoint flag
2017-07-02 19:42:45 +02:00
m_checkpoints_updating.clear();
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
void core::set_cryptonote_protocol(i_cryptonote_protocol* pprotocol)
{
if(pprotocol)
m_pprotocol = pprotocol;
else
m_pprotocol = &m_protocol_stub;
}
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 07:15:28 +02:00
//-----------------------------------------------------------------------------------------------
Rollback chain when receiving conflicting checkpoints (#726)
2019-07-11 06:16:55 +02:00
bool core::update_checkpoints_from_json_file()
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 07:15:28 +02:00
{
DNS Checkpoint updating-related fixes/changes Only one thread will be doing the updating. Two valid responses must match, and the first two that match will be used.
2015-04-22 10:36:39 +02:00
if (m_checkpoints_updating.test_and_set()) return true;
Service Node Checkpoint Storage (#579) * Add functions for storing checkpoints to the DB * Allocate the DB entry on the stack instead of heap * Add virtual overrides for new checkpoint functions * Clean up for pull request, simplify some logic * Revise API to include height in checkpoint header * Move log to top of function even if early exit * Begin moving checkpoints to db * Allow storing of checkpoints to DB * Cleanup for code reviewer, fix unit tests * Fix tests, fix casting issues * Don't use DUPSORT, use height->checkpoint mapping in DB * Remove if 0 disabling checkpoint vote, we already check HF12 * Fix unit test infinite loop * Update db schemas to match blk_checkpoint_header * Code review
2019-05-01 07:57:41 +02:00
// load json checkpoints every 10min and verify them with respect to what blocks we already have
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 07:01:58 +02:00
bool res = true;
Merge dev into upstream
2019-05-01 08:01:17 +02:00
if (time(NULL) - m_last_json_checkpoints_update >= 600)
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 07:15:28 +02:00
{
Rollback chain when receiving conflicting checkpoints (#726)
2019-07-11 06:16:55 +02:00
res = m_blockchain_storage.update_checkpoints_from_json_file(m_checkpoints_path);
various changes to runtime checkpoint updating json checkpoints will be checked every 10 minutes, dns every 60. json checkpoints always enforced, dns still with flag. conflicting checkpoints is hard fail, but soft if dns enforce flag not set and dns checkpoints are wonky.
2014-09-29 22:30:47 +02:00
m_last_json_checkpoints_update = time(NULL);
}
DNS Checkpoint updating-related fixes/changes Only one thread will be doing the updating. Two valid responses must match, and the first two that match will be used.
2015-04-22 10:36:39 +02:00
m_checkpoints_updating.clear();
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
// if anything fishy happened getting new checkpoints, bring down the house
if (!res)
{
graceful_exit();
}
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 07:01:58 +02:00
return res;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------
RPC calls for background daemon added in The RPC calls the daemon executable uses to talk to the running daemon instance have mostly been added back in. Rate limiting has not been added in upstream, but is on its way in a separate effort, so those calls are still NOPed out.
2015-02-05 10:11:20 +01:00
void core::stop()
{
daemon: stop miner before we bring the whole thing down This avoids the miner erroring out trying to submit blocks to a core that's already shut down (and avoids pegging the CPU while we're busy shutting down).
2018-11-30 15:53:38 +01:00
m_miner.stop();
Fix delayed exit when syncing
2016-12-04 13:27:45 +01:00
m_blockchain_storage.cancel();
core: make update download cancellable
2017-02-26 22:00:38 +01:00
tools::download_async_handle handle;
{
boost::lock_guard<boost::mutex> lock(m_update_mutex);
handle = m_update_download;
m_update_download = 0;
}
if (handle)
tools::download_cancel(handle);
RPC calls for background daemon added in The RPC calls the daemon executable uses to talk to the running daemon instance have mostly been added back in. Rate limiting has not been added in upstream, but is on its way in a separate effort, so those calls are still NOPed out.
2015-02-05 10:11:20 +01:00
}
//-----------------------------------------------------------------------------------
Register daemon command line arguments to core if they're used in core This fixes coretests, which does not register daemon specific arguments, but uses core, which uses those arguments. Also gets rid of an unwanted dependency on daemon code from core.
2015-12-09 00:06:29 +01:00
void core::init_options(boost::program_options::options_description& desc)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
options: remove testnet-* options
2018-01-21 16:29:55 +01:00
command_line::add_arg(desc, arg_data_dir);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
command_line::add_arg(desc, arg_test_drop_download);
command_line::add_arg(desc, arg_test_drop_download_height);
command_line::add_arg(desc, arg_testnet_on);
Stagenet
2018-02-16 12:04:04 +01:00
command_line::add_arg(desc, arg_stagenet_on);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
command_line::add_arg(desc, arg_regtest_on);
Add a --keep-fakechain option to keep fakechain databases This is handy when doing tests that generate a lot of transactions, since that takes time it's preferable to re-use the database for future runs.
2019-11-01 16:16:37 +01:00
command_line::add_arg(desc, arg_keep_fakechain);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
command_line::add_arg(desc, arg_fixed_difficulty);
Add --dev-allow-local-ip flag for allowing local network public-ip This is very useful for testing. Also adds a suitable scary warning at startup when used.
2019-09-17 03:20:15 +02:00
command_line::add_arg(desc, arg_dev_allow_local);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
command_line::add_arg(desc, arg_prep_blocks_threads);
command_line::add_arg(desc, arg_fast_block_sync);
command_line::add_arg(desc, arg_show_time_stats);
command_line::add_arg(desc, arg_block_sync_size);
command_line::add_arg(desc, arg_check_updates);
command_line::add_arg(desc, arg_test_dbg_lock_sleep);
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
command_line::add_arg(desc, arg_offline);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
command_line::add_arg(desc, arg_block_download_max_size);
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
command_line::add_arg(desc, arg_max_txpool_weight);
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
command_line::add_arg(desc, arg_service_node);
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
command_line::add_arg(desc, arg_public_ip);
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
command_line::add_arg(desc, arg_storage_server_port);
command_line::add_arg(desc, arg_quorumnet_port);
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 23:27:25 +01:00
command_line::add_arg(desc, arg_pad_transactions);
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 22:18:08 +02:00
command_line::add_arg(desc, arg_block_notify);
Disable pruning in Loki, it is not supported yet
2019-10-23 04:44:56 +02:00
#if 0 // TODO(loki): Pruning not supported because of Service Node List
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
command_line::add_arg(desc, arg_prune_blockchain);
Disable pruning in Loki, it is not supported yet
2019-10-23 04:44:56 +02:00
#endif
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 15:29:28 +01:00
command_line::add_arg(desc, arg_reorg_notify);
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
command_line::add_arg(desc, arg_block_rate_notify);
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
command_line::add_arg(desc, arg_keep_alt_blocks);
core: add p2p bind port options from net_node They're now used by core to determine the data directory to use for the txpool directory. This fixes an assert in the core tests, which don't use the RPC server, which normally initializes the P2P code.
2017-02-09 22:29:35 +01:00
Add flag to recalculate difficulty per block (#598) * Beging adding functions to recalculate difficulty * Add command line args to utility for recalculating difficulty * Exception safety for recalculating difficulty * Update help text for recalc difficulty * Add recalc flag on the daemon * Make context be const, signify intent for var++ to 1
2019-05-12 01:40:57 +02:00
command_line::add_arg(desc, arg_recalculate_difficulty);
(Low priority) Add disabled-by-default quorum storage support (#702) * Add disabled-by-default quorum storage support This adds support for storing N blocks of recent expired quorum state history in lokid (and dumping to/from the lmdb). This isn't useful for regular nodes (and that's why we don't store it) but is incredibly useful for the block explorer to be able to report *which* node got deregistered/decommissioned/etc. by a given state_change tx. * Fix quorum states copy
2019-07-17 08:01:54 +02:00
command_line::add_arg(desc, arg_store_quorum_history);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
#if defined(LOKI_ENABLE_INTEGRATION_TEST_HOOKS)
Rewrite the integration tests to use pipes (#1045)
2020-02-21 04:48:25 +01:00
command_line::add_arg(desc, integration_test::arg_hardforks_override);
command_line::add_arg(desc, integration_test::arg_pipe_name);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
#endif
output_blacklist: Remove redundant return, report exception
2020-06-02 07:50:27 +02:00
command_line::add_arg(desc, arg_lmq_quorumnet_public);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
Fix core_tests breaking on startup You're wondering how this fixes core tests, aren't you... It prevents the miner (initialized by cryptonote::core) from breaking trying to access arguments that were not added. Since the tests don't use the miner directly, it makes more sense to have cryptonote_core add those, since it also uses the miner.
2017-02-21 16:06:07 +01:00
miner::init_options(desc);
move db specific options to BlockchainDB Avoids common depending on blockchain_db, which can cause link errors.
2017-08-29 12:14:42 +02:00
BlockchainDB::init_options(desc);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
bool core::handle_command_line(const boost::program_options::variables_map& vm)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Stagenet
2018-02-16 12:04:04 +01:00
if (m_nettype != FAKECHAIN)
{
const bool testnet = command_line::get_arg(vm, arg_testnet_on);
const bool stagenet = command_line::get_arg(vm, arg_stagenet_on);
m_nettype = testnet ? TESTNET : stagenet ? STAGENET : MAINNET;
}
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
options: remove testnet-* options
2018-01-21 16:29:55 +01:00
m_config_folder = command_line::get_arg(vm, arg_data_dir);
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
test_drop_download_height(command_line::get_arg(vm, arg_test_drop_download_height));
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 23:27:25 +01:00
m_pad_transactions = get_arg(vm, arg_pad_transactions);
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
m_offline = get_arg(vm, arg_offline);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
if (command_line::get_arg(vm, arg_test_drop_download) == true)
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
test_drop_download();
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
Add --dev-allow-local-ip flag for allowing local network public-ip This is very useful for testing. Also adds a suitable scary warning at startup when used.
2019-09-17 03:20:15 +02:00
if (command_line::get_arg(vm, arg_dev_allow_local))
m_service_node_list.debug_allow_local_ips = true;
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
m_service_node = command_line::get_arg(vm, arg_service_node);
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (m_service_node) {
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
/// TODO: parse these options early, before we start p2p server etc?
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
m_storage_port = command_line::get_arg(vm, arg_storage_server_port);
m_quorumnet_port = command_line::get_arg(vm, arg_quorumnet_port);
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
bool storage_ok = true;
if (m_storage_port == 0) {
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
MERROR("Please specify the port on which the storage server is listening with: '--" << arg_storage_server_port.name << " <port>'");
storage_ok = false;
}
if (m_quorumnet_port == 0) {
MERROR("Quorumnet port cannot be 0; please specify a valid port to listen on with: '--" << arg_quorumnet_port.name << " <port>'");
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
storage_ok = false;
}
const std::string pub_ip = command_line::get_arg(vm, arg_public_ip);
Make --service-node give more verbose failure flags (#630)
2019-06-06 09:39:54 +02:00
if (pub_ip.size())
{
if (!epee::string_tools::get_ip_int32_from_string(m_sn_public_ip, pub_ip)) {
MERROR("Unable to parse IPv4 public address from: " << pub_ip);
storage_ok = false;
}
Tighten allowed IPv4 ranges epee's is_ip_local is missing two ranges that are commonly found: link-local auto-config addresses (169.254.0.0/16) that are sometimes used as a fallback when DHCP isn't present; and the carrier-grade NAT range (100.64.0.0/10) reserved for carriers who impose NAT on their customers. There are also other ranges that aren't exactly "local" but aren't public either: 0.0.0.0/8 isn't a valid destination address; and 224.0.0.0/3 (includes includes both the 224/4 multicast range, and the reserved (but will most likely never be used) 240.0.0.0/4 range). These are now added to a new `is_ip_public` function that returns true if it's not local or loopback, and not one of these special ranges. This also simplifies some convoluted netmask logic. (The simplification would look better except that epee took the extremely bizarre and wrong decision to store IPv4 addresses in little-endian order).
2019-06-25 18:31:38 +02:00
if (!epee::net_utils::is_ip_public(m_sn_public_ip)) {
Add --dev-allow-local-ip flag for allowing local network public-ip This is very useful for testing. Also adds a suitable scary warning at startup when used.
2019-09-17 03:20:15 +02:00
if (m_service_node_list.debug_allow_local_ips) {
MWARNING("Address given for public-ip is not public; allowing it because dev-allow-local-ips was specified. This service node WILL NOT WORK ON THE PUBLIC LOKI NETWORK!");
} else {
MERROR("Address given for public-ip is not public: " << epee::string_tools::get_ip_string_from_int32(m_sn_public_ip));
storage_ok = false;
}
Make --service-node give more verbose failure flags (#630)
2019-06-06 09:39:54 +02:00
}
}
else
{
MERROR("Please specify an IPv4 public address which the service node & storage server is accessible from with: '--" << arg_public_ip.name << " <ip address>'");
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
storage_ok = false;
}
if (!storage_ok) {
Make --service-node give more verbose failure flags (#630)
2019-06-06 09:39:54 +02:00
MERROR("IMPORTANT: All service node operators are now required to run the loki storage "
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
<< "server and provide the public ip and ports on which it can be accessed on the internet.");
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
return false;
}
MGINFO("Storage server endpoint is set to: "
<< (epee::net_utils::ipv4_network_address{ m_sn_public_ip, m_storage_port }).str());
Add storage server lokimq port (#1056) * Add storage server lokimq port * Don't hash storage lmq port until HF15 * Parse storage lmq port from ping
2020-03-11 02:35:36 +01:00
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
}
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
epee::debug::g_test_dbg_lock_sleep() = command_line::get_arg(vm, arg_test_dbg_lock_sleep);
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
uint64_t core::get_current_blockchain_height() const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_current_blockchain_height();
}
//-----------------------------------------------------------------------------------------------
get_blockchain_top now returns void It was always returning true, and could not be foreseen to usefully return errors in the future. This silences CID 162652 as well as saves some checking code in a few places.
2017-09-09 13:06:24 +02:00
void core::get_blockchain_top(uint64_t& height, crypto::hash& top_id) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
top_id = m_blockchain_storage.get_tail_id(height);
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_blocks(uint64_t start_offset, size_t count, std::vector<std::pair<cryptonote::blobdata,block>>& blocks, std::vector<cryptonote::blobdata>& txs) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_blocks(start_offset, count, blocks, txs);
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_blocks(uint64_t start_offset, size_t count, std::vector<std::pair<cryptonote::blobdata,block>>& blocks) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_blocks(start_offset, count, blocks);
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_blocks(uint64_t start_offset, size_t count, std::vector<block>& blocks) const
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<std::pair<cryptonote::blobdata, cryptonote::block>> bs;
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
if (!m_blockchain_storage.get_blocks(start_offset, count, bs))
return false;
for (const auto &b: bs)
blocks.push_back(b.second);
return true;
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_transactions(const std::vector<crypto::hash>& txs_ids, std::vector<cryptonote::blobdata>& txs, std::vector<crypto::hash>& missed_txs) const
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
{
return m_blockchain_storage.get_transactions_blobs(txs_ids, txs, missed_txs);
}
//-----------------------------------------------------------------------------------------------
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
bool core::get_split_transactions_blobs(const std::vector<crypto::hash>& txs_ids, std::vector<std::tuple<crypto::hash, cryptonote::blobdata, crypto::hash, cryptonote::blobdata>>& txs, std::vector<crypto::hash>& missed_txs) const
{
return m_blockchain_storage.get_split_transactions_blobs(txs_ids, txs, missed_txs);
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_transactions(const std::vector<crypto::hash>& txs_ids, std::vector<transaction>& txs, std::vector<crypto::hash>& missed_txs) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_transactions(txs_ids, txs, missed_txs);
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
bool core::get_alternative_blocks(std::vector<block>& blocks) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_alternative_blocks(blocks);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
size_t core::get_alternative_blocks_count() const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_alternative_blocks_count();
}
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
static std::string time_ago_str(time_t now, time_t then) {
if (then >= now)
return "now"s;
if (then == 0)
return "never"s;
int seconds = now - then;
if (seconds >= 60)
return std::to_string(seconds / 60) + "m" + std::to_string(seconds % 60) + "s";
return std::to_string(seconds % 60) + "s";
}
// Returns a string for systemd status notifications such as:
// Height: 1234567, SN: active, proof: 55m12s, storage: 4m48s, lokinet: 47s
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
std::string core::get_status_string() const
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
{
std::string s;
s.reserve(128);
Add version to beginning of systemd status string (#1092) Lokinet and storage server have this and it's nice to be able to tell at a glance at the service status what lokid version is running.
2020-03-18 09:28:06 +01:00
s += 'v'; s += LOKI_VERSION_STR;
s += "; Height: ";
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
s += std::to_string(get_blockchain_storage().get_current_blockchain_height());
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
s += ", SN: ";
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (!service_node())
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
s += "no";
else
{
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
auto& snl = get_service_node_list();
const auto& pubkey = get_service_keys().pub;
auto states = snl.get_service_node_list_state({ pubkey });
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
if (states.empty())
s += "not registered";
else
{
auto &info = *states[0].info;
if (!info.is_fully_funded())
s += "awaiting contr.";
else if (info.is_active())
s += "active";
else if (info.is_decommissioned())
s += "decomm.";
uint64_t last_proof = 0;
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
snl.access_proof(pubkey, [&](auto& proof) { last_proof = proof.timestamp; });
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
s += ", proof: ";
time_t now = std::time(nullptr);
s += time_ago_str(now, last_proof);
s += ", storage: ";
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
s += time_ago_str(now, m_last_storage_server_ping);
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
s += ", lokinet: ";
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
s += time_ago_str(now, m_last_lokinet_ping);
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
}
}
return s;
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
//-----------------------------------------------------------------------------------------------
remove some unused code Found by codacy.com
2018-11-23 14:47:51 +01:00
bool core::init(const boost::program_options::variables_map& vm, const cryptonote::test_options *test_options, const GetCheckpointsCallback& get_checkpoints/* = nullptr */)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Add start_time to get_info methods and show uptime
2017-01-09 00:50:29 +01:00
start_time = std::time(nullptr);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
#if defined(LOKI_ENABLE_INTEGRATION_TEST_HOOKS)
Rewrite the integration tests to use pipes (#1045)
2020-02-21 04:48:25 +01:00
const std::string arg_hardforks_override = command_line::get_arg(vm, integration_test::arg_hardforks_override);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
std::vector<std::pair<uint8_t, uint64_t>> integration_test_hardforks;
Rewrite the integration tests to use pipes (#1045)
2020-02-21 04:48:25 +01:00
if (!arg_hardforks_override.empty())
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
{
// Expected format: <fork_version>:<fork_height>, ...
// Example: 7:0, 8:10, 9:20, 10:100
Rewrite the integration tests to use pipes (#1045)
2020-02-21 04:48:25 +01:00
char const *ptr = arg_hardforks_override.c_str();
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
while (ptr[0])
{
int hf_version = atoi(ptr);
while(ptr[0] != ':') ptr++;
++ptr;
int hf_height = atoi(ptr);
while(ptr[0] && ptr[0] != ',') ptr++;
integration_test_hardforks.push_back(std::make_pair(static_cast<uint8_t>(hf_version), static_cast<uint64_t>(hf_height)));
if (!ptr[0]) break;
ptr++;
}
}
cryptonote::test_options integration_hardfork_override = {integration_test_hardforks};
Rewrite the integration tests to use pipes (#1045)
2020-02-21 04:48:25 +01:00
if (!arg_hardforks_override.empty())
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
test_options = &integration_hardfork_override;
Add ability to handle multiple daemon/wallets
2018-11-23 05:18:08 +01:00
{
Rewrite the integration tests to use pipes (#1045)
2020-02-21 04:48:25 +01:00
const std::string arg_pipe_name = command_line::get_arg(vm, integration_test::arg_pipe_name);
integration_test::init(arg_pipe_name);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
}
#endif
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
const bool regtest = command_line::get_arg(vm, arg_regtest_on);
if (test_options != NULL || regtest)
Stagenet
2018-02-16 12:04:04 +01:00
{
m_nettype = FAKECHAIN;
}
Add support for fakechain in wallet
2018-11-21 00:43:09 +01:00
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
bool r = handle_command_line(vm);
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
/// Currently terminating before blockchain is initialized results in a crash
/// during deinitialization... TODO: fix that
CHECK_AND_ASSERT_MES(r, false, "Failed to apply command line options.");
Separate data per P2P port poolstate.bin and p2pstate.bin are stored in .bitmonero/ if the default P2P port is being used. If another port is used both files are stored in .bitmonero/PORTNUMBER/.
2017-01-28 01:36:39 +01:00
move db specific options to BlockchainDB Avoids common depending on blockchain_db, which can cause link errors.
2017-08-29 12:14:42 +02:00
std::string db_sync_mode = command_line::get_arg(vm, cryptonote::arg_db_sync_mode);
bool db_salvage = command_line::get_arg(vm, cryptonote::arg_db_salvage) != 0;
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
bool fast_sync = command_line::get_arg(vm, arg_fast_block_sync) != 0;
uint64_t blocks_threads = command_line::get_arg(vm, arg_prep_blocks_threads);
std::string check_updates_string = command_line::get_arg(vm, arg_check_updates);
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
size_t max_txpool_weight = command_line::get_arg(vm, arg_max_txpool_weight);
Don't query prune blockchain arg boost cast failure Removing the prune blockchain arg from the command_line and trying to query it causesd a boost any_cast failuree
2019-12-04 06:27:51 +01:00
bool const prune_blockchain = false; /* command_line::get_arg(vm, arg_prune_blockchain); */
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
bool keep_alt_blocks = command_line::get_arg(vm, arg_keep_alt_blocks);
Add a --keep-fakechain option to keep fakechain databases This is handy when doing tests that generate a lot of transactions, since that takes time it's preferable to re-use the database for future runs.
2019-11-01 16:16:37 +01:00
bool keep_fakechain = command_line::get_arg(vm, arg_keep_fakechain);
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
r = init_service_keys();
CHECK_AND_ASSERT_MES(r, false, "Failed to create or load service keys");
if (m_service_node)
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
{
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
// Only use our service keys for our service node if we are running in SN mode:
m_service_node_list.set_my_service_node_keys(&m_service_keys);
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
}
Nicer looking exit when blockchain.bin is found Do not print the exception message, and write the important bit in red, since people will only read the last line otherwise.
2015-12-31 18:09:00 +01:00
boost::filesystem::path folder(m_config_folder);
Stagenet
2018-02-16 12:04:04 +01:00
if (m_nettype == FAKECHAIN)
Nicer looking exit when blockchain.bin is found Do not print the exception message, and write the important bit in red, since people will only read the last line otherwise.
2015-12-31 18:09:00 +01:00
folder /= "fake";
core: move the db lock to the data directory Locking just one db turns out to not have been a good idea, since the pool and p2p state fdles have to be used anyway. Also ensure the directory exists before tring to lock.
2016-02-04 19:09:45 +01:00
// make sure the data directory exists, and try to lock it
CHECK_AND_ASSERT_MES (boost::filesystem::exists(folder) || boost::filesystem::create_directories(folder), false,
std::string("Failed to create directory ").append(folder.string()).c_str());
Removed Berkeley DB and db switching logic
2019-06-26 16:39:41 +02:00
std::unique_ptr<BlockchainDB> db(new_db());
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
if (db == NULL)
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
{
Removed Berkeley DB and db switching logic
2019-06-26 16:39:41 +02:00
LOG_ERROR("Failed to initialize a database");
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
return false;
}
folder /= db->get_db_name();
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MGINFO("Loading blockchain from folder " << folder.string() << " ...");
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
const std::string filename = folder.string();
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
// default to fast:async:1 if overridden
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 20:36:51 +02:00
blockchain_db_sync_mode sync_mode = db_defaultsync;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
bool sync_on_blocks = true;
uint64_t sync_threshold = 1;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
Add support for Loki Name Service on the backend
2019-11-01 01:58:48 +01:00
std::string const lns_db_file_path = m_config_folder + "/lns.db";
#if !defined(LOKI_ENABLE_INTEGRATION_TEST_HOOKS) // In integration mode, don't delete the DB. This should be explicitly done in the tests. Otherwise the more likely behaviour is persisting the DB across multiple daemons in the same test.
Add a --keep-fakechain option to keep fakechain databases This is handy when doing tests that generate a lot of transactions, since that takes time it's preferable to re-use the database for future runs.
2019-11-01 16:16:37 +01:00
if (m_nettype == FAKECHAIN && !keep_fakechain)
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
{
// reset the db by removing the database file before opening it
if (!db->remove_data_file(filename))
{
MERROR("Failed to remove data file in " << filename);
return false;
}
Add support for Loki Name Service on the backend
2019-11-01 01:58:48 +01:00
boost::filesystem::remove(lns_db_file_path);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
}
Add support for Loki Name Service on the backend
2019-11-01 01:58:48 +01:00
#endif
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
try
{
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
uint64_t db_flags = 0;
std::vector<std::string> options;
boost::trim(db_sync_mode);
boost::split(options, db_sync_mode, boost::is_any_of(" :"));
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
const bool db_sync_mode_is_default = command_line::is_arg_defaulted(vm, cryptonote::arg_db_sync_mode);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
for(const auto &option : options)
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MDEBUG("option: " << option);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 04:38:55 +01:00
// default to fast:async:1
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
uint64_t DEFAULT_FLAGS = DBF_FAST;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
if(options.size() == 0)
{
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 04:38:55 +01:00
// default to fast:async:1
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
db_flags = DEFAULT_FLAGS;
}
bool safemode = false;
if(options.size() >= 1)
{
if(options[0] == "safe")
{
safemode = true;
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
db_flags = DBF_SAFE;
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_nosync;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
}
else if(options[0] == "fast")
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 20:36:51 +02:00
{
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
db_flags = DBF_FAST;
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 20:36:51 +02:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
else if(options[0] == "fastest")
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 04:38:55 +01:00
{
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
db_flags = DBF_FASTEST;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
sync_threshold = 1000; // default to fastest:async:1000
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 04:38:55 +01:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
else
db_flags = DEFAULT_FLAGS;
}
if(options.size() >= 2 && !safemode)
{
if(options[1] == "sync")
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_sync;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
else if(options[1] == "async")
core: fix automatic safe db sync mode switching
2018-05-28 13:12:19 +02:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
}
if(options.size() >= 3 && !safemode)
{
Remove blocks_per_sync limits The code used to cap at 5000 blocks per sync. It also treated 0 as 1. Remove these checks; if specified as 0 do no periodic syncs at all. Then the user is responsible for syncing in some external process.
2016-08-29 17:42:52 +02:00
char *endptr;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
uint64_t threshold = strtoull(options[2].c_str(), &endptr, 0);
if (*endptr == '\0' || !strcmp(endptr, "blocks"))
{
sync_on_blocks = true;
sync_threshold = threshold;
}
else if (!strcmp(endptr, "bytes"))
{
sync_on_blocks = false;
sync_threshold = threshold;
}
else
{
LOG_ERROR("Invalid db sync mode: " << options[2]);
return false;
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
}
core: add --db-salvage command line flag Use to load the database when the primary meta page is corrupted
2017-07-31 00:59:52 +02:00
if (db_salvage)
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 16:27:13 +02:00
db_flags |= DBF_SALVAGE;
core: add --db-salvage command line flag Use to load the database when the primary meta page is corrupted
2017-07-31 00:59:52 +02:00
Rescan difficulty periodically, fix incorrect timestamps/difficulties in rescan (#835) * Rescan difficulty periodically, fix incorrect timestamps/difficulties in rescan * Code review
2019-09-13 07:46:15 +02:00
db->open(filename, m_nettype, db_flags);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
if(!db->m_open)
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
return false;
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
}
catch (const DB_ERROR& e)
{
Moved logging to target functions rather than caller
2016-10-03 23:11:00 +02:00
LOG_ERROR("Error opening database: " << e.what());
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
return false;
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
m_blockchain_storage.set_user_options(blocks_threads,
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
sync_on_blocks, sync_threshold, sync_mode, fast_sync);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 22:18:08 +02:00
try
{
if (!command_line::is_arg_defaulted(vm, arg_block_notify))
m_blockchain_storage.set_block_notify(std::shared_ptr<tools::Notify>(new tools::Notify(command_line::get_arg(vm, arg_block_notify).c_str())));
}
catch (const std::exception &e)
{
MERROR("Failed to parse block notify spec");
}
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 15:29:28 +01:00
try
{
if (!command_line::is_arg_defaulted(vm, arg_reorg_notify))
m_blockchain_storage.set_reorg_notify(std::shared_ptr<tools::Notify>(new tools::Notify(command_line::get_arg(vm, arg_reorg_notify).c_str())));
}
catch (const std::exception &e)
{
MERROR("Failed to parse reorg notify spec");
}
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
try
{
if (!command_line::is_arg_defaulted(vm, arg_block_rate_notify))
m_block_rate_notify.reset(new tools::Notify(command_line::get_arg(vm, arg_block_rate_notify).c_str()));
}
catch (const std::exception &e)
{
MERROR("Failed to parse block rate notify spec");
}
Fix blink test coin generation The --regtest/fakechain mode that blink uses no longer generates enough mining rewards to submit blink registrations under the LRC-6 change (with HF16 mining rewards ta 0). This commit changes two things: - fakechain now goes hf7-8-9-...-16 instead of jumping immediately from hf7 to hf16 at height 2; this ensures that there are *some* mining rewards (before hf16) that can be used to register SNs and thus get additional rewards. - the registration logic is adapted to work with the more blocks needed under HF16 SN rewards before submitting registrations.
2020-03-11 06:23:44 +01:00
std::vector<std::pair<uint8_t, uint64_t>> regtest_hard_forks;
for (uint8_t hf = cryptonote::network_version_7; hf < cryptonote::network_version_count; hf++)
regtest_hard_forks.emplace_back(hf, regtest_hard_forks.size() + 1);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
const cryptonote::test_options regtest_test_options = {
Fix blink test coin generation The --regtest/fakechain mode that blink uses no longer generates enough mining rewards to submit blink registrations under the LRC-6 change (with HF16 mining rewards ta 0). This commit changes two things: - fakechain now goes hf7-8-9-...-16 instead of jumping immediately from hf7 to hf16 at height 2; this ensures that there are *some* mining rewards (before hf16) that can be used to register SNs and thus get additional rewards. - the registration logic is adapted to work with the more blocks needed under HF16 SN rewards before submitting registrations.
2020-03-11 06:23:44 +01:00
std::move(regtest_hard_forks),
ArticMine's new block weight algorithm This curbs runaway growth while still allowing substantial spikes in block weight Original specification from ArticMine: here is the scaling proposal Define: LongTermBlockWeight Before fork: LongTermBlockWeight = BlockWeight At or after fork: LongTermBlockWeight = min(BlockWeight, 1.4*LongTermEffectiveMedianBlockWeight) Note: To avoid possible consensus issues over rounding the LongTermBlockWeight for a given block should be calculated to the nearest byte, and stored as a integer in the block itself. The stored LongTermBlockWeight is then used for future calculations of the LongTermEffectiveMedianBlockWeight and not recalculated each time. Define: LongTermEffectiveMedianBlockWeight LongTermEffectiveMedianBlockWeight = max(300000, MedianOverPrevious100000Blocks(LongTermBlockWeight)) Change Definition of EffectiveMedianBlockWeight From (current definition) EffectiveMedianBlockWeight = max(300000, MedianOverPrevious100Blocks(BlockWeight)) To (proposed definition) EffectiveMedianBlockWeight = min(max(300000, MedianOverPrevious100Blocks(BlockWeight)), 50*LongTermEffectiveMedianBlockWeight) Notes: 1) There are no other changes to the existing penalty formula, median calculation, fees etc. 2) There is the requirement to store the LongTermBlockWeight of a block unencrypted in the block itself. This is to avoid possible consensus issues over rounding and also to prevent the calculations from becoming unwieldy as we move away from the fork. 3) When the EffectiveMedianBlockWeight cap is reached it is still possible to mine blocks up to 2x the EffectiveMedianBlockWeight by paying the corresponding penalty.
2019-01-21 18:18:50 +01:00
0
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
};
Merge commit '025187e' into LokiMergeUpstream20180821
2018-09-19 06:05:07 +02:00
Service Node Checkpoint Storage (#579) * Add functions for storing checkpoints to the DB * Allocate the DB entry on the stack instead of heap * Add virtual overrides for new checkpoint functions * Clean up for pull request, simplify some logic * Revise API to include height in checkpoint header * Move log to top of function even if early exit * Begin moving checkpoints to db * Allow storing of checkpoints to DB * Cleanup for code reviewer, fix unit tests * Fix tests, fix casting issues * Don't use DUPSORT, use height->checkpoint mapping in DB * Remove if 0 disabling checkpoint vote, we already check HF12 * Fix unit test infinite loop * Update db schemas to match blk_checkpoint_header * Code review
2019-05-01 07:57:41 +02:00
// Service Nodes
{
(Low priority) Add disabled-by-default quorum storage support (#702) * Add disabled-by-default quorum storage support This adds support for storing N blocks of recent expired quorum state history in lokid (and dumping to/from the lmdb). This isn't useful for regular nodes (and that's why we don't store it) but is incredibly useful for the block explorer to be able to report *which* node got deregistered/decommissioned/etc. by a given state_change tx. * Fix quorum states copy
2019-07-17 08:01:54 +02:00
m_service_node_list.set_quorum_history_storage(command_line::get_arg(vm, arg_store_quorum_history));
Incorporate checkpoint validation into syncing step Fix checkpoint verification check and add messages
2019-08-28 07:54:02 +02:00
// NOTE: Implicit dependency. Service node list needs to be hooked before checkpoints.
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
m_blockchain_storage.hook_blockchain_detached(m_service_node_list);
m_blockchain_storage.hook_init(m_service_node_list);
m_blockchain_storage.hook_validate_miner_tx(m_service_node_list);
Add alt block added hook Move service node list methods to state_t methods Add querying state from alt blocks and put key image parsing into function Incorporate hash into state_t to find alt states Add a way to query alternate testing quorums Rebase cleanup
2019-07-18 09:42:15 +02:00
m_blockchain_storage.hook_alt_block_added(m_service_node_list);
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
// NOTE: There is an implicit dependency on service node lists being hooked first!
m_blockchain_storage.hook_init(m_quorum_cop);
m_blockchain_storage.hook_block_added(m_quorum_cop);
m_blockchain_storage.hook_blockchain_detached(m_quorum_cop);
Service Node Checkpoint Storage (#579) * Add functions for storing checkpoints to the DB * Allocate the DB entry on the stack instead of heap * Add virtual overrides for new checkpoint functions * Clean up for pull request, simplify some logic * Revise API to include height in checkpoint header * Move log to top of function even if early exit * Begin moving checkpoints to db * Allow storing of checkpoints to DB * Cleanup for code reviewer, fix unit tests * Fix tests, fix casting issues * Don't use DUPSORT, use height->checkpoint mapping in DB * Remove if 0 disabling checkpoint vote, we already check HF12 * Fix unit test infinite loop * Update db schemas to match blk_checkpoint_header * Code review
2019-05-01 07:57:41 +02:00
}
// Checkpoints
{
auto data_dir = boost::filesystem::path(m_config_folder);
boost::filesystem::path json(JSON_HASH_FILE_NAME);
boost::filesystem::path checkpoint_json_hashfile_fullpath = data_dir / json;
m_checkpoints_path = checkpoint_json_hashfile_fullpath.string();
}
LNS: Respect read-only flag
2020-05-15 03:11:59 +02:00
sqlite3 *lns_db = lns::init_loki_name_system(lns_db_file_path.c_str(), db->is_read_only());
Add support for Loki Name Service on the backend
2019-11-01 01:58:48 +01:00
if (!lns_db) return false;
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
Initialize lokimq before blockchain The block-added hook requires lokimq, so initializing it late was causing a segfault when adding the genesis block.
2020-05-11 21:39:20 +02:00
init_lokimq(vm);
Add support for Loki Name Service on the backend
2019-11-01 01:58:48 +01:00
const difficulty_type fixed_difficulty = command_line::get_arg(vm, arg_fixed_difficulty);
r = m_blockchain_storage.init(db.release(), lns_db, m_nettype, m_offline, regtest ? &regtest_test_options : test_options, fixed_difficulty, get_checkpoints);
Check blockchain init failure before r is overwritten (#937)
2019-11-28 02:11:46 +01:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize blockchain storage");
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
Fix --recalculate-difficulty to be default-enablable This code was only entered when the argument was non-defaulted, but then accepted `--recalculate-difficulty=0` and rewrote it to recalculate from height 1 in the lmdb. This changes the argument handling so that `--recalculate-difficulty=0` now disables the recalculation, and the default value of 1 now does the recalculation.
2020-05-05 19:30:50 +02:00
uint64_t recalc_diff_from_block = command_line::get_arg(vm, arg_recalculate_difficulty);
if (recalc_diff_from_block > 0)
Add flag to recalculate difficulty per block (#598) * Beging adding functions to recalculate difficulty * Add command line args to utility for recalculating difficulty * Exception safety for recalculating difficulty * Update help text for recalc difficulty * Add recalc flag on the daemon * Make context be const, signify intent for var++ to 1
2019-05-12 01:40:57 +02:00
{
cryptonote::BlockchainDB::fixup_context context = {};
context.type = cryptonote::BlockchainDB::fixup_type::calculate_difficulty;
context.calculate_difficulty_params.start_height = recalc_diff_from_block;
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
m_blockchain_storage.get_db().fixup(context);
Add flag to recalculate difficulty per block (#598) * Beging adding functions to recalculate difficulty * Add command line args to utility for recalculating difficulty * Exception safety for recalculating difficulty * Update help text for recalc difficulty * Add recalc flag on the daemon * Make context be const, signify intent for var++ to 1
2019-05-12 01:40:57 +02:00
}
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
r = m_mempool.init(max_txpool_weight);
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize memory pool");
Fix V1/V2 use of hard fork related parameters Some of it uses hardcoded height, which will need some thinking for next (voted upon) fork.
2016-01-29 16:09:17 +01:00
// now that we have a valid m_blockchain_storage, we can clean out any
// transactions in the pool that do not conform to the current fork
m_mempool.validate(m_blockchain_storage.get_current_hard_fork_version());
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
bool show_time_stats = command_line::get_arg(vm, arg_show_time_stats) != 0;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
m_blockchain_storage.set_show_time_stats(show_time_stats);
moved all stuff to github
2014-03-03 23:07:58 +01:00
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
block_sync_size = command_line::get_arg(vm, arg_block_sync_size);
RandomX integration Support RandomX PoW algorithm
2019-04-23 21:32:27 +02:00
if (block_sync_size > BLOCKS_SYNCHRONIZING_MAX_COUNT)
Updated RandomX integration from upstream Support RandomX PoW algorithm
2019-04-23 21:32:27 +02:00
MERROR("Error --block-sync-size cannot be greater than " << BLOCKS_SYNCHRONIZING_MAX_COUNT);
core: make the sync chunk block count overridable
2016-09-24 18:00:19 +02:00
core: add a message when loading checkpoints at init time This uses DNS, which can take a while, so it's useful to know this is the culprit when loading pauses
2017-07-31 07:43:13 +02:00
MGINFO("Loading checkpoints");
Rollback chain when receiving conflicting checkpoints (#726)
2019-07-11 06:16:55 +02:00
CHECK_AND_ASSERT_MES(update_checkpoints_from_json_file(), false, "One or more checkpoints loaded from json conflicted with existing checkpoints.");
Fixed segfault with checkpoints loading Bounds checking on blockchain_storage' m_blocks.size() when validating against checkpoints. Also moved initial json & DNS checkpoints load to after blockchain init.
2014-09-27 23:46:12 +02:00
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
// DNS versions checking
if (check_updates_string == "disabled")
check_updates_level = UPDATES_DISABLED;
else if (check_updates_string == "notify")
check_updates_level = UPDATES_NOTIFY;
else if (check_updates_string == "download")
check_updates_level = UPDATES_DOWNLOAD;
else if (check_updates_string == "update")
check_updates_level = UPDATES_UPDATE;
else {
MERROR("Invalid argument to --dns-versions-check: " << check_updates_string);
return false;
}
Stagenet
2018-02-16 12:04:04 +01:00
r = m_miner.init(vm, m_nettype);
Merge pull request #749 bfd4a28 Update BlockchainDB documentation (Thomas Winget) 797357e Change Doxyfile, Blockchain not blockchain_storage (Thomas Winget) c835215 remove defunct code from cryptonote::core (Thomas Winget) 50dba6d cryptonote::core doxygen documentation (Thomas Winget) 8ac329d doxygen documentation for difficulty functions (Thomas Winget) 540a76c Move checkpoint functions into checkpoints class (Thomas Winget) 1b0c98e doxygen documentation for checkpoints.{h,cpp} (Thomas Winget) 89c24ac Remove unnecessary or defunct code (Thomas Winget) ab0ed14 doxygen include private and static members (Thomas Winget) 3a48449 Updated documentation for blockchain.* (Thomas Winget)
2016-03-25 07:22:06 +01:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize miner instance");
moved all stuff to github
2014-03-03 23:07:58 +01:00
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
if (!keep_alt_blocks && !m_blockchain_storage.get_db().is_read_only())
m_blockchain_storage.get_db().drop_alt_blocks();
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
if (prune_blockchain)
{
// display a message if the blockchain is not pruned yet
core: fix --prune-blockchain not pruning if no blockchain exists
2019-06-17 18:26:09 +02:00
if (!m_blockchain_storage.get_blockchain_pruning_seed())
core: update pruning if using --prune-blockchain on a pruned blockchain Avoids a massive amount of spurious warnings if the last update before the daemon exited was a while ago and the daemon was syncing
2019-05-28 11:09:30 +02:00
{
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
MGINFO("Pruning blockchain...");
core: update pruning if using --prune-blockchain on a pruned blockchain Avoids a massive amount of spurious warnings if the last update before the daemon exited was a while ago and the daemon was syncing
2019-05-28 11:09:30 +02:00
CHECK_AND_ASSERT_MES(m_blockchain_storage.prune_blockchain(), false, "Failed to prune blockchain");
}
else
{
CHECK_AND_ASSERT_MES(m_blockchain_storage.update_blockchain_pruning(), false, "Failed to update blockchain pruning");
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
}
Remove useless method. Five years into the future this pointless method is still not used. Time to go.
2019-09-17 04:57:15 +02:00
return true;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
/// Loads a key pair from disk, if it exists, otherwise generates a new key pair and saves it to
/// disk.
///
/// get_pubkey - a function taking (privkey &, pubkey &) that sets the pubkey from the privkey;
/// returns true for success/false for failure
/// generate_pair - a void function taking (privkey &, pubkey &) that sets them to the generated values
template <typename Privkey, typename Pubkey, typename GetPubkey, typename GeneratePair>
bool init_key(const std::string &keypath, Privkey &privkey, Pubkey &pubkey, GetPubkey get_pubkey, GeneratePair generate_pair) {
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
if (epee::file_io_utils::is_file_exist(keypath))
{
std::string keystr;
bool r = epee::file_io_utils::load_file_to_string(keypath, keystr);
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
memcpy(&unwrap(unwrap(privkey)), keystr.data(), sizeof(privkey));
memwipe(&keystr[0], keystr.size());
CHECK_AND_ASSERT_MES(r, false, "failed to load service node key from " + keypath);
CHECK_AND_ASSERT_MES(keystr.size() == sizeof(privkey), false,
"service node key file " + keypath + " has an invalid size");
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
r = get_pubkey(privkey, pubkey);
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
CHECK_AND_ASSERT_MES(r, false, "failed to generate pubkey from secret key");
}
else
{
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
generate_pair(privkey, pubkey);
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
std::string keystr(reinterpret_cast<const char *>(&privkey), sizeof(privkey));
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
bool r = epee::file_io_utils::save_string_to_file(keypath, keystr);
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
memwipe(&keystr[0], keystr.size());
CHECK_AND_ASSERT_MES(r, false, "failed to save service node key to " + keypath);
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
using namespace boost::filesystem;
permissions(keypath, owner_read);
}
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
return true;
}
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
//-----------------------------------------------------------------------------------------------
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
bool core::init_service_keys()
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
{
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
auto& keys = m_service_keys;
Don't generate SN pubkey when not a SN We still generate the ed/x keys, but don't need the Monero key for anything.
2020-05-11 22:00:58 +02:00
// Primary SN pubkey (curve25519-based cryptography, but not Ed25519 because it doesn't do the
// EdDSA part of Ed25519, and thus can't be used with tools that do proper Ed25519
// cryptography). We only use this for SN registrations and proof signatures but the various
// external interfaces (lokinet, storage server, lokimq) use a secondary, standard Ed25519 key.
if (m_service_node) {
if (!init_key(m_config_folder + "/key", keys.key, keys.pub,
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
crypto::secret_key_to_public_key,
[](crypto::secret_key &key, crypto::public_key &pubkey) {
cryptonote::keypair keypair = keypair::generate(hw::get_device("default"));
key = keypair.sec;
pubkey = keypair.pub;
})
Don't generate SN pubkey when not a SN We still generate the ed/x keys, but don't need the Monero key for anything.
2020-05-11 22:00:58 +02:00
)
return false;
} else {
keys.key = crypto::null_skey;
keys.pub = crypto::null_pkey;
}
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
static_assert(
sizeof(crypto::ed25519_public_key) == crypto_sign_ed25519_PUBLICKEYBYTES &&
sizeof(crypto::ed25519_secret_key) == crypto_sign_ed25519_SECRETKEYBYTES &&
sizeof(crypto::ed25519_signature) == crypto_sign_BYTES &&
sizeof(crypto::x25519_public_key) == crypto_scalarmult_curve25519_BYTES &&
sizeof(crypto::x25519_secret_key) == crypto_scalarmult_curve25519_BYTES,
"Invalid ed25519/x25519 sizes");
// Secondary standard ed25519 key, usable in tools wanting standard ed25519 keys
//
// TODO(loki) - eventually it would be nice to make this become the only key pair that gets used
// for new registrations instead of the above. We'd still need to keep the above for
// compatibility with existing stakes registered before the relevant fork height, but we could
// then avoid needing to include this secondary key in uptime proofs for new SN registrations.
if (!init_key(m_config_folder + "/key_ed25519", keys.key_ed25519, keys.pub_ed25519,
[](crypto::ed25519_secret_key &sk, crypto::ed25519_public_key &pk) { crypto_sign_ed25519_sk_to_pk(pk.data, sk.data); return true; },
[](crypto::ed25519_secret_key &sk, crypto::ed25519_public_key &pk) { crypto_sign_ed25519_keypair(pk.data, sk.data); })
)
return false;
// Standard x25519 keys generated from the ed25519 keypair, used for encrypted communication between SNs
int rc = crypto_sign_ed25519_pk_to_curve25519(keys.pub_x25519.data, keys.pub_ed25519.data);
CHECK_AND_ASSERT_MES(rc == 0, false, "failed to convert ed25519 pubkey to x25519");
crypto_sign_ed25519_sk_to_curve25519(keys.key_x25519.data, keys.key_ed25519.data);
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (m_service_node) {
MGINFO_YELLOW("Service node public keys:");
C++17 Switch loki dev branch to C++17 compilation, and update the code with various C++17 niceties. - stop including the (deprecated) lokimq/string_view.h header and instead switch everything to use std::string_view and `""sv` instead of `""_sv`. - std::string_view is much nicer than epee::span, so updated various loki-specific code to use it instead. - made epee "portable storage" serialization accept a std::string_view instead of const lvalue std::string so that we can avoid copying. - switched from mapbox::variant to std::variant - use `auto [a, b] = whatever()` instead of `T1 a; T2 b; std::tie(a, b) = whatever()` in a couple places (in the wallet code). - switch to std::lock(...) instead of boost::lock(...) for simultaneous lock acquisition. boost::lock() won't compile in C++17 mode when given locks of different types. - removed various pre-C++17 workarounds, e.g. for fold expressions, unused argument attributes, and byte-spannable object detection. - class template deduction means lock types no longer have to specify the mutex, so `std::unique_lock<std::mutex> lock{mutex}` can become `std::unique_lock lock{mutex}`. This will make switching any mutex types (e.g. from boost to std mutexes) far easier as you just have to update the type in the header and everything should work. This also makes the tools::unique_lock and tools::shared_lock methods redundant (which were a sort of poor-mans-pre-C++17 way to eliminate the redundancy) so they are now gone and replaced with direct unique_lock or shared_lock constructions. - Redid the LNS validation using a string_view; instead of using raw char pointers the code now uses a string view and chops off parts of the view as it validates. So, for instance, it starts with "abcd.loki", validates the ".loki" and chops the view to "abcd", then validates the first character and chops to "bcd", validates the last and chops to "bc", then can just check everything remaining for is-valid-middle-char. - LNS validation gained a couple minor validation checks in the process: - slightly tightened the requirement on lokinet addresses to require that the last character of the mapped address is 'y' or 'o' (the last base32z char holds only one significant bit). - In parse_owner_to_generic_owner made sure that the owner value has the correct size (otherwise we could up end not filling or overfilling the pubkey buffer). - Replaced base32z/base64/hex conversions with lokimq's versions which have a nicer interface, are better optimized, and don't depend on epee.
2020-05-13 20:12:49 +02:00
MGINFO_YELLOW("- primary: " << lokimq::to_hex(tools::view_guts(keys.pub)));
MGINFO_YELLOW("- ed25519: " << lokimq::to_hex(tools::view_guts(keys.pub_ed25519)));
// .snode address is the ed25519 pubkey, encoded with base32z and with .snode appended:
MGINFO_YELLOW("- lokinet: " << lokimq::to_base32z(tools::view_guts(keys.pub_ed25519)) << ".snode");
MGINFO_YELLOW("- x25519: " << lokimq::to_hex(tools::view_guts(keys.pub_x25519)));
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
} else {
Don't generate SN pubkey when not a SN We still generate the ed/x keys, but don't need the Monero key for anything.
2020-05-11 22:00:58 +02:00
// Only print the x25519 version because it's the only thing useful for a non-SN (for
// encrypted LMQ RPC connections).
C++17 Switch loki dev branch to C++17 compilation, and update the code with various C++17 niceties. - stop including the (deprecated) lokimq/string_view.h header and instead switch everything to use std::string_view and `""sv` instead of `""_sv`. - std::string_view is much nicer than epee::span, so updated various loki-specific code to use it instead. - made epee "portable storage" serialization accept a std::string_view instead of const lvalue std::string so that we can avoid copying. - switched from mapbox::variant to std::variant - use `auto [a, b] = whatever()` instead of `T1 a; T2 b; std::tie(a, b) = whatever()` in a couple places (in the wallet code). - switch to std::lock(...) instead of boost::lock(...) for simultaneous lock acquisition. boost::lock() won't compile in C++17 mode when given locks of different types. - removed various pre-C++17 workarounds, e.g. for fold expressions, unused argument attributes, and byte-spannable object detection. - class template deduction means lock types no longer have to specify the mutex, so `std::unique_lock<std::mutex> lock{mutex}` can become `std::unique_lock lock{mutex}`. This will make switching any mutex types (e.g. from boost to std mutexes) far easier as you just have to update the type in the header and everything should work. This also makes the tools::unique_lock and tools::shared_lock methods redundant (which were a sort of poor-mans-pre-C++17 way to eliminate the redundancy) so they are now gone and replaced with direct unique_lock or shared_lock constructions. - Redid the LNS validation using a string_view; instead of using raw char pointers the code now uses a string view and chops off parts of the view as it validates. So, for instance, it starts with "abcd.loki", validates the ".loki" and chops the view to "abcd", then validates the first character and chops to "bcd", validates the last and chops to "bc", then can just check everything remaining for is-valid-middle-char. - LNS validation gained a couple minor validation checks in the process: - slightly tightened the requirement on lokinet addresses to require that the last character of the mapped address is 'y' or 'o' (the last base32z char holds only one significant bit). - In parse_owner_to_generic_owner made sure that the owner value has the correct size (otherwise we could up end not filling or overfilling the pubkey buffer). - Replaced base32z/base64/hex conversions with lokimq's versions which have a nicer interface, are better optimized, and don't depend on epee.
2020-05-13 20:12:49 +02:00
MGINFO_YELLOW("x25519 public key: " << lokimq::to_hex(tools::view_guts(keys.pub_x25519)));
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
}
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
return true;
}
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
static constexpr el::Level easylogging_level(lokimq::LogLevel level) {
using namespace lokimq;
switch (level) {
case LogLevel::fatal: return el::Level::Fatal;
case LogLevel::error: return el::Level::Error;
case LogLevel::warn: return el::Level::Warning;
case LogLevel::info: return el::Level::Info;
case LogLevel::debug: return el::Level::Debug;
case LogLevel::trace: return el::Level::Trace;
default: return el::Level::Unknown;
}
}
lokimq::AuthLevel core::lmq_check_access(const crypto::x25519_public_key& pubkey) const {
auto it = m_lmq_auth.find(pubkey);
if (it != m_lmq_auth.end())
return it->second;
return lokimq::AuthLevel::denied;
}
// Builds an allow function; takes `*this`, the default auth level, and whether this connection
// should allow incoming SN connections.
//
// default_auth should be AuthLevel::denied if only pre-approved connections may connect,
// AuthLevel::basic for public RPC, AuthLevel::admin for a (presumably localhost) unrestricted
// port, and AuthLevel::none for a super restricted mode (generally this is useful when there are
// also SN-restrictions on commands, i.e. for quorumnet).
//
// check_sn is whether we check an incoming key against known service nodes (and thus return
// "true" for the service node access if it checks out).
//
C++17 Switch loki dev branch to C++17 compilation, and update the code with various C++17 niceties. - stop including the (deprecated) lokimq/string_view.h header and instead switch everything to use std::string_view and `""sv` instead of `""_sv`. - std::string_view is much nicer than epee::span, so updated various loki-specific code to use it instead. - made epee "portable storage" serialization accept a std::string_view instead of const lvalue std::string so that we can avoid copying. - switched from mapbox::variant to std::variant - use `auto [a, b] = whatever()` instead of `T1 a; T2 b; std::tie(a, b) = whatever()` in a couple places (in the wallet code). - switch to std::lock(...) instead of boost::lock(...) for simultaneous lock acquisition. boost::lock() won't compile in C++17 mode when given locks of different types. - removed various pre-C++17 workarounds, e.g. for fold expressions, unused argument attributes, and byte-spannable object detection. - class template deduction means lock types no longer have to specify the mutex, so `std::unique_lock<std::mutex> lock{mutex}` can become `std::unique_lock lock{mutex}`. This will make switching any mutex types (e.g. from boost to std mutexes) far easier as you just have to update the type in the header and everything should work. This also makes the tools::unique_lock and tools::shared_lock methods redundant (which were a sort of poor-mans-pre-C++17 way to eliminate the redundancy) so they are now gone and replaced with direct unique_lock or shared_lock constructions. - Redid the LNS validation using a string_view; instead of using raw char pointers the code now uses a string view and chops off parts of the view as it validates. So, for instance, it starts with "abcd.loki", validates the ".loki" and chops the view to "abcd", then validates the first character and chops to "bcd", validates the last and chops to "bc", then can just check everything remaining for is-valid-middle-char. - LNS validation gained a couple minor validation checks in the process: - slightly tightened the requirement on lokinet addresses to require that the last character of the mapped address is 'y' or 'o' (the last base32z char holds only one significant bit). - In parse_owner_to_generic_owner made sure that the owner value has the correct size (otherwise we could up end not filling or overfilling the pubkey buffer). - Replaced base32z/base64/hex conversions with lokimq's versions which have a nicer interface, are better optimized, and don't depend on epee.
2020-05-13 20:12:49 +02:00
lokimq::AuthLevel core::lmq_allow(std::string_view ip, std::string_view x25519_pubkey_str, lokimq::AuthLevel default_auth) {
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
using namespace lokimq;
AuthLevel auth = default_auth;
if (x25519_pubkey_str.size() == sizeof(crypto::x25519_public_key)) {
crypto::x25519_public_key x25519_pubkey;
std::memcpy(x25519_pubkey.data, x25519_pubkey_str.data(), x25519_pubkey_str.size());
auto user_auth = lmq_check_access(x25519_pubkey);
if (user_auth >= AuthLevel::basic) {
if (user_auth > auth)
auth = user_auth;
MCINFO("lmq", "Incoming " << auth << "-authenticated connection");
}
MCINFO("lmq", "Incoming [" << auth << "] curve connection from " << ip << "/" << x25519_pubkey);
}
else {
MCINFO("lmq", "Incoming [" << auth << "] plain connection from " << ip);
}
return auth;
}
void core::init_lokimq(const boost::program_options::variables_map& vm) {
using namespace lokimq;
MGINFO("Starting lokimq");
m_lmq = std::make_unique<LokiMQ>(
tools::copy_guts(m_service_keys.pub_x25519),
tools::copy_guts(m_service_keys.key_x25519),
m_service_node,
C++17 Switch loki dev branch to C++17 compilation, and update the code with various C++17 niceties. - stop including the (deprecated) lokimq/string_view.h header and instead switch everything to use std::string_view and `""sv` instead of `""_sv`. - std::string_view is much nicer than epee::span, so updated various loki-specific code to use it instead. - made epee "portable storage" serialization accept a std::string_view instead of const lvalue std::string so that we can avoid copying. - switched from mapbox::variant to std::variant - use `auto [a, b] = whatever()` instead of `T1 a; T2 b; std::tie(a, b) = whatever()` in a couple places (in the wallet code). - switch to std::lock(...) instead of boost::lock(...) for simultaneous lock acquisition. boost::lock() won't compile in C++17 mode when given locks of different types. - removed various pre-C++17 workarounds, e.g. for fold expressions, unused argument attributes, and byte-spannable object detection. - class template deduction means lock types no longer have to specify the mutex, so `std::unique_lock<std::mutex> lock{mutex}` can become `std::unique_lock lock{mutex}`. This will make switching any mutex types (e.g. from boost to std mutexes) far easier as you just have to update the type in the header and everything should work. This also makes the tools::unique_lock and tools::shared_lock methods redundant (which were a sort of poor-mans-pre-C++17 way to eliminate the redundancy) so they are now gone and replaced with direct unique_lock or shared_lock constructions. - Redid the LNS validation using a string_view; instead of using raw char pointers the code now uses a string view and chops off parts of the view as it validates. So, for instance, it starts with "abcd.loki", validates the ".loki" and chops the view to "abcd", then validates the first character and chops to "bcd", validates the last and chops to "bc", then can just check everything remaining for is-valid-middle-char. - LNS validation gained a couple minor validation checks in the process: - slightly tightened the requirement on lokinet addresses to require that the last character of the mapped address is 'y' or 'o' (the last base32z char holds only one significant bit). - In parse_owner_to_generic_owner made sure that the owner value has the correct size (otherwise we could up end not filling or overfilling the pubkey buffer). - Replaced base32z/base64/hex conversions with lokimq's versions which have a nicer interface, are better optimized, and don't depend on epee.
2020-05-13 20:12:49 +02:00
[this](std::string_view x25519_pk) { return m_service_node_list.remote_lookup(x25519_pk); },
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
[](LogLevel level, const char *file, int line, std::string msg) {
// What a lovely interface (<-- sarcasm)
if (ELPP->vRegistry()->allowed(easylogging_level(level), "lmq"))
el::base::Writer(easylogging_level(level), file, line, ELPP_FUNC, el::base::DispatchAction::NormalLog).construct("lmq") << msg;
},
lokimq::LogLevel::trace
);
// ping.ping: a simple debugging target for pinging the lmq listener
m_lmq->add_category("ping", Access{AuthLevel::none})
.add_request_command("ping", [](Message& m) {
MCINFO("lmq", "Received ping from " << m.conn);
m.send_reply("pong");
})
;
if (m_service_node)
{
// Service nodes always listen for quorumnet data on the p2p IP, quorumnet port
std::string listen_ip = vm["p2p-bind-ip"].as<std::string>();
if (listen_ip.empty())
listen_ip = "0.0.0.0";
std::string qnet_listen = "tcp://" + listen_ip + ":" + std::to_string(m_quorumnet_port);
MGINFO("- listening on " << qnet_listen << " (quorumnet)");
m_lmq->listen_curve(qnet_listen,
C++17 Switch loki dev branch to C++17 compilation, and update the code with various C++17 niceties. - stop including the (deprecated) lokimq/string_view.h header and instead switch everything to use std::string_view and `""sv` instead of `""_sv`. - std::string_view is much nicer than epee::span, so updated various loki-specific code to use it instead. - made epee "portable storage" serialization accept a std::string_view instead of const lvalue std::string so that we can avoid copying. - switched from mapbox::variant to std::variant - use `auto [a, b] = whatever()` instead of `T1 a; T2 b; std::tie(a, b) = whatever()` in a couple places (in the wallet code). - switch to std::lock(...) instead of boost::lock(...) for simultaneous lock acquisition. boost::lock() won't compile in C++17 mode when given locks of different types. - removed various pre-C++17 workarounds, e.g. for fold expressions, unused argument attributes, and byte-spannable object detection. - class template deduction means lock types no longer have to specify the mutex, so `std::unique_lock<std::mutex> lock{mutex}` can become `std::unique_lock lock{mutex}`. This will make switching any mutex types (e.g. from boost to std mutexes) far easier as you just have to update the type in the header and everything should work. This also makes the tools::unique_lock and tools::shared_lock methods redundant (which were a sort of poor-mans-pre-C++17 way to eliminate the redundancy) so they are now gone and replaced with direct unique_lock or shared_lock constructions. - Redid the LNS validation using a string_view; instead of using raw char pointers the code now uses a string view and chops off parts of the view as it validates. So, for instance, it starts with "abcd.loki", validates the ".loki" and chops the view to "abcd", then validates the first character and chops to "bcd", validates the last and chops to "bc", then can just check everything remaining for is-valid-middle-char. - LNS validation gained a couple minor validation checks in the process: - slightly tightened the requirement on lokinet addresses to require that the last character of the mapped address is 'y' or 'o' (the last base32z char holds only one significant bit). - In parse_owner_to_generic_owner made sure that the owner value has the correct size (otherwise we could up end not filling or overfilling the pubkey buffer). - Replaced base32z/base64/hex conversions with lokimq's versions which have a nicer interface, are better optimized, and don't depend on epee.
2020-05-13 20:12:49 +02:00
[this, public_=command_line::get_arg(vm, arg_lmq_quorumnet_public)](std::string_view ip, std::string_view pk, bool) {
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
return lmq_allow(ip, pk, public_ ? AuthLevel::basic : AuthLevel::none);
});
m_quorumnet_state = quorumnet_new(*this);
}
}
Update LMQ SN list before starting This needs to be set before start() gets called to be able to authenticate connections we establish (otherwise we'd have nothing until the next block arrives).
2020-05-11 23:37:24 +02:00
void core::start_lokimq() {
update_lmq_sns(); // Ensure we have SNs set for the current block before starting
m_lmq->start();
}
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
//-----------------------------------------------------------------------------------------------
moved all stuff to github
2014-03-03 23:07:58 +01:00
bool core::set_genesis_block(const block& b)
{
return m_blockchain_storage.reset_and_set_genesis_block(b);
}
//-----------------------------------------------------------------------------------------------
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
void core::deinit()
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
#ifdef ENABLE_SYSTEMD
sd_notify(0, "STOPPING=1\nSTATUS=Shutting down");
#endif
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (m_quorumnet_state)
quorumnet_delete(m_quorumnet_state);
m_lmq.reset();
Long polling QOL, handle errors gracefully avoid lock contention Handle errors better when long polling is disabled instead of endlessly spamming logs. Avoid lock contention when set_daemon is called. Instead of immediately affecting the long polling thread (which could be engaging the mutex until RPC timeout, meaning the program stalls for that duration), update the address on the next iteration of the long polling thread. Wallets handle daemons that disable long polling better by sleeping.
2020-01-10 03:05:18 +01:00
m_long_poll_wake_up_clients.notify_all();
Fix destruction of service node list locking destroyed mutex
2019-01-18 06:16:23 +01:00
m_service_node_list.store();
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
m_miner.stop();
m_mempool.deinit();
Removed all code related to fast_exit
2016-10-03 03:06:55 +02:00
m_blockchain_storage.deinit();
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 20:59:39 +01:00
//-----------------------------------------------------------------------------------------------
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
void core::test_drop_download()
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 20:59:39 +01:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
m_test_drop_download = false;
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 20:59:39 +01:00
}
//-----------------------------------------------------------------------------------------------
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
void core::test_drop_download_height(uint64_t height)
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 20:59:39 +01:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
m_test_drop_download_height = height;
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::get_test_drop_download() const
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
return m_test_drop_download;
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::get_test_drop_download_height() const
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
if (m_test_drop_download_height == 0)
return true;
if (get_blockchain_storage().get_current_blockchain_height() <= m_test_drop_download_height)
return true;
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
return false;
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 20:59:39 +01:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
//-----------------------------------------------------------------------------------------------
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
void core::parse_incoming_tx_pre(tx_verification_batch_info &tx_info)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if(tx_info.blob->size() > get_max_tx_size())
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
LOG_PRINT_L1("WRONG TRANSACTION BLOB, too big size " << tx_info.blob->size() << ", rejected");
tx_info.tvc.m_verifivation_failed = true;
tx_info.tvc.m_too_big = true;
return;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
tx_info.parsed = parse_tx_from_blob(tx_info.tx, tx_info.tx_hash, *tx_info.blob);
if(!tx_info.parsed)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
moved non-critical warnings and errors to log level 1
2014-09-09 11:32:00 +02:00
LOG_PRINT_L1("WRONG TRANSACTION BLOB, Failed to parse, rejected");
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
tx_info.tvc.m_verifivation_failed = true;
return;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//std::cout << "!"<< tx.vin.size() << std::endl;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
std::lock_guard<boost::mutex> lock(bad_semantics_txes_lock);
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
for (int idx = 0; idx < 2; ++idx)
core: cache tx hashes of failing semantics txes
2017-01-22 17:13:31 +01:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (bad_semantics_txes[idx].find(tx_info.tx_hash) != bad_semantics_txes[idx].end())
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
{
LOG_PRINT_L1("Transaction already seen with bad semantics, rejected");
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
tx_info.tvc.m_verifivation_failed = true;
return;
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
}
core: cache tx hashes of failing semantics txes
2017-01-22 17:13:31 +01:00
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
tx_info.result = true;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
//-----------------------------------------------------------------------------------------------
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
void core::set_semantics_failed(const crypto::hash &tx_hash)
{
LOG_PRINT_L1("WRONG TRANSACTION BLOB, Failed to check tx " << tx_hash << " semantic, rejected");
bad_semantics_txes_lock.lock();
bad_semantics_txes[0].insert(tx_hash);
if (bad_semantics_txes[0].size() >= BAD_SEMANTICS_TXES_MAX_SIZE)
{
std::swap(bad_semantics_txes[0], bad_semantics_txes[1]);
bad_semantics_txes[0].clear();
}
bad_semantics_txes_lock.unlock();
}
//-----------------------------------------------------------------------------------------------
require canonical multi output bulletproof layout
2018-03-31 11:49:53 +02:00
static bool is_canonical_bulletproof_layout(const std::vector<rct::Bulletproof> &proofs)
{
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
if (proofs.size() != 1)
return false;
const size_t sz = proofs[0].V.size();
if (sz == 0 || sz > BULLETPROOF_MAX_OUTPUTS)
return false;
require canonical multi output bulletproof layout
2018-03-31 11:49:53 +02:00
return true;
}
//-----------------------------------------------------------------------------------------------
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
void core::parse_incoming_tx_accumulated_batch(std::vector<tx_verification_batch_info> &tx_info, bool kept_by_block)
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (kept_by_block && get_blockchain_storage().is_within_compiled_block_hash_area())
blockchain: skip checking tx semantics in embedded block hash range If the txes are bad, this'll be picked up by the block hash mismatch since the tx merkle root is part of the block hash.
2017-07-30 10:45:55 +02:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
MTRACE("Skipping semantics check for txs kept by block in embedded hash area");
return;
blockchain: skip checking tx semantics in embedded block hash range If the txes are bad, this'll be picked up by the block hash mismatch since the tx merkle root is part of the block hash.
2017-07-30 10:45:55 +02:00
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
std::vector<const rct::rctSig*> rvv;
for (size_t n = 0; n < tx_info.size(); ++n)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (!tx_info[n].result || tx_info[n].already_have)
continue;
if (!check_tx_semantic(tx_info[n].tx, kept_by_block))
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
{
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
continue;
}
Build/core test fixes, incoporate staking changes from LNS - Fix assert to use version_t::_count in service_node_list.cpp - Incoporate staking changes from LNS which revamp construct tx to derive more of the parameters from hf_version and type. This removes extraneous parameters that can be derived elsewhere. Also delegate setting loki_construct_tx_params into wallet2 atleast, so that callers into the wallet logic, like rpc server, simplewallet don't need bend backwards to get the HF version whereas the wallet has dedicated functions for determining the HF.
2019-11-29 04:36:43 +01:00
if (!tx_info[n].tx.is_transfer())
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
continue;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
const rct::rctSig &rv = tx_info[n].tx.rct_signatures;
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
switch (rv.type) {
case rct::RCTTypeNull:
// coinbase should not come here, so we reject for all other types
MERROR_VER("Unexpected Null rctSig type");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
case rct::RCTTypeSimple:
if (!rct::verRctSemanticsSimple(rv))
{
MERROR_VER("rct signature semantics check failed");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
break;
case rct::RCTTypeFull:
if (!rct::verRct(rv, true))
{
MERROR_VER("rct signature semantics check failed");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
break;
case rct::RCTTypeBulletproof:
add a bulletproof version, new bulletproof type, and rct config This makes it easier to modify the bulletproof format
2019-01-06 14:47:16 +01:00
case rct::RCTTypeBulletproof2:
require canonical multi output bulletproof layout
2018-03-31 11:49:53 +02:00
if (!is_canonical_bulletproof_layout(rv.p.bulletproofs))
{
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
MERROR_VER("Bulletproof does not have canonical form");
require canonical multi output bulletproof layout
2018-03-31 11:49:53 +02:00
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
rvv.push_back(&rv); // delayed batch verification
break;
default:
MERROR_VER("Unknown rct type: " << rv.type);
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
}
if (!rvv.empty() && !rct::verRctSemanticsSimple(rvv))
{
LOG_PRINT_L1("One transaction among this group has bad semantics, verifying one at a time");
const bool assumed_bad = rvv.size() == 1; // if there's only one tx, it must be the bad one
for (size_t n = 0; n < tx_info.size(); ++n)
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (!tx_info[n].result || tx_info[n].already_have)
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
continue;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (tx_info[n].tx.rct_signatures.type != rct::RCTTypeBulletproof && tx_info[n].tx.rct_signatures.type != rct::RCTTypeBulletproof2)
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
continue;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (assumed_bad || !rct::verRctSemanticsSimple(tx_info[n].tx.rct_signatures))
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
{
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
}
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
//-----------------------------------------------------------------------------------------------
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
std::vector<core::tx_verification_batch_info> core::parse_incoming_txs(const std::vector<blobdata>& tx_blobs, const tx_pool_options &opts)
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
{
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
// Caller needs to do this around both this *and* handle_parsed_txs
//auto lock = incoming_tx_lock();
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
std::vector<tx_verification_batch_info> tx_info(tx_blobs.size());
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
core: remove threadpool dependency from header
2018-04-23 19:55:58 +02:00
tools::threadpool& tpool = tools::threadpool::getInstance();
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
tools::threadpool::waiter waiter;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
for (size_t i = 0; i < tx_blobs.size(); i++) {
tx_info[i].blob = &tx_blobs[i];
tpool.submit(&waiter, [this, &info = tx_info[i]] {
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
try
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
parse_incoming_tx_pre(info);
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
}
catch (const std::exception &e)
{
MERROR_VER("Exception in handle_incoming_tx_pre: " << e.what());
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
info.tvc.m_verifivation_failed = true;
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
}
});
}
threadpool: allow leaf functions to run concurrently Decrease the number of worker threads by one to account for the fact the calling thread acts as a worker thread now
2018-04-26 12:44:47 +02:00
waiter.wait(&tpool);
Remove check_tx_syntax unused since 2015
2019-10-23 06:13:55 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
for (auto &info : tx_info) {
if (!info.result)
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
continue;
Remove check_tx_syntax unused since 2015
2019-10-23 06:13:55 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if(m_mempool.have_tx(info.tx_hash))
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
LOG_PRINT_L2("tx " << info.tx_hash << " already have transaction in tx_pool");
info.already_have = true;
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
else if(m_blockchain_storage.have_tx(info.tx_hash))
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
LOG_PRINT_L2("tx " << info.tx_hash << " already have transaction in blockchain");
info.already_have = true;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
}
Remove check_tx_syntax unused since 2015
2019-10-23 06:13:55 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
parse_incoming_tx_accumulated_batch(tx_info, opts.kept_by_block);
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
return tx_info;
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
bool core::handle_parsed_txs(std::vector<tx_verification_batch_info> &parsed_txs, const tx_pool_options &opts,
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
uint64_t *blink_rollback_height)
{
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
// Caller needs to do this around both this *and* parse_incoming_txs
//auto lock = incoming_tx_lock();
Add wallet long polling to wallet-cli
2019-12-18 02:49:56 +01:00
uint8_t version = m_blockchain_storage.get_current_hard_fork_version();
bool ok = true;
bool tx_pool_changed = false;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (blink_rollback_height)
*blink_rollback_height = 0;
tx_pool_options tx_opts;
for (size_t i = 0; i < parsed_txs.size(); i++) {
auto &info = parsed_txs[i];
if (!info.result)
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
ok = false; // Propagate failures (so this can be chained with parse_incoming_txs without an intermediate check)
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
continue;
}
Generic burn fee checking + blink burn fee checking This adds the ability for check_fee() to also check the burn amount. This requires passing extra info through `add_tx()` (and the various things that call it), so I took the: bool keeped_by_block, bool relayed, bool do_not_relay argument triplet, moved it into a struct in tx_pool.h, then added the other fee options there (along with some static factory functions for generating the typical sets of option). The majority of this commit is chasing that change through the codebase and test suite. This is used by blink but should also help LNS and other future burn transactions to verify a burn amount simply when adding the transation to the mempool. It supports a fixed burn amount, a burn amount as a multiple of the minimum tx fee, and also allows you to increase the minimum tx fee (so that, for example, we could require blink txes to pay miners 250% of the usual minimum (unimportant) priority tx fee. - Removed a useless core::add_new_tx() overload that wasn't used anywhere. Blink-specific changes: (I'd normally separate these into a separate commit, but they got interwoven fairly heavily with the above change). - changed the way blink burning is specified so that we have three knobs for fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee). The fixed amount is currently 0, base fee is 400%, and require miner tx fee is simply 100% (i.e. no different than a normal transaction). This is the same as before this commit, but is changing how they are being specified in cryptonote_config.h. - blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked before signing a blink tx. (These fee checks don't apply to anyone else -- when propagating over the network only the miner tx fee is checked). - Added a couple of checks for blink quorums: 1) make sure they have reached the blink hf; 2) make sure the submitted tx version conforms to the current hf min/max tx version. - print blink fee information in simplewallet's `fee` output - add "typical" fee calculations in the `fee` output: [wallet T6SCwL (has locked stakes)]: fee Current fee is 0.000000850 loki per byte + 0.020000000 loki per output No backlog at priority 1 No backlog at priority 2 No backlog at priority 3 No backlog at priority 4 Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink) where "small" here is the same tx size (2500 bytes + 2 outputs) used to estimate backlogs.
2019-11-09 04:14:15 +01:00
if (opts.kept_by_block)
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
get_blockchain_storage().on_new_tx_from_block(info.tx);
if (info.already_have)
continue; // Not a failure
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
const size_t weight = get_transaction_weight(info.tx, info.blob->size());
const tx_pool_options *local_opts = &opts;
if (blink_rollback_height && info.approved_blink)
Fix TXBuilder default hf value causing regression in core_tests
2018-11-21 06:30:16 +01:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
// If this is an approved blink then pass a copy of the options with the flag added
tx_opts = opts;
tx_opts.approved_blink = true;
local_opts = &tx_opts;
Fix TXBuilder default hf value causing regression in core_tests
2018-11-21 06:30:16 +01:00
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (m_mempool.add_tx(info.tx, info.tx_hash, *info.blob, weight, info.tvc, *local_opts, version, blink_rollback_height))
Add wallet long polling to wallet-cli
2019-12-18 02:49:56 +01:00
{
tx_pool_changed |= info.tvc.m_added_to_pool;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
MDEBUG("tx added: " << info.tx_hash);
Add wallet long polling to wallet-cli
2019-12-18 02:49:56 +01:00
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
else
Fix TXBuilder default hf value causing regression in core_tests
2018-11-21 06:30:16 +01:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
ok = false;
if (info.tvc.m_verifivation_failed)
MERROR_VER("Transaction verification failed: " << info.tx_hash);
else if (info.tvc.m_verifivation_impossible)
MERROR_VER("Transaction verification impossible: " << info.tx_hash);
Fix TXBuilder default hf value causing regression in core_tests
2018-11-21 06:30:16 +01:00
}
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
Rate limit long polling connections
2019-12-18 04:39:21 +01:00
if (tx_pool_changed) m_long_poll_wake_up_clients.notify_all();
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
return ok;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
//-----------------------------------------------------------------------------------------------
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
std::vector<core::tx_verification_batch_info> core::handle_incoming_txs(const std::vector<blobdata>& tx_blobs, const tx_pool_options &opts)
{
auto lock = incoming_tx_lock();
auto parsed = parse_incoming_txs(tx_blobs, opts);
handle_parsed_txs(parsed, opts);
return parsed;
}
//-----------------------------------------------------------------------------------------------
Generic burn fee checking + blink burn fee checking This adds the ability for check_fee() to also check the burn amount. This requires passing extra info through `add_tx()` (and the various things that call it), so I took the: bool keeped_by_block, bool relayed, bool do_not_relay argument triplet, moved it into a struct in tx_pool.h, then added the other fee options there (along with some static factory functions for generating the typical sets of option). The majority of this commit is chasing that change through the codebase and test suite. This is used by blink but should also help LNS and other future burn transactions to verify a burn amount simply when adding the transation to the mempool. It supports a fixed burn amount, a burn amount as a multiple of the minimum tx fee, and also allows you to increase the minimum tx fee (so that, for example, we could require blink txes to pay miners 250% of the usual minimum (unimportant) priority tx fee. - Removed a useless core::add_new_tx() overload that wasn't used anywhere. Blink-specific changes: (I'd normally separate these into a separate commit, but they got interwoven fairly heavily with the above change). - changed the way blink burning is specified so that we have three knobs for fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee). The fixed amount is currently 0, base fee is 400%, and require miner tx fee is simply 100% (i.e. no different than a normal transaction). This is the same as before this commit, but is changing how they are being specified in cryptonote_config.h. - blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked before signing a blink tx. (These fee checks don't apply to anyone else -- when propagating over the network only the miner tx fee is checked). - Added a couple of checks for blink quorums: 1) make sure they have reached the blink hf; 2) make sure the submitted tx version conforms to the current hf min/max tx version. - print blink fee information in simplewallet's `fee` output - add "typical" fee calculations in the `fee` output: [wallet T6SCwL (has locked stakes)]: fee Current fee is 0.000000850 loki per byte + 0.020000000 loki per output No backlog at priority 1 No backlog at priority 2 No backlog at priority 3 No backlog at priority 4 Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink) where "small" here is the same tx size (2500 bytes + 2 outputs) used to estimate backlogs.
2019-11-09 04:14:15 +01:00
bool core::handle_incoming_tx(const blobdata& tx_blob, tx_verification_context& tvc, const tx_pool_options &opts)
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
{
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
const std::vector<cryptonote::blobdata> tx_blobs{{tx_blob}};
auto parsed = handle_incoming_txs(tx_blobs, opts);
parsed[0].blob = &tx_blob; // Update pointer to the input rather than the copy in case the caller wants to use it for some reason
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
tvc = parsed[0].tvc;
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
return parsed[0].result && (parsed[0].already_have || tvc.m_added_to_pool);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
//-----------------------------------------------------------------------------------------------
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
std::pair<std::vector<std::shared_ptr<blink_tx>>, std::unordered_set<crypto::hash>>
core::parse_incoming_blinks(const std::vector<serializable_blink_metadata> &blinks)
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
std::pair<std::vector<std::shared_ptr<blink_tx>>, std::unordered_set<crypto::hash>> results;
auto &new_blinks = results.first;
auto &missing_txs = results.second;
Add blink HF version checks
2019-11-18 23:38:16 +01:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (m_blockchain_storage.get_current_hard_fork_version() < HF_VERSION_BLINK)
return results;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
std::vector<uint8_t> want(blinks.size(), false); // Really bools, but std::vector<bool> is broken.
size_t want_count = 0;
// Step 1: figure out which referenced transactions we want to keep:
// - unknown tx (typically an incoming blink)
// - in mempool without blink sigs (it's possible to get the tx before the blink signatures)
// - in a recent, still-mutable block with blink sigs (can happen when syncing blocks before
// retrieving blink signatures)
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
{
std::vector<crypto::hash> hashes;
hashes.reserve(blinks.size());
for (auto &bm : blinks)
hashes.emplace_back(bm.tx_hash);
Review fixes - set the proper promise - reuse reference rather than repeating lookups - fix unneeded mempool lock
2019-11-26 16:53:16 +01:00
std::unique_lock<Blockchain> lock(m_blockchain_storage);
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
auto tx_block_heights = m_blockchain_storage.get_transactions_heights(hashes);
auto immutable_height = m_blockchain_storage.get_immutable_height();
auto &db = m_blockchain_storage.get_db();
for (size_t i = 0; i < blinks.size(); i++) {
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (tx_block_heights[i] == 0 /*mempool or unknown*/ || tx_block_heights[i] > immutable_height /*mined but not yet immutable*/)
{
want[i] = true;
want_count++;
}
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
}
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
MDEBUG("Want " << want_count << " of " << blinks.size() << " incoming blink signature sets after filtering out immutable txes");
if (!want_count) return results;
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
// Step 2: filter out any transactions for which we already have a blink signature
{
auto mempool_lock = m_mempool.blink_shared_lock();
for (size_t i = 0; i < blinks.size(); i++)
Fix TXBuilder default hf value causing regression in core_tests
2018-11-21 06:30:16 +01:00
{
Fix rare blink pool deadlock cryptonote_protocol_handler calls `pool.get_blink(hash)` while already holding a blink shared lock, which should have been `pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own lock. That double lock is undefined behaviour and can cause a deadlock on the mutex, although it appears rare that it actually does. If it does, however, this eventually backs up into vote relaying during the idle loop, which then stalls the idle loop so we stop sending out uptime proofs (since that is also in the idle loop). A simple fix here is to add the `true` argument, but on reconsideration this extra argument to take or not take a lock is messy and error prone, so this commit instead removes the second argument entirely and instead documents which call must and must not hold a lock, getting rid of the three methods (get_blink, has_blink, and add_existing_blink) that had the `have_lock` argument. This ends up having only a small impact on calling code - the vast majority of callers already hold a lock, and the few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
if (want[i] && m_mempool.has_blink(blinks[i].tx_hash))
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
{
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
MDEBUG("Ignoring blink data for " << blinks[i].tx_hash << ": already have blink signatures");
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
want[i] = false; // Already have it, move along
want_count--;
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
}
}
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
MDEBUG("Want " << want_count << " of " << blinks.size() << " incoming blink signature sets after filtering out existing blink sigs");
if (!want_count) return results;
// Step 3: create new blink_tx objects for txes and add the blink signatures. We can do all of
// this without a lock since these are (for now) just local instances.
new_blinks.reserve(want_count);
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
std::unordered_map<uint64_t, std::shared_ptr<const service_nodes::quorum>> quorum_cache;
for (size_t i = 0; i < blinks.size(); i++)
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (!want[i])
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
continue;
auto &bdata = blinks[i];
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
new_blinks.push_back(std::make_shared<blink_tx>(bdata.height, bdata.tx_hash));
auto &blink = *new_blinks.back();
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
// Data structure checks (we have more stringent checks for validity later, but if these fail
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
// now then there's no point of even trying to do signature validation.
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
if (bdata.signature.size() != bdata.position.size() || // Each signature must have an associated quorum position
bdata.signature.size() != bdata.quorum.size() || // and quorum index
bdata.signature.size() < service_nodes::BLINK_MIN_VOTES * tools::enum_count<blink_tx::subquorum> || // too few signatures for possible validity
bdata.signature.size() > service_nodes::BLINK_SUBQUORUM_SIZE * tools::enum_count<blink_tx::subquorum> || // too many signatures
blink_tx::quorum_height(bdata.height, blink_tx::subquorum::base) == 0 || // Height is too early (no blink quorum height)
std::any_of(bdata.position.begin(), bdata.position.end(), [](const auto &p) { return p >= service_nodes::BLINK_SUBQUORUM_SIZE; }) || // invalid position
std::any_of(bdata.quorum.begin(), bdata.quorum.end(), [](const auto &qi) { return qi >= tools::enum_count<blink_tx::subquorum>; }) // invalid quorum index
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
) {
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
MINFO("Invalid blink tx " << bdata.tx_hash << ": invalid signature data");
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
continue;
Fix TXBuilder default hf value causing regression in core_tests
2018-11-21 06:30:16 +01:00
}
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
Fix crash when receiving blocks for unknown quorum This can occur when syncing if we get a blink tx before the blocks that let us determine the quorum. Just ignore it at this point; we'll pick it up at the next once-per-minute sync run.
2019-12-24 04:01:48 +01:00
bool no_quorum = false;
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
std::array<const std::vector<crypto::public_key> *, tools::enum_count<blink_tx::subquorum>> validators;
for (uint8_t qi = 0; qi < tools::enum_count<blink_tx::subquorum>; qi++)
Fix TXBuilder default hf value causing regression in core_tests
2018-11-21 06:30:16 +01:00
{
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
auto q_height = blink.quorum_height(static_cast<blink_tx::subquorum>(qi));
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
auto &q = quorum_cache[q_height];
if (!q)
q = get_quorum(service_nodes::quorum_type::blink, q_height);
Fix crash when receiving blocks for unknown quorum This can occur when syncing if we get a blink tx before the blocks that let us determine the quorum. Just ignore it at this point; we'll pick it up at the next once-per-minute sync run.
2019-12-24 04:01:48 +01:00
if (!q)
{
MINFO("Don't have a quorum for height " << q_height << " (yet?), ignoring this blink");
no_quorum = true;
break;
}
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
validators[qi] = &q->validators;
Fix TXBuilder default hf value causing regression in core_tests
2018-11-21 06:30:16 +01:00
}
Fix crash when receiving blocks for unknown quorum This can occur when syncing if we get a blink tx before the blocks that let us determine the quorum. Just ignore it at this point; we'll pick it up at the next once-per-minute sync run.
2019-12-24 04:01:48 +01:00
if (no_quorum)
continue;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
std::vector<std::pair<size_t, std::string>> failures;
for (size_t s = 0; s < bdata.signature.size(); s++)
{
try {
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
blink.add_signature(static_cast<blink_tx::subquorum>(bdata.quorum[s]), bdata.position[s], true /*approved*/, bdata.signature[s],
validators[bdata.quorum[s]]->at(bdata.position[s]));
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
} catch (const std::exception &e) {
failures.emplace_back(s, e.what());
}
}
if (blink.approved())
{
MINFO("Blink tx " << bdata.tx_hash << " blink signatures approved with " << failures.size() << " signature validation failures");
for (auto &f : failures)
MDEBUG("- failure for quorum " << int(bdata.quorum[f.first]) << ", position " << int(bdata.position[f.first]) << ": " << f.second);
}
else
{
std::ostringstream os;
os << "Blink validation failed:";
for (auto &f : failures)
os << " [" << int(bdata.quorum[f.first]) << ":" << int(bdata.position[f.first]) << "]: " << f.second;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
MINFO("Invalid blink tx " << bdata.tx_hash << ": " << os.str());
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
}
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
return results;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
int core::add_blinks(const std::vector<std::shared_ptr<blink_tx>> &blinks)
{
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
int added = 0;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (blinks.empty())
return added;
auto lock = m_mempool.blink_unique_lock();
for (auto &b : blinks)
if (b->approved())
Fix rare blink pool deadlock cryptonote_protocol_handler calls `pool.get_blink(hash)` while already holding a blink shared lock, which should have been `pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own lock. That double lock is undefined behaviour and can cause a deadlock on the mutex, although it appears rare that it actually does. If it does, however, this eventually backs up into vote relaying during the idle loop, which then stalls the idle loop so we stop sending out uptime proofs (since that is also in the idle loop). A simple fix here is to add the `true` argument, but on reconsideration this extra argument to take or not take a lock is messy and error prone, so this commit instead removes the second argument entirely and instead documents which call must and must not hold a lock, getting rid of the three methods (get_blink, has_blink, and add_existing_blink) that had the `have_lock` argument. This ends up having only a small impact on calling code - the vast majority of callers already hold a lock, and the few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
if (m_mempool.add_existing_blink(b))
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
added++;
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
MINFO("Added blink signatures for " << added << " blinks");
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
return added;
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
}
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
//-----------------------------------------------------------------------------------------------
Blink This is the bulk of the work for blink. There is two pieces yet to come which will follow shortly, which are: the p2p communication of blink transactions (which needs to be fully synchronized, not just shared, unlike regular mempool txes); and an implementation of fee burning. Blink approval, multi-quorum signing, cli wallet and node support for submission denial are all implemented here. This overhauls and fixes various parts of the SNNetwork interface to fix some issues (particularly around non-SN communication with SNs, which wasn't working). There are also a few sundry FIXME's and TODO's of other minor details that will follow shortly under cleanup/testing/etc.
2019-10-27 23:47:19 +01:00
std::future<std::pair<blink_result, std::string>> core::handle_blink_tx(const std::string &tx_blob)
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
{
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
return quorumnet_send_blink(*this, tx_blob);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::check_tx_semantic(const transaction& tx, bool keeped_by_block) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Build/core test fixes, incoporate staking changes from LNS - Fix assert to use version_t::_count in service_node_list.cpp - Incoporate staking changes from LNS which revamp construct tx to derive more of the parameters from hf_version and type. This removes extraneous parameters that can be derived elsewhere. Also delegate setting loki_construct_tx_params into wallet2 atleast, so that callers into the wallet logic, like rpc server, simplewallet don't need bend backwards to get the HF version whereas the wallet has dedicated functions for determining the HF.
2019-11-29 04:36:43 +01:00
if (tx.is_transfer())
Check that inputs/outputs size is 0 for deregister
2018-07-25 03:13:38 +02:00
{
Build/core test fixes, incoporate staking changes from LNS - Fix assert to use version_t::_count in service_node_list.cpp - Incoporate staking changes from LNS which revamp construct tx to derive more of the parameters from hf_version and type. This removes extraneous parameters that can be derived elsewhere. Also delegate setting loki_construct_tx_params into wallet2 atleast, so that callers into the wallet logic, like rpc server, simplewallet don't need bend backwards to get the HF version whereas the wallet has dedicated functions for determining the HF.
2019-11-29 04:36:43 +01:00
if (tx.vin.empty())
Check that inputs/outputs size is 0 for deregister
2018-07-25 03:13:38 +02:00
{
Build/core test fixes, incoporate staking changes from LNS - Fix assert to use version_t::_count in service_node_list.cpp - Incoporate staking changes from LNS which revamp construct tx to derive more of the parameters from hf_version and type. This removes extraneous parameters that can be derived elsewhere. Also delegate setting loki_construct_tx_params into wallet2 atleast, so that callers into the wallet logic, like rpc server, simplewallet don't need bend backwards to get the HF version whereas the wallet has dedicated functions for determining the HF.
2019-11-29 04:36:43 +01:00
MERROR_VER("tx with empty inputs, rejected for tx id= " << get_transaction_hash(tx));
Check that inputs/outputs size is 0 for deregister
2018-07-25 03:13:38 +02:00
return false;
}
}
Build/core test fixes, incoporate staking changes from LNS - Fix assert to use version_t::_count in service_node_list.cpp - Incoporate staking changes from LNS which revamp construct tx to derive more of the parameters from hf_version and type. This removes extraneous parameters that can be derived elsewhere. Also delegate setting loki_construct_tx_params into wallet2 atleast, so that callers into the wallet logic, like rpc server, simplewallet don't need bend backwards to get the HF version whereas the wallet has dedicated functions for determining the HF.
2019-11-29 04:36:43 +01:00
else
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Build/core test fixes, incoporate staking changes from LNS - Fix assert to use version_t::_count in service_node_list.cpp - Incoporate staking changes from LNS which revamp construct tx to derive more of the parameters from hf_version and type. This removes extraneous parameters that can be derived elsewhere. Also delegate setting loki_construct_tx_params into wallet2 atleast, so that callers into the wallet logic, like rpc server, simplewallet don't need bend backwards to get the HF version whereas the wallet has dedicated functions for determining the HF.
2019-11-29 04:36:43 +01:00
if (tx.vin.size() != 0)
{
MERROR_VER("tx type: " << tx.type << " must have 0 inputs, received: " << tx.vin.size() << ", rejected for tx id = " << get_transaction_hash(tx));
return false;
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
if(!check_inputs_types_supported(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("unsupported input types for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
if(!check_outs_valid(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx with invalid outputs, rejected for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
Make tx type and version scoped enums This converts the transaction type and version to scoped enum, giving type safety and making the tx type assignment less error prone because there is no implicit conversion or comparison with raw integers that has to be worried about. This ends up converting any use of `cryptonote::transaction::type_xyz` to `cryptonote::transaction::txtype::xyz`. For version, names like `transaction::version_v4` become `cryptonote::txversion::v4_tx_types`. This also allows/includes various other simplifications related to or enabled by this change: - handle `is_deregister` dynamically in serialization code (setting `type::standard` or `type::deregister` rather than using a version-determined union) - `get_type()` is no longer needed with the above change: it is now much simpler to directly access `type` which will always have the correct value (even for v2 or v3 transaction types). And though there was an assertion on the enum value, `get_type()` was being used only sporadically: many places accessed `.type` directly. - the old unscoped enum didn't have a type but was assumed castable to/from `uint16_t`, which technically meant there was potential undefined behaviour when deserializing any type values >= 8. - tx type range checks weren't being done in all serialization paths; they are now. Because `get_type()` was not used everywhere (lots of places simply accessed `.type` directory) these might not have been caught. - `set_type()` is not needed; it was only being used in a single place (wallet2.cpp) and only for v4 txes, so the version protection code was never doing anything. - added a std::ostream << operator for the enum types so that they can be output with `<< tx_type <<` rather than needing to wrap it in `type_to_string(tx_type)` everywhere. For the versions, you get the annotated version string (e.g. 4_tx_types) rather than just the number 4.
2019-06-11 20:53:46 +02:00
if (tx.version >= txversion::v2_ringct)
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
{
if (tx.rct_signatures.outPk.size() != tx.vout.size())
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx with mismatched vout/outPk count, rejected for tx id= " << get_transaction_hash(tx));
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
return false;
}
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
if(!check_money_overflow(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx has money overflow, rejected for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
Make tx type and version scoped enums This converts the transaction type and version to scoped enum, giving type safety and making the tx type assignment less error prone because there is no implicit conversion or comparison with raw integers that has to be worried about. This ends up converting any use of `cryptonote::transaction::type_xyz` to `cryptonote::transaction::txtype::xyz`. For version, names like `transaction::version_v4` become `cryptonote::txversion::v4_tx_types`. This also allows/includes various other simplifications related to or enabled by this change: - handle `is_deregister` dynamically in serialization code (setting `type::standard` or `type::deregister` rather than using a version-determined union) - `get_type()` is no longer needed with the above change: it is now much simpler to directly access `type` which will always have the correct value (even for v2 or v3 transaction types). And though there was an assertion on the enum value, `get_type()` was being used only sporadically: many places accessed `.type` directly. - the old unscoped enum didn't have a type but was assumed castable to/from `uint16_t`, which technically meant there was potential undefined behaviour when deserializing any type values >= 8. - tx type range checks weren't being done in all serialization paths; they are now. Because `get_type()` was not used everywhere (lots of places simply accessed `.type` directory) these might not have been caught. - `set_type()` is not needed; it was only being used in a single place (wallet2.cpp) and only for v4 txes, so the version protection code was never doing anything. - added a std::ostream << operator for the enum types so that they can be output with `<< tx_type <<` rather than needing to wrap it in `type_to_string(tx_type)` everywhere. For the versions, you get the annotated version string (e.g. 4_tx_types) rather than just the number 4.
2019-06-11 20:53:46 +02:00
if (tx.version == txversion::v1)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
uint64_t amount_in = 0;
get_inputs_money_amount(tx, amount_in);
uint64_t amount_out = get_outs_money_amount(tx);
if(amount_in <= amount_out)
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx with wrong amounts: ins " << amount_in << ", outs " << amount_out << ", rejected for tx id= " << get_transaction_hash(tx));
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
return false;
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
if(!keeped_by_block && get_transaction_weight(tx) >= m_blockchain_storage.get_current_cumulative_block_weight_limit() - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
MERROR_VER("tx is too large " << get_transaction_weight(tx) << ", expected not bigger than " << m_blockchain_storage.get_current_cumulative_block_weight_limit() - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE);
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
if(!check_tx_inputs_keyimages_diff(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MERROR_VER("tx uses a single key image more than once");
moved all stuff to github
2014-03-03 23:07:58 +01:00
return false;
}
core: forbid duplicate ring members from v6 This avoids someone adding what amounts to mixin 0 rings in practice, as there is no other good reason to allow this.
2017-07-18 23:38:26 +02:00
if (!check_tx_inputs_ring_members_diff(tx))
{
MERROR_VER("tx uses duplicate ring members");
return false;
}
core: test key images against validity domain
2017-02-20 18:41:39 +01:00
if (!check_tx_inputs_keyimages_domain(tx))
{
MERROR_VER("tx uses key image not in the valid domain");
return false;
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::is_key_image_spent(const crypto::key_image &key_image) const
Add a is_key_image_spent daemon command and RPC call
2015-08-11 11:49:15 +02:00
{
return m_blockchain_storage.have_tx_keyimg_as_spent(key_image);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::are_key_images_spent(const std::vector<crypto::key_image>& key_im, std::vector<bool> &spent) const
Add a is_key_image_spent daemon command and RPC call
2015-08-11 11:49:15 +02:00
{
spent.clear();
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 21:38:10 +01:00
for(auto& ki: key_im)
Add a is_key_image_spent daemon command and RPC call
2015-08-11 11:49:15 +02:00
{
spent.push_back(m_blockchain_storage.have_tx_keyimg_as_spent(ki));
}
return true;
}
//-----------------------------------------------------------------------------------------------
cryptonote_protocol: large block sync size before v4
2017-08-16 22:19:53 +02:00
size_t core::get_block_sync_size(uint64_t height) const
{
if (block_sync_size > 0)
return block_sync_size;
Removed unused/always-obsolete macros This removes a bunch of macros from cryptonote_config.h that either aren't used at all, or have never applied to Loki, along with removing/simplifying some of the dead code touching these macros. A few of these are still used only in the test suite, so I moved them there instead. One of these sounds sort of scary -- ALLOW_DEBUG_COMMANDS -- but this has *always* been forcibly enabled with no way to disable it going back to the very first monero git commit. DYNAMIC_FEE_PER_KB_BASE_FEE_V5 was defined in a very strange way that doesn't make a lot of sense (including using a constant that is not otherwise applied in loki), so I just replaced it with the expanded value.
2019-08-23 02:32:34 +02:00
return BLOCKS_SYNCHRONIZING_DEFAULT_COUNT;
cryptonote_protocol: large block sync size before v4
2017-08-16 22:19:53 +02:00
}
//-----------------------------------------------------------------------------------------------
json serialization for rpc-relevant monero types Structured {de-,}serialization methods for (many new) types which are used for requests or responses in the RPC. New types include RPC requests and responses, and structs which compose types within those. # Conflicts: # src/cryptonote_core/blockchain.cpp
2017-09-05 18:20:27 +02:00
bool core::are_key_images_spent_in_pool(const std::vector<crypto::key_image>& key_im, std::vector<bool> &spent) const
{
spent.clear();
return m_mempool.check_for_key_images(key_im, spent);
}
//-----------------------------------------------------------------------------------------------
Calculate total burnt loki Adds the total burnt loki as a return value to the get_coinbase_tx_sum as it will commonly be used in relation to the other variables.
2020-03-12 10:53:24 +01:00
std::tuple<uint64_t, uint64_t, uint64_t> core::get_coinbase_tx_sum(const uint64_t start_offset, const size_t count)
added print_coinbase_tx_sum option
2016-10-10 21:45:51 +02:00
{
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-11 01:55:18 +02:00
uint64_t emission_amount = 0;
uint64_t total_fee_amount = 0;
Calculate total burnt loki Adds the total burnt loki as a return value to the get_coinbase_tx_sum as it will commonly be used in relation to the other variables.
2020-03-12 10:53:24 +01:00
uint64_t burnt_loki = 0;
Don't copy blockchain for coinbase_tx_sum Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:29:51 +02:00
if (count)
added print_coinbase_tx_sum option
2016-10-10 21:45:51 +02:00
{
Don't copy blockchain for coinbase_tx_sum Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:29:51 +02:00
const uint64_t end = start_offset + count - 1;
m_blockchain_storage.for_blocks_range(start_offset, end,
Calculate total burnt loki Adds the total burnt loki as a return value to the get_coinbase_tx_sum as it will commonly be used in relation to the other variables.
2020-03-12 10:53:24 +01:00
[this, &emission_amount, &total_fee_amount, &burnt_loki](uint64_t, const crypto::hash& hash, const block& b){
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<transaction> txs;
std::vector<crypto::hash> missed_txs;
core: fix recalculation of emission/fees It was counting things many, many times
2016-12-21 12:18:33 +01:00
uint64_t coinbase_amount = get_outs_money_amount(b.miner_tx);
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-11 01:55:18 +02:00
this->get_transactions(b.tx_hashes, txs, missed_txs);
core: fix recalculation of emission/fees It was counting things many, many times
2016-12-21 12:18:33 +01:00
uint64_t tx_fee_amount = 0;
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 21:38:10 +01:00
for(const auto& tx: txs)
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-11 01:55:18 +02:00
{
Add fee burning This adds a tx extra field that specifies an amount of the tx fee that must be burned; the miner can claim only (txnFee - burnFee) when including the block. This will be used for the extra, burned part of blink fees and LNS fees and any other transaction that requires fee burning in the future.
2019-10-30 22:28:47 +01:00
tx_fee_amount += get_tx_miner_fee(tx, b.major_version >= HF_VERSION_FEE_BURNING);
Calculate total burnt loki Adds the total burnt loki as a return value to the get_coinbase_tx_sum as it will commonly be used in relation to the other variables.
2020-03-12 10:53:24 +01:00
if(b.major_version >= HF_VERSION_FEE_BURNING)
{
burnt_loki += get_burned_amount_from_tx_extra(tx.extra);
}
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-11 01:55:18 +02:00
}
emission_amount += coinbase_amount - tx_fee_amount;
total_fee_amount += tx_fee_amount;
Don't copy blockchain for coinbase_tx_sum Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:29:51 +02:00
return true;
});
added print_coinbase_tx_sum option
2016-10-10 21:45:51 +02:00
}
Calculate total burnt loki Adds the total burnt loki as a return value to the get_coinbase_tx_sum as it will commonly be used in relation to the other variables.
2020-03-12 10:53:24 +01:00
return std::tuple<uint64_t, uint64_t, uint64_t>(emission_amount, total_fee_amount, burnt_loki);
added print_coinbase_tx_sum option
2016-10-10 21:45:51 +02:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::check_tx_inputs_keyimages_diff(const transaction& tx) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
std::unordered_set<crypto::key_image> ki;
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 21:38:10 +01:00
for(const auto& in: tx.vin)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
CHECKED_GET_SPECIFIC_VARIANT(in, const txin_to_key, tokey_in, false);
if(!ki.insert(tokey_in.k_image).second)
return false;
}
return true;
}
//-----------------------------------------------------------------------------------------------
core: forbid duplicate ring members from v6 This avoids someone adding what amounts to mixin 0 rings in practice, as there is no other good reason to allow this.
2017-07-18 23:38:26 +02:00
bool core::check_tx_inputs_ring_members_diff(const transaction& tx) const
{
const uint8_t version = m_blockchain_storage.get_current_hard_fork_version();
if (version >= 6)
{
for(const auto& in: tx.vin)
{
CHECKED_GET_SPECIFIC_VARIANT(in, const txin_to_key, tokey_in, false);
for (size_t n = 1; n < tokey_in.key_offsets.size(); ++n)
if (tokey_in.key_offsets[n] == 0)
return false;
}
}
return true;
}
//-----------------------------------------------------------------------------------------------
core: test key images against validity domain
2017-02-20 18:41:39 +01:00
bool core::check_tx_inputs_keyimages_domain(const transaction& tx) const
{
std::unordered_set<crypto::key_image> ki;
for(const auto& in: tx.vin)
{
CHECKED_GET_SPECIFIC_VARIANT(in, const txin_to_key, tokey_in, false);
if (!(rct::scalarmultKey(rct::ki2rct(tokey_in.k_image), rct::curveOrder()) == rct::identity()))
return false;
}
return true;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
size_t core::get_blockchain_total_transactions() const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_total_transactions();
}
//-----------------------------------------------------------------------------------------------
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
bool core::relay_txpool_transactions()
{
// we attempt to relay txes that should be relayed, but were not
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<std::pair<crypto::hash, cryptonote::blobdata>> txs;
core: don't try to re-relay an empty set of pool transactions
2017-03-12 15:51:32 +01:00
if (m_mempool.get_relayable_transactions(txs) && !txs.empty())
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
{
Remove boost::value_initialized and BOOST_FOREACH (#921) Neither of these have a place in modern C++11; boost::value_initialized is entirely superseded by `Type var{};` which does value initialization (or default construction if a default constructor is defined). More problematically, each `boost::value_initialized<T>` requires instantiation of another wrapping templated type which is a pointless price to pay the compiler in C++11 or newer. Also removed is the AUTO_VAL_INIT macro (which is just a simple macro around constructing a boost::value_initialized<T>). BOOST_FOREACH is a similarly massive pile of code to implement C++11-style for-each loops. (And bizarrely it *doesn't* appear to fall back to C++ for-each loops even when under a C++11 compiler!) This removes both entirely from the codebase.
2019-10-31 23:26:58 +01:00
cryptonote_connection_context fake_context{};
tx_verification_context tvc{};
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
NOTIFY_NEW_TRANSACTIONS::request r{};
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
for (auto it = txs.begin(); it != txs.end(); ++it)
{
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
r.txs.push_back(it->second);
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
}
get_protocol()->relay_transactions(r, fake_context);
m_mempool.set_relayed(txs);
}
return true;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
quorum_cop: added quorum cop and service node keys to lokid Submit deregistration autonomously, prune uptime_proofs Move pruner to relayer, fix inf. loop, quorum participation bool return find_if, fix reorg logic, make 2hrs check a variable Add iterator assign on erase, adjust detached check qacop: review iteration, changed warnings and std::find call Accept proofs excluding the added time buffer
2018-07-12 03:58:17 +02:00
bool core::submit_uptime_proof()
{
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (!m_service_node)
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
return true;
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
NOTIFY_UPTIME_PROOF::request req = m_service_node_list.generate_uptime_proof(m_sn_public_ip, m_storage_port, m_storage_lmq_port, m_quorumnet_port);
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
Remove boost::value_initialized and BOOST_FOREACH (#921) Neither of these have a place in modern C++11; boost::value_initialized is entirely superseded by `Type var{};` which does value initialization (or default construction if a default constructor is defined). More problematically, each `boost::value_initialized<T>` requires instantiation of another wrapping templated type which is a pointless price to pay the compiler in C++11 or newer. Also removed is the AUTO_VAL_INIT macro (which is just a simple macro around constructing a boost::value_initialized<T>). BOOST_FOREACH is a similarly massive pile of code to implement C++11-style for-each loops. (And bizarrely it *doesn't* appear to fall back to C++ for-each loops even when under a C++11 compiler!) This removes both entirely from the codebase.
2019-10-31 23:26:58 +01:00
cryptonote_connection_context fake_context{};
Remove sychronization check on receipt of uptime If a peer views the destination peer as not synchronizing, then the destination peer should just accept the uptime proof, rather than accept it and then conditionally relay it depending on whether or not you are synchronizing at the point of attempting to relay (which you could transition into synchronizing state interim between accepting and attempting to relay the proof).
2019-09-18 07:36:00 +02:00
bool relayed = get_protocol()->relay_uptime_proof(req, fake_context);
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
if (relayed)
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
MGINFO("Submitted uptime-proof for Service Node (yours): " << m_service_keys.pub);
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
return true;
}
//-----------------------------------------------------------------------------------------------
Remove sychronization check on receipt of uptime If a peer views the destination peer as not synchronizing, then the destination peer should just accept the uptime proof, rather than accept it and then conditionally relay it depending on whether or not you are synchronizing at the point of attempting to relay (which you could transition into synchronizing state interim between accepting and attempting to relay the proof).
2019-09-18 07:36:00 +02:00
bool core::handle_uptime_proof(const NOTIFY_UPTIME_PROOF::request &proof, bool &my_uptime_proof_confirmation)
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
{
Relay proofs back to source, but don't relay your proof again Otherwise we get into a ping-pong situation as follows Node1 sends uptime -> Node2 receives uptime and relays it back to Node1 for acknowledgement -> Node1 receives it, handle_uptime_proof returns true to acknowledge -> Node1 tries to resend to the same peers again Instead, if we receive our own uptime proof, then acknowledge but don't send on. If the we are missing an uptime proof it will have been submitted automatically by the daemon itself instead of using my own proof relayed by other nodes.
2019-09-18 07:22:27 +02:00
return m_service_node_list.handle_uptime_proof(proof, my_uptime_proof_confirmation);
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
}
//-----------------------------------------------------------------------------------------------
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
crypto::hash core::on_transaction_relayed(const cryptonote::blobdata& tx_blob)
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<std::pair<crypto::hash, cryptonote::blobdata>> txs;
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
cryptonote::transaction tx;
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
crypto::hash tx_hash;
if (!parse_and_validate_tx_from_blob(tx_blob, tx, tx_hash))
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
{
Fix a few minor typos
2016-12-04 14:13:54 +01:00
LOG_ERROR("Failed to parse relayed transaction");
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
return crypto::null_hash;
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
}
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
txs.push_back(std::make_pair(tx_hash, std::move(tx_blob)));
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
m_mempool.set_relayed(txs);
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
return tx_hash;
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
}
//-----------------------------------------------------------------------------------------------
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
bool core::relay_service_node_votes()
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
{
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
auto height = get_current_blockchain_height();
Keep checkpoint votes over p2p Checkpoint votes were only going over quorumnet, which was quite broken as they need to go out universally: random nodes piece together the individual checkpoints to create checkpoints on their own, and so need to see all of the votes. With the current code only service nodes that participated in the specific quorum ever saw the checkpoint votes. This commit returns checkpoint vote distribution to distribution over p2p. We could, in theory, do checkpoint vote collection over quorumnet and then relay the set of votes as a pack to reduce p2p traffic a bit, but this wouldn't be a huge optimization since we'd still have to distribute all the votes over p2p at some point anyway, so leaving that as a future optimization for now. (Obligations votes, in contrast, don't need to be distributed at all -- if a vote results in a state change, the quorum members themselves can produce and distribute the state change tx).
2019-12-10 19:57:36 +01:00
auto hf_version = get_hard_fork_version(height);
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
Keep checkpoint votes over p2p Checkpoint votes were only going over quorumnet, which was quite broken as they need to go out universally: random nodes piece together the individual checkpoints to create checkpoints on their own, and so need to see all of the votes. With the current code only service nodes that participated in the specific quorum ever saw the checkpoint votes. This commit returns checkpoint vote distribution to distribution over p2p. We could, in theory, do checkpoint vote collection over quorumnet and then relay the set of votes as a pack to reduce p2p traffic a bit, but this wouldn't be a huge optimization since we'd still have to distribute all the votes over p2p at some point anyway, so leaving that as a future optimization for now. (Obligations votes, in contrast, don't need to be distributed at all -- if a vote results in a state change, the quorum members themselves can produce and distribute the state change tx).
2019-12-10 19:57:36 +01:00
auto quorum_votes = m_quorum_cop.get_relayable_votes(height, hf_version, true);
auto p2p_votes = m_quorum_cop.get_relayable_votes(height, hf_version, false);
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (!quorum_votes.empty() && m_quorumnet_state && m_service_node)
quorumnet_relay_obligation_votes(m_quorumnet_state, quorum_votes);
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
Keep checkpoint votes over p2p Checkpoint votes were only going over quorumnet, which was quite broken as they need to go out universally: random nodes piece together the individual checkpoints to create checkpoints on their own, and so need to see all of the votes. With the current code only service nodes that participated in the specific quorum ever saw the checkpoint votes. This commit returns checkpoint vote distribution to distribution over p2p. We could, in theory, do checkpoint vote collection over quorumnet and then relay the set of votes as a pack to reduce p2p traffic a bit, but this wouldn't be a huge optimization since we'd still have to distribute all the votes over p2p at some point anyway, so leaving that as a future optimization for now. (Obligations votes, in contrast, don't need to be distributed at all -- if a vote results in a state change, the quorum members themselves can produce and distribute the state change tx).
2019-12-10 19:57:36 +01:00
if (!p2p_votes.empty())
{
Remove pre-HF14 p2p vote relaying command
2020-01-18 17:12:29 +01:00
NOTIFY_NEW_SERVICE_NODE_VOTE::request req{};
req.votes = std::move(p2p_votes);
cryptonote_connection_context fake_context{};
get_protocol()->relay_service_node_votes(req, fake_context);
Merge dev into upstream
2019-05-01 08:01:17 +02:00
}
return true;
}
Keep checkpoint votes over p2p Checkpoint votes were only going over quorumnet, which was quite broken as they need to go out universally: random nodes piece together the individual checkpoints to create checkpoints on their own, and so need to see all of the votes. With the current code only service nodes that participated in the specific quorum ever saw the checkpoint votes. This commit returns checkpoint vote distribution to distribution over p2p. We could, in theory, do checkpoint vote collection over quorumnet and then relay the set of votes as a pack to reduce p2p traffic a bit, but this wouldn't be a huge optimization since we'd still have to distribute all the votes over p2p at some point anyway, so leaving that as a future optimization for now. (Obligations votes, in contrast, don't need to be distributed at all -- if a vote results in a state change, the quorum members themselves can produce and distribute the state change tx).
2019-12-10 19:57:36 +01:00
void core::set_service_node_votes_relayed(const std::vector<service_nodes::quorum_vote_t> &votes)
{
m_quorum_cop.set_votes_relayed(votes);
}
Merge dev into upstream
2019-05-01 08:01:17 +02:00
//-----------------------------------------------------------------------------------------------
Add expected total reward to RPC "getblocktemplate". Only works from V5 fork onward - returns 0 before that block.
2017-03-28 16:55:38 +02:00
bool core::get_block_template(block& b, const account_public_address& adr, difficulty_type& diffic, uint64_t& height, uint64_t& expected_reward, const blobdata& ex_nonce)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Add expected total reward to RPC "getblocktemplate". Only works from V5 fork onward - returns 0 before that block.
2017-03-28 16:55:38 +02:00
return m_blockchain_storage.create_block_template(b, adr, diffic, height, expected_reward, ex_nonce);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
functional_tests: add more blockchain related tests Related to emission, reorgs, getting tx data back, output distribution and histogram
2019-03-23 17:20:08 +01:00
bool core::get_block_template(block& b, const crypto::hash *prev_block, const account_public_address& adr, difficulty_type& diffic, uint64_t& height, uint64_t& expected_reward, const blobdata& ex_nonce)
{
return m_blockchain_storage.create_block_template(b, prev_block, adr, diffic, height, expected_reward, ex_nonce);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, NOTIFY_RESPONSE_CHAIN_ENTRY::request& resp) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.find_blockchain_supplement(qblock_ids, resp);
}
//-----------------------------------------------------------------------------------------------
rpc: rework to avoid repeated calculations in get_blocks.bin
2018-04-16 21:16:00 +02:00
bool core::find_blockchain_supplement(const uint64_t req_start_block, const std::list<crypto::hash>& qblock_ids, std::vector<std::pair<std::pair<cryptonote::blobdata, crypto::hash>, std::vector<std::pair<crypto::hash, cryptonote::blobdata> > > >& blocks, uint64_t& total_height, uint64_t& start_height, bool pruned, bool get_miner_tx_hash, size_t max_count) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
rpc: rework to avoid repeated calculations in get_blocks.bin
2018-04-16 21:16:00 +02:00
return m_blockchain_storage.find_blockchain_supplement(req_start_block, qblock_ids, blocks, total_height, start_height, pruned, get_miner_tx_hash, max_count);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
bool core::get_outs(const rpc::GET_OUTPUTS_BIN::request& req, rpc::GET_OUTPUTS_BIN::response& res) const
Fake outs set is now decided by the wallet This plugs a privacy leak from the wallet to the daemon, as the daemon could previously see what input is included as a transaction input, which the daemon hadn't previously supplied. Now, the wallet requests a particular set of outputs, including the real one. This can result in transactions that can't be accepted if the wallet happens to select too many outputs with non standard unlock times. The daemon could know this and select another output, but the wallet is blind to it. It's currently very unlikely since I don't think anything uses non default unlock times. The wallet requests more outputs than necessary so it can use spares if any of the returns outputs are still locked. If there are not enough spares to reach the desired mixin, the transaction will fail.
2016-08-02 22:48:09 +02:00
{
return m_blockchain_storage.get_outs(req, res);
}
//-----------------------------------------------------------------------------------------------
speed up get_output_distribution (and precalc common case)
2018-05-16 23:41:50 +02:00
bool core::get_output_distribution(uint64_t amount, uint64_t from_height, uint64_t to_height, uint64_t &start_height, std::vector<uint64_t> &distribution, uint64_t &base) const
add RPC to get a histogram of outputs of a given amount
2018-02-19 12:15:15 +01:00
{
speed up get_output_distribution (and precalc common case)
2018-05-16 23:41:50 +02:00
return m_blockchain_storage.get_output_distribution(amount, from_height, to_height, start_height, distribution, base);
add RPC to get a histogram of outputs of a given amount
2018-02-19 12:15:15 +01:00
}
//-----------------------------------------------------------------------------------------------
output_blacklist: Remove redundant return, report exception
2020-06-02 07:50:27 +02:00
void core::get_output_blacklist(std::vector<uint64_t> &blacklist) const
Infinite Staking Part 2 (#406) * Cleanup and undoing some protocol breakages * Simplify expiration of nodes * Request unlock schedules entire node for expiration * Fix off by one in expiring nodes * Undo expiring code for pre v10 nodes * Fix RPC returning register as unlock height and not checking 0 * Rename key image unlock height const * Undo testnet hardfork debug changes * Remove is_type for get_type, fix missing var rename * Move serialisable data into public namespace * Serialise tx types properly * Fix typo in no service node known msg * Code review * Fix == to >= on serialising tx type * Code review 2 * Fix tests and key image unlock * Add command to print locked key images * Update ui to display lock stakes, query in print cmd blacklist * Modify print stakes to be less slow * Remove autostaking code * Refactor staking into sweep functions It appears staking was derived off stake_main written separately at implementation at the beginning. This merges them back into a common code path, after removing autostake there's only some minor differences. It also makes sure that any changes to sweeping upstream are going to be considered in the staking process which we want. * Display unlock height for stakes * Begin creating output blacklist * Make blacklist output a migration step * Implement get_output_blacklist for lmdb * In wallet output selection ignore blacklisted outputs * Apply blacklisted outputs to output selection * Fix broken tests, switch key image unlock * Fix broken unit_tests * Begin change to limit locked key images to 4 globally * Revamp prepare registration for new min contribution rules * Fix up old back case in prepare registration * Remove debug code * Cleanup debug code and some unecessary changes * Fix migration step on mainnet db * Fix blacklist outputs for pre-existing DB's * Remove irrelevant note * Tweak scanning addresses for locked stakes Since we only now allow contributions from the primary address we can skip checking all subaddress + lookahead to speed up wallet scanning * Define macro for SCNu64 for Mingw * Fix failure on empty DB * Add missing error msg, remove contributor from stake * Improve staking messages * Flush prompt to always display * Return the msg from stake failure and fix stake parsing error * Tweak fork rules for smaller bulletproofs * Tweak pooled nodes minimum amounts * Fix crash on exit, there's no need to store on destructor Since all information about service nodes is derived from the blockchain and we store state every time we receive a block, storing in the destructor is redundant as there is no new information to store. * Make prompt be consistent with CLI * Check max number of key images from per user to node * Implement error message on get_output_blacklist failure * Remove resolved TODO's/comments * Handle infinite staking in print_sn * Atoi->strtol, fix prepare_registration, virtual override, stale msgs
2019-02-14 02:12:57 +01:00
{
output_blacklist: Remove redundant return, report exception
2020-06-02 07:50:27 +02:00
m_blockchain_storage.get_output_blacklist(blacklist);
Infinite Staking Part 2 (#406) * Cleanup and undoing some protocol breakages * Simplify expiration of nodes * Request unlock schedules entire node for expiration * Fix off by one in expiring nodes * Undo expiring code for pre v10 nodes * Fix RPC returning register as unlock height and not checking 0 * Rename key image unlock height const * Undo testnet hardfork debug changes * Remove is_type for get_type, fix missing var rename * Move serialisable data into public namespace * Serialise tx types properly * Fix typo in no service node known msg * Code review * Fix == to >= on serialising tx type * Code review 2 * Fix tests and key image unlock * Add command to print locked key images * Update ui to display lock stakes, query in print cmd blacklist * Modify print stakes to be less slow * Remove autostaking code * Refactor staking into sweep functions It appears staking was derived off stake_main written separately at implementation at the beginning. This merges them back into a common code path, after removing autostake there's only some minor differences. It also makes sure that any changes to sweeping upstream are going to be considered in the staking process which we want. * Display unlock height for stakes * Begin creating output blacklist * Make blacklist output a migration step * Implement get_output_blacklist for lmdb * In wallet output selection ignore blacklisted outputs * Apply blacklisted outputs to output selection * Fix broken tests, switch key image unlock * Fix broken unit_tests * Begin change to limit locked key images to 4 globally * Revamp prepare registration for new min contribution rules * Fix up old back case in prepare registration * Remove debug code * Cleanup debug code and some unecessary changes * Fix migration step on mainnet db * Fix blacklist outputs for pre-existing DB's * Remove irrelevant note * Tweak scanning addresses for locked stakes Since we only now allow contributions from the primary address we can skip checking all subaddress + lookahead to speed up wallet scanning * Define macro for SCNu64 for Mingw * Fix failure on empty DB * Add missing error msg, remove contributor from stake * Improve staking messages * Flush prompt to always display * Return the msg from stake failure and fix stake parsing error * Tweak fork rules for smaller bulletproofs * Tweak pooled nodes minimum amounts * Fix crash on exit, there's no need to store on destructor Since all information about service nodes is derived from the blockchain and we store state every time we receive a block, storing in the destructor is redundant as there is no new information to store. * Make prompt be consistent with CLI * Check max number of key images from per user to node * Implement error message on get_output_blacklist failure * Remove resolved TODO's/comments * Handle infinite staking in print_sn * Atoi->strtol, fix prepare_registration, virtual override, stale msgs
2019-02-14 02:12:57 +01:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_tx_outputs_gindexs(tx_id, indexs);
}
//-----------------------------------------------------------------------------------------------
blockchain_db: speedup tx output gathering We know all the data we'll want for getblocks.bin is contiguous
2018-12-16 14:28:49 +01:00
bool core::get_tx_outputs_gindexs(const crypto::hash& tx_id, size_t n_txes, std::vector<std::vector<uint64_t>>& indexs) const
{
return m_blockchain_storage.get_tx_outputs_gindexs(tx_id, n_txes, indexs);
}
//-----------------------------------------------------------------------------------------------
moved all stuff to github
2014-03-03 23:07:58 +01:00
void core::pause_mine()
{
m_miner.pause();
}
//-----------------------------------------------------------------------------------------------
void core::resume_mine()
{
m_miner.resume();
}
//-----------------------------------------------------------------------------------------------
core: call {prepare|cleanup}_handle_incoming_blocks when adding a mined block This ensures that a batch can't also be started/stopped out of sync by another thread and us getting in the middle
2017-03-19 11:25:12 +01:00
block_complete_entry get_block_complete_entry(block& b, tx_memory_pool &pool)
{
Merge commit '1ba9bafd3331f6534242e682bafbd6d863be42ed' into MergeUpstream3
2020-05-25 03:48:32 +02:00
block_complete_entry bce = {};
bce.block = cryptonote::block_to_blob(b);
core: call {prepare|cleanup}_handle_incoming_blocks when adding a mined block This ensures that a batch can't also be started/stopped out of sync by another thread and us getting in the middle
2017-03-19 11:25:12 +01:00
for (const auto &tx_hash: b.tx_hashes)
{
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
cryptonote::blobdata txblob;
CHECK_AND_ASSERT_THROW_MES(pool.get_transaction(tx_hash, txblob), "Transaction not found in pool");
bce.txs.push_back(txblob);
core: call {prepare|cleanup}_handle_incoming_blocks when adding a mined block This ensures that a batch can't also be started/stopped out of sync by another thread and us getting in the middle
2017-03-19 11:25:12 +01:00
}
return bce;
}
//-----------------------------------------------------------------------------------------------
functional_tests: add more blockchain related tests Related to emission, reorgs, getting tx data back, output distribution and histogram
2019-03-23 17:20:08 +01:00
bool core::handle_block_found(block& b, block_verification_context &bvc)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Remove boost::value_initialized and BOOST_FOREACH (#921) Neither of these have a place in modern C++11; boost::value_initialized is entirely superseded by `Type var{};` which does value initialization (or default construction if a default constructor is defined). More problematically, each `boost::value_initialized<T>` requires instantiation of another wrapping templated type which is a pointless price to pay the compiler in C++11 or newer. Also removed is the AUTO_VAL_INIT macro (which is just a simple macro around constructing a boost::value_initialized<T>). BOOST_FOREACH is a similarly massive pile of code to implement C++11-style for-each loops. (And bizarrely it *doesn't* appear to fall back to C++ for-each loops even when under a C++11 compiler!) This removes both entirely from the codebase.
2019-10-31 23:26:58 +01:00
bvc = {};
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<block_complete_entry> blocks;
Handle multi state changes in the pool (#781) * Update state transition check to account for height and universally set timestamp on recommission Reject invalidated state changes by their height after HF13 * Prune invalidated state changes on blockchain increment Simplify check_tx_inputs for state_changes by using service node list Instead of querying the last 60 historical blocks for every transaction, use the service node list and determine the state of the service node and if it can transition to its new state. We also now enforce at hardfork 13, that the network cannot commit transactions to the network if they would have been invalidated by a newer state change that already is already on the blockchain. This is backwards compatible all the way back to hardfork 9. Greatly simplify state change tx pruning on block added Use the new stricter rules for pruning state changes in the txpool We can do so because pruning the TX pool won't cause issues at the protocol level, but the more people we can upgrade the better network behaviour we get in terms of propagating more intrinsically correct ordering of state changes to other peers. * Don't generate state changes if not valid, disallow voting if node is non-votable
2019-08-12 01:48:47 +02:00
m_miner.pause();
cryptonote: avoid double parsing blocks when syncing
2018-11-23 14:05:48 +01:00
{
Handle multi state changes in the pool (#781) * Update state transition check to account for height and universally set timestamp on recommission Reject invalidated state changes by their height after HF13 * Prune invalidated state changes on blockchain increment Simplify check_tx_inputs for state_changes by using service node list Instead of querying the last 60 historical blocks for every transaction, use the service node list and determine the state of the service node and if it can transition to its new state. We also now enforce at hardfork 13, that the network cannot commit transactions to the network if they would have been invalidated by a newer state change that already is already on the blockchain. This is backwards compatible all the way back to hardfork 9. Greatly simplify state change tx pruning on block added Use the new stricter rules for pruning state changes in the txpool We can do so because pruning the TX pool won't cause issues at the protocol level, but the more people we can upgrade the better network behaviour we get in terms of propagating more intrinsically correct ordering of state changes to other peers. * Don't generate state changes if not valid, disallow voting if node is non-votable
2019-08-12 01:48:47 +02:00
LOKI_DEFER { m_miner.resume(); };
try
{
blocks.push_back(get_block_complete_entry(b, m_mempool));
}
catch (const std::exception &e)
{
return false;
}
std::vector<block> pblocks;
if (!prepare_handle_incoming_blocks(blocks, pblocks))
{
MERROR("Block found, but failed to prepare to add");
return false;
}
add_new_block(b, bvc, nullptr /*checkpoint*/);
cleanup_handle_incoming_blocks(true);
m_miner.on_block_chain_update();
cryptonote: avoid double parsing blocks when syncing
2018-11-23 14:05:48 +01:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
CHECK_AND_ASSERT_MES(!bvc.m_verifivation_failed, false, "mined block failed verification");
if(bvc.m_added_to_main_chain)
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<crypto::hash> missed_txs;
std::vector<cryptonote::blobdata> txs;
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
m_blockchain_storage.get_transactions_blobs(b.tx_hashes, txs, missed_txs);
moved all stuff to github
2014-03-03 23:07:58 +01:00
if(missed_txs.size() && m_blockchain_storage.get_block_id_by_height(get_block_height(b)) != get_block_hash(b))
{
moved non-critical warnings and errors to log level 1
2014-09-09 11:32:00 +02:00
LOG_PRINT_L1("Block found but, seems that reorganize just happened after that, do not relay this block");
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
}
Fix typos in various files
2018-03-01 12:36:19 +01:00
CHECK_AND_ASSERT_MES(txs.size() == b.tx_hashes.size() && !missed_txs.size(), false, "can't find some transactions in found block:" << get_block_hash(b) << " txs.size()=" << txs.size()
moved all stuff to github
2014-03-03 23:07:58 +01:00
<< ", b.tx_hashes.size()=" << b.tx_hashes.size() << ", missed_txs.size()" << missed_txs.size());
Remove boost::value_initialized and BOOST_FOREACH (#921) Neither of these have a place in modern C++11; boost::value_initialized is entirely superseded by `Type var{};` which does value initialization (or default construction if a default constructor is defined). More problematically, each `boost::value_initialized<T>` requires instantiation of another wrapping templated type which is a pointless price to pay the compiler in C++11 or newer. Also removed is the AUTO_VAL_INIT macro (which is just a simple macro around constructing a boost::value_initialized<T>). BOOST_FOREACH is a similarly massive pile of code to implement C++11-style for-each loops. (And bizarrely it *doesn't* appear to fall back to C++ for-each loops even when under a C++11 compiler!) This removes both entirely from the codebase.
2019-10-31 23:26:58 +01:00
cryptonote_connection_context exclude_context{};
NOTIFY_NEW_FLUFFY_BLOCK::request arg{};
Remove non-fluffy blockchain syncing path (#618) Obsolete since forever, fluffy blocks is by default- it's more space efficient when syncing the blockchain.
2019-06-04 04:28:56 +02:00
arg.current_blockchain_height = m_blockchain_storage.get_current_blockchain_height();
arg.b = blocks[0];
moved all stuff to github
2014-03-03 23:07:58 +01:00
m_pprotocol->relay_block(arg, exclude_context);
}
functional_tests: add more blockchain related tests Related to emission, reorgs, getting tx data back, output distribution and histogram
2019-03-23 17:20:08 +01:00
return true;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
void core::on_synchronized()
{
m_miner.on_synchronized();
}
//-----------------------------------------------------------------------------------------------
Cleanup test impact of adding safesyncmode() method
2017-08-22 15:54:18 +02:00
void core::safesyncmode(const bool onoff)
{
m_blockchain_storage.safesyncmode(onoff);
}
//-----------------------------------------------------------------------------------------------
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
bool core::add_new_block(const block& b, block_verification_context& bvc, checkpoint_t const *checkpoint)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
bool result = m_blockchain_storage.add_new_block(b, bvc, checkpoint);
if (result)
{
// TODO(loki): PERF(loki): This causes perf problems in integration mode, so in real-time operation it may not be
// noticeable but could bubble up and cause slowness if the runtime variables align up undesiredly.
relay_service_node_votes(); // NOTE: nop if synchronising due to not accepting votes whilst syncing
}
return result;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
//-----------------------------------------------------------------------------------------------
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
bool core::prepare_handle_incoming_blocks(const std::vector<block_complete_entry> &blocks_entry, std::vector<block> &blocks)
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
{
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
m_incoming_tx_lock.lock();
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
if (!m_blockchain_storage.prepare_handle_incoming_blocks(blocks_entry, blocks))
cryptonote: avoid double parsing blocks when syncing
2018-11-23 14:05:48 +01:00
{
cleanup_handle_incoming_blocks(false);
return false;
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
return true;
}
//-----------------------------------------------------------------------------------------------
bool core::cleanup_handle_incoming_blocks(bool force_sync)
{
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 16:43:32 +02:00
bool success = false;
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
try {
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 16:43:32 +02:00
success = m_blockchain_storage.cleanup_handle_incoming_blocks(force_sync);
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
}
catch (...) {}
m_incoming_tx_lock.unlock();
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 16:43:32 +02:00
return success;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
//-----------------------------------------------------------------------------------------------
Enforce ordering of votes in HF13
2019-08-14 06:52:51 +02:00
bool core::handle_incoming_block(const blobdata& block_blob, const block *b, block_verification_context& bvc, checkpoint_t *checkpoint, bool update_miner_blocktemplate)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
TRY_ENTRY();
Remove boost::value_initialized and BOOST_FOREACH (#921) Neither of these have a place in modern C++11; boost::value_initialized is entirely superseded by `Type var{};` which does value initialization (or default construction if a default constructor is defined). More problematically, each `boost::value_initialized<T>` requires instantiation of another wrapping templated type which is a pointless price to pay the compiler in C++11 or newer. Also removed is the AUTO_VAL_INIT macro (which is just a simple macro around constructing a boost::value_initialized<T>). BOOST_FOREACH is a similarly massive pile of code to implement C++11-style for-each loops. (And bizarrely it *doesn't* appear to fall back to C++ for-each loops even when under a C++11 compiler!) This removes both entirely from the codebase.
2019-10-31 23:26:58 +01:00
bvc = {};
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
if (!check_incoming_block_size(block_blob))
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
bvc.m_verifivation_failed = true;
return false;
}
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
if (((size_t)-1) <= 0xffffffff && block_blob.size() >= 0x3fffffff)
MWARNING("This block's size is " << block_blob.size() << ", closing on the 32 bit limit");
Rollback chain when receiving conflicting checkpoints (#726)
2019-07-11 06:16:55 +02:00
CHECK_AND_ASSERT_MES(update_checkpoints_from_json_file(), false, "One or more checkpoints loaded from json conflicted with existing checkpoints.");
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
core: avoid double parsing blocks after hoh
2018-11-23 12:15:04 +01:00
block lb;
if (!b)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
cryptonote: block parsing + hash calculation speedup This saves a duplicate serialization step
2018-12-13 20:47:47 +01:00
crypto::hash block_hash;
if(!parse_and_validate_block_from_blob(block_blob, lb, block_hash))
core: avoid double parsing blocks after hoh
2018-11-23 12:15:04 +01:00
{
LOG_PRINT_L1("Failed to parse and validate new block");
bvc.m_verifivation_failed = true;
return false;
}
b = &lb;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Enforce ordering of votes in HF13
2019-08-14 06:52:51 +02:00
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
add_new_block(*b, bvc, checkpoint);
moved all stuff to github
2014-03-03 23:07:58 +01:00
if(update_miner_blocktemplate && bvc.m_added_to_main_chain)
Inline update_miner_block_template and remove from core header
2019-08-07 02:30:46 +02:00
m_miner.on_block_chain_update();
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
CATCH_ENTRY_L0("core::handle_incoming_block()", false);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
Fix for orphan fix (check blocksize)
2014-06-11 23:32:53 +02:00
// Used by the RPC server to check the size of an incoming
// block_blob
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::check_incoming_block_size(const blobdata& block_blob) const
Fix for orphan fix (check blocksize)
2014-06-11 23:32:53 +02:00
{
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
// note: we assume block weight is always >= block blob size, so we check incoming
// blob size against the block weight limit, which acts as a sanity check without
// having to parse/weigh first; in fact, since the block blob is the block header
// plus the tx hashes, the weight will typically be much larger than the blob size
if(block_blob.size() > m_blockchain_storage.get_current_cumulative_block_weight_limit() + BLOCK_SIZE_SANITY_LEEWAY)
Fix for orphan fix (check blocksize)
2014-06-11 23:32:53 +02:00
{
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
LOG_PRINT_L1("WRONG BLOCK BLOB, sanity check failed on size " << block_blob.size() << ", rejected");
Fix for orphan fix (check blocksize)
2014-06-11 23:32:53 +02:00
return false;
}
return true;
}
Update for LokiMQ 1.1.0's changes (#1122) With the 1.1.0 update, rather than LokiMQ getting a callback that it invokes on connection to get auth level for the duration of that connection, it now checks authentication each time a command is invoked. This improves message reliability (because on an invocation failure the remote doesn't have to reconnect to be reauthenticated). This helps storage server greatly (which has some tricky initialization issue that requires it to initialize lokimq before it has the SN list); it is less needed for quorumnet communications (which have been all using connection-time authenciation for a while now), but this change does some one "weirdness" that a remote client may be unable to issue a command that they *can* issue if they reconnect. The lokid update here piggypacks the call into quorumnet in quorum_cop's `add_block` callback to make the call; this would be far cleaner if we replaced the callbacks with stateful std::function's instead of inherited base class pointers, but I didn't want to go that far in this PR.
2020-04-15 01:22:18 +02:00
void core::update_lmq_sns()
{
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
// TODO: let callers (e.g. lokinet, ss) subscribe to callbacks when this fires
lokimq::pubkey_set active_sns;
m_service_node_list.copy_active_x25519_pubkeys(std::inserter(active_sns, active_sns.end()));
m_lmq->set_active_sns(std::move(active_sns));
Update for LokiMQ 1.1.0's changes (#1122) With the 1.1.0 update, rather than LokiMQ getting a callback that it invokes on connection to get auth level for the duration of that connection, it now checks authentication each time a command is invoked. This improves message reliability (because on an invocation failure the remote doesn't have to reconnect to be reauthenticated). This helps storage server greatly (which has some tricky initialization issue that requires it to initialize lokimq before it has the SN list); it is less needed for quorumnet communications (which have been all using connection-time authenciation for a while now), but this change does some one "weirdness" that a remote client may be unable to issue a command that they *can* issue if they reconnect. The lokid update here piggypacks the call into quorumnet in quorum_cop's `add_block` callback to make the call; this would be far cleaner if we replaced the callbacks with stateful std::function's instead of inherited base class pointers, but I didn't want to go that far in this PR.
2020-04-15 01:22:18 +02:00
}
Fix for orphan fix (check blocksize)
2014-06-11 23:32:53 +02:00
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
crypto::hash core::get_tail_id() const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_tail_id();
}
//-----------------------------------------------------------------------------------------------
protocol: pass blockchain cumulative difficulty when syncing Not used yet.
2017-08-12 16:24:39 +02:00
difficulty_type core::get_block_cumulative_difficulty(uint64_t height) const
{
return m_blockchain_storage.get_db().get_block_cumulative_difficulty(height);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
bool core::have_block(const crypto::hash& id) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.have_block(id);
}
//-----------------------------------------------------------------------------------------------
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
bool core::parse_tx_from_blob(transaction& tx, crypto::hash& tx_hash, const blobdata& blob) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
return parse_and_validate_tx_from_blob(blob, tx, tx_hash);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 12:25:57 +02:00
crypto::hash core::get_block_id_by_height(uint64_t height) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
return m_blockchain_storage.get_block_id_by_height(height);
}
//-----------------------------------------------------------------------------------------------
rpc: fix orphan_status when getting blocks It was always set to false, even for orphan blocks
2017-01-22 13:20:55 +01:00
bool core::get_block_by_hash(const crypto::hash &h, block &blk, bool *orphan) const
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
{
rpc: fix orphan_status when getting blocks It was always set to false, even for orphan blocks
2017-01-22 13:20:55 +01:00
return m_blockchain_storage.get_block_by_hash(h, blk, orphan);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
//-----------------------------------------------------------------------------------------------
Add lokinet ping rpc endpoint This abstracts much of the storage server ping code to work for both storage server and lokinet.
2019-11-26 23:43:15 +01:00
static bool check_external_ping(time_t last_ping, time_t lifetime, const char *what)
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
{
Add lokinet ping rpc endpoint This abstracts much of the storage server ping code to work for both storage server and lokinet.
2019-11-26 23:43:15 +01:00
const auto elapsed = std::time(nullptr) - last_ping;
if (elapsed > lifetime)
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
{
Add lokinet ping rpc endpoint This abstracts much of the storage server ping code to work for both storage server and lokinet.
2019-11-26 23:43:15 +01:00
MWARNING("Have not heard from " << what << " " <<
(!last_ping ? "since starting" :
Wait for storage server (#874) * Fix elapsed time in storage server warning message This was passing the time value rather than the number of seconds so basically always printed "a long time" instead of the elapsed time. * Removed pre-HF12 code * Wait for storage server before sending proof On startup we send a proof immediately, but this is misleading to the operator - they may see a proof broadcast over the network suggesting everything is good even when the storage server is still down. This makes lokid wait for a first ping before the first proof so that the user doesn't get mislead. It also adds the storage server last ping into the `lokid status` message, such as: Height: 375634/375634 (100.0%) on mainnet, not mining, net hash 63.00 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 4s SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: NO PING RECEIVED or: Height: 375663/375663 (100.0%) on mainnet, not mining, net hash 76.17 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 32s SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: last ping 11 seconds ago
2019-10-08 06:40:11 +02:00
"for more than " + tools::get_human_readable_timespan(std::chrono::seconds(elapsed))));
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
return false;
}
return true;
}
Replace once_a_time_seconds; send proofs faster This replaces the horrible, horrible, badly misused templated once_a_time_seconds and once_a_time_milliseconds with a `periodic_task` that works the same way but takes parameters as constructor arguments instead of template parameters. It also makes various small improvements: - uses std::chrono::steady_clock instead of ifdef'ing platform dependent timer code. - takes a std::chrono duration rather than a template integer and scaling parameter. - timers can be reset to trigger on the next invocation, and this is thread-safe. - timer intervals can be changed at run-time. This all then gets used to reset the proof timer immediately upon receiving a ping (initially or after expiring) from storage server and lokinet so that we send proofs out faster.
2019-12-20 19:21:24 +01:00
void core::reset_proof_interval()
{
m_check_uptime_proof_interval.reset();
}
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
//-----------------------------------------------------------------------------------------------
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
void core::do_uptime_proof_call()
{
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
std::vector<service_nodes::service_node_pubkey_info> const states = get_service_node_list_state({ m_service_keys.pub });
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
Fix crash when receiving blocks for unknown quorum This can occur when syncing if we get a blink tx before the blocks that let us determine the quorum. Just ignore it at this point; we'll pick it up at the next once-per-minute sync run.
2019-12-24 04:01:48 +01:00
// wait one block before starting uptime proofs.
Use shared_ptr storage for service_node_info This converts the stored service_node_info value into a `shared_ptr<const service_node_info>` rather than a plain `service_node_info`. This yields a huge performance benefit by significantly eliminating the vast majority of service_node_info construction, destruction, and copying. Most of the time when we copy a service_node_info nothing in it has changed, which means we're storing exactly the same thing; this means an extra construction for every SN info on every block *and* an extra destruction when we cull old stored history. By using a shared_ptr, the vast majority of those constructions and destructions are eliminated. The immediately previous commit (upon which this one builds) already reduced a full rescan from 180s to 171s; this commit further reduces that time to 104s, or about 42% reduced from the rescan time required before this pair of commits. (All timings are from the dev.lokinet.org box, tested over multiple runs with the entire lmdb cached in memory). With the shared_ptr approach, we only make a copy when a change is actually needed: because of infrequent (at the per-SN level) events like a state_change, received reward, contribution, etc. The contained reference is deliberately `const` so that values are not changeable; there's a new function that does an explicit copying duplication, returning the new non-const and storing the const ref in the shared pointer. Related to this is a small change (and fix) to how proof info and public_ip/storage_port are stored: rather than store the values in the service_node_info struct itself, they now gets stored in a shared_ptr inside the service_node_info that intentionally gets shared among all copies of the service_node_info (that is, a SN info copy deliberately copies the pointer rather than the values). This also moves the ip/port values into the proof struct, since that seemed much easier than maintaining a separate shared_ptr for each value. Previously, because these were stored as values in the service_node_info they would actually get rolled back in the event of a reorg, but that seems highly undesirable: you would end up rolling back to the old values of the uptime proof and ip address (for example), but that should not happen: those values are not dependent on the blockchain and so should not be affected by a reorg/rollback. With this change they aren't since there is only one actual proof stored. Note that the shared storage here only applies to in-memory states; states loaded from the db will still be duplicated.
2019-08-11 18:19:24 +02:00
if (!states.empty() && (states[0].info->registration_height + 1) < get_current_blockchain_height())
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
{
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
m_check_uptime_proof_interval.do_call([this]() {
Uptime proof timer tweak + debug option This tweaks uptime proofs to go out on the first timer tick in the 58.5-62.5 interval (rather than 60-65) which should result in proofs usually being very close to the 60 minute mark. (Currently a fair percentage end up at the 65m mark because the timer fires just before the 60m mark). It also tweaks the first uptime proof to go out after 2 minutes instead of 5 minutes, and uses separate constants (UPTIME_PROOF_INITIAL_DELAY_SECONDS, UPTIME_PROOF_TIMER_SECONDS) for times that were previously overloading the UPTIME_PROOF_BUFFER_IN_SECONDS constant. For debugging this also adds a cmake option LOKI_DEBUG_SHORT_PROOFS that, when enabled, makes the whole proof cycle (sending and acceptance) 20x faster, which is very useful for local testnet debugging.
2019-10-27 23:10:22 +01:00
// This timer is not perfectly precise and can leak seconds slightly, so send the uptime
// proof if we are within half a tick of the target time. (Essentially our target proof
// window becomes the first time this triggers in the 57.5-62.5 minute window).
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
uint64_t next_proof_time = 0;
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
m_service_node_list.access_proof(m_service_keys.pub, [&](auto &proof) { next_proof_time = proof.timestamp; });
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
next_proof_time += UPTIME_PROOF_FREQUENCY_IN_SECONDS - UPTIME_PROOF_TIMER_SECONDS/2;
if ((uint64_t) std::time(nullptr) < next_proof_time)
Replace once_a_time_seconds; send proofs faster This replaces the horrible, horrible, badly misused templated once_a_time_seconds and once_a_time_milliseconds with a `periodic_task` that works the same way but takes parameters as constructor arguments instead of template parameters. It also makes various small improvements: - uses std::chrono::steady_clock instead of ifdef'ing platform dependent timer code. - takes a std::chrono duration rather than a template integer and scaling parameter. - timers can be reset to trigger on the next invocation, and this is thread-safe. - timer intervals can be changed at run-time. This all then gets used to reset the proof timer immediately upon receiving a ping (initially or after expiring) from storage server and lokinet so that we send proofs out faster.
2019-12-20 19:21:24 +01:00
return;
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
auto pubkey = m_service_node_list.get_pubkey_from_x25519(m_service_keys.pub_x25519);
if (pubkey != crypto::null_pkey && pubkey != m_service_keys.pub)
Warn and refuse to submit proofs if we detect duplicate ed/x25519 keys (#1121)
2020-04-14 02:53:40 +02:00
{
MGINFO_RED(
"Failed to submit uptime proof: another service node on the network is using the same ed/x25519 keys as "
"this service node. This typically means both have the same 'key_ed25519' private key file.");
return;
}
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
if (!check_external_ping(m_last_storage_server_ping, STORAGE_SERVER_PING_LIFETIME, "the storage server"))
{
MGINFO_RED(
"Failed to submit uptime proof: have not heard from the storage server recently. Make sure that it "
"is running! It is required to run alongside the Loki daemon");
Replace once_a_time_seconds; send proofs faster This replaces the horrible, horrible, badly misused templated once_a_time_seconds and once_a_time_milliseconds with a `periodic_task` that works the same way but takes parameters as constructor arguments instead of template parameters. It also makes various small improvements: - uses std::chrono::steady_clock instead of ifdef'ing platform dependent timer code. - takes a std::chrono duration rather than a template integer and scaling parameter. - timers can be reset to trigger on the next invocation, and this is thread-safe. - timer intervals can be changed at run-time. This all then gets used to reset the proof timer immediately upon receiving a ping (initially or after expiring) from storage server and lokinet so that we send proofs out faster.
2019-12-20 19:21:24 +01:00
return;
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
}
uint8_t hf_version = get_blockchain_storage().get_current_hard_fork_version();
if (!check_external_ping(m_last_lokinet_ping, LOKINET_PING_LIFETIME, "Lokinet"))
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
{
Rebase fix, blink_lns -> blink, v15 and reserve types for protocol
2020-01-21 07:16:08 +01:00
if (hf_version >= cryptonote::network_version_14_blink)
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
{
Wait for storage server (#874) * Fix elapsed time in storage server warning message This was passing the time value rather than the number of seconds so basically always printed "a long time" instead of the elapsed time. * Removed pre-HF12 code * Wait for storage server before sending proof On startup we send a proof immediately, but this is misleading to the operator - they may see a proof broadcast over the network suggesting everything is good even when the storage server is still down. This makes lokid wait for a first ping before the first proof so that the user doesn't get mislead. It also adds the storage server last ping into the `lokid status` message, such as: Height: 375634/375634 (100.0%) on mainnet, not mining, net hash 63.00 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 4s SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: NO PING RECEIVED or: Height: 375663/375663 (100.0%) on mainnet, not mining, net hash 76.17 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 32s SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: last ping 11 seconds ago
2019-10-08 06:40:11 +02:00
MGINFO_RED(
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
"Failed to submit uptime proof: have not heard from lokinet recently. Make sure that it "
Wait for storage server (#874) * Fix elapsed time in storage server warning message This was passing the time value rather than the number of seconds so basically always printed "a long time" instead of the elapsed time. * Removed pre-HF12 code * Wait for storage server before sending proof On startup we send a proof immediately, but this is misleading to the operator - they may see a proof broadcast over the network suggesting everything is good even when the storage server is still down. This makes lokid wait for a first ping before the first proof so that the user doesn't get mislead. It also adds the storage server last ping into the `lokid status` message, such as: Height: 375634/375634 (100.0%) on mainnet, not mining, net hash 63.00 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 4s SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: NO PING RECEIVED or: Height: 375663/375663 (100.0%) on mainnet, not mining, net hash 76.17 MH/s, v12, up to date, 1(out)+0(in) connections, uptime 0d 0h 0m 32s SN: f4558f60b1c4075e469a15411f12d5a747c1c62b44bcbc8523f1a90becc80475 not registered, s.server: last ping 11 seconds ago
2019-10-08 06:40:11 +02:00
"is running! It is required to run alongside the Loki daemon");
Replace once_a_time_seconds; send proofs faster This replaces the horrible, horrible, badly misused templated once_a_time_seconds and once_a_time_milliseconds with a `periodic_task` that works the same way but takes parameters as constructor arguments instead of template parameters. It also makes various small improvements: - uses std::chrono::steady_clock instead of ifdef'ing platform dependent timer code. - takes a std::chrono duration rather than a template integer and scaling parameter. - timers can be reset to trigger on the next invocation, and this is thread-safe. - timer intervals can be changed at run-time. This all then gets used to reset the proof timer immediately upon receiving a ping (initially or after expiring) from storage server and lokinet so that we send proofs out faster.
2019-12-20 19:21:24 +01:00
return;
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
}
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
else
Add lokinet ping rpc endpoint This abstracts much of the storage server ping code to work for both storage server and lokinet.
2019-11-26 23:43:15 +01:00
{
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
MGINFO_RED(
"Have not heard from lokinet recently. Make sure that it is running! "
"It is required to run alongside the Loki daemon after hard fork 14");
Add lokinet ping rpc endpoint This abstracts much of the storage server ping code to work for both storage server and lokinet.
2019-11-26 23:43:15 +01:00
}
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
}
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
submit_uptime_proof();
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
});
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
}
else
{
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
// reset the interval so that we're ready when we register, OR if we get deregistered this primes us up for re-registration in the same session
Replace once_a_time_seconds; send proofs faster This replaces the horrible, horrible, badly misused templated once_a_time_seconds and once_a_time_milliseconds with a `periodic_task` that works the same way but takes parameters as constructor arguments instead of template parameters. It also makes various small improvements: - uses std::chrono::steady_clock instead of ifdef'ing platform dependent timer code. - takes a std::chrono duration rather than a template integer and scaling parameter. - timers can be reset to trigger on the next invocation, and this is thread-safe. - timer intervals can be changed at run-time. This all then gets used to reset the proof timer immediately upon receiving a ping (initially or after expiring) from storage server and lokinet so that we send proofs out faster.
2019-12-20 19:21:24 +01:00
m_check_uptime_proof_interval.reset();
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
}
}
//-----------------------------------------------------------------------------------------------
moved all stuff to github
2014-03-03 23:07:58 +01:00
bool core::on_idle()
{
if(!m_starter_message_showed)
{
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
std::string main_message;
if (m_offline)
First pass Monero to Loki refs
2018-04-10 06:49:20 +02:00
main_message = "The daemon is running offline and will not attempt to sync to the Loki network.";
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
else
main_message = "The daemon will start synchronizing with the network. This may take a long time to complete.";
"Remove namespace pollution" << ENDL Removes all "using namespace epee;" and "using namespace std;" from the code and fixes up the various crappy places where unnamespaced types were being used. Also removes the ENDL macro (which was defined to be `std::endl`) because it is retarded, and because even using std::endl instead of a plain "\n" is usually a mistake (`<< std::endl` is equivalent to `<< "\n" << std::flush`, and that explicit flush is rarely desirable).
2020-04-03 18:30:11 +02:00
MGINFO_YELLOW("\n**********************************************************************\n"
<< main_message << "\n"
<< "\n"
<< "You can set the level of process detailization through \"set_log <level|categories>\" command,\n"
<< "where <level> is between 0 (no details) and 4 (very verbose), or custom category based levels (eg, *:WARNING).\n"
<< "\n"
<< "Use the \"help\" command to see the list of available commands.\n"
<< "Use \"help <command>\" to see a command's documentation.\n"
<< "**********************************************************************\n");
moved all stuff to github
2014-03-03 23:07:58 +01:00
m_starter_message_showed = true;
}
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
m_fork_moaner.do_call(boost::bind(&core::check_fork_time, this));
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
m_txpool_auto_relayer.do_call(boost::bind(&core::relay_txpool_transactions, this));
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
m_service_node_vote_relayer.do_call(boost::bind(&core::relay_service_node_votes, this));
Merge commit '4a652d6' into LokiMergeUpstreamUntil_20180911_e6d36c1
2018-10-10 04:48:01 +02:00
// m_check_updates_interval.do_call(boost::bind(&core::check_updates, this));
core: warn when free disk space is low
2017-11-12 00:25:12 +01:00
m_check_disk_space_interval.do_call(boost::bind(&core::check_disk_space, this));
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
m_block_rate_interval.do_call(boost::bind(&core::check_block_rate, this));
Store uptime proof data separately from states This extracts uptime proof data entirely from service node states, instead storing (some) proof data as its own first-class object in the code and backed by the database. We now persistently store: - timestamp - version - ip & ports - ed25519 pubkey and update it every time a proof is received. Upon restart, we load all the proofs from the database, which means we no longer lose last proof received times on restart, and always have the most recently received ip and port information (rather than only having whatever it was in the most recently stored state, which could be ~12 blocks ago). It also means we don't need to spam the network with proofs for up to half an hour after a restart because we now know when we last sent (and received) our own proof before restarting. This separation required some restructuring: most notably changing `state_t` to be constructed with a `service_node_list` pointer, which it needs both directly and for BlockchainDB access. Related to this is also eliminating the separate BlockchainDB pointer stored in service_node_list in favour of just accessing it via m_blockchain (which now has a new `has_db()` method to detect when it has been initialized).
2019-12-04 21:50:28 +01:00
m_sn_proof_cleanup_interval.do_call([&snl=m_service_node_list] { snl.cleanup_proofs(); return true; });
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
time_t const lifetime = time(nullptr) - get_start_time();
--regtest fixes and mine n blocks ability `--regtest` didn't work in some edge cases, this fixes various things: - the genesis block wasn't accepted because it needed to be v7, not vMax - reduce initial uptime proof delay to 5s in regtest mode - add --regtest flag to the wallet so that it can talk to a daemon in --regtest mode. This also adds two new mining options, available via rpc: - slow_mining - this avoids the RandomX initialization. It is much slower, but for regtest with fixed difficulty of 1 that is perfectly fine. - `num_blocks` - instruct the miner to mine for the given number of blocks, then stop. (This can overmine if mining with multiple threads at a low difficulty, but that's fine).
2019-12-24 04:06:30 +01:00
int proof_delay = m_nettype == FAKECHAIN ? 5 : UPTIME_PROOF_INITIAL_DELAY_SECONDS;
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (m_service_node && lifetime > proof_delay) // Give us some time to connect to peers before sending uptimes
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
{
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
do_uptime_proof_call();
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
m_blockchain_pruning_interval.do_call(boost::bind(&core::update_blockchain_pruning, this));
moved all stuff to github
2014-03-03 23:07:58 +01:00
m_miner.on_idle();
proper tx_pool handling from CryptoZoidberg / BBR
2014-06-15 09:48:13 +02:00
m_mempool.on_idle();
Revamp to use semaphores for IPC Delay hooks into stdin/out until core is idle for daemons Disable logging for daemon (super problematic)
2019-01-15 10:09:17 +01:00
#if defined(LOKI_ENABLE_INTEGRATION_TEST_HOOKS)
Rewrite the integration tests to use pipes (#1045)
2020-02-21 04:48:25 +01:00
integration_test::state.core_is_idle = true;
Revamp to use semaphores for IPC Delay hooks into stdin/out until core is idle for daemons Disable logging for daemon (super problematic)
2019-01-15 10:09:17 +01:00
#endif
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
#ifdef ENABLE_SYSTEMD
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
m_systemd_notify_interval.do_call([this] { sd_notify(0, ("WATCHDOG=1\nSTATUS=" + get_status_string()).c_str()); });
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
#endif
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
bool core::check_fork_time()
{
core: don't check block rate nor fork time in regtest mode
2019-04-07 15:29:21 +02:00
if (m_nettype == FAKECHAIN)
return true;
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
HardFork::State state = m_blockchain_storage.get_hard_fork_state();
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
const el::Level level = el::Level::Warning;
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
switch (state) {
case HardFork::LikelyForked:
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
MCLOG_RED(level, "global", "**********************************************************************");
MCLOG_RED(level, "global", "Last scheduled hard fork is too far in the past.");
MCLOG_RED(level, "global", "We are most likely forked from the network. Daemon update needed now.");
MCLOG_RED(level, "global", "**********************************************************************");
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 19:09:57 +02:00
break;
case HardFork::UpdateNeeded:
break;
default:
break;
}
return true;
}
//-----------------------------------------------------------------------------------------------
Only log an error if fork version is higher AND is not known.
2018-04-16 04:02:26 +02:00
uint8_t core::get_ideal_hard_fork_version() const
{
return get_blockchain_storage().get_ideal_hard_fork_version();
}
//-----------------------------------------------------------------------------------------------
tests: fix tests build Add get_fork_version and add_ideal_fork_version to core so cryptonote_protocol does not have to need the Blockchain class directly, as it's not in its dependencies, and add those to the fake core classes in tests too.
2017-08-10 11:10:04 +02:00
uint8_t core::get_ideal_hard_fork_version(uint64_t height) const
{
return get_blockchain_storage().get_ideal_hard_fork_version(height);
}
//-----------------------------------------------------------------------------------------------
uint8_t core::get_hard_fork_version(uint64_t height) const
{
return get_blockchain_storage().get_hard_fork_version(height);
}
//-----------------------------------------------------------------------------------------------
core: add get_earliest_ideal_height_for_version()
2018-03-20 12:30:09 +01:00
uint64_t core::get_earliest_ideal_height_for_version(uint8_t version) const
{
return get_blockchain_storage().get_earliest_ideal_height_for_version(version);
}
//-----------------------------------------------------------------------------------------------
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
bool core::check_updates()
{
Update cmakelists and misc refs to Loki
2018-04-25 08:13:38 +02:00
static const char software[] = "loki";
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
#ifdef BUILD_TAG
static const char buildtag[] = BOOST_PP_STRINGIZE(BUILD_TAG);
Source updates are in a source subdirectory rather than in the same directory as the prebuilt versions
2017-09-22 22:48:19 +02:00
static const char subdir[] = "cli"; // because it can never be simple
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
#else
static const char buildtag[] = "source";
Source updates are in a source subdirectory rather than in the same directory as the prebuilt versions
2017-09-22 22:48:19 +02:00
static const char subdir[] = "source"; // because it can never be simple
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
#endif
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
if (m_offline)
return true;
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
if (check_updates_level == UPDATES_DISABLED)
return true;
std::string version, hash;
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
MCDEBUG("updates", "Checking for a new " << software << " version for " << buildtag);
updates: remove testnet case It need not be any different
2017-02-22 12:19:20 +01:00
if (!tools::check_updates(software, buildtag, version, hash))
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
return false;
Store loki version as std::array<u16,3> everywhere Currently we store it as various different things: 3 separate ints, 2 u16s, 3 separate u16s, and a vector of u16s. This unifies all version values to a `std::array<uint16_t,3>`. - LOKI_VERSION_{MAJOR,MINOR,PATCH} are now just LOKI_VERSION - The previous LOKI_VERSION (C-string of the version) is now renamed LOKI_VERSION_STR A related change included here is that the restricted RPC now returns the major version in the get_info rpc call instead of an empty string (e.g. "5" instead of ""). There is almost certainly enough difference in the RPC results to distinguish major versions already so this doesn't seem like it actually leaks anything significant.
2019-10-15 22:31:46 +02:00
if (tools::vercmp(version.c_str(), LOKI_VERSION_STR) <= 0)
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 16:31:34 +02:00
{
m_update_available = false;
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
return true;
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 16:31:34 +02:00
}
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
updates: we now have a user URL, and an automatic one So the user can use https, while the automatic does not have to
2017-02-22 12:11:45 +01:00
std::string url = tools::get_update_url(software, subdir, buildtag, version, true);
core: print the "new update found" message in cyan, for visibility
2017-03-04 19:29:18 +01:00
MCLOG_CYAN(el::Level::Info, "global", "Version " << version << " of " << software << " for " << buildtag << " is available: " << url << ", SHA256 hash " << hash);
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 16:31:34 +02:00
m_update_available = true;
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
if (check_updates_level == UPDATES_NOTIFY)
return true;
updates: we now have a user URL, and an automatic one So the user can use https, while the automatic does not have to
2017-02-22 12:11:45 +01:00
url = tools::get_update_url(software, subdir, buildtag, version, false);
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
std::string filename;
const char *slash = strrchr(url.c_str(), '/');
if (slash)
filename = slash + 1;
else
filename = std::string(software) + "-update-" + version;
boost::filesystem::path path(epee::string_tools::get_current_module_folder());
path /= filename;
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
core: make update download cancellable
2017-02-26 22:00:38 +01:00
boost::unique_lock<boost::mutex> lock(m_update_mutex);
if (m_update_download != 0)
{
MCDEBUG("updates", "Already downloading update");
return true;
}
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
crypto::hash file_hash;
Improve checkpoint loading code; make sha256sum less dangerous - replace the gross cmake shell script tools being used to create the data file with cmake code and a single configure_file - change checkpoint data to string_view - tools::sha256sum() had two different overloads that did very different things, yet presents two overloads: one that takes a pointer+size, the other that takes a string lvalue ref, and yet the string lvalue reference is *entirely* different (it reads a file). Append _str and _file suffixes to make it less dangerous.
2020-05-29 23:23:39 +02:00
if (!tools::sha256sum_file(path.string(), file_hash) || (hash != epee::string_tools::pod_to_hex(file_hash)))
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
{
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
MCDEBUG("updates", "We don't have that file already, downloading");
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
const std::string tmppath = path.string() + ".tmp";
if (epee::file_io_utils::is_file_exist(tmppath))
{
MCDEBUG("updates", "We have part of the file already, resuming download");
}
core: make update download cancellable
2017-02-26 22:00:38 +01:00
m_last_update_length = 0;
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
m_update_download = tools::download_async(tmppath, url, [this, hash, path](const std::string &tmppath, const std::string &uri, bool success) {
bool remove = false, good = true;
core: make update download cancellable
2017-02-26 22:00:38 +01:00
if (success)
{
crypto::hash file_hash;
Improve checkpoint loading code; make sha256sum less dangerous - replace the gross cmake shell script tools being used to create the data file with cmake code and a single configure_file - change checkpoint data to string_view - tools::sha256sum() had two different overloads that did very different things, yet presents two overloads: one that takes a pointer+size, the other that takes a string lvalue ref, and yet the string lvalue reference is *entirely* different (it reads a file). Append _str and _file suffixes to make it less dangerous.
2020-05-29 23:23:39 +02:00
if (!tools::sha256sum_file(tmppath, file_hash))
core: make update download cancellable
2017-02-26 22:00:38 +01:00
{
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
MCERROR("updates", "Failed to hash " << tmppath);
remove = true;
good = false;
core: make update download cancellable
2017-02-26 22:00:38 +01:00
}
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
else if (hash != epee::string_tools::pod_to_hex(file_hash))
core: make update download cancellable
2017-02-26 22:00:38 +01:00
{
MCERROR("updates", "Download from " << uri << " does not match the expected hash");
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
remove = true;
good = false;
core: make update download cancellable
2017-02-26 22:00:38 +01:00
}
}
else
{
MCERROR("updates", "Failed to download " << uri);
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
good = false;
core: make update download cancellable
2017-02-26 22:00:38 +01:00
}
boost::unique_lock<boost::mutex> lock(m_update_mutex);
m_update_download = 0;
resumption support for updates using range requests
2017-12-15 11:27:03 +01:00
if (success && !remove)
{
std::error_code e = tools::replace_file(tmppath, path.string());
if (e)
{
MCERROR("updates", "Failed to rename downloaded file");
good = false;
}
}
else if (remove)
{
if (!boost::filesystem::remove(tmppath))
{
MCERROR("updates", "Failed to remove invalid downloaded file");
good = false;
}
}
if (good)
MCLOG_CYAN(el::Level::Info, "updates", "New version downloaded to " << path.string());
core: make update download cancellable
2017-02-26 22:00:38 +01:00
}, [this](const std::string &path, const std::string &uri, size_t length, ssize_t content_length) {
if (length >= m_last_update_length + 1024 * 1024 * 10)
{
m_last_update_length = length;
MCDEBUG("updates", "Downloaded " << length << "/" << (content_length ? std::to_string(content_length) : "unknown"));
}
return true;
});
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
}
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
else
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
{
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
MCDEBUG("updates", "We already have " << path << " with expected hash");
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
}
core: make update download cancellable
2017-02-26 22:00:38 +01:00
lock.unlock();
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 21:48:36 +01:00
if (check_updates_level == UPDATES_DOWNLOAD)
return true;
core: do not download an update we already downloaded
2017-02-21 23:23:59 +01:00
MCERROR("updates", "Download/update not implemented yet");
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 17:30:29 +01:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: warn when free disk space is low
2017-11-12 00:25:12 +01:00
bool core::check_disk_space()
{
uint64_t free_space = get_free_space();
if (free_space < 1ull * 1024 * 1024 * 1024) // 1 GB
{
const el::Level level = el::Level::Warning;
MCLOG_RED(level, "global", "Free space is below 1 GB on " << m_config_folder);
}
return true;
}
//-----------------------------------------------------------------------------------------------
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
double factorial(unsigned int n)
{
if (n <= 1)
return 1.0;
double f = n;
while (n-- > 1)
f *= n;
return f;
}
//-----------------------------------------------------------------------------------------------
core: improve block rate monitor trigger probabilities The original intent of one false positive a week on average was not met, since what we really want is not the probability of having N blocks in T seconds, but either N blocks of fewer in T seconds, or N blocks or more in T seconds. Some of this could be cached since it calculates the same fairly complex floating point values, but it seems pretty fast already.
2019-03-19 17:30:17 +01:00
static double probability1(unsigned int blocks, unsigned int expected)
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
{
// https://www.umass.edu/wsp/resources/poisson/#computing
return pow(expected, blocks) / (factorial(blocks) * exp(expected));
}
//-----------------------------------------------------------------------------------------------
core: improve block rate monitor trigger probabilities The original intent of one false positive a week on average was not met, since what we really want is not the probability of having N blocks in T seconds, but either N blocks of fewer in T seconds, or N blocks or more in T seconds. Some of this could be cached since it calculates the same fairly complex floating point values, but it seems pretty fast already.
2019-03-19 17:30:17 +01:00
static double probability(unsigned int blocks, unsigned int expected)
{
double p = 0.0;
if (blocks <= expected)
{
for (unsigned int b = 0; b <= blocks; ++b)
p += probability1(b, expected);
}
else if (blocks > expected)
{
for (unsigned int b = blocks; b <= expected * 3 /* close enough */; ++b)
p += probability1(b, expected);
}
return p;
}
//-----------------------------------------------------------------------------------------------
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
bool core::check_block_rate()
{
core: do not complain about low block rate if disconnected In that case, we'll still keep the "Monero is now disconnected from the network" near the end of the log
2019-06-13 12:24:18 +02:00
if (m_offline || m_nettype == FAKECHAIN || m_target_blockchain_height > get_current_blockchain_height() || m_target_blockchain_height == 0)
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
{
MDEBUG("Not checking block rate, offline or syncing");
return true;
}
Disable block rate and min life time in integration mode
2018-11-29 08:20:31 +01:00
#if defined(LOKI_ENABLE_INTEGRATION_TEST_HOOKS)
MDEBUG("Not checking block rate, integration test mode");
return true;
#endif
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
static constexpr double threshold = 1. / (864000 / DIFFICULTY_TARGET_V2); // one false positive every 10 days
core: point out when we hit the block rate visibility limit
2019-10-28 17:04:43 +01:00
static constexpr unsigned int max_blocks_checked = 150;
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
const time_t now = time(NULL);
core: point out when we hit the block rate visibility limit
2019-10-28 17:04:43 +01:00
const std::vector<time_t> timestamps = m_blockchain_storage.get_last_block_timestamps(max_blocks_checked);
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
core: add a few more block rate window sizes The 10 minute one will never trigger for 0 blocks, as it's still fairly likely to happen even without the actual hash rate changing much, so we add a 20 minute window, where it will (for 0 blocks) and a one hour window.
2019-02-06 20:04:47 +01:00
static const unsigned int seconds[] = { 5400, 3600, 1800, 1200, 600 };
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
for (size_t n = 0; n < sizeof(seconds)/sizeof(seconds[0]); ++n)
{
unsigned int b = 0;
cryptonote_core: avoid gratuitous recalculations in check_block_rate()
2018-11-05 01:13:51 +01:00
const time_t time_boundary = now - static_cast<time_t>(seconds[n]);
for (time_t ts: timestamps) b += ts >= time_boundary;
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
const double p = probability(b, seconds[n] / DIFFICULTY_TARGET_V2);
MDEBUG("blocks in the last " << seconds[n] / 60 << " minutes: " << b << " (probability " << p << ")");
if (p < threshold)
{
Merge commit 'c4f75fe' into MergeUpstream3
2020-05-28 07:29:00 +02:00
MWARNING("There were " << b << (b == max_blocks_checked ? " or more" : "") << " blocks in the last " << seconds[n] / 60 << " minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Loki network or under attack, or your computer's time is off. Or it could be just sheer bad luck.");
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
std::shared_ptr<tools::Notify> block_rate_notify = m_block_rate_notify;
if (block_rate_notify)
{
auto expected = seconds[n] / DIFFICULTY_TARGET_V2;
block_rate_notify->notify("%t", std::to_string(seconds[n] / 60).c_str(), "%b", std::to_string(b).c_str(), "%e", std::to_string(expected).c_str(), NULL);
}
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
break; // no need to look further
}
}
return true;
}
//-----------------------------------------------------------------------------------------------
Add storage server reachability field; test it with quorums (#820) * Add storage server reachability field; test it with quorums * Improve method names * Address review comments
2019-09-02 05:06:15 +02:00
bool core::set_storage_server_peer_reachable(crypto::public_key const &pubkey, bool value)
{
return m_service_node_list.set_storage_server_peer_reachable(pubkey, value);
}
//-----------------------------------------------------------------------------------------------
rpc: add a flush_cache RPC This allows flushing internal caches (for now, the bad tx cache, which will allow debugging a stuck monerod after it has failed to verify a transaction in a block, since it would otherwise not try again, making subsequent log changes pointless)
2019-10-22 17:28:08 +02:00
void core::flush_bad_txs_cache()
{
bad_semantics_txes_lock.lock();
for (int idx = 0; idx < 2; ++idx)
bad_semantics_txes[idx].clear();
bad_semantics_txes_lock.unlock();
}
//-----------------------------------------------------------------------------------------------
rpc: add bad-blocks to flush_cache RPC Flushes m_invalid_blocks in Blockchain.
2019-12-02 19:51:28 +01:00
void core::flush_invalid_blocks()
{
m_blockchain_storage.flush_invalid_blocks();
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
bool core::update_blockchain_pruning()
{
return m_blockchain_storage.update_blockchain_pruning();
}
//-----------------------------------------------------------------------------------------------
bool core::check_blockchain_pruning()
{
return m_blockchain_storage.check_blockchain_pruning();
}
//-----------------------------------------------------------------------------------------------
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
void core::set_target_blockchain_height(uint64_t target_blockchain_height)
{
Revert "Fix fake 'network synchronized, begin using' messages" This reverts commit 78035d2b6c9922f4cd730df0766aa74f4854ccb2. The patch doesn't work, and causes constant SYNCHRONIZED OK spam.
2016-09-25 20:08:31 +02:00
m_target_blockchain_height = target_blockchain_height;
'getinfo' daemon HTTP-RPC returns 'target_height' for progress estimations
2014-06-04 22:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
uint64_t core::get_target_blockchain_height() const
{
'getinfo' daemon HTTP-RPC returns 'target_height' for progress estimations
2014-06-04 22:50:13 +02:00
return m_target_blockchain_height;
}
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
uint64_t core::prevalidate_block_hashes(uint64_t height, const std::vector<crypto::hash> &hashes)
precomputed block hashes are now in blocks of N (currently 256) This shaves a lot of space off binaries
2017-09-13 22:03:04 +02:00
{
return get_blockchain_storage().prevalidate_block_hashes(height, hashes);
}
//-----------------------------------------------------------------------------------------------
core: warn when free disk space is low
2017-11-12 00:25:12 +01:00
uint64_t core::get_free_space() const
{
boost::filesystem::path path(m_config_folder);
boost::filesystem::space_info si = boost::filesystem::space(path);
return si.available;
}
//-----------------------------------------------------------------------------------------------
Rename `testing_quorum` class to just `quorum` testing_quorum is a bit of a misnomer: it only does testing when used for an obligations quorum, but it is also used for checkpointing and soon Blink.
2019-10-15 19:52:49 +02:00
std::shared_ptr<const service_nodes::quorum> core::get_quorum(service_nodes::quorum_type type, uint64_t height, bool include_old, std::vector<std::shared_ptr<const service_nodes::quorum>> *alt_states) const
Merge dev into upstream
2019-05-01 08:01:17 +02:00
{
Rename `testing_quorum` class to just `quorum` testing_quorum is a bit of a misnomer: it only does testing when used for an obligations quorum, but it is also used for checkpointing and soon Blink.
2019-10-15 19:52:49 +02:00
return m_service_node_list.get_quorum(type, height, include_old, alt_states);
Making service nodes list and quorum cop thread-safe (#285) * Remove dependency on service node list in quorum cop * making service node list therad-safe * improve thread-safety of quorum cop
2018-10-10 07:24:01 +02:00
}
//-----------------------------------------------------------------------------------------------
Relax deregistration rules The replaces the deregistration mechanism with a new state change mechanism (beginning at the v12 fork) which can change a service node's network status via three potential values (and is extensible in the future to handle more): - deregistered -- this is the same as the existing deregistration; the SN is instantly removed from the SN list. - decommissioned -- this is a sort of temporary deregistration: your SN remains in the service node list, but is removed from the rewards list and from any network duties. - recommissioned -- this tx is sent by a quorum if they observe a decommissioned SN sending uptime proofs again. Upon reception, the SN is reactivated and put on the end of the reward list. Since this is broadening the quorum use, this also renames the relevant quorum to a "obligations" quorum (since it validates SN obligations), while the transactions are "state_change" transactions (since they change the state of a registered SN). The new parameters added to service_node_rules.h control how this works: // Service node decommissioning: as service nodes stay up they earn "credits" (measured in blocks) // towards a future outage. A new service node starts out with INITIAL_CREDIT, and then builds up // CREDIT_PER_DAY for each day the service node remains active up to a maximum of // DECOMMISSION_MAX_CREDIT. // // If a service node stops sending uptime proofs, a quorum will consider whether the service node // has built up enough credits (at least MINIMUM): if so, instead of submitting a deregistration, // it instead submits a decommission. This removes the service node from the list of active // service nodes both for rewards and for any active network duties. If the service node comes // back online (i.e. starts sending the required performance proofs again) before the credits run // out then a quorum will reinstate the service node using a recommission transaction, which adds // the service node back to the bottom of the service node reward list, and resets its accumulated // credits to 0. If it does not come back online within the required number of blocks (i.e. the // accumulated credit at the point of decommissioning) then a quorum will send a permanent // deregistration transaction to the network, starting a 30-day deregistration count down. This commit currently includes values (which are not necessarily finalized): - 8 hours (240 blocks) of credit required for activation of a decommission (rather than a deregister) - 0 initial credits at registration - a maximum of 24 hours (720 blocks) of credits - credits accumulate at a rate that you hit 24 hours of credits after 30 days of operation. Miscellaneous other details of this PR: - a new TX extra tag is used for the state change (including deregistrations). The old extra tag has no version or type tag, so couldn't be reused. The data in the new tag is slightly more efficiently packed than the old deregistration transaction, so it gets used for deregistrations (starting at the v12 fork) as well. - Correct validator/worker selection required generalizing the shuffle function to be able to shuffle just part of a vector. This lets us stick any down service nodes at the end of the potential list, then select validators by only shuffling the part of the index vector that contains active service indices. Once the validators are selected, the remainder of the list (this time including decommissioned SN indices) is shuffled to select quorum workers to check, thus allowing decommisioned nodes to be randomly included in the nodes to check without being selected as a validator. - Swarm recalculation was not quite right: swarms were recalculated on SN registrations, even if those registrations were include shared node registrations, but *not* recalculated on stakes. Starting with the upgrade this behaviour is fixed (swarms aren't actually used currently and aren't consensus-relevant so recalculating early won't hurt anything). - Details on decomm/dereg are added to RPC info and print_sn/print_sn_status - Slightly improves the % of reward output in the print_sn output by rounding it to two digits, and reserves space in the output string to avoid excessive reallocations. - Adds various debugging at higher debug levels to quorum voting (into all of voting itself, vote transmission, and vote reception). - Reset service node list internal data structure version to 0. The SN list has to be rescanned anyway at upgrade (its size has changed), so we might as well reset the version and remove the version-dependent serialization code. (Note that the affected code here is for SN states in lmdb storage, not for SN-to-SN communication serialization).
2019-06-18 23:57:02 +02:00
bool core::is_service_node(const crypto::public_key& pubkey, bool require_active) const
Making service nodes list and quorum cop thread-safe (#285) * Remove dependency on service node list in quorum cop * making service node list therad-safe * improve thread-safety of quorum cop
2018-10-10 07:24:01 +02:00
{
Relax deregistration rules The replaces the deregistration mechanism with a new state change mechanism (beginning at the v12 fork) which can change a service node's network status via three potential values (and is extensible in the future to handle more): - deregistered -- this is the same as the existing deregistration; the SN is instantly removed from the SN list. - decommissioned -- this is a sort of temporary deregistration: your SN remains in the service node list, but is removed from the rewards list and from any network duties. - recommissioned -- this tx is sent by a quorum if they observe a decommissioned SN sending uptime proofs again. Upon reception, the SN is reactivated and put on the end of the reward list. Since this is broadening the quorum use, this also renames the relevant quorum to a "obligations" quorum (since it validates SN obligations), while the transactions are "state_change" transactions (since they change the state of a registered SN). The new parameters added to service_node_rules.h control how this works: // Service node decommissioning: as service nodes stay up they earn "credits" (measured in blocks) // towards a future outage. A new service node starts out with INITIAL_CREDIT, and then builds up // CREDIT_PER_DAY for each day the service node remains active up to a maximum of // DECOMMISSION_MAX_CREDIT. // // If a service node stops sending uptime proofs, a quorum will consider whether the service node // has built up enough credits (at least MINIMUM): if so, instead of submitting a deregistration, // it instead submits a decommission. This removes the service node from the list of active // service nodes both for rewards and for any active network duties. If the service node comes // back online (i.e. starts sending the required performance proofs again) before the credits run // out then a quorum will reinstate the service node using a recommission transaction, which adds // the service node back to the bottom of the service node reward list, and resets its accumulated // credits to 0. If it does not come back online within the required number of blocks (i.e. the // accumulated credit at the point of decommissioning) then a quorum will send a permanent // deregistration transaction to the network, starting a 30-day deregistration count down. This commit currently includes values (which are not necessarily finalized): - 8 hours (240 blocks) of credit required for activation of a decommission (rather than a deregister) - 0 initial credits at registration - a maximum of 24 hours (720 blocks) of credits - credits accumulate at a rate that you hit 24 hours of credits after 30 days of operation. Miscellaneous other details of this PR: - a new TX extra tag is used for the state change (including deregistrations). The old extra tag has no version or type tag, so couldn't be reused. The data in the new tag is slightly more efficiently packed than the old deregistration transaction, so it gets used for deregistrations (starting at the v12 fork) as well. - Correct validator/worker selection required generalizing the shuffle function to be able to shuffle just part of a vector. This lets us stick any down service nodes at the end of the potential list, then select validators by only shuffling the part of the index vector that contains active service indices. Once the validators are selected, the remainder of the list (this time including decommissioned SN indices) is shuffled to select quorum workers to check, thus allowing decommisioned nodes to be randomly included in the nodes to check without being selected as a validator. - Swarm recalculation was not quite right: swarms were recalculated on SN registrations, even if those registrations were include shared node registrations, but *not* recalculated on stakes. Starting with the upgrade this behaviour is fixed (swarms aren't actually used currently and aren't consensus-relevant so recalculating early won't hurt anything). - Details on decomm/dereg are added to RPC info and print_sn/print_sn_status - Slightly improves the % of reward output in the print_sn output by rounding it to two digits, and reserves space in the output string to avoid excessive reallocations. - Adds various debugging at higher debug levels to quorum voting (into all of voting itself, vote transmission, and vote reception). - Reset service node list internal data structure version to 0. The SN list has to be rescanned anyway at upgrade (its size has changed), so we might as well reset the version and remove the version-dependent serialization code. (Note that the affected code here is for SN states in lmdb storage, not for SN-to-SN communication serialization).
2019-06-18 23:57:02 +02:00
return m_service_node_list.is_service_node(pubkey, require_active);
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
}
//-----------------------------------------------------------------------------------------------
Infinite Staking Part 1 (#387) * Remove dead branches in hot-path check_tx_inputs Also renames #define for mixins to better match naming convention * Shuffle around some more code into common branches * Fix min/max tx version rules, since there 1 tx v2 on v9 fork * First draft infinite staking implementation * Actually generate the right key image and expire appropriately * Add framework to lock key images after expiry * Return locked key images for nodes, add request unlock option * Introduce transaction types for key image unlock * Update validation steps to accept tx types, key_image_unlock * Add mapping for lockable key images to amounts * Change inconsistent naming scheme of contributors * Create key image unlock transaction type and process it * Update tx params to allow v4 types and as a result construct_tx* * Fix some serialisation issues not sending all the information * Fix dupe tx extra tag causing incorrect deserialisation * Add warning comments * Fix key image unlocks parsing error * Simplify key image proof checks * Fix rebase errors * Correctly calculate the key image unlock times * Blacklist key image on deregistration * Serialise key image blacklist * Rollback blacklisted key images * Fix expiry logic error * Disallow requesting stake unlock if already unlocked client side * Add double spend checks for key image unlocks * Rename get_staking_requirement_lock_blocks To staking_initial_num_lock_blocks * Begin modifying output selection to not use locked outputs * Modify output selection to avoid locked/blacklisted key images * Cleanup and undoing some protocol breakages * Simplify expiration of nodes * Request unlock schedules entire node for expiration * Fix off by one in expiring nodes * Undo expiring code for pre v10 nodes * Fix RPC returning register as unlock height and not checking 0 * Rename key image unlock height const * Undo testnet hardfork debug changes * Remove is_type for get_type, fix missing var rename * Move serialisable data into public namespace * Serialise tx types properly * Fix typo in no service node known msg * Code review * Fix == to >= on serialising tx type * Code review 2 * Fix tests and key image unlock * Add additional test, fix assert * Remove debug code in wallet * Fix merge dev problem
2019-01-25 04:15:52 +01:00
const std::vector<service_nodes::key_image_blacklist_entry> &core::get_service_node_blacklisted_key_images() const
{
Optimizations and simplifications This commit makes various simplifications and optimizations, mainly in the service node list code. All in all, this shaves about 5% of the CPU time used for re-processing the entire service node list. In particular: - changed m_state_history from a std::vector to a std::set that sorts on height. This is responsible for the bulk of the CPU reduction by significant reducing the amount of work required for checkpoint culling, which has to shuffle a lot of `state_t`s around when removing from the midde of a vector. - the above also allows replacing the binary-search `std::lower_bound` complexity with a much simpler `find()`. - since the history is now always sorted, removed the error messages related to sorting that either can't happen (on store) or don't matter (on load). - Added some converting constructors to simplify code (for example, a `state_t` can now be very usefully constructed from an r-value `state_serialized`). - Many construct + moves (and a couple construct + copy) were replaced with in-place constructions. - removed some unused variables
2019-08-11 05:46:59 +02:00
return m_service_node_list.get_blacklisted_key_images();
Infinite Staking Part 1 (#387) * Remove dead branches in hot-path check_tx_inputs Also renames #define for mixins to better match naming convention * Shuffle around some more code into common branches * Fix min/max tx version rules, since there 1 tx v2 on v9 fork * First draft infinite staking implementation * Actually generate the right key image and expire appropriately * Add framework to lock key images after expiry * Return locked key images for nodes, add request unlock option * Introduce transaction types for key image unlock * Update validation steps to accept tx types, key_image_unlock * Add mapping for lockable key images to amounts * Change inconsistent naming scheme of contributors * Create key image unlock transaction type and process it * Update tx params to allow v4 types and as a result construct_tx* * Fix some serialisation issues not sending all the information * Fix dupe tx extra tag causing incorrect deserialisation * Add warning comments * Fix key image unlocks parsing error * Simplify key image proof checks * Fix rebase errors * Correctly calculate the key image unlock times * Blacklist key image on deregistration * Serialise key image blacklist * Rollback blacklisted key images * Fix expiry logic error * Disallow requesting stake unlock if already unlocked client side * Add double spend checks for key image unlocks * Rename get_staking_requirement_lock_blocks To staking_initial_num_lock_blocks * Begin modifying output selection to not use locked outputs * Modify output selection to avoid locked/blacklisted key images * Cleanup and undoing some protocol breakages * Simplify expiration of nodes * Request unlock schedules entire node for expiration * Fix off by one in expiring nodes * Undo expiring code for pre v10 nodes * Fix RPC returning register as unlock height and not checking 0 * Rename key image unlock height const * Undo testnet hardfork debug changes * Remove is_type for get_type, fix missing var rename * Move serialisable data into public namespace * Serialise tx types properly * Fix typo in no service node known msg * Code review * Fix == to >= on serialising tx type * Code review 2 * Fix tests and key image unlock * Add additional test, fix assert * Remove debug code in wallet * Fix merge dev problem
2019-01-25 04:15:52 +01:00
}
//-----------------------------------------------------------------------------------------------
Service Nodes States Endpoint (#145) * Add get_service_node_list_state command for analytics * Update service_node_list_state to search particular pubkey * Service node list state sorts display results by longest waiting * Fix up leftover todos/unused data structures * Change get_service_node_list_state to print_sn
2018-08-15 04:59:05 +02:00
std::vector<service_nodes::service_node_pubkey_info> core::get_service_node_list_state(const std::vector<crypto::public_key> &service_node_pubkeys) const
{
Optimizations and simplifications This commit makes various simplifications and optimizations, mainly in the service node list code. All in all, this shaves about 5% of the CPU time used for re-processing the entire service node list. In particular: - changed m_state_history from a std::vector to a std::set that sorts on height. This is responsible for the bulk of the CPU reduction by significant reducing the amount of work required for checkpoint culling, which has to shuffle a lot of `state_t`s around when removing from the midde of a vector. - the above also allows replacing the binary-search `std::lower_bound` complexity with a much simpler `find()`. - since the history is now always sorted, removed the error messages related to sorting that either can't happen (on store) or don't matter (on load). - Added some converting constructors to simplify code (for example, a `state_t` can now be very usefully constructed from an r-value `state_serialized`). - Many construct + moves (and a couple construct + copy) were replaced with in-place constructions. - removed some unused variables
2019-08-11 05:46:59 +02:00
return m_service_node_list.get_service_node_list_state(service_node_pubkeys);
Service Nodes States Endpoint (#145) * Add get_service_node_list_state command for analytics * Update service_node_list_state to search particular pubkey * Service node list state sorts display results by longest waiting * Fix up leftover todos/unused data structures * Change get_service_node_list_state to print_sn
2018-08-15 04:59:05 +02:00
}
//-----------------------------------------------------------------------------------------------
Unify and move responsibility of voting to quorum_cop (#615) * Unify checkpointing and uptime quorums * Begin making checkpoints cull old votes/checkpoints * Begin rehaul of service node code out of core, to assist checkpoints * Begin overhaul of votes to move resposibility into quorum_cop * Update testing suite to work with the new system * Remove vote culling from checkpoints and into voting_pool * Fix bugs making integration deregistration fail * Votes don't always specify an index in the validators * Update tests for validator index member change * Rename deregister to voting, fix subtle hashing bug Update the deregister hash derivation to use uint32_t as originally set not uint64_t otherwise this affects the result and produces different results. * Remove un-needed nettype from vote pool * PR review, use <algorithms> * Rename uptime_deregister/uptime quorums to just deregister quorums * Remove unused add_deregister_vote, move side effect out of macro
2019-05-31 03:06:42 +02:00
bool core::add_service_node_vote(const service_nodes::quorum_vote_t& vote, vote_verification_context &vvc)
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
{
Optimizations and simplifications This commit makes various simplifications and optimizations, mainly in the service node list code. All in all, this shaves about 5% of the CPU time used for re-processing the entire service node list. In particular: - changed m_state_history from a std::vector to a std::set that sorts on height. This is responsible for the bulk of the CPU reduction by significant reducing the amount of work required for checkpoint culling, which has to shuffle a lot of `state_t`s around when removing from the midde of a vector. - the above also allows replacing the binary-search `std::lower_bound` complexity with a much simpler `find()`. - since the history is now always sorted, removed the error messages related to sorting that either can't happen (on store) or don't matter (on load). - Added some converting constructors to simplify code (for example, a `state_t` can now be very usefully constructed from an r-value `state_serialized`). - Many construct + moves (and a couple construct + copy) were replaced with in-place constructions. - removed some unused variables
2019-08-11 05:46:59 +02:00
return m_quorum_cop.handle_vote(vote, vvc);
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
}
First service node test (#190) * core_tests: added service node tests * core_tests: check balance after registration tx
2018-08-29 07:53:13 +02:00
//-----------------------------------------------------------------------------------------------
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
uint32_t core::get_blockchain_pruning_seed() const
{
return get_blockchain_storage().get_blockchain_pruning_seed();
}
//-----------------------------------------------------------------------------------------------
bool core::prune_blockchain(uint32_t pruning_seed)
{
return get_blockchain_storage().prune_blockchain(pruning_seed);
}
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
//-----------------------------------------------------------------------------------------------
Add start_time to get_info methods and show uptime
2017-01-09 00:50:29 +01:00
std::time_t core::get_start_time() const
{
return start_time;
}
//-----------------------------------------------------------------------------------------------
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
void core::graceful_exit()
{
raise(SIGTERM);
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Reference in a new issue Copy permalink