oxen-io/oxen-core
2
1
Fork 0
mirror of https://github.com/oxen-io/oxen-core.git synced 2023-12-14 02:22:56 +01:00
Code Issues Projects Releases Wiki Activity
oxen-core/src/cryptonote_core/cryptonote_core.cpp

2698 lines
117 KiB
C++
Raw Permalink Normal View History

Update monero copyright to 2019 pre-emptively to make merge simpler
2019-04-12 06:36:43 +02:00
// Copyright (c) 2014-2019, The Monero Project
First pass Monero to Loki refs
2018-04-10 06:49:20 +02:00
// Copyright (c) 2018, The Loki Project
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// All rights reserved.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// Redistribution and use in source and binary forms, with or without modification, are
// permitted provided that the following conditions are met:
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// 1. Redistributions of source code must retain the above copyright notice, this list of
// conditions and the following disclaimer.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
// of conditions and the following disclaimer in the documentation and/or other
// materials provided with the distribution.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// 3. Neither the name of the copyright holder nor the names of its contributors may be
// used to endorse or promote products derived from this software without specific
// prior written permission.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
//
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
moved all stuff to github
2014-03-03 23:07:58 +01:00
run clang format
2023-04-13 15:50:13 +02:00
#include <oxenc/base32z.h>
move includes around to lessen overall load
2017-11-25 23:25:05 +01:00
#include <boost/algorithm/string.hpp>
run clang format
2023-04-13 15:50:13 +02:00
#include <iomanip>
#include <unordered_set>
move includes around to lessen overall load
2017-11-25 23:25:05 +01:00
Move epee includes under "epee/..." This ends epee's include pollution.
2020-10-24 00:49:42 +02:00
#include "epee/string_tools.h"
moved all stuff to github
2014-03-03 23:07:58 +01:00
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
extern "C" {
#include <sodium.h>
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
#ifdef ENABLE_SYSTEMD
run clang format
2023-04-13 15:50:13 +02:00
#include <systemd/sd-daemon.h>
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
#endif
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
#include <fmt/color.h>
#include <oxenmq/fmt.h>
Remove vendored SQLite, rely on system installed lib
2020-02-03 03:03:32 +01:00
#include <sqlite3.h>
run clang format
2023-04-13 15:50:13 +02:00
#include <csignal>
#include "blockchain_db/blockchain_db.h"
#include "blockchain_db/sqlite/db_sqlite.h"
#include "checkpoints/checkpoints.h"
#include "common/base58.h"
#include "common/command_line.h"
Split up common/util.h common/util.h has become something of a dumping ground of random functions. This splits them up a little by moving the filesystem bits to common/file.h, the sha256sum functions to common/sha256sum.h, and the (singleton) signal handler to common/signal_handler.h.
2020-05-30 06:06:16 +02:00
#include "common/file.h"
Eliminate most << output operators Replace (nearly) everything with fmt formatting. Some crap in wallet2 remains that I'm not going to bother with.
2022-10-08 04:30:29 +02:00
#include "common/fs-format.h"
run clang format
2023-04-13 15:50:13 +02:00
#include "common/hex.h"
#include "common/i18n.h"
#include "common/notify.h"
Split up common/util.h common/util.h has become something of a dumping ground of random functions. This splits them up a little by moving the filesystem bits to common/file.h, the sha256sum functions to common/sha256sum.h, and the (singleton) signal handler to common/signal_handler.h.
2020-05-30 06:06:16 +02:00
#include "common/sha256sum.h"
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
#include "common/threadpool.h"
moved all stuff to github
2014-03-03 23:07:58 +01:00
#include "crypto/crypto.h"
Add snode revision soft forks & drop hard fork voting code Snode revisions are a secondary version that let us put out a mandatory update for snodes that isn't a hardfork (and so isn't mandatory for wallets/exchanges/etc.). The main point of this is to let us make a 9.2.0 release that includes new mandatory minimums of future versions of storage server (2.2.0) and lokinet (0.9.4) to bring upgrades to the network. This slightly changes the HF7 blocks to 0 (instead of 1) because, apparently, we weren't properly checking the HF value of the pre-first-hf genesis block at all before. (In practice this changes nothing because genesis blocks are v7 anyway). This also changes (slightly) how we check for hard forks: now if we skip some hard forks then we still want to know the height when a hard fork triggers. For example, if the hf tables contains {7,14} then we still need to know that the HF14 block height also is the height that activates HF9, 10, etc.
2021-06-18 01:06:51 +02:00
#include "cryptonote_basic/hardfork.h"
run clang format
2023-04-13 15:50:13 +02:00
#include "cryptonote_config.h"
#include "cryptonote_core.h"
Move epee includes under "epee/..." This ends epee's include pollution.
2020-10-24 00:49:42 +02:00
#include "epee/memwipe.h"
#include "epee/net/local_ip.h"
run clang format
2023-04-13 15:50:13 +02:00
#include "epee/warnings.h"
Logging Refactor This replaces the current epee logging system with our oxen::log library. It replaces the easylogging library with spdlog, removes the macros and replaces with functions and standardises how we call the logs.
2022-09-13 08:57:56 +02:00
#include "logging/oxen_logger.h"
run clang format
2023-04-13 15:50:13 +02:00
#include "ringct/rctSigs.h"
#include "ringct/rctTypes.h"
#include "uptime_proof.h"
#include "version.h"
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 17:34:23 +01:00
moved all stuff to github
2014-03-03 23:07:58 +01:00
DISABLE_VS_WARNINGS(4355)
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 21:15:52 +01:00
#define BAD_SEMANTICS_TXES_MAX_SIZE 100
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
// basically at least how many bytes the block itself serializes to without the miner tx
#define BLOCK_SIZE_SANITY_LEEWAY 100
run clang format
2023-04-13 15:50:13 +02:00
namespace cryptonote {
static auto logcat = log::Cat("cn");
static auto omqlogcat = log::Cat("omq");
const command_line::arg_descriptor<bool, false> arg_testnet_on = {
"testnet", "Run on testnet. The wallet must be launched with --testnet flag.", false};
const command_line::arg_descriptor<bool, false> arg_devnet_on = {
"devnet", "Run on devnet. The wallet must be launched with --devnet flag.", false};
const command_line::arg_descriptor<bool> arg_regtest_on = {
"regtest", "Run in a regression testing mode.", false};
const command_line::arg_descriptor<bool> arg_keep_fakechain = {
"keep-fakechain", "Don't delete any existing database when in fakechain mode.", false};
const command_line::arg_descriptor<difficulty_type> arg_fixed_difficulty = {
"fixed-difficulty", "Fixed difficulty used for testing.", 0};
const command_line::arg_descriptor<bool> arg_dev_allow_local = {
"dev-allow-local-ips",
"Allow a local IPs for local and received service node public IP (for local testing only)",
false};
const command_line::arg_descriptor<std::string, false, true, 2> arg_data_dir = {
"data-dir",
"Specify data directory",
tools::get_default_data_dir().u8string(),
{{&arg_testnet_on, &arg_devnet_on}},
[](std::array<bool, 2> testnet_devnet, bool defaulted, std::string val) -> std::string {
if (testnet_devnet[0])
return (fs::u8path(val) / "testnet").u8string();
else if (testnet_devnet[1])
return (fs::u8path(val) / "devnet").u8string();
return val;
}};
const command_line::arg_descriptor<bool> arg_offline = {
"offline", "Do not listen for peers, nor connect to any"};
const command_line::arg_descriptor<size_t> arg_block_download_max_size = {
"block-download-max-size",
"Set maximum size of block download queue in bytes (0 for default)",
0};
static const command_line::arg_descriptor<bool> arg_test_drop_download = {
"test-drop-download",
"For net tests: in download, discard ALL blocks instead checking/saving them (very fast)"};
static const command_line::arg_descriptor<uint64_t> arg_test_drop_download_height = {
"test-drop-download-height",
"Like test-drop-download but discards only after around certain height",
0};
static const command_line::arg_descriptor<uint64_t> arg_fast_block_sync = {
"fast-block-sync", "Sync up most of the way by using embedded, known block hashes.", 1};
static const command_line::arg_descriptor<uint64_t> arg_prep_blocks_threads = {
"prep-blocks-threads",
"Max number of threads to use when preparing block hashes in groups.",
4};
static const command_line::arg_descriptor<uint64_t> arg_show_time_stats = {
"show-time-stats",
"Show time-stats when processing blocks/txs and disk synchronization.",
0};
static const command_line::arg_descriptor<size_t> arg_block_sync_size = {
"block-sync-size",
"How many blocks to sync at once during chain synchronization (0 = adaptive).",
0};
static const command_line::arg_descriptor<bool> arg_pad_transactions = {
"pad-transactions",
"Pad relayed transactions to help defend against traffic volume analysis",
false};
static const command_line::arg_descriptor<size_t> arg_max_txpool_weight = {
"max-txpool-weight", "Set maximum txpool weight in bytes.", DEFAULT_MEMPOOL_MAX_WEIGHT};
static const command_line::arg_descriptor<bool> arg_service_node = {
"service-node", "Run as a service node, option 'service-node-public-ip' must be set"};
static const command_line::arg_descriptor<std::string> arg_public_ip = {
"service-node-public-ip",
"Public IP address on which this service node's services (such as the Loki "
"storage server) are accessible. This IP address will be advertised to the "
"network via the service node uptime proofs. Required if operating as a "
"service node."};
static const command_line::arg_descriptor<uint16_t> arg_storage_server_port = {
"storage-server-port", "Deprecated option, ignored.", 0};
static const command_line::arg_descriptor<uint16_t, false, true, 2> arg_quorumnet_port = {
"quorumnet-port",
"The port on which this service node listen for direct connections from other "
"service nodes for quorum messages. The port must be publicly reachable "
"on the `--service-node-public-ip' address and binds to the p2p IP address."
" Only applies when running as a service node.",
config::QNET_DEFAULT_PORT,
{{&cryptonote::arg_testnet_on, &cryptonote::arg_devnet_on}},
[](std::array<bool, 2> testnet_devnet, bool defaulted, uint16_t val) -> uint16_t {
return defaulted && testnet_devnet[0] ? config::testnet::QNET_DEFAULT_PORT
: defaulted && testnet_devnet[1] ? config::devnet::QNET_DEFAULT_PORT
: val;
}};
static const command_line::arg_descriptor<bool> arg_omq_quorumnet_public{
"lmq-public-quorumnet",
"Allow the curve-enabled quorumnet address (for a Service Node) to be used for public RPC "
"commands as if passed to --lmq-curve-public. "
"Note that even without this option the quorumnet port can be used for RPC commands by "
"--lmq-admin and --lmq-user pubkeys.",
false};
static const command_line::arg_descriptor<std::string> arg_block_notify = {
"block-notify",
"Run a program for each new block, '%s' will be replaced by the block hash",
""};
static const command_line::arg_descriptor<bool> arg_prune_blockchain = {
"prune-blockchain", "Prune blockchain", false};
static const command_line::arg_descriptor<std::string> arg_reorg_notify = {
"reorg-notify",
"Run a program for each reorg, '%s' will be replaced by the split height, "
"'%h' will be replaced by the new blockchain height, and '%n' will be "
"replaced by the number of new blocks in the new chain",
""};
static const command_line::arg_descriptor<bool> arg_keep_alt_blocks = {
"keep-alt-blocks", "Keep alternative blocks on restart", false};
static const command_line::arg_descriptor<uint64_t> arg_store_quorum_history = {
"store-quorum-history",
"Store the service node quorum history for the last N blocks to allow historic quorum "
"lookups "
"(e.g. by a block explorer). Specify the number of blocks of history to store, or 1 to "
"store "
"the entire history. Requires considerably more memory and block chain storage.",
0};
// Loads stubs that fail if invoked. The stubs are replaced in the
// cryptonote_protocol/quorumnet.cpp glue code.
[[noreturn]] static void need_core_init(std::string_view stub_name) {
throw std::logic_error(
"Internal error: core callback initialization was not performed for "s +
std::string(stub_name));
}
void (*long_poll_trigger)(tx_memory_pool& pool) = [](tx_memory_pool&) {
need_core_init("long_poll_trigger"sv);
};
quorumnet_new_proc* quorumnet_new = [](core&) -> void* { need_core_init("quorumnet_new"sv); };
quorumnet_init_proc* quorumnet_init = [](core&, void*) { need_core_init("quorumnet_init"sv); };
quorumnet_delete_proc* quorumnet_delete = [](void*&) { need_core_init("quorumnet_delete"sv); };
quorumnet_relay_obligation_votes_proc* quorumnet_relay_obligation_votes =
[](void*, const std::vector<service_nodes::quorum_vote_t>&) {
need_core_init("quorumnet_relay_obligation_votes"sv);
};
quorumnet_send_blink_proc* quorumnet_send_blink =
[](core&, const std::string&) -> std::future<std::pair<blink_result, std::string>> {
need_core_init("quorumnet_send_blink"sv);
};
quorumnet_pulse_relay_message_to_quorum_proc* quorumnet_pulse_relay_message_to_quorum =
[](void*, pulse::message const&, service_nodes::quorum const&, bool) -> void {
need_core_init("quorumnet_pulse_relay_message_to_quorum"sv);
};
//-----------------------------------------------------------------------------------------------
core::core() :
m_mempool(m_blockchain_storage),
m_service_node_list(m_blockchain_storage),
m_blockchain_storage(m_mempool, m_service_node_list),
m_quorum_cop(*this),
m_miner(this,
[this](const cryptonote::block& b,
uint64_t height,
unsigned int threads,
crypto::hash& hash) {
hash = cryptonote::get_block_longhash_w_blockchain(
m_nettype, &m_blockchain_storage, b, height, threads);
return true;
}),
m_pprotocol(&m_protocol_stub),
m_starter_message_showed(false),
m_target_blockchain_height(0),
m_last_json_checkpoints_update(0),
m_nettype(network_type::UNDEFINED),
m_last_storage_server_ping(0),
m_last_lokinet_ping(0),
m_pad_transactions(false),
ss_version{0},
lokinet_version{0} {
cryptonote_core: initialize checkpoint flag
2017-07-02 19:42:45 +02:00
m_checkpoints_updating.clear();
run clang format
2023-04-13 15:50:13 +02:00
}
void core::set_cryptonote_protocol(i_cryptonote_protocol* pprotocol) {
if (pprotocol)
m_pprotocol = pprotocol;
moved all stuff to github
2014-03-03 23:07:58 +01:00
else
run clang format
2023-04-13 15:50:13 +02:00
m_pprotocol = &m_protocol_stub;
}
//-----------------------------------------------------------------------------------------------
bool core::update_checkpoints_from_json_file() {
if (m_checkpoints_updating.test_and_set())
return true;
DNS Checkpoint updating-related fixes/changes Only one thread will be doing the updating. Two valid responses must match, and the first two that match will be used.
2015-04-22 10:36:39 +02:00
Service Node Checkpoint Storage (#579) * Add functions for storing checkpoints to the DB * Allocate the DB entry on the stack instead of heap * Add virtual overrides for new checkpoint functions * Clean up for pull request, simplify some logic * Revise API to include height in checkpoint header * Move log to top of function even if early exit * Begin moving checkpoints to db * Allow storing of checkpoints to DB * Cleanup for code reviewer, fix unit tests * Fix tests, fix casting issues * Don't use DUPSORT, use height->checkpoint mapping in DB * Remove if 0 disabling checkpoint vote, we already check HF12 * Fix unit test infinite loop * Update db schemas to match blk_checkpoint_header * Code review
2019-05-01 07:57:41 +02:00
// load json checkpoints every 10min and verify them with respect to what blocks we already have
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 07:01:58 +02:00
bool res = true;
run clang format
2023-04-13 15:50:13 +02:00
if (time(NULL) - m_last_json_checkpoints_update >= 600) {
res = m_blockchain_storage.update_checkpoints_from_json_file(m_checkpoints_path);
m_last_json_checkpoints_update = time(NULL);
various changes to runtime checkpoint updating json checkpoints will be checked every 10 minutes, dns every 60. json checkpoints always enforced, dns still with flag. conflicting checkpoints is hard fail, but soft if dns enforce flag not set and dns checkpoints are wonky.
2014-09-29 22:30:47 +02:00
}
DNS Checkpoint updating-related fixes/changes Only one thread will be doing the updating. Two valid responses must match, and the first two that match will be used.
2015-04-22 10:36:39 +02:00
m_checkpoints_updating.clear();
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
// if anything fishy happened getting new checkpoints, bring down the house
run clang format
2023-04-13 15:50:13 +02:00
if (!res) {
graceful_exit();
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
}
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 07:01:58 +02:00
return res;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------
void core::stop() {
daemon: stop miner before we bring the whole thing down This avoids the miner erroring out trying to submit blocks to a core that's already shut down (and avoids pegging the CPU while we're busy shutting down).
2018-11-30 15:53:38 +01:00
m_miner.stop();
Fix delayed exit when syncing
2016-12-04 13:27:45 +01:00
m_blockchain_storage.cancel();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------
void core::init_options(boost::program_options::options_description& desc) {
options: remove testnet-* options
2018-01-21 16:29:55 +01:00
command_line::add_arg(desc, arg_data_dir);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
command_line::add_arg(desc, arg_test_drop_download);
command_line::add_arg(desc, arg_test_drop_download_height);
command_line::add_arg(desc, arg_testnet_on);
Rename stagenet to devnet
2020-08-11 23:41:54 +02:00
command_line::add_arg(desc, arg_devnet_on);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
command_line::add_arg(desc, arg_regtest_on);
Add a --keep-fakechain option to keep fakechain databases This is handy when doing tests that generate a lot of transactions, since that takes time it's preferable to re-use the database for future runs.
2019-11-01 16:16:37 +01:00
command_line::add_arg(desc, arg_keep_fakechain);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
command_line::add_arg(desc, arg_fixed_difficulty);
Add --dev-allow-local-ip flag for allowing local network public-ip This is very useful for testing. Also adds a suitable scary warning at startup when used.
2019-09-17 03:20:15 +02:00
command_line::add_arg(desc, arg_dev_allow_local);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
command_line::add_arg(desc, arg_prep_blocks_threads);
command_line::add_arg(desc, arg_fast_block_sync);
command_line::add_arg(desc, arg_show_time_stats);
command_line::add_arg(desc, arg_block_sync_size);
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
command_line::add_arg(desc, arg_offline);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
command_line::add_arg(desc, arg_block_download_max_size);
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
command_line::add_arg(desc, arg_max_txpool_weight);
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
command_line::add_arg(desc, arg_service_node);
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
command_line::add_arg(desc, arg_public_ip);
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
command_line::add_arg(desc, arg_storage_server_port);
command_line::add_arg(desc, arg_quorumnet_port);
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 23:27:25 +01:00
command_line::add_arg(desc, arg_pad_transactions);
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 22:18:08 +02:00
command_line::add_arg(desc, arg_block_notify);
run clang format
2023-04-13 15:50:13 +02:00
#if 0 // TODO(oxen): Pruning not supported because of Service Node List
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
command_line::add_arg(desc, arg_prune_blockchain);
Disable pruning in Loki, it is not supported yet
2019-10-23 04:44:56 +02:00
#endif
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 15:29:28 +01:00
command_line::add_arg(desc, arg_reorg_notify);
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
command_line::add_arg(desc, arg_keep_alt_blocks);
core: add p2p bind port options from net_node They're now used by core to determine the data directory to use for the txpool directory. This fixes an assert in the core tests, which don't use the RPC server, which normally initializes the P2P code.
2017-02-09 22:29:35 +01:00
(Low priority) Add disabled-by-default quorum storage support (#702) * Add disabled-by-default quorum storage support This adds support for storing N blocks of recent expired quorum state history in lokid (and dumping to/from the lmdb). This isn't useful for regular nodes (and that's why we don't store it) but is incredibly useful for the block explorer to be able to report *which* node got deregistered/decommissioned/etc. by a given state_change tx. * Fix quorum states copy
2019-07-17 08:01:54 +02:00
command_line::add_arg(desc, arg_store_quorum_history);
lmq -> omq internal rename More rebrand updates to rename lmq (lokimq) internals with omq (oxenmq).
2021-04-08 18:38:51 +02:00
command_line::add_arg(desc, arg_omq_quorumnet_public);
Allow passing custom hardfork heights through cmdline Expected format: <fork_version>:<fork_height>, ... Example: 7:0, 8:10, 9:20, 10:100
2018-11-15 07:17:51 +01:00
Fix core_tests breaking on startup You're wondering how this fixes core tests, aren't you... It prevents the miner (initialized by cryptonote::core) from breaking trying to access arguments that were not added. Since the tests don't use the miner directly, it makes more sense to have cryptonote_core add those, since it also uses the miner.
2017-02-21 16:06:07 +01:00
miner::init_options(desc);
move db specific options to BlockchainDB Avoids common depending on blockchain_db, which can cause link errors.
2017-08-29 12:14:42 +02:00
BlockchainDB::init_options(desc);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::handle_command_line(const boost::program_options::variables_map& vm) {
if (m_nettype != network_type::FAKECHAIN) {
const bool testnet = command_line::get_arg(vm, arg_testnet_on);
const bool devnet = command_line::get_arg(vm, arg_devnet_on);
m_nettype = testnet ? network_type::TESTNET
: devnet ? network_type::DEVNET
: network_type::MAINNET;
Stagenet
2018-02-16 12:04:04 +01:00
}
Avoid calling get_net_config() too early m_nettype won't be set properly during construction so we can't call get_net_config() in the constructor arguments here; set a reasonable value and then update during actual initialization instead.
2021-03-25 16:44:30 +01:00
m_check_uptime_proof_interval.interval(get_net_config().UPTIME_PROOF_CHECK_INTERVAL);
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
m_config_folder = fs::u8path(command_line::get_arg(vm, arg_data_dir));
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
test_drop_download_height(command_line::get_arg(vm, arg_test_drop_download_height));
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 23:27:25 +01:00
m_pad_transactions = get_arg(vm, arg_pad_transactions);
core: make --offline also disable DNS lookups
2017-11-30 16:35:52 +01:00
m_offline = get_arg(vm, arg_offline);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
if (command_line::get_arg(vm, arg_test_drop_download) == true)
run clang format
2023-04-13 15:50:13 +02:00
test_drop_download();
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
Add --dev-allow-local-ip flag for allowing local network public-ip This is very useful for testing. Also adds a suitable scary warning at startup when used.
2019-09-17 03:20:15 +02:00
if (command_line::get_arg(vm, arg_dev_allow_local))
run clang format
2023-04-13 15:50:13 +02:00
m_service_node_list.debug_allow_local_ips = true;
Add --dev-allow-local-ip flag for allowing local network public-ip This is very useful for testing. Also adds a suitable scary warning at startup when used.
2019-09-17 03:20:15 +02:00
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
m_service_node = command_line::get_arg(vm, arg_service_node);
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (m_service_node) {
run clang format
2023-04-13 15:50:13 +02:00
/// TODO: parse these options early, before we start p2p server etc?
m_quorumnet_port = command_line::get_arg(vm, arg_quorumnet_port);
bool args_okay = true;
if (m_quorumnet_port == 0) {
log::error(
logcat,
"Quorumnet port cannot be 0; please specify a valid port to listen on with: "
"'--{} <port>'",
arg_quorumnet_port.name);
args_okay = false;
Make --service-node give more verbose failure flags (#630)
2019-06-06 09:39:54 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
const std::string pub_ip = command_line::get_arg(vm, arg_public_ip);
if (pub_ip.size()) {
if (!epee::string_tools::get_ip_int32_from_string(m_sn_public_ip, pub_ip)) {
log::error(logcat, "Unable to parse IPv4 public address from: {}", pub_ip);
args_okay = false;
}
if (!epee::net_utils::is_ip_public(m_sn_public_ip)) {
if (m_service_node_list.debug_allow_local_ips) {
log::warning(
logcat,
"Address given for public-ip is not public; allowing it because "
"dev-allow-local-ips was specified. This service node WILL NOT WORK ON "
"THE PUBLIC OXEN NETWORK!");
} else {
log::error(
logcat,
"Address given for public-ip is not public: {}",
epee::string_tools::get_ip_string_from_int32(m_sn_public_ip));
args_okay = false;
}
}
} else {
log::error(
logcat,
"Please specify an IPv4 public address which the service node & storage server "
"is accessible from with: '--{} <ip address>'",
arg_public_ip.name);
lmq -> omq internal rename More rebrand updates to rename lmq (lokimq) internals with omq (oxenmq).
2021-04-08 18:38:51 +02:00
args_okay = false;
Make --service-node give more verbose failure flags (#630)
2019-06-06 09:39:54 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
if (!args_okay) {
log::error(
logcat,
"IMPORTANT: One or more required service node-related configuration "
"settings/options were omitted or invalid please fix them and restart oxend.");
return false;
}
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
uint64_t core::get_current_blockchain_height() const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.get_current_blockchain_height();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
std::pair<uint64_t, crypto::hash> core::get_blockchain_top() const {
Minor api improvement: return pair instead of two output params
2021-08-05 02:27:32 +02:00
std::pair<uint64_t, crypto::hash> result;
result.second = m_blockchain_storage.get_tail_id(result.first);
return result;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_blocks(
uint64_t start_offset,
size_t count,
std::vector<std::pair<std::string, block>>& blocks,
std::vector<std::string>& txs) const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.get_blocks(start_offset, count, blocks, txs);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_blocks(
uint64_t start_offset,
size_t count,
std::vector<std::pair<std::string, block>>& blocks) const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.get_blocks(start_offset, count, blocks);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_blocks(uint64_t start_offset, size_t count, std::vector<block>& blocks) const {
Pulse: Pull blocks by header in rescan, avoid extra copies
2020-09-02 06:21:44 +02:00
return m_blockchain_storage.get_blocks_only(start_offset, count, blocks);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_blocks(
const std::vector<crypto::hash>& block_ids,
std::vector<std::pair<std::string, block>> blocks,
std::unordered_set<crypto::hash>* missed_bs) const {
RPC: add support for fetching transaction pool This drops the dedicated GET_TRANSACTION_POOL endpoint entirely and folds the txpool fetching into GET_TRANSACTIONS via a new `memory_pool` parameter. (This is backwards incompatible, of course, but the results may not be too harmful: the wallet doesn't actually use this endpoint, so it mainly affects internal oxend commands (fixed here) and the block explorer (which is fairly easily fixed). This also removes some of the useless GET_TRANSACTIONS bits such as decode_as_json. `fee` and `burned` are now always provided in the return (for both specific requested transactions and mempool transactions). As a side effect, this code ventured deep into core/blockchain in terms of dealing with missed transactions: previously they were always returned (via lvalue ref) as a vector, and always had to be specified. This changes to take via pointer to an unordered_set, which can be null if you don't care about the missed ones. This reduces several calls that indeed didn't care about collecting the missed values, and improved virtually all the places that *did* care which need to query which ones were missed rather than needing a specific order.
2021-08-19 05:43:26 +02:00
return m_blockchain_storage.get_blocks(block_ids, blocks, missed_bs);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_transactions(
const std::vector<crypto::hash>& txs_ids,
std::vector<std::string>& txs,
std::unordered_set<crypto::hash>* missed_txs) const {
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 17:05:55 +01:00
return m_blockchain_storage.get_transactions_blobs(txs_ids, txs, missed_txs);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_split_transactions_blobs(
const std::vector<crypto::hash>& txs_ids,
std::vector<std::tuple<crypto::hash, std::string, crypto::hash, std::string>>& txs,
std::unordered_set<crypto::hash>* missed_txs) const {
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
return m_blockchain_storage.get_split_transactions_blobs(txs_ids, txs, missed_txs);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_transactions(
const std::vector<crypto::hash>& txs_ids,
std::vector<transaction>& txs,
std::unordered_set<crypto::hash>* missed_txs) const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.get_transactions(txs_ids, txs, missed_txs);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_alternative_blocks(std::vector<block>& blocks) const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.get_alternative_blocks(blocks);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
size_t core::get_alternative_blocks_count() const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.get_alternative_blocks_count();
run clang format
2023-04-13 15:50:13 +02:00
}
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
run clang format
2023-04-13 15:50:13 +02:00
static std::string time_ago_str(time_t now, time_t then) {
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
if (then >= now)
run clang format
2023-04-13 15:50:13 +02:00
return "now"s;
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
if (then == 0)
run clang format
2023-04-13 15:50:13 +02:00
return "never"s;
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
int seconds = now - then;
if (seconds >= 60)
run clang format
2023-04-13 15:50:13 +02:00
return std::to_string(seconds / 60) + "m" + std::to_string(seconds % 60) + "s";
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
return std::to_string(seconds % 60) + "s";
run clang format
2023-04-13 15:50:13 +02:00
}
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
run clang format
2023-04-13 15:50:13 +02:00
// Returns a bool on whether the service node is currently active
bool core::is_active_sn() const {
quorumnet message for timestamp requests random sampling of service nodes, call timestamp lmq message checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user tracks external timesync status and timestamp participation of service nodes clean up includes new template struct for participation history, individual types for participation entry refactor checking participation update select_randomly, move the testing for variance overflow version locks, bump to 8.1.5 explicit casting for mac & clang note to remove after hard fork timestamp debugging log messages debugging messages for before timesync - before message sent logging errord with compiling print version and change add_command to add_request_command log if statement test std::to_string replaced with tools::view_guts for x25519 key check if my sn is active before sending timestamp requests logging the failures checking if statement for success of message more logging, if guards arn't passing more logging, successfully tests if service node might be out of sync more tests before we decide we are out of sync logging output if sn isn't passing tests if check_participation fails then disconnect print timestamp status remove saving variance from the participation history reduce MIN_TIME_IN_S_BEFORE_VOTING reset participation history on recommission undo reduction in startup time reduce log levels Set hardfork time in testnet
2020-12-01 01:56:43 +01:00
auto info = get_my_sn_info();
return (info && info->is_active());
run clang format
2023-04-13 15:50:13 +02:00
}
quorumnet message for timestamp requests random sampling of service nodes, call timestamp lmq message checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user tracks external timesync status and timestamp participation of service nodes clean up includes new template struct for participation history, individual types for participation entry refactor checking participation update select_randomly, move the testing for variance overflow version locks, bump to 8.1.5 explicit casting for mac & clang note to remove after hard fork timestamp debugging log messages debugging messages for before timesync - before message sent logging errord with compiling print version and change add_command to add_request_command log if statement test std::to_string replaced with tools::view_guts for x25519 key check if my sn is active before sending timestamp requests logging the failures checking if statement for success of message more logging, if guards arn't passing more logging, successfully tests if service node might be out of sync more tests before we decide we are out of sync logging output if sn isn't passing tests if check_participation fails then disconnect print timestamp status remove saving variance from the participation history reduce MIN_TIME_IN_S_BEFORE_VOTING reset participation history on recommission undo reduction in startup time reduce log levels Set hardfork time in testnet
2020-12-01 01:56:43 +01:00
run clang format
2023-04-13 15:50:13 +02:00
// Returns the service nodes info
std::shared_ptr<const service_nodes::service_node_info> core::get_my_sn_info() const {
quorumnet message for timestamp requests random sampling of service nodes, call timestamp lmq message checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user tracks external timesync status and timestamp participation of service nodes clean up includes new template struct for participation history, individual types for participation entry refactor checking participation update select_randomly, move the testing for variance overflow version locks, bump to 8.1.5 explicit casting for mac & clang note to remove after hard fork timestamp debugging log messages debugging messages for before timesync - before message sent logging errord with compiling print version and change add_command to add_request_command log if statement test std::to_string replaced with tools::view_guts for x25519 key check if my sn is active before sending timestamp requests logging the failures checking if statement for success of message more logging, if guards arn't passing more logging, successfully tests if service node might be out of sync more tests before we decide we are out of sync logging output if sn isn't passing tests if check_participation fails then disconnect print timestamp status remove saving variance from the participation history reduce MIN_TIME_IN_S_BEFORE_VOTING reset participation history on recommission undo reduction in startup time reduce log levels Set hardfork time in testnet
2020-12-01 01:56:43 +01:00
auto& snl = get_service_node_list();
const auto& pubkey = get_service_keys().pub;
run clang format
2023-04-13 15:50:13 +02:00
auto states = snl.get_service_node_list_state({pubkey});
quorumnet message for timestamp requests random sampling of service nodes, call timestamp lmq message checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user tracks external timesync status and timestamp participation of service nodes clean up includes new template struct for participation history, individual types for participation entry refactor checking participation update select_randomly, move the testing for variance overflow version locks, bump to 8.1.5 explicit casting for mac & clang note to remove after hard fork timestamp debugging log messages debugging messages for before timesync - before message sent logging errord with compiling print version and change add_command to add_request_command log if statement test std::to_string replaced with tools::view_guts for x25519 key check if my sn is active before sending timestamp requests logging the failures checking if statement for success of message more logging, if guards arn't passing more logging, successfully tests if service node might be out of sync more tests before we decide we are out of sync logging output if sn isn't passing tests if check_participation fails then disconnect print timestamp status remove saving variance from the participation history reduce MIN_TIME_IN_S_BEFORE_VOTING reset participation history on recommission undo reduction in startup time reduce log levels Set hardfork time in testnet
2020-12-01 01:56:43 +01:00
if (states.empty())
run clang format
2023-04-13 15:50:13 +02:00
return nullptr;
else {
return states[0].info;
quorumnet message for timestamp requests random sampling of service nodes, call timestamp lmq message checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user tracks external timesync status and timestamp participation of service nodes clean up includes new template struct for participation history, individual types for participation entry refactor checking participation update select_randomly, move the testing for variance overflow version locks, bump to 8.1.5 explicit casting for mac & clang note to remove after hard fork timestamp debugging log messages debugging messages for before timesync - before message sent logging errord with compiling print version and change add_command to add_request_command log if statement test std::to_string replaced with tools::view_guts for x25519 key check if my sn is active before sending timestamp requests logging the failures checking if statement for success of message more logging, if guards arn't passing more logging, successfully tests if service node might be out of sync more tests before we decide we are out of sync logging output if sn isn't passing tests if check_participation fails then disconnect print timestamp status remove saving variance from the participation history reduce MIN_TIME_IN_S_BEFORE_VOTING reset participation history on recommission undo reduction in startup time reduce log levels Set hardfork time in testnet
2020-12-01 01:56:43 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
}
quorumnet message for timestamp requests random sampling of service nodes, call timestamp lmq message checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user tracks external timesync status and timestamp participation of service nodes clean up includes new template struct for participation history, individual types for participation entry refactor checking participation update select_randomly, move the testing for variance overflow version locks, bump to 8.1.5 explicit casting for mac & clang note to remove after hard fork timestamp debugging log messages debugging messages for before timesync - before message sent logging errord with compiling print version and change add_command to add_request_command log if statement test std::to_string replaced with tools::view_guts for x25519 key check if my sn is active before sending timestamp requests logging the failures checking if statement for success of message more logging, if guards arn't passing more logging, successfully tests if service node might be out of sync more tests before we decide we are out of sync logging output if sn isn't passing tests if check_participation fails then disconnect print timestamp status remove saving variance from the participation history reduce MIN_TIME_IN_S_BEFORE_VOTING reset participation history on recommission undo reduction in startup time reduce log levels Set hardfork time in testnet
2020-12-01 01:56:43 +01:00
run clang format
2023-04-13 15:50:13 +02:00
// Returns a string for systemd status notifications such as:
// Height: 1234567, SN: active, proof: 55m12s, storage: 4m48s, lokinet: 47s
std::string core::get_status_string() const {
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
std::string s;
s.reserve(128);
run clang format
2023-04-13 15:50:13 +02:00
s += 'v';
s += OXEN_VERSION_STR;
Add version to beginning of systemd status string (#1092) Lokinet and storage server have this and it's nice to be able to tell at a glance at the service status what lokid version is running.
2020-03-18 09:28:06 +01:00
s += "; Height: ";
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
s += std::to_string(get_blockchain_storage().get_current_blockchain_height());
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
s += ", SN: ";
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (!service_node())
run clang format
2023-04-13 15:50:13 +02:00
s += "no";
else {
auto& snl = get_service_node_list();
const auto& pubkey = get_service_keys().pub;
auto states = snl.get_service_node_list_state({pubkey});
if (states.empty())
s += "not registered";
else {
auto& info = *states[0].info;
if (!info.is_fully_funded())
s += "awaiting contr.";
else if (info.is_active())
s += "active";
else if (info.is_decommissioned())
s += "decomm.";
uint64_t last_proof = 0;
snl.access_proof(pubkey, [&](auto& proof) { last_proof = proof.timestamp; });
s += ", proof: ";
time_t now = std::time(nullptr);
s += time_ago_str(now, last_proof);
s += ", storage: ";
s += time_ago_str(now, m_last_storage_server_ping);
s += ", lokinet: ";
s += time_ago_str(now, m_last_lokinet_ping);
}
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
}
return s;
run clang format
2023-04-13 15:50:13 +02:00
}
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
run clang format
2023-04-13 15:50:13 +02:00
//-----------------------------------------------------------------------------------------------
bool core::init(
const boost::program_options::variables_map& vm,
const cryptonote::test_options* test_options,
const GetCheckpointsCallback& get_checkpoints /* = nullptr */) {
Add start_time to get_info methods and show uptime
2017-01-09 00:50:29 +01:00
start_time = std::time(nullptr);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
const bool regtest = command_line::get_arg(vm, arg_regtest_on);
run clang format
2023-04-13 15:50:13 +02:00
if (test_options != NULL || regtest) {
m_nettype = network_type::FAKECHAIN;
Stagenet
2018-02-16 12:04:04 +01:00
}
Add support for fakechain in wallet
2018-11-21 00:43:09 +01:00
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 23:10:53 +01:00
bool r = handle_command_line(vm);
Incorporate service node ip address into uptime proofs (#622) * Incorporate service node ip address into uptime proofs; expose them using rpc * Check that storage server port is specified in service-node mode * Remove problematic const, rename argument name for storage port, update comments * Validate ip address when receive uptime proof * Better argument names and descriptions
2019-06-04 06:15:16 +02:00
/// Currently terminating before blockchain is initialized results in a crash
/// during deinitialization... TODO: fix that
CHECK_AND_ASSERT_MES(r, false, "Failed to apply command line options.");
Separate data per P2P port poolstate.bin and p2pstate.bin are stored in .bitmonero/ if the default P2P port is being used. If another port is used both files are stored in .bitmonero/PORTNUMBER/.
2017-01-28 01:36:39 +01:00
move db specific options to BlockchainDB Avoids common depending on blockchain_db, which can cause link errors.
2017-08-29 12:14:42 +02:00
std::string db_sync_mode = command_line::get_arg(vm, cryptonote::arg_db_sync_mode);
bool db_salvage = command_line::get_arg(vm, cryptonote::arg_db_salvage) != 0;
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
bool fast_sync = command_line::get_arg(vm, arg_fast_block_sync) != 0;
uint64_t blocks_threads = command_line::get_arg(vm, arg_prep_blocks_threads);
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
size_t max_txpool_weight = command_line::get_arg(vm, arg_max_txpool_weight);
Don't query prune blockchain arg boost cast failure Removing the prune blockchain arg from the command_line and trying to query it causesd a boost any_cast failuree
2019-12-04 06:27:51 +01:00
bool const prune_blockchain = false; /* command_line::get_arg(vm, arg_prune_blockchain); */
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
bool keep_alt_blocks = command_line::get_arg(vm, arg_keep_alt_blocks);
Add a --keep-fakechain option to keep fakechain databases This is handy when doing tests that generate a lot of transactions, since that takes time it's preferable to re-use the database for future runs.
2019-11-01 16:16:37 +01:00
bool keep_fakechain = command_line::get_arg(vm, arg_keep_fakechain);
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
r = init_service_keys();
CHECK_AND_ASSERT_MES(r, false, "Failed to create or load service keys");
run clang format
2023-04-13 15:50:13 +02:00
if (m_service_node) {
// Only use our service keys for our service node if we are running in SN mode:
m_service_node_list.set_my_service_node_keys(&m_service_keys);
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
}
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
auto folder = m_config_folder;
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
if (m_nettype == network_type::FAKECHAIN)
run clang format
2023-04-13 15:50:13 +02:00
folder /= "fake";
core: move the db lock to the data directory Locking just one db turns out to not have been a good idea, since the pool and p2p state fdles have to be used anyway. Also ensure the directory exists before tring to lock.
2016-02-04 19:09:45 +01:00
// make sure the data directory exists, and try to lock it
run clang format
2023-04-13 15:50:13 +02:00
if (std::error_code ec;
!fs::is_directory(folder, ec) && !fs::create_directories(folder, ec) && ec) {
log::error(
logcat,
"Failed to create directory " + folder.u8string() +
(ec ? ": " + ec.message() : ""s));
return false;
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
}
core: move the db lock to the data directory Locking just one db turns out to not have been a good idea, since the pool and p2p state fdles have to be used anyway. Also ensure the directory exists before tring to lock.
2016-02-04 19:09:45 +01:00
Removed Berkeley DB and db switching logic
2019-06-26 16:39:41 +02:00
std::unique_ptr<BlockchainDB> db(new_db());
run clang format
2023-04-13 15:50:13 +02:00
if (!db) {
log::error(logcat, "Failed to initialize a database");
return false;
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
}
rebrand lns -> ons
2021-02-12 05:19:30 +01:00
auto ons_db_file_path = folder / "ons.db";
run clang format
2023-04-13 15:50:13 +02:00
if (fs::exists(folder / "lns.db"))
ons_db_file_path = folder / "lns.db";
rename ons_owners_by_names merge wallet and session name check
2021-04-14 09:25:53 +02:00
Batching of service node rewards This updates the coinbase transactions to reward service nodes periodically rather than every block. If you recieve a service node reward this reward will be delayed x blocks, if you receive another reward to the same wallet before those blocks have been completed it will be added to your total and all will be paid out after those x blocks has passed. For example if our batching interval is 2 blocks: Block 1 - Address A receives reward of 10 oxen - added to batch Block 2 - Address A receives reward of 10 oxen - added to batch Block 3 - Address A is paid out 20 oxen. Batching accumulates a small reward for all nodes every block The batching of service node rewards allows us to drip feed rewards to service nodes. Rather than accruing each service node 16.5 oxen every time they are pulse block leader we now reward every node the 16.5 / num_service_nodes every block and pay each wallet the full amount that has been accrued after a period of time (Likely 3.5 days). To spread each payment evenly we now pay the rewards based on the address of the recipient. This modulus of their address determines which block the address should be paid and by setting the interval to our service_node_batching interval we can guarantee they will be paid out regularly and evenly distribute the payments for all wallets over this
2022-04-29 01:51:14 +02:00
auto sqlite_db_file_path = folder / "sqlite.db";
run clang format
2023-04-13 15:50:13 +02:00
if (m_nettype == network_type::FAKECHAIN) {
sqlite_db_file_path = ":memory:";
Batching of service node rewards This updates the coinbase transactions to reward service nodes periodically rather than every block. If you recieve a service node reward this reward will be delayed x blocks, if you receive another reward to the same wallet before those blocks have been completed it will be added to your total and all will be paid out after those x blocks has passed. For example if our batching interval is 2 blocks: Block 1 - Address A receives reward of 10 oxen - added to batch Block 2 - Address A receives reward of 10 oxen - added to batch Block 3 - Address A is paid out 20 oxen. Batching accumulates a small reward for all nodes every block The batching of service node rewards allows us to drip feed rewards to service nodes. Rather than accruing each service node 16.5 oxen every time they are pulse block leader we now reward every node the 16.5 / num_service_nodes every block and pay each wallet the full amount that has been accrued after a period of time (Likely 3.5 days). To spread each payment evenly we now pay the rewards based on the address of the recipient. This modulus of their address determines which block the address should be paid and by setting the interval to our service_node_batching interval we can guarantee they will be paid out regularly and evenly distribute the payments for all wallets over this
2022-04-29 01:51:14 +02:00
}
auto sqliteDB = std::make_shared<cryptonote::BlockchainSQLite>(m_nettype, sqlite_db_file_path);
Don't overwrite mainnet lns.db for test suite/fakechain (#1249) Currently running the test suite obliterates a mainnet lns.db; this fixes it to put the fakechain one under fake/ instead.
2020-09-10 02:02:40 +02:00
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
folder /= db->get_db_name();
Change most oxen::log::whatever to log::whatever oxen::log::info(...), etc. are a bit too verbose; this simplifies them to just `log::info(...)`, etc. by aliasing the `oxen::log` namespace into most of the common namespaces we use in core. This result is usage that is shorter but also reads better: oxen::log::info(logcat, "blah: {}", 42); log::info(logcat, "blah: {}", 42);
2022-09-23 01:27:12 +02:00
log::info(logcat, "Loading blockchain from folder {} ...", folder);
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
// default to fast:async:1 if overridden
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 20:36:51 +02:00
blockchain_db_sync_mode sync_mode = db_defaultsync;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
bool sync_on_blocks = true;
uint64_t sync_threshold = 1;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
run clang format
2023-04-13 15:50:13 +02:00
if (m_nettype == network_type::FAKECHAIN && !keep_fakechain) {
// reset the db by removing the database file before opening it
if (!db->remove_data_file(folder)) {
log::error(logcat, "Failed to remove data file in {}", folder);
return false;
}
fs::remove(ons_db_file_path);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 21:11:49 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
try {
uint64_t db_flags = 0;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
run clang format
2023-04-13 15:50:13 +02:00
std::vector<std::string> options;
boost::trim(db_sync_mode);
boost::split(options, db_sync_mode, boost::is_any_of(" :"));
const bool db_sync_mode_is_default =
command_line::is_arg_defaulted(vm, cryptonote::arg_db_sync_mode);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
run clang format
2023-04-13 15:50:13 +02:00
for (const auto& option : options)
log::debug(logcat, "option: {}", option);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 04:38:55 +01:00
// default to fast:async:1
run clang format
2023-04-13 15:50:13 +02:00
uint64_t DEFAULT_FLAGS = DBF_FAST;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
run clang format
2023-04-13 15:50:13 +02:00
if (options.size() == 0) {
// default to fast:async:1
db_flags = DEFAULT_FLAGS;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
bool safemode = false;
if (options.size() >= 1) {
if (options[0] == "safe") {
safemode = true;
db_flags = DBF_SAFE;
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_nosync;
} else if (options[0] == "fast") {
db_flags = DBF_FAST;
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
} else if (options[0] == "fastest") {
db_flags = DBF_FASTEST;
sync_threshold = 1000; // default to fastest:async:1000
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
} else
db_flags = DEFAULT_FLAGS;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
if (options.size() >= 2 && !safemode) {
if (options[1] == "sync")
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_sync;
else if (options[1] == "async")
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
if (options.size() >= 3 && !safemode) {
char* endptr;
uint64_t threshold = strtoull(options[2].c_str(), &endptr, 0);
if (*endptr == '\0' || !strcmp(endptr, "blocks")) {
sync_on_blocks = true;
sync_threshold = threshold;
} else if (!strcmp(endptr, "bytes")) {
sync_on_blocks = false;
sync_threshold = threshold;
} else {
log::error(logcat, "Invalid db sync mode: {}", options[2]);
return false;
}
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 12:09:35 +02:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
run clang format
2023-04-13 15:50:13 +02:00
if (db_salvage)
db_flags |= DBF_SALVAGE;
core: add --db-salvage command line flag Use to load the database when the primary meta page is corrupted
2017-07-31 00:59:52 +02:00
run clang format
2023-04-13 15:50:13 +02:00
db->open(folder, m_nettype, db_flags);
if (!db->m_open)
return false;
} catch (const DB_ERROR& e) {
log::error(logcat, "Error opening database: {}", e.what());
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
return false;
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 16:41:30 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
m_blockchain_storage.set_user_options(
blocks_threads, sync_on_blocks, sync_threshold, sync_mode, fast_sync);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
Rewrite trash tools::Notify This was nasty. Remove --block-rate-notify entirely; it's nearly useless on Oxen. block and reorg notify remain, but now use a post-block hook rather than shoving toxic Notify crap into cryptonote_core headers.
2022-07-27 22:02:12 +02:00
// We need this hook to get added before the block hook below, so that it fires first and
// catches the start of a reorg before the block hook fires for the block in the reorg.
run clang format
2023-04-13 15:50:13 +02:00
try {
if (!command_line::is_arg_defaulted(vm, arg_reorg_notify))
m_blockchain_storage.hook_block_post_add(
[this, notify = tools::Notify(command_line::get_arg(vm, arg_reorg_notify))](
const auto& info) {
if (!info.reorg)
return;
auto h = get_current_blockchain_height();
notify.notify(
"%s", info.split_height, "%h", h, "%n", h - info.split_height);
});
} catch (const std::exception& e) {
log::error(logcat, "Failed to parse reorg notify spec");
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 15:29:28 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
try {
if (!command_line::is_arg_defaulted(vm, arg_block_notify))
m_blockchain_storage.hook_block_post_add(
[notify = tools::Notify(command_line::get_arg(vm, arg_block_notify))](
const auto& info) {
notify.notify("%s", tools::type_to_hex(get_block_hash(info.block)));
});
} catch (const std::exception& e) {
log::error(logcat, "Failed to parse block rate notify spec");
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 18:09:28 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
Add snode revision soft forks & drop hard fork voting code Snode revisions are a secondary version that let us put out a mandatory update for snodes that isn't a hardfork (and so isn't mandatory for wallets/exchanges/etc.). The main point of this is to let us make a 9.2.0 release that includes new mandatory minimums of future versions of storage server (2.2.0) and lokinet (0.9.4) to bring upgrades to the network. This slightly changes the HF7 blocks to 0 (instead of 1) because, apparently, we weren't properly checking the HF value of the pre-first-hf genesis block at all before. (In practice this changes nothing because genesis blocks are v7 anyway). This also changes (slightly) how we check for hard forks: now if we skip some hard forks then we still want to know the height when a hard fork triggers. For example, if the hf tables contains {7,14} then we still need to know that the HF14 block height also is the height that activates HF9, 10, etc.
2021-06-18 01:06:51 +02:00
cryptonote::test_options regtest_test_options{};
run clang format
2023-04-13 15:50:13 +02:00
for (auto [it, end] = get_hard_forks(network_type::MAINNET); it != end; it++) {
regtest_test_options.hard_forks.push_back(hard_fork{
it->version,
it->snode_revision,
regtest_test_options.hard_forks.size(),
std::time(nullptr)});
Add snode revision soft forks & drop hard fork voting code Snode revisions are a secondary version that let us put out a mandatory update for snodes that isn't a hardfork (and so isn't mandatory for wallets/exchanges/etc.). The main point of this is to let us make a 9.2.0 release that includes new mandatory minimums of future versions of storage server (2.2.0) and lokinet (0.9.4) to bring upgrades to the network. This slightly changes the HF7 blocks to 0 (instead of 1) because, apparently, we weren't properly checking the HF value of the pre-first-hf genesis block at all before. (In practice this changes nothing because genesis blocks are v7 anyway). This also changes (slightly) how we check for hard forks: now if we skip some hard forks then we still want to know the height when a hard fork triggers. For example, if the hf tables contains {7,14} then we still need to know that the HF14 block height also is the height that activates HF9, 10, etc.
2021-06-18 01:06:51 +02:00
}
Merge commit '025187e' into LokiMergeUpstream20180821
2018-09-19 06:05:07 +02:00
Service Node Checkpoint Storage (#579) * Add functions for storing checkpoints to the DB * Allocate the DB entry on the stack instead of heap * Add virtual overrides for new checkpoint functions * Clean up for pull request, simplify some logic * Revise API to include height in checkpoint header * Move log to top of function even if early exit * Begin moving checkpoints to db * Allow storing of checkpoints to DB * Cleanup for code reviewer, fix unit tests * Fix tests, fix casting issues * Don't use DUPSORT, use height->checkpoint mapping in DB * Remove if 0 disabling checkpoint vote, we already check HF12 * Fix unit test infinite loop * Update db schemas to match blk_checkpoint_header * Code review
2019-05-01 07:57:41 +02:00
// Service Nodes
run clang format
2023-04-13 15:50:13 +02:00
m_service_node_list.set_quorum_history_storage(
command_line::get_arg(vm, arg_store_quorum_history));
Add and use post-block-added hooks; fix block notify race Add hooks that are called *after* a block is successfully added to the blockchain. This also fixes a race condition with OMQ notify.block subscriptions: because the notification was firing during (instead of after) block addition, the lokinet/storage server receiving the notify would race to fetch the block info, but that request could happen *before* the block addition is finished, ending up with lokinet/SS sometimes getting a stale block.
2022-07-27 21:58:13 +02:00
// NOTE: Implicit dependency. Service node list needs to be hooked before checkpoints.
run clang format
2023-04-13 15:50:13 +02:00
m_blockchain_storage.hook_blockchain_detached(
[this](const auto& info) { m_service_node_list.blockchain_detached(info.height); });
Add and use post-block-added hooks; fix block notify race Add hooks that are called *after* a block is successfully added to the blockchain. This also fixes a race condition with OMQ notify.block subscriptions: because the notification was firing during (instead of after) block addition, the lokinet/storage server receiving the notify would race to fetch the block info, but that request could happen *before* the block addition is finished, ending up with lokinet/SS sometimes getting a stale block.
2022-07-27 21:58:13 +02:00
m_blockchain_storage.hook_init([this] { m_service_node_list.init(); });
run clang format
2023-04-13 15:50:13 +02:00
m_blockchain_storage.hook_validate_miner_tx(
[this](const auto& info) { m_service_node_list.validate_miner_tx(info); });
m_blockchain_storage.hook_alt_block_add(
[this](const auto& info) { m_service_node_list.alt_block_add(info); });
m_blockchain_storage.hook_blockchain_detached([this](const auto& info) {
m_blockchain_storage.sqlite_db()->blockchain_detached(info.height);
});
Add and use post-block-added hooks; fix block notify race Add hooks that are called *after* a block is successfully added to the blockchain. This also fixes a race condition with OMQ notify.block subscriptions: because the notification was firing during (instead of after) block addition, the lokinet/storage server receiving the notify would race to fetch the block info, but that request could happen *before* the block addition is finished, ending up with lokinet/SS sometimes getting a stale block.
2022-07-27 21:58:13 +02:00
// NOTE: There is an implicit dependency on service node lists being hooked first!
m_blockchain_storage.hook_init([this] { m_quorum_cop.init(); });
run clang format
2023-04-13 15:50:13 +02:00
m_blockchain_storage.hook_block_add(
[this](const auto& info) { m_quorum_cop.block_add(info.block, info.txs); });
m_blockchain_storage.hook_blockchain_detached([this](const auto& info) {
m_quorum_cop.blockchain_detached(info.height, info.by_pop_blocks);
});
Add and use post-block-added hooks; fix block notify race Add hooks that are called *after* a block is successfully added to the blockchain. This also fixes a race condition with OMQ notify.block subscriptions: because the notification was firing during (instead of after) block addition, the lokinet/storage server receiving the notify would race to fetch the block info, but that request could happen *before* the block addition is finished, ending up with lokinet/SS sometimes getting a stale block.
2022-07-27 21:58:13 +02:00
run clang format
2023-04-13 15:50:13 +02:00
m_blockchain_storage.hook_block_post_add([this](const auto&) { update_omq_sns(); });
Service Node Checkpoint Storage (#579) * Add functions for storing checkpoints to the DB * Allocate the DB entry on the stack instead of heap * Add virtual overrides for new checkpoint functions * Clean up for pull request, simplify some logic * Revise API to include height in checkpoint header * Move log to top of function even if early exit * Begin moving checkpoints to db * Allow storing of checkpoints to DB * Cleanup for code reviewer, fix unit tests * Fix tests, fix casting issues * Don't use DUPSORT, use height->checkpoint mapping in DB * Remove if 0 disabling checkpoint vote, we already check HF12 * Fix unit test infinite loop * Update db schemas to match blk_checkpoint_header * Code review
2019-05-01 07:57:41 +02:00
// Checkpoints
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
m_checkpoints_path = m_config_folder / fs::u8path(JSON_HASH_FILE_NAME);
Service Node Checkpoint Storage (#579) * Add functions for storing checkpoints to the DB * Allocate the DB entry on the stack instead of heap * Add virtual overrides for new checkpoint functions * Clean up for pull request, simplify some logic * Revise API to include height in checkpoint header * Move log to top of function even if early exit * Begin moving checkpoints to db * Allow storing of checkpoints to DB * Cleanup for code reviewer, fix unit tests * Fix tests, fix casting issues * Don't use DUPSORT, use height->checkpoint mapping in DB * Remove if 0 disabling checkpoint vote, we already check HF12 * Fix unit test infinite loop * Update db schemas to match blk_checkpoint_header * Code review
2019-05-01 07:57:41 +02:00
run clang format
2023-04-13 15:50:13 +02:00
sqlite3* ons_db = ons::init_oxen_name_system(ons_db_file_path, db->is_read_only());
if (!ons_db)
return false;
Batching of service node rewards This updates the coinbase transactions to reward service nodes periodically rather than every block. If you recieve a service node reward this reward will be delayed x blocks, if you receive another reward to the same wallet before those blocks have been completed it will be added to your total and all will be paid out after those x blocks has passed. For example if our batching interval is 2 blocks: Block 1 - Address A receives reward of 10 oxen - added to batch Block 2 - Address A receives reward of 10 oxen - added to batch Block 3 - Address A is paid out 20 oxen. Batching accumulates a small reward for all nodes every block The batching of service node rewards allows us to drip feed rewards to service nodes. Rather than accruing each service node 16.5 oxen every time they are pulse block leader we now reward every node the 16.5 / num_service_nodes every block and pay each wallet the full amount that has been accrued after a period of time (Likely 3.5 days). To spread each payment evenly we now pay the rewards based on the address of the recipient. This modulus of their address determines which block the address should be paid and by setting the interval to our service_node_batching interval we can guarantee they will be paid out regularly and evenly distribute the payments for all wallets over this
2022-04-29 01:51:14 +02:00
Rename lokimq -> oxenmq
2021-01-14 21:28:50 +01:00
init_oxenmq(vm);
Initialize lokimq before blockchain The block-added hook requires lokimq, so initializing it late was causing a segfault when adding the genesis block.
2020-05-11 21:39:20 +02:00
Add support for Loki Name Service on the backend
2019-11-01 01:58:48 +01:00
const difficulty_type fixed_difficulty = command_line::get_arg(vm, arg_fixed_difficulty);
run clang format
2023-04-13 15:50:13 +02:00
r = m_blockchain_storage.init(
db.release(),
ons_db,
std::move(sqliteDB),
m_nettype,
m_offline,
regtest ? &regtest_test_options : test_options,
fixed_difficulty,
get_checkpoints);
Check blockchain init failure before r is overwritten (#937)
2019-11-28 02:11:46 +01:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize blockchain storage");
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
r = m_mempool.init(max_txpool_weight);
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize memory pool");
Fix V1/V2 use of hard fork related parameters Some of it uses hardcoded height, which will need some thinking for next (voted upon) fork.
2016-01-29 16:09:17 +01:00
// now that we have a valid m_blockchain_storage, we can clean out any
// transactions in the pool that do not conform to the current fork
Add snode revision soft forks & drop hard fork voting code Snode revisions are a secondary version that let us put out a mandatory update for snodes that isn't a hardfork (and so isn't mandatory for wallets/exchanges/etc.). The main point of this is to let us make a 9.2.0 release that includes new mandatory minimums of future versions of storage server (2.2.0) and lokinet (0.9.4) to bring upgrades to the network. This slightly changes the HF7 blocks to 0 (instead of 1) because, apparently, we weren't properly checking the HF value of the pre-first-hf genesis block at all before. (In practice this changes nothing because genesis blocks are v7 anyway). This also changes (slightly) how we check for hard forks: now if we skip some hard forks then we still want to know the height when a hard fork triggers. For example, if the hf tables contains {7,14} then we still need to know that the HF14 block height also is the height that activates HF9, 10, etc.
2021-06-18 01:06:51 +02:00
m_mempool.validate(m_blockchain_storage.get_network_version());
Fix V1/V2 use of hard fork related parameters Some of it uses hardcoded height, which will need some thinking for next (voted upon) fork.
2016-01-29 16:09:17 +01:00
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
bool show_time_stats = command_line::get_arg(vm, arg_show_time_stats) != 0;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
m_blockchain_storage.set_show_time_stats(show_time_stats);
moved all stuff to github
2014-03-03 23:07:58 +01:00
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 17:06:43 +02:00
block_sync_size = command_line::get_arg(vm, arg_block_sync_size);
RandomX integration Support RandomX PoW algorithm
2019-04-23 21:32:27 +02:00
if (block_sync_size > BLOCKS_SYNCHRONIZING_MAX_COUNT)
run clang format
2023-04-13 15:50:13 +02:00
log::error(
logcat,
"Error --block-sync-size cannot be greater than {}",
BLOCKS_SYNCHRONIZING_MAX_COUNT);
core: make the sync chunk block count overridable
2016-09-24 18:00:19 +02:00
Change most oxen::log::whatever to log::whatever oxen::log::info(...), etc. are a bit too verbose; this simplifies them to just `log::info(...)`, etc. by aliasing the `oxen::log` namespace into most of the common namespaces we use in core. This result is usage that is shorter but also reads better: oxen::log::info(logcat, "blah: {}", 42); log::info(logcat, "blah: {}", 42);
2022-09-23 01:27:12 +02:00
log::info(logcat, "Loading checkpoints");
run clang format
2023-04-13 15:50:13 +02:00
CHECK_AND_ASSERT_MES(
update_checkpoints_from_json_file(),
false,
"One or more checkpoints loaded from json conflicted with existing checkpoints.");
Fixed segfault with checkpoints loading Bounds checking on blockchain_storage' m_blocks.size() when validating against checkpoints. Also moved initial json & DNS checkpoints load to after blockchain init.
2014-09-27 23:46:12 +02:00
Stagenet
2018-02-16 12:04:04 +01:00
r = m_miner.init(vm, m_nettype);
Merge pull request #749 bfd4a28 Update BlockchainDB documentation (Thomas Winget) 797357e Change Doxyfile, Blockchain not blockchain_storage (Thomas Winget) c835215 remove defunct code from cryptonote::core (Thomas Winget) 50dba6d cryptonote::core doxygen documentation (Thomas Winget) 8ac329d doxygen documentation for difficulty functions (Thomas Winget) 540a76c Move checkpoint functions into checkpoints class (Thomas Winget) 1b0c98e doxygen documentation for checkpoints.{h,cpp} (Thomas Winget) 89c24ac Remove unnecessary or defunct code (Thomas Winget) ab0ed14 doxygen include private and static members (Thomas Winget) 3a48449 Updated documentation for blockchain.* (Thomas Winget)
2016-03-25 07:22:06 +01:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize miner instance");
moved all stuff to github
2014-03-03 23:07:58 +01:00
blockchain: keep alternative blocks in LMDB Alternative blocks are cleared on startup unless --keep-alt-blocks is passed on the command line
2019-05-08 13:11:23 +02:00
if (!keep_alt_blocks && !m_blockchain_storage.get_db().is_read_only())
run clang format
2023-04-13 15:50:13 +02:00
m_blockchain_storage.get_db().drop_alt_blocks();
if (prune_blockchain) {
// display a message if the blockchain is not pruned yet
if (!m_blockchain_storage.get_blockchain_pruning_seed()) {
log::info(logcat, "Pruning blockchain...");
CHECK_AND_ASSERT_MES(
m_blockchain_storage.prune_blockchain(), false, "Failed to prune blockchain");
} else {
CHECK_AND_ASSERT_MES(
m_blockchain_storage.update_blockchain_pruning(),
false,
"Failed to update blockchain pruning");
}
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
}
Remove useless method. Five years into the future this pointless method is still not used. Time to go.
2019-09-17 04:57:15 +02:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
run clang format
2023-04-13 15:50:13 +02:00
/// Loads a key pair from disk, if it exists, otherwise generates a new key pair and saves it to
/// disk.
///
/// get_pubkey - a function taking (privkey &, pubkey &) that sets the pubkey from the privkey;
/// returns true for success/false for failure
/// generate_pair - a void function taking (privkey &, pubkey &) that sets them to the generated
/// values; can throw on error.
template <typename Privkey, typename Pubkey, typename GetPubkey, typename GeneratePair>
bool init_key(
const fs::path& keypath,
Privkey& privkey,
Pubkey& pubkey,
GetPubkey get_pubkey,
GeneratePair generate_pair) {
std::error_code ec;
if (fs::exists(keypath, ec)) {
std::string keystr;
bool r = tools::slurp_file(keypath, keystr);
memcpy(&unwrap(unwrap(privkey)), keystr.data(), sizeof(privkey));
memwipe(&keystr[0], keystr.size());
CHECK_AND_ASSERT_MES(
r, false, "failed to load service node key from " + keypath.u8string());
CHECK_AND_ASSERT_MES(
keystr.size() == sizeof(privkey),
false,
"service node key file " + keypath.u8string() + " has an invalid size");
r = get_pubkey(privkey, pubkey);
CHECK_AND_ASSERT_MES(r, false, "failed to generate pubkey from secret key");
} else {
try {
generate_pair(privkey, pubkey);
} catch (const std::exception& e) {
log::error(logcat, "failed to generate keypair {}", e.what());
return false;
}
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
run clang format
2023-04-13 15:50:13 +02:00
bool r = tools::dump_file(keypath, tools::view_guts(privkey));
CHECK_AND_ASSERT_MES(r, false, "failed to save service node key to " + keypath.u8string());
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
run clang format
2023-04-13 15:50:13 +02:00
fs::permissions(keypath, fs::perms::owner_read, ec);
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
}
Add unified key container This puts the SN pubkey/privkey into a single struct (which have ed25519/x25519 keys added to it in the future), which simplifies various places to just pass the struct rather than storing and passing the pubkey and privkey separately.
2019-10-07 17:16:20 +02:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
run clang format
2023-04-13 15:50:13 +02:00
//-----------------------------------------------------------------------------------------------
bool core::init_service_keys() {
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
auto& keys = m_service_keys;
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
static_assert(
run clang format
2023-04-13 15:50:13 +02:00
sizeof(crypto::ed25519_public_key) == crypto_sign_ed25519_PUBLICKEYBYTES &&
sizeof(crypto::ed25519_secret_key) == crypto_sign_ed25519_SECRETKEYBYTES &&
sizeof(crypto::ed25519_signature) == crypto_sign_BYTES &&
sizeof(crypto::x25519_public_key) == crypto_scalarmult_curve25519_BYTES &&
sizeof(crypto::x25519_secret_key) == crypto_scalarmult_curve25519_BYTES,
"Invalid ed25519/x25519 sizes");
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
Use only ed25519 key by default We currently generate (for service nodes) both a /key and /key_ed25519 file: the former is a SN's primary monero privkey (i.e. the private key point of an Ed25519 key, but with no seed so that it can't do full Ed25519 signatures), the latter is a sodium standard seed+pubkey concatenated file. This commit makes lokid stop generate a new /key file: instead, if it doesn't already exist, we calculate the private key point from the ed25519 key and set that as the monero-style private key. (The pubkeys will be the same). When a /key file exists, we continue to use it for the primary SN keypair so that existing SN keys won't be affected.
2020-08-29 21:56:19 +02:00
// <data>/key_ed25519: Standard ed25519 secret key. We always have this, and generate one if it
// doesn't exist.
//
// As of Loki 8.x, if this exists and `key` doesn't, we use this key for everything. For
// compatibility with earlier versions we also allow `key` to contain a separate monero privkey
// for the SN keypair. (The main difference is that the Monero keypair is unclamped and that it
// only contains the private key value but not the secret key value that we need for full
// Ed25519 signing).
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
//
run clang format
2023-04-13 15:50:13 +02:00
if (!init_key(
m_config_folder / "key_ed25519",
keys.key_ed25519,
keys.pub_ed25519,
[](crypto::ed25519_secret_key& sk, crypto::ed25519_public_key& pk) {
crypto_sign_ed25519_sk_to_pk(pk.data(), sk.data());
return true;
},
[](crypto::ed25519_secret_key& sk, crypto::ed25519_public_key& pk) {
crypto_sign_ed25519_keypair(pk.data(), sk.data());
}))
return false;
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
run clang format
2023-04-13 15:50:13 +02:00
// Standard x25519 keys generated from the ed25519 keypair, used for encrypted communication
// between SNs
Overhaul and fix crypto::{public_key,ec_point,etc.} types - Remove implicit `operator bool` from ec_point/public_key/etc. which was causing all sorts of implicit conversion mess and bugs. - Change ec_point/public_key/etc. to use a `std::array<unsigned char, 32>` (via a base type) rather than a C-array of char that has to be reinterpret_cast<>'ed all over the place. - Add methods to ec_point/public_key/etc. that make it work more like a container of bytes (`.data()`, `.size()`, `operator[]`, `begin()`, `end()`). - Make a generic `crypto::null<T>` that is a constexpr all-0 `T`, rather than the mishmash `crypto::null_hash`, crypto::null_pkey, crypto::hash::null(), and so on. - Replace three metric tons of `crypto::hash blahblah = crypto::null_hash;` with the much simpler `crypto::hash blahblah{};`, because there's no need to make a copy of a null hash in all these cases. (Likewise for a few other null_whatevers). - Remove a whole bunch of `if (blahblah == crypto::null_hash)` and `if (blahblah != crypto::null_hash)` with the more concise `if (!blahblah)` and `if (blahblah)` (which are fine via the newly *explicit* bool conversion operators). - `crypto::signature` becomes a 64-byte container (as above) but with `c()` and `r()` to get the c() and r() data pointers. (Previously `.c` and `.r` were `ec_scalar`s). - Delete with great prejudice CRYPTO_MAKE_COMPARABLE and CRYPTO_MAKE_HASHABLE and all the other utter trash in `crypto/generic-ops.h`. - De-inline functions in very common crypto/*.h files so that they don't have to get compiled 300 times. - Remove the disgusting include-a-C-header-inside-a-C++-namespace garbage from some crypto headers trying to be both a C and *different* C++ header at once. - Remove the toxic, disgusting, shameful `operator&` on ec_scalar, etc. that replace `&x` with `reinterpret_cast x into an unsigned char*`. This was pure toxic waste. - changed some `<<` outputs to fmt - Random other small changes encountered while fixing everything that cascaded out of the above changes.
2022-10-15 03:22:44 +02:00
int rc = crypto_sign_ed25519_pk_to_curve25519(keys.pub_x25519.data(), keys.pub_ed25519.data());
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
CHECK_AND_ASSERT_MES(rc == 0, false, "failed to convert ed25519 pubkey to x25519");
Overhaul and fix crypto::{public_key,ec_point,etc.} types - Remove implicit `operator bool` from ec_point/public_key/etc. which was causing all sorts of implicit conversion mess and bugs. - Change ec_point/public_key/etc. to use a `std::array<unsigned char, 32>` (via a base type) rather than a C-array of char that has to be reinterpret_cast<>'ed all over the place. - Add methods to ec_point/public_key/etc. that make it work more like a container of bytes (`.data()`, `.size()`, `operator[]`, `begin()`, `end()`). - Make a generic `crypto::null<T>` that is a constexpr all-0 `T`, rather than the mishmash `crypto::null_hash`, crypto::null_pkey, crypto::hash::null(), and so on. - Replace three metric tons of `crypto::hash blahblah = crypto::null_hash;` with the much simpler `crypto::hash blahblah{};`, because there's no need to make a copy of a null hash in all these cases. (Likewise for a few other null_whatevers). - Remove a whole bunch of `if (blahblah == crypto::null_hash)` and `if (blahblah != crypto::null_hash)` with the more concise `if (!blahblah)` and `if (blahblah)` (which are fine via the newly *explicit* bool conversion operators). - `crypto::signature` becomes a 64-byte container (as above) but with `c()` and `r()` to get the c() and r() data pointers. (Previously `.c` and `.r` were `ec_scalar`s). - Delete with great prejudice CRYPTO_MAKE_COMPARABLE and CRYPTO_MAKE_HASHABLE and all the other utter trash in `crypto/generic-ops.h`. - De-inline functions in very common crypto/*.h files so that they don't have to get compiled 300 times. - Remove the disgusting include-a-C-header-inside-a-C++-namespace garbage from some crypto headers trying to be both a C and *different* C++ header at once. - Remove the toxic, disgusting, shameful `operator&` on ec_scalar, etc. that replace `&x` with `reinterpret_cast x into an unsigned char*`. This was pure toxic waste. - changed some `<<` outputs to fmt - Random other small changes encountered while fixing everything that cascaded out of the above changes.
2022-10-15 03:22:44 +02:00
crypto_sign_ed25519_sk_to_curve25519(keys.key_x25519.data(), keys.key_ed25519.data());
Add SN ed25519/x25519 pubkeys + quorumnet port in proofs This generates a ed25519 keypair (and from it derives a x25519 keypair) and broadcasts the ed25519 pubkey in HF13 uptime proofs. This auxiliary key will be used both inside lokid (starting in HF14) in places like the upcoming quorumnet code where we need a standard pub/priv keypair that is usable in external tools (e.g. sodium) without having to reimplement the incompatible (though still 25519-based) Monero pubkey format. This pulls it back into HF13 from the quorumnet code because the generation code is ready now, and because there may be opportunities to use this outside of lokid (e.g. in the storage server and in lokinet) before HF14. Broadcasting it earlier also allows us to be ready to go as soon as HF14 hits rather than having to wait for every node to have sent a post-HF14 block uptime proof. For a similar reason this adds a placeholder for the quorumnet port in the uptime proof: currently the value is just set to 0 and ignored, but allowing it to be passed will allow upgraded loki 6.x nodes to start sending it to each other without having to wait for the fork height so that they can start using it immediately when HF14 begins.
2019-10-07 19:23:16 +02:00
Use only ed25519 key by default We currently generate (for service nodes) both a /key and /key_ed25519 file: the former is a SN's primary monero privkey (i.e. the private key point of an Ed25519 key, but with no seed so that it can't do full Ed25519 signatures), the latter is a sodium standard seed+pubkey concatenated file. This commit makes lokid stop generate a new /key file: instead, if it doesn't already exist, we calculate the private key point from the ed25519 key and set that as the monero-style private key. (The pubkeys will be the same). When a /key file exists, we continue to use it for the primary SN keypair so that existing SN keys won't be affected.
2020-08-29 21:56:19 +02:00
// Legacy primary SN key file; we only load this if it exists, otherwise we use `key_ed25519`
// for the primary SN keypair. (This key predates the Ed25519 keys and so is needed for
// backwards compatibility with existing active service nodes.) The legacy key consists of
// *just* the private point, but not the seed, and so cannot be used for full Ed25519 signatures
// (which rely on the seed for signing).
if (m_service_node) {
run clang format
2023-04-13 15:50:13 +02:00
if (std::error_code ec; !fs::exists(m_config_folder / "key", ec)) {
epee::wipeable_string privkey_signhash;
privkey_signhash.resize(crypto_hash_sha512_BYTES);
unsigned char* pk_sh_data = reinterpret_cast<unsigned char*>(privkey_signhash.data());
crypto_hash_sha512(pk_sh_data, keys.key_ed25519.data(), 32 /* first 32 bytes are the seed to be SHA512 hashed (the last 32 are just the pubkey) */);
// Clamp private key (as libsodium does and expects -- see
// https://www.jcraige.com/an-explainer-on-ed25519-clamping if you want the broader
// reasons)
pk_sh_data[0] &= 248;
pk_sh_data[31] &= 63; // (some implementations put 127 here, but with the |64 in the
// next line it is the same thing)
pk_sh_data[31] |= 64;
// Monero crypto requires a pointless check that the secret key is < basepoint, so
// calculate it mod basepoint to make it happy:
sc_reduce32(pk_sh_data);
std::memcpy(keys.key.data(), pk_sh_data, 32);
if (!crypto::secret_key_to_public_key(keys.key, keys.pub))
throw std::runtime_error{"Failed to derive primary key from ed25519 key"};
if (std::memcmp(keys.pub.data(), keys.pub_ed25519.data(), 32))
throw std::runtime_error{
"Internal error: unexpected primary pubkey and ed25519 pubkey mismatch"};
} else if (!init_key(
m_config_folder / "key",
keys.key,
keys.pub,
crypto::secret_key_to_public_key,
[](crypto::secret_key& key, crypto::public_key& pubkey) {
throw std::runtime_error{
"Internal error: old-style public keys are no longer "
"generated"};
}))
return false;
Use only ed25519 key by default We currently generate (for service nodes) both a /key and /key_ed25519 file: the former is a SN's primary monero privkey (i.e. the private key point of an Ed25519 key, but with no seed so that it can't do full Ed25519 signatures), the latter is a sodium standard seed+pubkey concatenated file. This commit makes lokid stop generate a new /key file: instead, if it doesn't already exist, we calculate the private key point from the ed25519 key and set that as the monero-style private key. (The pubkeys will be the same). When a /key file exists, we continue to use it for the primary SN keypair so that existing SN keys won't be affected.
2020-08-29 21:56:19 +02:00
} else {
run clang format
2023-04-13 15:50:13 +02:00
keys.key.zero();
keys.pub.zero();
Use only ed25519 key by default We currently generate (for service nodes) both a /key and /key_ed25519 file: the former is a SN's primary monero privkey (i.e. the private key point of an Ed25519 key, but with no seed so that it can't do full Ed25519 signatures), the latter is a sodium standard seed+pubkey concatenated file. This commit makes lokid stop generate a new /key file: instead, if it doesn't already exist, we calculate the private key point from the ed25519 key and set that as the monero-style private key. (The pubkeys will be the same). When a /key file exists, we continue to use it for the primary SN keypair so that existing SN keys won't be affected.
2020-08-29 21:56:19 +02:00
}
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (m_service_node) {
run clang format
2023-04-13 15:50:13 +02:00
log::info(logcat, fg(fmt::terminal_color::yellow), "Service node public keys:");
log::info(
logcat,
fg(fmt::terminal_color::yellow),
"- primary: {}",
tools::type_to_hex(keys.pub));
log::info(
logcat,
fg(fmt::terminal_color::yellow),
"- ed25519: {}",
tools::type_to_hex(keys.pub_ed25519));
// .snode address is the ed25519 pubkey, encoded with base32z and with .snode appended:
log::info(
logcat,
fg(fmt::terminal_color::yellow),
"- lokinet: {}.snode",
oxenc::to_base32z(tools::view_guts(keys.pub_ed25519)));
log::info(
logcat,
fg(fmt::terminal_color::yellow),
"- x25519: {}",
tools::type_to_hex(keys.pub_x25519));
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
} else {
run clang format
2023-04-13 15:50:13 +02:00
// Only print the x25519 version because it's the only thing useful for a non-SN (for
// encrypted OMQ RPC connections).
log::info(
logcat,
fg(fmt::terminal_color::yellow),
"x25519 public key: {}",
tools::type_to_hex(keys.pub_x25519));
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
}
core, service_node_list: separated address from service node pubkey
2018-07-12 04:07:34 +02:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
run clang format
2023-04-13 15:50:13 +02:00
oxenmq::AuthLevel core::omq_check_access(const crypto::x25519_public_key& pubkey) const {
lmq -> omq internal rename More rebrand updates to rename lmq (lokimq) internals with omq (oxenmq).
2021-04-08 18:38:51 +02:00
auto it = m_omq_auth.find(pubkey);
if (it != m_omq_auth.end())
run clang format
2023-04-13 15:50:13 +02:00
return it->second;
Rename lokimq -> oxenmq
2021-01-14 21:28:50 +01:00
return oxenmq::AuthLevel::denied;
run clang format
2023-04-13 15:50:13 +02:00
}
// Builds an allow function; takes `*this`, the default auth level, and whether this connection
// should allow incoming SN connections.
//
// default_auth should be AuthLevel::denied if only pre-approved connections may connect,
// AuthLevel::basic for public RPC, AuthLevel::admin for a (presumably localhost) unrestricted
// port, and AuthLevel::none for a super restricted mode (generally this is useful when there are
// also SN-restrictions on commands, i.e. for quorumnet).
//
// check_sn is whether we check an incoming key against known service nodes (and thus return
// "true" for the service node access if it checks out).
//
oxenmq::AuthLevel core::omq_allow(
std::string_view ip, std::string_view x25519_pubkey_str, oxenmq::AuthLevel default_auth) {
Rename lokimq -> oxenmq
2021-01-14 21:28:50 +01:00
using namespace oxenmq;
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
AuthLevel auth = default_auth;
if (x25519_pubkey_str.size() == sizeof(crypto::x25519_public_key)) {
run clang format
2023-04-13 15:50:13 +02:00
crypto::x25519_public_key x25519_pubkey;
std::memcpy(x25519_pubkey.data(), x25519_pubkey_str.data(), x25519_pubkey_str.size());
auto user_auth = omq_check_access(x25519_pubkey);
if (user_auth >= AuthLevel::basic) {
if (user_auth > auth)
auth = user_auth;
log::info(log::Cat("omq"), "Incoming {}-authenticated connection", auth);
}
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
run clang format
2023-04-13 15:50:13 +02:00
log::info(
log::Cat("omq"),
"Incoming [{}] curve connection from {}/{}",
auth,
ip,
x25519_pubkey);
} else {
log::info(log::Cat("omq"), "Incoming [{}] plain connection from {}", auth, ip);
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
}
return auth;
run clang format
2023-04-13 15:50:13 +02:00
}
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
run clang format
2023-04-13 15:50:13 +02:00
void core::init_oxenmq(const boost::program_options::variables_map& vm) {
Rename lokimq -> oxenmq
2021-01-14 21:28:50 +01:00
using namespace oxenmq;
Change most oxen::log::whatever to log::whatever oxen::log::info(...), etc. are a bit too verbose; this simplifies them to just `log::info(...)`, etc. by aliasing the `oxen::log` namespace into most of the common namespaces we use in core. This result is usage that is shorter but also reads better: oxen::log::info(logcat, "blah: {}", 42); log::info(logcat, "blah: {}", 42);
2022-09-23 01:27:12 +02:00
log::info(omqlogcat, "Starting oxenmq");
lmq -> omq internal rename More rebrand updates to rename lmq (lokimq) internals with omq (oxenmq).
2021-04-08 18:38:51 +02:00
m_omq = std::make_unique<OxenMQ>(
run clang format
2023-04-13 15:50:13 +02:00
tools::copy_guts(m_service_keys.pub_x25519),
tools::copy_guts(m_service_keys.key_x25519),
m_service_node,
[this](std::string_view x25519_pk) {
return m_service_node_list.remote_lookup(x25519_pk);
},
[](LogLevel omqlevel, const char* file, int line, std::string msg) {
auto level = *oxen::logging::parse_level(omqlevel);
if (omqlogcat->should_log(level))
omqlogcat->log({file, line, "omq"}, level, "{}", msg);
},
oxenmq::LogLevel::trace);
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
lmq -> omq internal rename More rebrand updates to rename lmq (lokimq) internals with omq (oxenmq).
2021-04-08 18:38:51 +02:00
// ping.ping: a simple debugging target for pinging the omq listener
m_omq->add_category("ping", Access{AuthLevel::none})
run clang format
2023-04-13 15:50:13 +02:00
.add_request_command("ping", [](Message& m) {
log::info(log::Cat("omq"), "Received ping from {}", m.conn);
m.send_reply("pong");
});
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
run clang format
2023-04-13 15:50:13 +02:00
if (m_service_node) {
// Service nodes always listen for quorumnet data on the p2p IP, quorumnet port
std::string listen_ip = vm["p2p-bind-ip"].as<std::string>();
if (listen_ip.empty())
listen_ip = "0.0.0.0";
std::string qnet_listen = "tcp://" + listen_ip + ":" + std::to_string(m_quorumnet_port);
log::info(logcat, "- listening on {} (quorumnet)", qnet_listen);
m_omq->listen_curve(
qnet_listen,
[this, public_ = command_line::get_arg(vm, arg_omq_quorumnet_public)](
std::string_view ip, std::string_view pk, bool) {
return omq_allow(ip, pk, public_ ? AuthLevel::basic : AuthLevel::none);
});
m_quorumnet_state = quorumnet_new(*this);
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
}
Backport Fix bl.* commands from pulse to dev bl.* weren't being registered for non-service nodes. This fixes it by moving endpoint setup from quorumnet_new (SN-only) to a new quorumnet_init that does endpoint setup (for both SNs and non-SNs). (This is a backport of the changes applied on the pulse branch to get these fixes into `dev` for testing).
2020-08-21 09:01:37 +02:00
quorumnet_init(*this, m_quorumnet_state);
run clang format
2023-04-13 15:50:13 +02:00
}
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
run clang format
2023-04-13 15:50:13 +02:00
void core::start_oxenmq() {
update_omq_sns(); // Ensure we have SNs set for the current block before starting
Pulse: Add main block gen/validator handshake synchronizing worker
2020-07-27 07:54:02 +02:00
run clang format
2023-04-13 15:50:13 +02:00
if (m_service_node) {
lmq -> omq internal rename More rebrand updates to rename lmq (lokimq) internals with omq (oxenmq).
2021-04-08 18:38:51 +02:00
m_pulse_thread_id = m_omq->add_tagged_thread("pulse");
run clang format
2023-04-13 15:50:13 +02:00
m_omq->add_timer(
[this]() { pulse::main(m_quorumnet_state, *this); },
std::chrono::milliseconds(500),
false,
m_pulse_thread_id);
m_omq->add_timer([this]() { this->check_service_node_time(); }, 5s, false);
}
m_omq->start();
}
//-----------------------------------------------------------------------------------------------
bool core::set_genesis_block(const block& b) {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.reset_and_set_genesis_block(b);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::deinit() {
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
#ifdef ENABLE_SYSTEMD
sd_notify(0, "STOPPING=1\nSTATUS=Shutting down");
#endif
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (m_quorumnet_state)
run clang format
2023-04-13 15:50:13 +02:00
quorumnet_delete(m_quorumnet_state);
lmq -> omq internal rename More rebrand updates to rename lmq (lokimq) internals with omq (oxenmq).
2021-04-08 18:38:51 +02:00
m_omq.reset();
Fix destruction of service node list locking destroyed mutex
2019-01-18 06:16:23 +01:00
m_service_node_list.store();
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
m_miner.stop();
m_mempool.deinit();
Removed all code related to fast_exit
2016-10-03 03:06:55 +02:00
m_blockchain_storage.deinit();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::test_drop_download() {
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
m_test_drop_download = false;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::test_drop_download_height(uint64_t height) {
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
m_test_drop_download_height = height;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_test_drop_download() const {
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
return m_test_drop_download;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_test_drop_download_height() const {
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
if (m_test_drop_download_height == 0)
run clang format
2023-04-13 15:50:13 +02:00
return true;
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
if (get_blockchain_storage().get_current_blockchain_height() <= m_test_drop_download_height)
run clang format
2023-04-13 15:50:13 +02:00
return true;
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 22:28:03 +01:00
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 05:54:39 +01:00
return false;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::parse_incoming_tx_pre(tx_verification_batch_info& tx_info) {
if (tx_info.blob->size() > MAX_TX_SIZE) {
log::info(
logcat, "WRONG TRANSACTION BLOB, too big size {}, rejected", tx_info.blob->size());
tx_info.tvc.m_verifivation_failed = true;
tx_info.tvc.m_too_big = true;
return;
} else if (tx_info.blob->empty()) {
log::info(logcat, "WRONG TRANSACTION BLOB, blob is empty, rejected");
tx_info.tvc.m_verifivation_failed = true;
return;
Detect and don't try to parse empty txes If we pass the blob into parse_and_validate_tx_from_blob we get `deserialization or varint failed` error logs, which are harmless bug annoying. This catches the empty tx blob case earlier and logs at a lesser severity.
2022-06-01 20:50:19 +02:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
Remove useless method parse_tx_from_blob is only ever called from one place and does absolutely nothing except call some other function which can just be done directly.
2020-05-31 19:37:05 +02:00
tx_info.parsed = parse_and_validate_tx_from_blob(*tx_info.blob, tx_info.tx, tx_info.tx_hash);
run clang format
2023-04-13 15:50:13 +02:00
if (!tx_info.parsed) {
log::info(logcat, "WRONG TRANSACTION BLOB, Failed to parse, rejected");
tx_info.tvc.m_verifivation_failed = true;
return;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
// std::cout << "!"<< tx.vin.size() << std::endl;
moved all stuff to github
2014-03-03 23:07:58 +01:00
boost->std: mutex, locks; C++17 lock vars Changes all boost mutexes, locks, and condition_variables to their stl equivalents. Changes all lock_guard/unique_lock/shared_lock to not specify the mutex type (C++17), e.g. std::lock_guard foo{mutex}; instead of std::lock_guard<oh::um::what::mutex> foo{mutex}; Also changes some related boost::thread calls to std::thread, and some related boost chrono calls to stl chrono. boost::thread isn't changed here to std::thread because some of the instances rely on some boost thread extensions.
2020-06-02 01:01:24 +02:00
std::lock_guard lock{bad_semantics_txes_lock};
run clang format
2023-04-13 15:50:13 +02:00
for (int idx = 0; idx < 2; ++idx) {
if (bad_semantics_txes[idx].find(tx_info.tx_hash) != bad_semantics_txes[idx].end()) {
log::info(logcat, "Transaction already seen with bad semantics, rejected");
tx_info.tvc.m_verifivation_failed = true;
return;
}
core: cache tx hashes of failing semantics txes
2017-01-22 17:13:31 +01:00
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
tx_info.result = true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::set_semantics_failed(const crypto::hash& tx_hash) {
Change most oxen::log::whatever to log::whatever oxen::log::info(...), etc. are a bit too verbose; this simplifies them to just `log::info(...)`, etc. by aliasing the `oxen::log` namespace into most of the common namespaces we use in core. This result is usage that is shorter but also reads better: oxen::log::info(logcat, "blah: {}", 42); log::info(logcat, "blah: {}", 42);
2022-09-23 01:27:12 +02:00
log::info(logcat, "WRONG TRANSACTION BLOB, Failed to check tx {} semantic, rejected", tx_hash);
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
bad_semantics_txes_lock.lock();
bad_semantics_txes[0].insert(tx_hash);
run clang format
2023-04-13 15:50:13 +02:00
if (bad_semantics_txes[0].size() >= BAD_SEMANTICS_TXES_MAX_SIZE) {
std::swap(bad_semantics_txes[0], bad_semantics_txes[1]);
bad_semantics_txes[0].clear();
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
}
bad_semantics_txes_lock.unlock();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
static bool is_canonical_bulletproof_layout(const std::vector<rct::Bulletproof>& proofs) {
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
if (proofs.size() != 1)
run clang format
2023-04-13 15:50:13 +02:00
return false;
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 23:24:53 +02:00
const size_t sz = proofs[0].V.size();
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
if (sz == 0 || sz > TX_BULLETPROOF_MAX_OUTPUTS)
run clang format
2023-04-13 15:50:13 +02:00
return false;
require canonical multi output bulletproof layout
2018-03-31 11:49:53 +02:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::parse_incoming_tx_accumulated_batch(
std::vector<tx_verification_batch_info>& tx_info, bool kept_by_block) {
if (kept_by_block && get_blockchain_storage().is_within_compiled_block_hash_area()) {
log::trace(logcat, "Skipping semantics check for txs kept by block in embedded hash area");
return;
blockchain: skip checking tx semantics in embedded block hash range If the txes are bad, this'll be picked up by the block hash mismatch since the tx merkle root is part of the block hash.
2017-07-30 10:45:55 +02:00
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
std::vector<const rct::rctSig*> rvv;
run clang format
2023-04-13 15:50:13 +02:00
for (size_t n = 0; n < tx_info.size(); ++n) {
if (!tx_info[n].result || tx_info[n].already_have)
continue;
if (!check_tx_semantic(tx_info[n].tx, kept_by_block)) {
require canonical multi output bulletproof layout
2018-03-31 11:49:53 +02:00
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
run clang format
2023-04-13 15:50:13 +02:00
continue;
}
if (!tx_info[n].tx.is_transfer())
continue;
const rct::rctSig& rv = tx_info[n].tx.rct_signatures;
switch (rv.type) {
case rct::RCTType::Null:
// coinbase should not come here, so we reject for all other types
log::error(log::Cat("verify"), "Unexpected Null rctSig type");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
case rct::RCTType::Simple:
if (!rct::verRctSemanticsSimple(rv)) {
log::error(log::Cat("verify"), "rct signature semantics check failed");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
break;
case rct::RCTType::Full:
if (!rct::verRct(rv, true)) {
log::error(log::Cat("verify"), "rct signature semantics check failed");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
break;
case rct::RCTType::Bulletproof:
case rct::RCTType::Bulletproof2:
case rct::RCTType::CLSAG:
if (!is_canonical_bulletproof_layout(rv.p.bulletproofs)) {
log::error(log::Cat("verify"), "Bulletproof does not have canonical form");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
rvv.push_back(&rv); // delayed batch verification
break;
default:
log::error(log::Cat("verify"), "Unknown rct type: {}", (int)rv.type);
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
}
if (!rvv.empty() && !rct::verRctSemanticsSimple(rvv)) {
log::info(
logcat,
"One transaction among this group has bad semantics, verifying one at a time");
const bool assumed_bad = rvv.size() == 1; // if there's only one tx, it must be the bad one
for (size_t n = 0; n < tx_info.size(); ++n) {
if (!tx_info[n].result || tx_info[n].already_have)
continue;
if (!rct::is_rct_bulletproof(tx_info[n].tx.rct_signatures.type))
continue;
if (assumed_bad || !rct::verRctSemanticsSimple(tx_info[n].tx.rct_signatures)) {
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
std::vector<cryptonote::tx_verification_batch_info> core::parse_incoming_txs(
const std::vector<std::string>& tx_blobs, const tx_pool_options& opts) {
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
// Caller needs to do this around both this *and* handle_parsed_txs
run clang format
2023-04-13 15:50:13 +02:00
// auto lock = incoming_tx_lock();
Batching of service node rewards This updates the coinbase transactions to reward service nodes periodically rather than every block. If you recieve a service node reward this reward will be delayed x blocks, if you receive another reward to the same wallet before those blocks have been completed it will be added to your total and all will be paid out after those x blocks has passed. For example if our batching interval is 2 blocks: Block 1 - Address A receives reward of 10 oxen - added to batch Block 2 - Address A receives reward of 10 oxen - added to batch Block 3 - Address A is paid out 20 oxen. Batching accumulates a small reward for all nodes every block The batching of service node rewards allows us to drip feed rewards to service nodes. Rather than accruing each service node 16.5 oxen every time they are pulse block leader we now reward every node the 16.5 / num_service_nodes every block and pay each wallet the full amount that has been accrued after a period of time (Likely 3.5 days). To spread each payment evenly we now pay the rewards based on the address of the recipient. This modulus of their address determines which block the address should be paid and by setting the interval to our service_node_batching interval we can guarantee they will be paid out regularly and evenly distribute the payments for all wallets over this
2022-04-29 01:51:14 +02:00
std::vector<cryptonote::tx_verification_batch_info> tx_info(tx_blobs.size());
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
core: remove threadpool dependency from header
2018-04-23 19:55:58 +02:00
tools::threadpool& tpool = tools::threadpool::getInstance();
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
tools::threadpool::waiter waiter;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
for (size_t i = 0; i < tx_blobs.size(); i++) {
run clang format
2023-04-13 15:50:13 +02:00
tx_info[i].blob = &tx_blobs[i];
tpool.submit(&waiter, [this, &info = tx_info[i]] {
try {
parse_incoming_tx_pre(info);
} catch (const std::exception& e) {
log::error(log::Cat("verify"), "Exception in handle_incoming_tx_pre: {}", e.what());
info.tvc.m_verifivation_failed = true;
}
});
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 05:39:37 +02:00
}
threadpool: allow leaf functions to run concurrently Decrease the number of worker threads by one to account for the fact the calling thread acts as a worker thread now
2018-04-26 12:44:47 +02:00
waiter.wait(&tpool);
Remove check_tx_syntax unused since 2015
2019-10-23 06:13:55 +02:00
run clang format
2023-04-13 15:50:13 +02:00
for (auto& info : tx_info) {
if (!info.result)
continue;
Remove check_tx_syntax unused since 2015
2019-10-23 06:13:55 +02:00
run clang format
2023-04-13 15:50:13 +02:00
if (m_mempool.have_tx(info.tx_hash)) {
log::debug(logcat, "tx {} already has a transaction in tx_pool", info.tx_hash);
info.already_have = true;
} else if (m_blockchain_storage.have_tx(info.tx_hash)) {
log::debug(logcat, "tx {} already has a transaction in tx_pool", info.tx_hash);
info.already_have = true;
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
}
Remove check_tx_syntax unused since 2015
2019-10-23 06:13:55 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
parse_incoming_tx_accumulated_batch(tx_info, opts.kept_by_block);
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
return tx_info;
run clang format
2023-04-13 15:50:13 +02:00
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 21:29:42 +02:00
run clang format
2023-04-13 15:50:13 +02:00
bool core::handle_parsed_txs(
std::vector<tx_verification_batch_info>& parsed_txs,
const tx_pool_options& opts,
uint64_t* blink_rollback_height) {
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
// Caller needs to do this around both this *and* parse_incoming_txs
run clang format
2023-04-13 15:50:13 +02:00
// auto lock = incoming_tx_lock();
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
auto version = m_blockchain_storage.get_network_version();
bool ok = true;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (blink_rollback_height)
run clang format
2023-04-13 15:50:13 +02:00
*blink_rollback_height = 0;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
tx_pool_options tx_opts;
for (size_t i = 0; i < parsed_txs.size(); i++) {
run clang format
2023-04-13 15:50:13 +02:00
auto& info = parsed_txs[i];
if (!info.result) {
ok = false; // Propagate failures (so this can be chained with parse_incoming_txs
// without an intermediate check)
continue;
}
if (opts.kept_by_block)
get_blockchain_storage().on_new_tx_from_block(info.tx);
if (info.already_have)
continue; // Not a failure
const size_t weight = get_transaction_weight(info.tx, info.blob->size());
const tx_pool_options* local_opts = &opts;
if (blink_rollback_height && info.approved_blink) {
// If this is an approved blink then pass a copy of the options with the flag added
tx_opts = opts;
tx_opts.approved_blink = true;
local_opts = &tx_opts;
}
if (m_mempool.add_tx(
info.tx,
info.tx_hash,
*info.blob,
weight,
info.tvc,
*local_opts,
version,
blink_rollback_height)) {
log::debug(logcat, "tx added: {}", info.tx_hash);
} else {
ok = false;
if (info.tvc.m_verifivation_failed)
log::error(log::Cat("verify"), "Transaction verification failed: {}", info.tx_hash);
else if (info.tvc.m_verifivation_impossible)
log::error(
log::Cat("verify"),
"Transaction verification impossible: {}",
info.tx_hash);
}
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
return ok;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
std::vector<cryptonote::tx_verification_batch_info> core::handle_incoming_txs(
const std::vector<std::string>& tx_blobs, const tx_pool_options& opts) {
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
auto lock = incoming_tx_lock();
auto parsed = parse_incoming_txs(tx_blobs, opts);
handle_parsed_txs(parsed, opts);
return parsed;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::handle_incoming_tx(
const std::string& tx_blob, tx_verification_context& tvc, const tx_pool_options& opts) {
Remove stupid typedef
2022-05-20 23:29:48 +02:00
const std::vector<std::string> tx_blobs{{tx_blob}};
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
auto parsed = handle_incoming_txs(tx_blobs, opts);
run clang format
2023-04-13 15:50:13 +02:00
parsed[0].blob = &tx_blob; // Update pointer to the input rather than the copy in case the
// caller wants to use it for some reason
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
tvc = parsed[0].tvc;
Readd handle_incoming_txs & locking fixes Rename handle_incoming_txs to handle_parsed_txs, and re-add a handle_incoming_txs that does the parse+handle_parsed call with the appropriate lock held for code that doesn't care about being able to split it into two steps. Clarifies the locking requirements with a comment in the code, and adds an incoming tx lock over the blink receiving process in cryptonote_protocol_handler.inl. Also fixes a bug in the weight calculation happening because the blobdata pointer was being invalidated when going through the singular `handle_parsed_tx()`.
2019-11-25 19:22:30 +01:00
return parsed[0].result && (parsed[0].already_have || tvc.m_added_to_pool);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
std::pair<std::vector<std::shared_ptr<blink_tx>>, std::unordered_set<crypto::hash>>
core::parse_incoming_blinks(const std::vector<serializable_blink_metadata>& blinks) {
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
std::pair<std::vector<std::shared_ptr<blink_tx>>, std::unordered_set<crypto::hash>> results;
run clang format
2023-04-13 15:50:13 +02:00
auto& new_blinks = results.first;
auto& missing_txs = results.second;
Add blink HF version checks
2019-11-18 23:38:16 +01:00
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
if (m_blockchain_storage.get_network_version() < feature::BLINK)
run clang format
2023-04-13 15:50:13 +02:00
return results;
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
run clang format
2023-04-13 15:50:13 +02:00
std::vector<uint8_t> want(
blinks.size(), false); // Really bools, but std::vector<bool> is broken.
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
size_t want_count = 0;
// Step 1: figure out which referenced transactions we want to keep:
// - unknown tx (typically an incoming blink)
// - in mempool without blink sigs (it's possible to get the tx before the blink signatures)
// - in a recent, still-mutable block with blink sigs (can happen when syncing blocks before
// retrieving blink signatures)
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
{
run clang format
2023-04-13 15:50:13 +02:00
std::vector<crypto::hash> hashes;
hashes.reserve(blinks.size());
for (auto& bm : blinks)
hashes.emplace_back(bm.tx_hash);
std::unique_lock<Blockchain> lock(m_blockchain_storage);
auto tx_block_heights = m_blockchain_storage.get_transactions_heights(hashes);
auto immutable_height = m_blockchain_storage.get_immutable_height();
auto& db = m_blockchain_storage.get_db();
for (size_t i = 0; i < blinks.size(); i++) {
if (tx_block_heights[i] == 0 /*mempool or unknown*/ ||
tx_block_heights[i] > immutable_height /*mined but not yet immutable*/) {
want[i] = true;
want_count++;
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
}
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
log::debug(
logcat,
"Want {} of {} incoming blink signature sets after filtering out immutable txes",
want_count,
blinks.size());
if (!want_count)
return results;
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
// Step 2: filter out any transactions for which we already have a blink signature
{
run clang format
2023-04-13 15:50:13 +02:00
auto mempool_lock = m_mempool.blink_shared_lock();
for (size_t i = 0; i < blinks.size(); i++) {
if (want[i] && m_mempool.has_blink(blinks[i].tx_hash)) {
log::debug(
logcat,
"Ignoring blink data for {}: already have blink signatures",
blinks[i].tx_hash);
want[i] = false; // Already have it, move along
want_count--;
}
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
}
}
run clang format
2023-04-13 15:50:13 +02:00
log::debug(
logcat,
"Want {} of {} incoming blink signature sets after filtering out existing blink sigs",
want_count,
blinks.size());
if (!want_count)
return results;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
// Step 3: create new blink_tx objects for txes and add the blink signatures. We can do all of
// this without a lock since these are (for now) just local instances.
new_blinks.reserve(want_count);
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
std::unordered_map<uint64_t, std::shared_ptr<const service_nodes::quorum>> quorum_cache;
run clang format
2023-04-13 15:50:13 +02:00
for (size_t i = 0; i < blinks.size(); i++) {
if (!want[i])
continue;
auto& bdata = blinks[i];
new_blinks.push_back(std::make_shared<blink_tx>(bdata.height, bdata.tx_hash));
auto& blink = *new_blinks.back();
// Data structure checks (we have more stringent checks for validity later, but if these
// fail now then there's no point of even trying to do signature validation.
if (bdata.signature.size() !=
bdata.position
.size() || // Each signature must have an associated quorum position
bdata.signature.size() != bdata.quorum.size() || // and quorum index
bdata.signature.size() <
service_nodes::BLINK_MIN_VOTES *
tools::enum_count<blink_tx::subquorum> || // too few signatures for
// possible validity
bdata.signature.size() >
service_nodes::BLINK_SUBQUORUM_SIZE *
tools::enum_count<blink_tx::subquorum> || // too many signatures
blink_tx::quorum_height(bdata.height, blink_tx::subquorum::base) ==
0 || // Height is too early (no blink quorum height)
std::any_of(
bdata.position.begin(),
bdata.position.end(),
[](const auto& p) {
return p >= service_nodes::BLINK_SUBQUORUM_SIZE;
}) || // invalid position
std::any_of(
bdata.quorum.begin(),
bdata.quorum.end(),
[](const auto& qi) {
return qi >= tools::enum_count<blink_tx::subquorum>;
}) // invalid quorum index
) {
log::info(logcat, "Invalid blink tx {}: invalid signature data", bdata.tx_hash);
continue;
Fix crash when receiving blocks for unknown quorum This can occur when syncing if we get a blink tx before the blocks that let us determine the quorum. Just ignore it at this point; we'll pick it up at the next once-per-minute sync run.
2019-12-24 04:01:48 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
bool no_quorum = false;
std::array<const std::vector<crypto::public_key>*, tools::enum_count<blink_tx::subquorum>>
validators;
for (uint8_t qi = 0; qi < tools::enum_count<blink_tx::subquorum>; qi++) {
auto q_height = blink.quorum_height(static_cast<blink_tx::subquorum>(qi));
auto& q = quorum_cache[q_height];
if (!q)
q = get_quorum(service_nodes::quorum_type::blink, q_height);
if (!q) {
log::trace(
logcat,
"Don't have a quorum for height {} (yet?), ignoring this blink",
q_height);
no_quorum = true;
break;
}
validators[qi] = &q->validators;
}
if (no_quorum)
continue;
std::vector<std::pair<size_t, std::string>> failures;
for (size_t s = 0; s < bdata.signature.size(); s++) {
try {
blink.add_signature(
static_cast<blink_tx::subquorum>(bdata.quorum[s]),
bdata.position[s],
true /*approved*/,
bdata.signature[s],
validators[bdata.quorum[s]]->at(bdata.position[s]));
} catch (const std::exception& e) {
failures.emplace_back(s, e.what());
}
}
if (blink.approved()) {
log::info(
logcat,
"Blink tx {} blink signatures approved with {} signature validation failures",
bdata.tx_hash,
failures.size());
for (auto& f : failures)
log::debug(
logcat,
"- failure for quorum {}, position {}: {}",
int(bdata.quorum[f.first]),
int(bdata.position[f.first]),
f.second);
} else {
std::string blink_error = "Blink validation failed:";
auto append = std::back_inserter(blink_error);
for (auto& f : failures)
fmt::format_to(
append,
" [{}:{}]: {}",
int(bdata.quorum[f.first]),
int(bdata.position[f.first]),
f.second);
log::info(logcat, "Invalid blink tx {}: {}", bdata.tx_hash, blink_error);
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
}
core: thread most of handle_incoming_tx
2017-07-10 16:38:56 +02:00
}
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
return results;
run clang format
2023-04-13 15:50:13 +02:00
}
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
run clang format
2023-04-13 15:50:13 +02:00
int core::add_blinks(const std::vector<std::shared_ptr<blink_tx>>& blinks) {
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
int added = 0;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
if (blinks.empty())
run clang format
2023-04-13 15:50:13 +02:00
return added;
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
auto lock = m_mempool.blink_unique_lock();
run clang format
2023-04-13 15:50:13 +02:00
for (auto& b : blinks)
if (b->approved())
if (m_mempool.add_existing_blink(b))
added++;
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
run clang format
2023-04-13 15:50:13 +02:00
if (added) {
log::info(logcat, "Added blink signatures for {} blinks", added);
long_poll_trigger(m_mempool);
Blink: Make wallet only long poll blinks - Sometimes the wallet would query TX's before they are blink approved and get cached into the wallet as a normal TX preventing external services from using 0-confirmation on transactions. We make the wallet only conduct long-polling for blinks in order to notice the transaction after blink. The downside here is that normal mempool transacations won't appear until they are placed into a block. With Pulse this is mitigated with reliable within sub-seconds of a target 2 min block time.
2020-10-01 10:08:19 +02:00
}
blink sync fixes - actually include the blink hashes in the core sync data - fix cleanup to delete heights in (0, immutable] instead of [0, immutable); we want to keep 0 because it signifies the mempool, and we only need blocks after immutable, not immutable itself. - fixed NOTIFY_REQUEST_GET_TXS to handle mempool txes properly (it was only searching the chain and ignored missed_txs, but missed_txs are ones we need to look up in the mempool) - Add a method to tx_pool (needed for the above) to grab multiple txes by hash (essentially a multi-tx version of `get_transaction()`), and change get_transaction() to use it with a single value. - Added various debugging statements - Added a bunch of comments to each condition of the preliminary blink data check condition. - Don't abort blink addition on a single signature failure: if there are enough valid signatures we should still accept it. - Check for blink signature approval when receiving blink signatures; it's not enough to know that all were added successfully, we also have to ask the blink tx if it is approved (which does additional checks on subquorum counts) once we add them all.
2019-11-20 01:24:48 +01:00
Add tx pool blink replacement This adds support for bumping conflicting transactions from the mempool upon receipt of a blink tx if those conflicting txs are not themselves blinks. This commit chops the handle_incoming_txs effectively in half into a parse_incoming_txs phase, followed by a handle_incoming_txs phase which is needed so that we can first parse the txes then, assuming they parse successfully, can flag them as blink txes if blink info was included with them, so that the actual insertion becomes forcing. There is a stub here for also allowing a rollback of the mutable part of the chain; implementation of actually doing that rollback will follow in another commit.
2019-11-25 05:57:15 +01:00
return added;
run clang format
2023-04-13 15:50:13 +02:00
}
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
run clang format
2023-04-13 15:50:13 +02:00
//-----------------------------------------------------------------------------------------------
std::future<std::pair<blink_result, std::string>> core::handle_blink_tx(
const std::string& tx_blob) {
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
return quorumnet_send_blink(*this, tx_blob);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::check_tx_semantic(const transaction& tx, bool keeped_by_block) const {
if (tx.is_transfer()) {
if (tx.vin.empty()) {
log::error(
log::Cat("verify"),
"tx with empty inputs, rejected for tx id= {}",
get_transaction_hash(tx));
return false;
}
} else {
if (tx.vin.size() != 0) {
log::error(
log::Cat("verify"),
"tx type: {} must have 0 inputs, received: {}, rejected for tx id = {}",
tx.type,
tx.vin.size(),
get_transaction_hash(tx));
return false;
}
Check that inputs/outputs size is 0 for deregister
2018-07-25 03:13:38 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
if (!check_inputs_types_supported(tx)) {
log::error(
log::Cat("verify"),
"unsupported input types for tx id= {}",
get_transaction_hash(tx));
Build/core test fixes, incoporate staking changes from LNS - Fix assert to use version_t::_count in service_node_list.cpp - Incoporate staking changes from LNS which revamp construct tx to derive more of the parameters from hf_version and type. This removes extraneous parameters that can be derived elsewhere. Also delegate setting loki_construct_tx_params into wallet2 atleast, so that callers into the wallet logic, like rpc server, simplewallet don't need bend backwards to get the HF version whereas the wallet has dedicated functions for determining the HF.
2019-11-29 04:36:43 +01:00
return false;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
if (!check_outs_valid(tx)) {
log::error(
log::Cat("verify"),
"tx with invalid outputs, rejected for tx id= {}",
get_transaction_hash(tx));
return false;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
if (tx.version >= txversion::v2_ringct) {
if (tx.rct_signatures.outPk.size() != tx.vout.size()) {
log::error(
log::Cat("verify"),
"tx with mismatched vout/outPk count, rejected for tx id= {}",
get_transaction_hash(tx));
return false;
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Service Node Deregister Part 5 (#89) * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * core, service_node_list: separated address from service node pubkey * Retrieve quorum list from height, reviewed * Setup data structures for de/register TX * Submit and validate partial/full deregisters * Add P2P relaying of partial deregistration votes * Code review adjustments for deregistration part 1 - Fix check_tx_semantic - Remove signature_pod as votes are now stored as blobs. Serialization overrides don't intefere with crypto::signature anymore. * deregistration_vote_pool - changed sign/verify interface and removed repeated code * Misc review, fix sign/verify api, vote threshold * Deregister/tx edge case handling for combinatoric votes * Store service node lists for the duration of deregister lifetimes * Quorum min/max bug, sort node list, fix node to test list * Change quorum to store acc pub address, fix oob bug * Code review for expiring votes, acc keys to pub_key, improve err msgs * Add early out for is_deregistration_tx and protect against quorum changes * Remove debug code, fix segfault * Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states Incorrect assumption that a transaction can be kept in the chain if it could eventually become invalid, because if it were the chain would be split and eventually these transaction would be dropped. But also that we should not override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
run clang format
2023-04-13 15:50:13 +02:00
if (!check_money_overflow(tx)) {
log::error(
log::Cat("verify"),
"tx has money overflow, rejected for tx id= {}",
get_transaction_hash(tx));
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-16 00:37:13 +02:00
return false;
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
run clang format
2023-04-13 15:50:13 +02:00
if (tx.version == txversion::v1) {
uint64_t amount_in = 0;
get_inputs_money_amount(tx, amount_in);
uint64_t amount_out = get_outs_money_amount(tx);
moved all stuff to github
2014-03-03 23:07:58 +01:00
run clang format
2023-04-13 15:50:13 +02:00
if (amount_in <= amount_out) {
log::error(
log::Cat("verify"),
"tx with wrong amounts: ins {}, outs {}, rejected for tx id= {}",
amount_in,
amount_out,
get_transaction_hash(tx));
return false;
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
if (!keeped_by_block &&
get_transaction_weight(tx) >=
m_blockchain_storage.get_current_cumulative_block_weight_limit() -
COINBASE_BLOB_RESERVED_SIZE) {
log::error(
log::Cat("verify"),
"tx is too large {}, expected not bigger than {}",
get_transaction_weight(tx),
m_blockchain_storage.get_current_cumulative_block_weight_limit() -
COINBASE_BLOB_RESERVED_SIZE);
return false;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
if (!check_tx_inputs_keyimages_diff(tx)) {
log::error(log::Cat("verify"), "tx uses a single key image more than once");
return false;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
if (!check_tx_inputs_ring_members_diff(tx)) {
log::error(log::Cat("verify"), "tx uses duplicate ring members");
return false;
core: forbid duplicate ring members from v6 This avoids someone adding what amounts to mixin 0 rings in practice, as there is no other good reason to allow this.
2017-07-18 23:38:26 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
if (!check_tx_inputs_keyimages_domain(tx)) {
log::error(log::Cat("verify"), "tx uses key image not in the valid domain");
return false;
core: test key images against validity domain
2017-02-20 18:41:39 +01:00
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::check_service_node_time() {
quorumnet message for timestamp requests random sampling of service nodes, call timestamp lmq message checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user tracks external timesync status and timestamp participation of service nodes clean up includes new template struct for participation history, individual types for participation entry refactor checking participation update select_randomly, move the testing for variance overflow version locks, bump to 8.1.5 explicit casting for mac & clang note to remove after hard fork timestamp debugging log messages debugging messages for before timesync - before message sent logging errord with compiling print version and change add_command to add_request_command log if statement test std::to_string replaced with tools::view_guts for x25519 key check if my sn is active before sending timestamp requests logging the failures checking if statement for success of message more logging, if guards arn't passing more logging, successfully tests if service node might be out of sync more tests before we decide we are out of sync logging output if sn isn't passing tests if check_participation fails then disconnect print timestamp status remove saving variance from the participation history reduce MIN_TIME_IN_S_BEFORE_VOTING reset participation history on recommission undo reduction in startup time reduce log levels Set hardfork time in testnet
2020-12-01 01:56:43 +01:00
run clang format
2023-04-13 15:50:13 +02:00
if (!is_active_sn()) {
return true;
}
quorumnet message for timestamp requests random sampling of service nodes, call timestamp lmq message checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user tracks external timesync status and timestamp participation of service nodes clean up includes new template struct for participation history, individual types for participation entry refactor checking participation update select_randomly, move the testing for variance overflow version locks, bump to 8.1.5 explicit casting for mac & clang note to remove after hard fork timestamp debugging log messages debugging messages for before timesync - before message sent logging errord with compiling print version and change add_command to add_request_command log if statement test std::to_string replaced with tools::view_guts for x25519 key check if my sn is active before sending timestamp requests logging the failures checking if statement for success of message more logging, if guards arn't passing more logging, successfully tests if service node might be out of sync more tests before we decide we are out of sync logging output if sn isn't passing tests if check_participation fails then disconnect print timestamp status remove saving variance from the participation history reduce MIN_TIME_IN_S_BEFORE_VOTING reset participation history on recommission undo reduction in startup time reduce log levels Set hardfork time in testnet
2020-12-01 01:56:43 +01:00
crypto::public_key pubkey = m_service_node_list.get_random_pubkey();
crypto::x25519_public_key x_pkey{0};
run clang format
2023-04-13 15:50:13 +02:00
constexpr std::array<uint16_t, 3> MIN_TIMESTAMP_VERSION{9, 1, 0};
std::array<uint16_t, 3> proofversion;
m_service_node_list.access_proof(pubkey, [&](auto& proof) {
x_pkey = proof.pubkey_x25519;
proofversion = proof.proof->version;
quorumnet message for timestamp requests random sampling of service nodes, call timestamp lmq message checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user tracks external timesync status and timestamp participation of service nodes clean up includes new template struct for participation history, individual types for participation entry refactor checking participation update select_randomly, move the testing for variance overflow version locks, bump to 8.1.5 explicit casting for mac & clang note to remove after hard fork timestamp debugging log messages debugging messages for before timesync - before message sent logging errord with compiling print version and change add_command to add_request_command log if statement test std::to_string replaced with tools::view_guts for x25519 key check if my sn is active before sending timestamp requests logging the failures checking if statement for success of message more logging, if guards arn't passing more logging, successfully tests if service node might be out of sync more tests before we decide we are out of sync logging output if sn isn't passing tests if check_participation fails then disconnect print timestamp status remove saving variance from the participation history reduce MIN_TIME_IN_S_BEFORE_VOTING reset participation history on recommission undo reduction in startup time reduce log levels Set hardfork time in testnet
2020-12-01 01:56:43 +01:00
});
if (proofversion >= MIN_TIMESTAMP_VERSION && x_pkey) {
run clang format
2023-04-13 15:50:13 +02:00
m_omq->request(
tools::view_guts(x_pkey),
"quorum.timestamp",
[this, pubkey](bool success, std::vector<std::string> data) {
const time_t local_seconds = time(nullptr);
log::debug(
logcat,
"Timestamp message received: {}, local time is: ",
data[0],
local_seconds);
if (success) {
int64_t received_seconds;
if (tools::parse_int(data[0], received_seconds)) {
uint16_t variance;
if (received_seconds > local_seconds + 65535 ||
received_seconds < local_seconds - 65535) {
variance = 65535;
} else {
variance = std::abs(local_seconds - received_seconds);
}
std::lock_guard<std::mutex> lk(m_sn_timestamp_mutex);
// Records the variance into the record of our performance (m_sn_times)
service_nodes::timesync_entry entry{
variance <= service_nodes::THRESHOLD_SECONDS_OUT_OF_SYNC};
m_sn_times.add(entry);
// Counts the number of times we have been out of sync
if (m_sn_times.failures() >
(m_sn_times.size() * service_nodes::MAXIMUM_EXTERNAL_OUT_OF_SYNC /
100)) {
log::warning(logcat, "service node time might be out of sync");
// If we are out of sync record the other service node as in sync
m_service_node_list.record_timesync_status(pubkey, true);
} else {
m_service_node_list.record_timesync_status(
pubkey,
variance <= service_nodes::THRESHOLD_SECONDS_OUT_OF_SYNC);
}
} else {
success = false;
}
}
m_service_node_list.record_timestamp_participation(pubkey, success);
});
quorumnet message for timestamp requests random sampling of service nodes, call timestamp lmq message checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user tracks external timesync status and timestamp participation of service nodes clean up includes new template struct for participation history, individual types for participation entry refactor checking participation update select_randomly, move the testing for variance overflow version locks, bump to 8.1.5 explicit casting for mac & clang note to remove after hard fork timestamp debugging log messages debugging messages for before timesync - before message sent logging errord with compiling print version and change add_command to add_request_command log if statement test std::to_string replaced with tools::view_guts for x25519 key check if my sn is active before sending timestamp requests logging the failures checking if statement for success of message more logging, if guards arn't passing more logging, successfully tests if service node might be out of sync more tests before we decide we are out of sync logging output if sn isn't passing tests if check_participation fails then disconnect print timestamp status remove saving variance from the participation history reduce MIN_TIME_IN_S_BEFORE_VOTING reset participation history on recommission undo reduction in startup time reduce log levels Set hardfork time in testnet
2020-12-01 01:56:43 +01:00
}
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::is_key_image_spent(const crypto::key_image& key_image) const {
Add a is_key_image_spent daemon command and RPC call
2015-08-11 11:49:15 +02:00
return m_blockchain_storage.have_tx_keyimg_as_spent(key_image);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::are_key_images_spent(
const std::vector<crypto::key_image>& key_im, std::vector<bool>& spent) const {
Add a is_key_image_spent daemon command and RPC call
2015-08-11 11:49:15 +02:00
spent.clear();
run clang format
2023-04-13 15:50:13 +02:00
for (auto& ki : key_im) {
spent.push_back(m_blockchain_storage.have_tx_keyimg_as_spent(ki));
Add a is_key_image_spent daemon command and RPC call
2015-08-11 11:49:15 +02:00
}
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
size_t core::get_block_sync_size(uint64_t height) const {
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
return block_sync_size > 0 ? block_sync_size : BLOCKS_SYNCHRONIZING_DEFAULT_COUNT;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::are_key_images_spent_in_pool(
const std::vector<crypto::key_image>& key_im, std::vector<bool>& spent) const {
json serialization for rpc-relevant monero types Structured {de-,}serialization methods for (many new) types which are used for requests or responses in the RPC. New types include RPC requests and responses, and structs which compose types within those. # Conflicts: # src/cryptonote_core/blockchain.cpp
2017-09-05 18:20:27 +02:00
spent.clear();
return m_mempool.check_for_key_images(key_im, spent);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
std::optional<std::tuple<int64_t, int64_t, int64_t>> core::get_coinbase_tx_sum(
uint64_t start_offset, size_t count) {
Fix negative coinbase emissions With batching, individual blocks can have a negative coinbase emission because the tx fee gets added to the batch rewards database and not paid out immediately, which then results in an negative overflow to a value close to 2^64. Thus a block with no payout and a tx fee will have an erroneous huge positive coinbase emission when queried via `get_coinbase_tx_sum`. For example block 1094068 queried with: {"jsonrpc":"2.0","id":"0","method":"get_coinbase_tx_sum","params":{"height": 1094068, "count": 1}} returns: { "jsonrpc": "2.0", "id": "0", "result": { "burn_amount": 0, "emission_amount": 18446744073699378616, "fee_amount": 10173000, "status": "OK" } } This commit fixes it by making the values signed (and also serves as an example of why unsigned integers are usually the wrong choice): { "jsonrpc": "2.0", "id": "0", "result": { "burn_amount": 0, "emission_amount": -10173000, "fee_amount": 10173000, "status": "OK" } }
2022-07-05 15:15:55 +02:00
std::optional<std::tuple<int64_t, int64_t, int64_t>> result{{0, 0, 0}};
Cache get_coinbase_tx_sum-from-0 result This caches the result of a get_coinbase_tx_sum to H-30 (if the last request started from 0 and retrieved up to at least H-30). This makes get_coinbase_tx_sum calls to get the full chain values massively faster for all but the first call. The "first call" is kind of tricky, though, because it can take a couple minutes, during which if we get multiple calls (e.g. from the block explorer) we might get multiple threads trying to create the cache all at once, and *each* of those takes minutes (and chew up an admin rpc thread). So this commit also blocks out other threads from getting a cacheable result while the cache is being built; instead those calls get a null optional back. Once the cache is built, requests start returning pretty much instantly (on my desktop system with the blockchain data cached in RAM I process around 5k blocks per second).
2020-08-13 21:40:54 +02:00
if (count == 0)
run clang format
2023-04-13 15:50:13 +02:00
return result;
Cache get_coinbase_tx_sum-from-0 result This caches the result of a get_coinbase_tx_sum to H-30 (if the last request started from 0 and retrieved up to at least H-30). This makes get_coinbase_tx_sum calls to get the full chain values massively faster for all but the first call. The "first call" is kind of tricky, though, because it can take a couple minutes, during which if we get multiple calls (e.g. from the block explorer) we might get multiple threads trying to create the cache all at once, and *each* of those takes minutes (and chew up an admin rpc thread). So this commit also blocks out other threads from getting a cacheable result while the cache is being built; instead those calls get a null optional back. Once the cache is built, requests start returning pretty much instantly (on my desktop system with the blockchain data cached in RAM I process around 5k blocks per second).
2020-08-13 21:40:54 +02:00
Fix negative coinbase emissions With batching, individual blocks can have a negative coinbase emission because the tx fee gets added to the batch rewards database and not paid out immediately, which then results in an negative overflow to a value close to 2^64. Thus a block with no payout and a tx fee will have an erroneous huge positive coinbase emission when queried via `get_coinbase_tx_sum`. For example block 1094068 queried with: {"jsonrpc":"2.0","id":"0","method":"get_coinbase_tx_sum","params":{"height": 1094068, "count": 1}} returns: { "jsonrpc": "2.0", "id": "0", "result": { "burn_amount": 0, "emission_amount": 18446744073699378616, "fee_amount": 10173000, "status": "OK" } } This commit fixes it by making the values signed (and also serves as an example of why unsigned integers are usually the wrong choice): { "jsonrpc": "2.0", "id": "0", "result": { "burn_amount": 0, "emission_amount": -10173000, "fee_amount": 10173000, "status": "OK" } }
2022-07-05 15:15:55 +02:00
auto& [emission_amount, total_fee_amount, burnt_oxen] = *result;
Cache get_coinbase_tx_sum-from-0 result This caches the result of a get_coinbase_tx_sum to H-30 (if the last request started from 0 and retrieved up to at least H-30). This makes get_coinbase_tx_sum calls to get the full chain values massively faster for all but the first call. The "first call" is kind of tricky, though, because it can take a couple minutes, during which if we get multiple calls (e.g. from the block explorer) we might get multiple threads trying to create the cache all at once, and *each* of those takes minutes (and chew up an admin rpc thread). So this commit also blocks out other threads from getting a cacheable result while the cache is being built; instead those calls get a null optional back. Once the cache is built, requests start returning pretty much instantly (on my desktop system with the blockchain data cached in RAM I process around 5k blocks per second).
2020-08-13 21:40:54 +02:00
// Caching.
//
// Requesting this value from the beginning of the chain is very slow, so we cache it. That
// still means the first request will be slow, but that's okay. To prevent a bunch of threads
// getting backed up trying to calculate this, we lock out more than one thread building the
// cache at a time if we're requesting a large number of block values at once. Any other thread
// requesting will get a nullopt back.
run clang format
2023-04-13 15:50:13 +02:00
constexpr uint64_t CACHE_LAG = 30; // We cache the values up to this many blocks ago; we lag so
// that we don't have to worry about small reorgs
constexpr uint64_t CACHE_EXCLUSIVE =
1000; // If we need to load more than this, we block out other threads
Cache get_coinbase_tx_sum-from-0 result This caches the result of a get_coinbase_tx_sum to H-30 (if the last request started from 0 and retrieved up to at least H-30). This makes get_coinbase_tx_sum calls to get the full chain values massively faster for all but the first call. The "first call" is kind of tricky, though, because it can take a couple minutes, during which if we get multiple calls (e.g. from the block explorer) we might get multiple threads trying to create the cache all at once, and *each* of those takes minutes (and chew up an admin rpc thread). So this commit also blocks out other threads from getting a cacheable result while the cache is being built; instead those calls get a null optional back. Once the cache is built, requests start returning pretty much instantly (on my desktop system with the blockchain data cached in RAM I process around 5k blocks per second).
2020-08-13 21:40:54 +02:00
// Check if we have a cacheable from-the-beginning result
uint64_t cache_to = 0;
std::chrono::steady_clock::time_point cache_build_started;
if (start_offset == 0) {
run clang format
2023-04-13 15:50:13 +02:00
uint64_t height = m_blockchain_storage.get_current_blockchain_height();
if (count > height)
count = height;
cache_to = height - std::min(CACHE_LAG, height);
{
std::shared_lock lock{m_coinbase_cache.mutex};
if (m_coinbase_cache.height && count >= m_coinbase_cache.height) {
emission_amount = m_coinbase_cache.emissions;
total_fee_amount = m_coinbase_cache.fees;
burnt_oxen = m_coinbase_cache.burnt;
start_offset = m_coinbase_cache.height + 1;
count -= m_coinbase_cache.height;
}
// else don't change anything; we need a subset of blocks that ends before the cache.
if (cache_to <= m_coinbase_cache.height)
cache_to = 0; // Cache doesn't need updating
Cache get_coinbase_tx_sum-from-0 result This caches the result of a get_coinbase_tx_sum to H-30 (if the last request started from 0 and retrieved up to at least H-30). This makes get_coinbase_tx_sum calls to get the full chain values massively faster for all but the first call. The "first call" is kind of tricky, though, because it can take a couple minutes, during which if we get multiple calls (e.g. from the block explorer) we might get multiple threads trying to create the cache all at once, and *each* of those takes minutes (and chew up an admin rpc thread). So this commit also blocks out other threads from getting a cacheable result while the cache is being built; instead those calls get a null optional back. Once the cache is built, requests start returning pretty much instantly (on my desktop system with the blockchain data cached in RAM I process around 5k blocks per second).
2020-08-13 21:40:54 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
// If we're loading a lot then acquire an exclusive lock, recheck our variables, and block
// out other threads until we're done. (We don't do this if we're only loading a few
// because even if we have some competing cache updates they don't hurt anything).
Cache get_coinbase_tx_sum-from-0 result This caches the result of a get_coinbase_tx_sum to H-30 (if the last request started from 0 and retrieved up to at least H-30). This makes get_coinbase_tx_sum calls to get the full chain values massively faster for all but the first call. The "first call" is kind of tricky, though, because it can take a couple minutes, during which if we get multiple calls (e.g. from the block explorer) we might get multiple threads trying to create the cache all at once, and *each* of those takes minutes (and chew up an admin rpc thread). So this commit also blocks out other threads from getting a cacheable result while the cache is being built; instead those calls get a null optional back. Once the cache is built, requests start returning pretty much instantly (on my desktop system with the blockchain data cached in RAM I process around 5k blocks per second).
2020-08-13 21:40:54 +02:00
if (cache_to > 0 && count > CACHE_EXCLUSIVE) {
run clang format
2023-04-13 15:50:13 +02:00
std::unique_lock lock{m_coinbase_cache.mutex};
if (m_coinbase_cache.building)
return std::nullopt; // Another thread is already updating the cache
if (m_coinbase_cache.height && m_coinbase_cache.height >= start_offset) {
// Someone else updated the cache while we were acquiring the unique lock, so update
// our variables
if (m_coinbase_cache.height >= start_offset + count) {
// The cache is now *beyond* us, which means we can't use it, so reset
// start/count back to what they were originally.
count += start_offset - 1;
start_offset = 0;
cache_to = 0;
} else {
// The cache is updated and we can still use it, so update our variables.
emission_amount = m_coinbase_cache.emissions;
total_fee_amount = m_coinbase_cache.fees;
burnt_oxen = m_coinbase_cache.burnt;
count -= m_coinbase_cache.height - start_offset + 1;
start_offset = m_coinbase_cache.height + 1;
}
}
if (cache_to > 0 && count > CACHE_EXCLUSIVE) {
cache_build_started = std::chrono::steady_clock::now();
m_coinbase_cache.building = true; // Block out other threads until we're done
log::info(
logcat,
"Starting slow cache build request for get_coinbase_tx_sum({}, {})",
start_offset,
count);
}
Cache get_coinbase_tx_sum-from-0 result This caches the result of a get_coinbase_tx_sum to H-30 (if the last request started from 0 and retrieved up to at least H-30). This makes get_coinbase_tx_sum calls to get the full chain values massively faster for all but the first call. The "first call" is kind of tricky, though, because it can take a couple minutes, during which if we get multiple calls (e.g. from the block explorer) we might get multiple threads trying to create the cache all at once, and *each* of those takes minutes (and chew up an admin rpc thread). So this commit also blocks out other threads from getting a cacheable result while the cache is being built; instead those calls get a null optional back. Once the cache is built, requests start returning pretty much instantly (on my desktop system with the blockchain data cached in RAM I process around 5k blocks per second).
2020-08-13 21:40:54 +02:00
}
}
const uint64_t end = start_offset + count - 1;
run clang format
2023-04-13 15:50:13 +02:00
m_blockchain_storage.for_blocks_range(
start_offset,
end,
[this, &cache_to, &result, &cache_build_started](
uint64_t height, const crypto::hash& hash, const block& b) {
auto& [emission_amount, total_fee_amount, burnt_oxen] = *result;
std::vector<transaction> txs;
auto coinbase_amount = static_cast<int64_t>(get_outs_money_amount(b.miner_tx));
get_transactions(b.tx_hashes, txs);
int64_t tx_fee_amount = 0;
for (const auto& tx : txs) {
tx_fee_amount += static_cast<int64_t>(
get_tx_miner_fee(tx, b.major_version >= feature::FEE_BURNING));
if (b.major_version >= feature::FEE_BURNING) {
burnt_oxen +=
static_cast<int64_t>(get_burned_amount_from_tx_extra(tx.extra));
}
}
emission_amount += coinbase_amount - tx_fee_amount;
total_fee_amount += tx_fee_amount;
if (cache_to && cache_to == height) {
std::unique_lock lock{m_coinbase_cache.mutex};
if (m_coinbase_cache.height < height) {
m_coinbase_cache.height = height;
m_coinbase_cache.emissions = emission_amount;
m_coinbase_cache.fees = total_fee_amount;
m_coinbase_cache.burnt = burnt_oxen;
}
if (m_coinbase_cache.building) {
m_coinbase_cache.building = false;
log::info(
logcat,
"Finishing cache build for get_coinbase_tx_sum in {} s",
std::chrono::duration<double>{
std::chrono::steady_clock::now() - cache_build_started}
.count());
}
cache_to = 0;
}
return true;
});
added print_coinbase_tx_sum option
2016-10-10 21:45:51 +02:00
Cache get_coinbase_tx_sum-from-0 result This caches the result of a get_coinbase_tx_sum to H-30 (if the last request started from 0 and retrieved up to at least H-30). This makes get_coinbase_tx_sum calls to get the full chain values massively faster for all but the first call. The "first call" is kind of tricky, though, because it can take a couple minutes, during which if we get multiple calls (e.g. from the block explorer) we might get multiple threads trying to create the cache all at once, and *each* of those takes minutes (and chew up an admin rpc thread). So this commit also blocks out other threads from getting a cacheable result while the cache is being built; instead those calls get a null optional back. Once the cache is built, requests start returning pretty much instantly (on my desktop system with the blockchain data cached in RAM I process around 5k blocks per second).
2020-08-13 21:40:54 +02:00
return result;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::check_tx_inputs_keyimages_diff(const transaction& tx) const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
std::unordered_set<crypto::key_image> ki;
run clang format
2023-04-13 15:50:13 +02:00
for (const auto& in : tx.vin) {
CHECKED_GET_SPECIFIC_VARIANT(in, txin_to_key, tokey_in, false);
if (!ki.insert(tokey_in.k_image).second)
return false;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::check_tx_inputs_ring_members_diff(const transaction& tx) const {
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
const auto version = m_blockchain_storage.get_network_version();
run clang format
2023-04-13 15:50:13 +02:00
for (const auto& in : tx.vin) {
CHECKED_GET_SPECIFIC_VARIANT(in, txin_to_key, tokey_in, false);
for (size_t n = 1; n < tokey_in.key_offsets.size(); ++n)
if (tokey_in.key_offsets[n] == 0)
return false;
core: forbid duplicate ring members from v6 This avoids someone adding what amounts to mixin 0 rings in practice, as there is no other good reason to allow this.
2017-07-18 23:38:26 +02:00
}
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::check_tx_inputs_keyimages_domain(const transaction& tx) const {
core: test key images against validity domain
2017-02-20 18:41:39 +01:00
std::unordered_set<crypto::key_image> ki;
run clang format
2023-04-13 15:50:13 +02:00
for (const auto& in : tx.vin) {
CHECKED_GET_SPECIFIC_VARIANT(in, txin_to_key, tokey_in, false);
if (!(rct::scalarmultKey(rct::ki2rct(tokey_in.k_image), rct::curveOrder()) ==
rct::identity()))
return false;
core: test key images against validity domain
2017-02-20 18:41:39 +01:00
}
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
size_t core::get_blockchain_total_transactions() const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.get_total_transactions();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::relay_txpool_transactions() {
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
// we attempt to relay txes that should be relayed, but were not
Remove stupid typedef
2022-05-20 23:29:48 +02:00
std::vector<std::pair<crypto::hash, std::string>> txs;
run clang format
2023-04-13 15:50:13 +02:00
if (m_mempool.get_relayable_transactions(txs) && !txs.empty()) {
cryptonote_connection_context fake_context{};
tx_verification_context tvc{};
NOTIFY_NEW_TRANSACTIONS::request r{};
for (auto it = txs.begin(); it != txs.end(); ++it) {
r.txs.push_back(it->second);
}
get_protocol()->relay_transactions(r, fake_context);
m_mempool.set_relayed(txs);
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 01:26:48 +01:00
}
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::submit_uptime_proof() {
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (!m_service_node)
run clang format
2023-04-13 15:50:13 +02:00
return true;
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
Remove boost::value_initialized and BOOST_FOREACH (#921) Neither of these have a place in modern C++11; boost::value_initialized is entirely superseded by `Type var{};` which does value initialization (or default construction if a default constructor is defined). More problematically, each `boost::value_initialized<T>` requires instantiation of another wrapping templated type which is a pointless price to pay the compiler in C++11 or newer. Also removed is the AUTO_VAL_INIT macro (which is just a simple macro around constructing a boost::value_initialized<T>). BOOST_FOREACH is a similarly massive pile of code to implement C++11-style for-each loops. (And bizarrely it *doesn't* appear to fall back to C++ for-each loops even when under a C++11 compiler!) This removes both entirely from the codebase.
2019-10-31 23:26:58 +01:00
cryptonote_connection_context fake_context{};
Serialize the uptime proof using btencoding
2020-12-17 06:52:16 +01:00
bool relayed;
auto height = get_current_blockchain_height();
9.1.2 hotfix -- fix proofs for older nodes with both pubkeys bt-encoded proofs have a bug for nodes with different legacy/ed25519 pubkeys that isn't easily solvable (it needs a fix on *both* sender and receiver ends). That fix is in here (in uptime_proof.cpp) but that isn't enough to solve it immediately. This works around the issue by submitting old-style proofs if we are a node with different legacy/ed25519 pubkeys.
2021-04-30 02:23:16 +02:00
run clang format
2023-04-13 15:50:13 +02:00
auto proof = m_service_node_list.generate_uptime_proof(
m_sn_public_ip,
storage_https_port(),
storage_omq_port(),
ss_version,
m_quorumnet_port,
lokinet_version);
9.1.2 hotfix -- fix proofs for older nodes with both pubkeys bt-encoded proofs have a bug for nodes with different legacy/ed25519 pubkeys that isn't easily solvable (it needs a fix on *both* sender and receiver ends). That fix is in here (in uptime_proof.cpp) but that isn't enough to solve it immediately. This works around the issue by submitting old-style proofs if we are a node with different legacy/ed25519 pubkeys.
2021-04-30 02:23:16 +02:00
NOTIFY_BTENCODED_UPTIME_PROOF::request req = proof.generate_request();
relayed = get_protocol()->relay_btencoded_uptime_proof(req, fake_context);
// TODO: remove after HF19
run clang format
2023-04-13 15:50:13 +02:00
if (relayed &&
tools::view_guts(m_service_keys.pub) != tools::view_guts(m_service_keys.pub_ed25519)) {
// Temp workaround: nodes with both pub and ed25519 are failing bt-encoded proofs, so send
// an old-style proof out as well as a workaround.
NOTIFY_UPTIME_PROOF::request req = m_service_node_list.generate_uptime_proof(
m_sn_public_ip, storage_https_port(), storage_omq_port(), m_quorumnet_port);
get_protocol()->relay_uptime_proof(req, fake_context);
Serialize the uptime proof using btencoding
2020-12-17 06:52:16 +01:00
}
9.1.2 hotfix -- fix proofs for older nodes with both pubkeys bt-encoded proofs have a bug for nodes with different legacy/ed25519 pubkeys that isn't easily solvable (it needs a fix on *both* sender and receiver ends). That fix is in here (in uptime_proof.cpp) but that isn't enough to solve it immediately. This works around the issue by submitting old-style proofs if we are a node with different legacy/ed25519 pubkeys.
2021-04-30 02:23:16 +02:00
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
if (relayed)
run clang format
2023-04-13 15:50:13 +02:00
log::info(
logcat, "Submitted uptime-proof for Service Node (yours): {}", m_service_keys.pub);
Require storage server running for uptime proofs (v.2) (#631) * Require storage server running for uptime proofs (v.2) * use atomic time_t; reset ping on startup
2019-06-07 09:03:11 +02:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::handle_uptime_proof(
const NOTIFY_UPTIME_PROOF::request& proof, bool& my_uptime_proof_confirmation) {
Pulse: Queue Block Gen code review - Rename find_named_argument -> find_prefixed_value - Update get_current_blockchain_height() comment to specify ability to lock - Update memcmp(address1, address2) to address1 == address2 - Make create_next_miner_block_template call create_miner_block_template - Update LMQ x25519 keys only when key has changed - construct_miner_tx: Use C++17 features - Pulse: Add log category "Pulse" - Pulse: Check shutdown before blockchain height - Pulse: Use get_human_readable_timespan - Pulse: Switch to MINFO, enable with --log-level +pulse:DEBUG - Pulse: Add height into context - Pulse: Remove some default switch cases hiding compiler warnings - SN: Update hardcoded '9' literals to network_version_9_service_nodes - Core: Use using namespace std::literals - Miner: Change num_blocks availability to debug - RPC: Gate test rpc commands to != MAINNET - Wallet: Store wallet before deinitialization instead of refresh
2020-08-06 09:21:00 +02:00
crypto::x25519_public_key pkey = {};
run clang format
2023-04-13 15:50:13 +02:00
bool result =
m_service_node_list.handle_uptime_proof(proof, my_uptime_proof_confirmation, pkey);
if (result && m_service_node_list.is_service_node(proof.pubkey, true /*require_active*/) &&
pkey) {
oxenmq::pubkey_set added;
added.insert(tools::copy_guts(pkey));
m_omq->update_active_sns(added, {} /*removed*/);
Pulse: Update LMQ active pubkeys when proof arrives - Fixes a edge case that is prevalent in testing when adding nodes to the network. Presuming an emty SN network, with Pulse once N nodes are added to the network, the network switches from PoW blocks to Pulse Blocks which require SN to SN communication. SN's can't communicate with each other until their uptime proofs are propagated with the IP bundled and currently LMQ only updates said communication info when blocks are added, meaning Pulse can never proceed. So instead on receipt of an uptime proof we add the comm data to LMQ if it was accepted and said Service Node is active.
2020-08-02 08:20:11 +02:00
}
return result;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::handle_btencoded_uptime_proof(
const NOTIFY_BTENCODED_UPTIME_PROOF::request& req, bool& my_uptime_proof_confirmation) {
Serialize the uptime proof using btencoding
2020-12-17 06:52:16 +01:00
crypto::x25519_public_key pkey = {};
shorten wire names
2021-01-28 01:07:57 +01:00
auto proof = std::make_unique<uptime_proof::Proof>(req.proof);
proof->sig = tools::make_from_guts<crypto::signature>(req.sig);
proof->sig_ed25519 = tools::make_from_guts<crypto::ed25519_signature>(req.ed_sig);
auto pubkey = proof->pubkey;
run clang format
2023-04-13 15:50:13 +02:00
bool result = m_service_node_list.handle_btencoded_uptime_proof(
std::move(proof), my_uptime_proof_confirmation, pkey);
if (result && m_service_node_list.is_service_node(pubkey, true /*require_active*/) && pkey) {
oxenmq::pubkey_set added;
added.insert(tools::copy_guts(pkey));
m_omq->update_active_sns(added, {} /*removed*/);
Serialize the uptime proof using btencoding
2020-12-17 06:52:16 +01:00
}
return result;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
crypto::hash core::on_transaction_relayed(const std::string& tx_blob) {
Remove stupid typedef
2022-05-20 23:29:48 +02:00
std::vector<std::pair<crypto::hash, std::string>> txs;
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
cryptonote::transaction tx;
core: avoid calculating tx prefix hash when we don't need it
2018-11-13 00:22:20 +01:00
crypto::hash tx_hash;
run clang format
2023-04-13 15:50:13 +02:00
if (!parse_and_validate_tx_from_blob(tx_blob, tx, tx_hash)) {
log::error(logcat, "Failed to parse relayed transaction");
return crypto::null<crypto::hash>;
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
}
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
txs.push_back(std::make_pair(tx_hash, std::move(tx_blob)));
core: notify the txpool when transactions are relayed
2016-10-22 21:46:51 +02:00
m_mempool.set_relayed(txs);
Blink synchronization - Adds blink signature synchronization and storage through the regular p2p network - Adds wallet support (though this is still currently buggy and needs additional fixes - it sees the tx when it arrives in the mempool but isn't properly updating when the blink tx gets mined.)
2019-11-06 07:28:33 +01:00
return tx_hash;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::relay_service_node_votes() {
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
auto height = get_current_blockchain_height();
Add snode revision soft forks & drop hard fork voting code Snode revisions are a secondary version that let us put out a mandatory update for snodes that isn't a hardfork (and so isn't mandatory for wallets/exchanges/etc.). The main point of this is to let us make a 9.2.0 release that includes new mandatory minimums of future versions of storage server (2.2.0) and lokinet (0.9.4) to bring upgrades to the network. This slightly changes the HF7 blocks to 0 (instead of 1) because, apparently, we weren't properly checking the HF value of the pre-first-hf genesis block at all before. (In practice this changes nothing because genesis blocks are v7 anyway). This also changes (slightly) how we check for hard forks: now if we skip some hard forks then we still want to know the height when a hard fork triggers. For example, if the hf tables contains {7,14} then we still need to know that the HF14 block height also is the height that activates HF9, 10, etc.
2021-06-18 01:06:51 +02:00
auto hf_version = get_network_version(m_nettype, height);
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
Keep checkpoint votes over p2p Checkpoint votes were only going over quorumnet, which was quite broken as they need to go out universally: random nodes piece together the individual checkpoints to create checkpoints on their own, and so need to see all of the votes. With the current code only service nodes that participated in the specific quorum ever saw the checkpoint votes. This commit returns checkpoint vote distribution to distribution over p2p. We could, in theory, do checkpoint vote collection over quorumnet and then relay the set of votes as a pack to reduce p2p traffic a bit, but this wouldn't be a huge optimization since we'd still have to distribute all the votes over p2p at some point anyway, so leaving that as a future optimization for now. (Obligations votes, in contrast, don't need to be distributed at all -- if a vote results in a state change, the quorum members themselves can produce and distribute the state change tx).
2019-12-10 19:57:36 +01:00
auto quorum_votes = m_quorum_cop.get_relayable_votes(height, hf_version, true);
run clang format
2023-04-13 15:50:13 +02:00
auto p2p_votes = m_quorum_cop.get_relayable_votes(height, hf_version, false);
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
if (!quorum_votes.empty() && m_quorumnet_state && m_service_node)
run clang format
2023-04-13 15:50:13 +02:00
quorumnet_relay_obligation_votes(m_quorumnet_state, quorum_votes);
Add quorumnet comms This adds vote relaying via quorumnet. - The main glue between existing code and quorumnet code is in cryptonote_protocol/quorumnet.{h,cpp} - Uses function pointers assigned at initialization to call into the glue without making cn_core depend on p2p - Adds ed25519 and quorumnet port to uptime proofs and serialization. - Added a second uptime proof signature using the ed25519 key to that SNs are proving they actually have it (otherwise they could spoof someone else's pubkey). - Adds quorumnet port, defaulting to zmq rpc port + 1. quorumnet listens on the p2p IPv4 address (but broadcasts the `public_ip` to the network). - Derives x25519 when received or deserialized. - Converted service_info_info::version_t into a scoped enum (with the same underlying type). - Added contribution_t::version_t scoped enum. It was using service_info_info::version for a 0 value which seemed rather wrong. Random small details: - Renamed internal `arg_sn_bind_port` for consistency - Moved default stagenet ports from 38153-38155 to 38056-38058, and add the default stagenet quorumnet port at 38059 (this keeps the range contiguous; otherwise the next stagenet default port would collide with the testnet p2p port).
2019-10-15 19:45:58 +02:00
run clang format
2023-04-13 15:50:13 +02:00
if (!p2p_votes.empty()) {
NOTIFY_NEW_SERVICE_NODE_VOTE::request req{};
req.votes = std::move(p2p_votes);
cryptonote_connection_context fake_context{};
get_protocol()->relay_service_node_votes(req, fake_context);
Merge dev into upstream
2019-05-01 08:01:17 +02:00
}
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
void core::set_service_node_votes_relayed(const std::vector<service_nodes::quorum_vote_t>& votes) {
Keep checkpoint votes over p2p Checkpoint votes were only going over quorumnet, which was quite broken as they need to go out universally: random nodes piece together the individual checkpoints to create checkpoints on their own, and so need to see all of the votes. With the current code only service nodes that participated in the specific quorum ever saw the checkpoint votes. This commit returns checkpoint vote distribution to distribution over p2p. We could, in theory, do checkpoint vote collection over quorumnet and then relay the set of votes as a pack to reduce p2p traffic a bit, but this wouldn't be a huge optimization since we'd still have to distribute all the votes over p2p at some point anyway, so leaving that as a future optimization for now. (Obligations votes, in contrast, don't need to be distributed at all -- if a vote results in a state change, the quorum members themselves can produce and distribute the state change tx).
2019-12-10 19:57:36 +01:00
m_quorum_cop.set_votes_relayed(votes);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::create_next_miner_block_template(
block& b,
const account_public_address& adr,
difficulty_type& diffic,
uint64_t& height,
uint64_t& expected_reward,
const std::string& ex_nonce) {
return m_blockchain_storage.create_next_miner_block_template(
b, adr, diffic, height, expected_reward, ex_nonce);
}
//-----------------------------------------------------------------------------------------------
bool core::create_miner_block_template(
block& b,
const crypto::hash* prev_block,
const account_public_address& adr,
difficulty_type& diffic,
uint64_t& height,
uint64_t& expected_reward,
const std::string& ex_nonce) {
return m_blockchain_storage.create_miner_block_template(
b, prev_block, adr, diffic, height, expected_reward, ex_nonce);
}
//-----------------------------------------------------------------------------------------------
bool core::find_blockchain_supplement(
const std::list<crypto::hash>& qblock_ids,
NOTIFY_RESPONSE_CHAIN_ENTRY::request& resp) const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.find_blockchain_supplement(qblock_ids, resp);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::find_blockchain_supplement(
const uint64_t req_start_block,
const std::list<crypto::hash>& qblock_ids,
std::vector<std::pair<
std::pair<std::string, crypto::hash>,
std::vector<std::pair<crypto::hash, std::string>>>>& blocks,
uint64_t& total_height,
uint64_t& start_height,
bool pruned,
bool get_miner_tx_hash,
size_t max_count) const {
return m_blockchain_storage.find_blockchain_supplement(
req_start_block,
qblock_ids,
blocks,
total_height,
start_height,
pruned,
get_miner_tx_hash,
max_count);
}
//-----------------------------------------------------------------------------------------------
bool core::get_outs(
const rpc::GET_OUTPUTS_BIN::request& req, rpc::GET_OUTPUTS_BIN::response& res) const {
Fake outs set is now decided by the wallet This plugs a privacy leak from the wallet to the daemon, as the daemon could previously see what input is included as a transaction input, which the daemon hadn't previously supplied. Now, the wallet requests a particular set of outputs, including the real one. This can result in transactions that can't be accepted if the wallet happens to select too many outputs with non standard unlock times. The daemon could know this and select another output, but the wallet is blind to it. It's currently very unlikely since I don't think anything uses non default unlock times. The wallet requests more outputs than necessary so it can use spares if any of the returns outputs are still locked. If there are not enough spares to reach the desired mixin, the transaction will fail.
2016-08-02 22:48:09 +02:00
return m_blockchain_storage.get_outs(req, res);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_output_distribution(
uint64_t amount,
uint64_t from_height,
uint64_t to_height,
uint64_t& start_height,
std::vector<uint64_t>& distribution,
uint64_t& base) const {
return m_blockchain_storage.get_output_distribution(
amount, from_height, to_height, start_height, distribution, base);
}
//-----------------------------------------------------------------------------------------------
void core::get_output_blacklist(std::vector<uint64_t>& blacklist) const {
output_blacklist: Remove redundant return, report exception
2020-06-02 07:50:27 +02:00
m_blockchain_storage.get_output_blacklist(blacklist);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.get_tx_outputs_gindexs(tx_id, indexs);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_tx_outputs_gindexs(
const crypto::hash& tx_id,
size_t n_txes,
std::vector<std::vector<uint64_t>>& indexs) const {
blockchain_db: speedup tx output gathering We know all the data we'll want for getblocks.bin is contiguous
2018-12-16 14:28:49 +01:00
return m_blockchain_storage.get_tx_outputs_gindexs(tx_id, n_txes, indexs);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::pause_mine() {
moved all stuff to github
2014-03-03 23:07:58 +01:00
m_miner.pause();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::resume_mine() {
moved all stuff to github
2014-03-03 23:07:58 +01:00
m_miner.resume();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
block_complete_entry get_block_complete_entry(block& b, tx_memory_pool& pool) {
Merge commit '1ba9bafd3331f6534242e682bafbd6d863be42ed' into MergeUpstream3
2020-05-25 03:48:32 +02:00
block_complete_entry bce = {};
run clang format
2023-04-13 15:50:13 +02:00
bce.block = cryptonote::block_to_blob(b);
for (const auto& tx_hash : b.tx_hashes) {
std::string txblob;
CHECK_AND_ASSERT_THROW_MES(
pool.get_transaction(tx_hash, txblob), "Transaction not found in pool");
bce.txs.push_back(txblob);
core: call {prepare|cleanup}_handle_incoming_blocks when adding a mined block This ensures that a batch can't also be started/stopped out of sync by another thread and us getting in the middle
2017-03-19 11:25:12 +01:00
}
return bce;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::handle_block_found(block& b, block_verification_context& bvc) {
Remove boost::value_initialized and BOOST_FOREACH (#921) Neither of these have a place in modern C++11; boost::value_initialized is entirely superseded by `Type var{};` which does value initialization (or default construction if a default constructor is defined). More problematically, each `boost::value_initialized<T>` requires instantiation of another wrapping templated type which is a pointless price to pay the compiler in C++11 or newer. Also removed is the AUTO_VAL_INIT macro (which is just a simple macro around constructing a boost::value_initialized<T>). BOOST_FOREACH is a similarly massive pile of code to implement C++11-style for-each loops. (And bizarrely it *doesn't* appear to fall back to C++ for-each loops even when under a C++11 compiler!) This removes both entirely from the codebase.
2019-10-31 23:26:58 +01:00
bvc = {};
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-16 01:16:02 +02:00
std::vector<block_complete_entry> blocks;
Handle multi state changes in the pool (#781) * Update state transition check to account for height and universally set timestamp on recommission Reject invalidated state changes by their height after HF13 * Prune invalidated state changes on blockchain increment Simplify check_tx_inputs for state_changes by using service node list Instead of querying the last 60 historical blocks for every transaction, use the service node list and determine the state of the service node and if it can transition to its new state. We also now enforce at hardfork 13, that the network cannot commit transactions to the network if they would have been invalidated by a newer state change that already is already on the blockchain. This is backwards compatible all the way back to hardfork 9. Greatly simplify state change tx pruning on block added Use the new stricter rules for pruning state changes in the txpool We can do so because pruning the TX pool won't cause issues at the protocol level, but the more people we can upgrade the better network behaviour we get in terms of propagating more intrinsically correct ordering of state changes to other peers. * Don't generate state changes if not valid, disallow voting if node is non-votable
2019-08-12 01:48:47 +02:00
m_miner.pause();
cryptonote: avoid double parsing blocks when syncing
2018-11-23 14:05:48 +01:00
{
run clang format
2023-04-13 15:50:13 +02:00
OXEN_DEFER {
m_miner.resume();
};
try {
blocks.push_back(get_block_complete_entry(b, m_mempool));
} catch (const std::exception& e) {
return false;
}
std::vector<block> pblocks;
if (!prepare_handle_incoming_blocks(blocks, pblocks)) {
log::error(logcat, "Block found, but failed to prepare to add");
return false;
}
// add_new_block will verify block and set bvc.m_verification_failed accordingly
add_new_block(b, bvc, nullptr /*checkpoint*/);
cleanup_handle_incoming_blocks(true);
m_miner.on_block_chain_update();
}
if (bvc.m_verifivation_failed) {
bool pulse = cryptonote::block_has_pulse_components(b);
log::error(
log::Cat("verify"),
"{} block failed verification\n{}",
(pulse ? "Pulse" : "Mined"),
cryptonote::obj_to_json_str(b));
Handle multi state changes in the pool (#781) * Update state transition check to account for height and universally set timestamp on recommission Reject invalidated state changes by their height after HF13 * Prune invalidated state changes on blockchain increment Simplify check_tx_inputs for state_changes by using service node list Instead of querying the last 60 historical blocks for every transaction, use the service node list and determine the state of the service node and if it can transition to its new state. We also now enforce at hardfork 13, that the network cannot commit transactions to the network if they would have been invalidated by a newer state change that already is already on the blockchain. This is backwards compatible all the way back to hardfork 9. Greatly simplify state change tx pruning on block added Use the new stricter rules for pruning state changes in the txpool We can do so because pruning the TX pool won't cause issues at the protocol level, but the more people we can upgrade the better network behaviour we get in terms of propagating more intrinsically correct ordering of state changes to other peers. * Don't generate state changes if not valid, disallow voting if node is non-votable
2019-08-12 01:48:47 +02:00
return false;
run clang format
2023-04-13 15:50:13 +02:00
} else if (bvc.m_added_to_main_chain) {
std::unordered_set<crypto::hash> missed_txs;
std::vector<std::string> txs;
m_blockchain_storage.get_transactions_blobs(b.tx_hashes, txs, &missed_txs);
if (missed_txs.size() &&
m_blockchain_storage.get_block_id_by_height(get_block_height(b)) != get_block_hash(b)) {
log::info(
logcat,
"Block found but, seems that reorganize just happened after that, do not relay "
"this block");
return true;
}
CHECK_AND_ASSERT_MES(
txs.size() == b.tx_hashes.size() && !missed_txs.size(),
false,
"can't find some transactions in found block:"
<< get_block_hash(b) << " txs.size()=" << txs.size()
<< ", b.tx_hashes.size()=" << b.tx_hashes.size() << ", missed_txs.size()"
<< missed_txs.size());
moved all stuff to github
2014-03-03 23:07:58 +01:00
run clang format
2023-04-13 15:50:13 +02:00
cryptonote_connection_context exclude_context{};
NOTIFY_NEW_FLUFFY_BLOCK::request arg{};
arg.current_blockchain_height = m_blockchain_storage.get_current_blockchain_height();
arg.b = blocks[0];
moved all stuff to github
2014-03-03 23:07:58 +01:00
run clang format
2023-04-13 15:50:13 +02:00
m_pprotocol->relay_block(arg, exclude_context);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
functional_tests: add more blockchain related tests Related to emission, reorgs, getting tx data back, output distribution and histogram
2019-03-23 17:20:08 +01:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::on_synchronized() {
moved all stuff to github
2014-03-03 23:07:58 +01:00
m_miner.on_synchronized();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::safesyncmode(const bool onoff) {
Cleanup test impact of adding safesyncmode() method
2017-08-22 15:54:18 +02:00
m_blockchain_storage.safesyncmode(onoff);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::add_new_block(
const block& b, block_verification_context& bvc, checkpoint_t const* checkpoint) {
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
bool result = m_blockchain_storage.add_new_block(b, bvc, checkpoint);
if (result)
run clang format
2023-04-13 15:50:13 +02:00
relay_service_node_votes(); // NOTE: nop if synchronising due to not accepting votes whilst
// syncing
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
return result;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::prepare_handle_incoming_blocks(
const std::vector<block_complete_entry>& blocks_entry, std::vector<block>& blocks) {
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
m_incoming_tx_lock.lock();
run clang format
2023-04-13 15:50:13 +02:00
if (!m_blockchain_storage.prepare_handle_incoming_blocks(blocks_entry, blocks)) {
cleanup_handle_incoming_blocks(false);
return false;
cryptonote: avoid double parsing blocks when syncing
2018-11-23 14:05:48 +01:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
run clang format
2023-04-13 15:50:13 +02:00
//-----------------------------------------------------------------------------------------------
bool core::cleanup_handle_incoming_blocks(bool force_sync) {
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 16:43:32 +02:00
bool success = false;
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
try {
run clang format
2023-04-13 15:50:13 +02:00
success = m_blockchain_storage.cleanup_handle_incoming_blocks(force_sync);
} catch (...) {
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 15:06:55 +02:00
}
m_incoming_tx_lock.unlock();
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 16:43:32 +02:00
return success;
run clang format
2023-04-13 15:50:13 +02:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 22:09:32 +02:00
run clang format
2023-04-13 15:50:13 +02:00
//-----------------------------------------------------------------------------------------------
bool core::handle_incoming_block(
const std::string& block_blob,
const block* b,
block_verification_context& bvc,
checkpoint_t* checkpoint,
bool update_miner_blocktemplate) {
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
TRY_ENTRY();
Remove boost::value_initialized and BOOST_FOREACH (#921) Neither of these have a place in modern C++11; boost::value_initialized is entirely superseded by `Type var{};` which does value initialization (or default construction if a default constructor is defined). More problematically, each `boost::value_initialized<T>` requires instantiation of another wrapping templated type which is a pointless price to pay the compiler in C++11 or newer. Also removed is the AUTO_VAL_INIT macro (which is just a simple macro around constructing a boost::value_initialized<T>). BOOST_FOREACH is a similarly massive pile of code to implement C++11-style for-each loops. (And bizarrely it *doesn't* appear to fall back to C++ for-each loops even when under a C++11 compiler!) This removes both entirely from the codebase.
2019-10-31 23:26:58 +01:00
bvc = {};
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
run clang format
2023-04-13 15:50:13 +02:00
if (!check_incoming_block_size(block_blob)) {
bvc.m_verifivation_failed = true;
return false;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
if (((size_t)-1) <= 0xffffffff && block_blob.size() >= 0x3fffffff)
run clang format
2023-04-13 15:50:13 +02:00
log::warning(
logcat, "This block's size is {}, closing on the 32 bit limit", block_blob.size());
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
run clang format
2023-04-13 15:50:13 +02:00
CHECK_AND_ASSERT_MES(
update_checkpoints_from_json_file(),
false,
"One or more checkpoints loaded from json conflicted with existing checkpoints.");
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
core: avoid double parsing blocks after hoh
2018-11-23 12:15:04 +01:00
block lb;
run clang format
2023-04-13 15:50:13 +02:00
if (!b) {
crypto::hash block_hash;
if (!parse_and_validate_block_from_blob(block_blob, lb, block_hash)) {
log::info(logcat, "Failed to parse and validate new block");
bvc.m_verifivation_failed = true;
return false;
}
b = &lb;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Enforce ordering of votes in HF13
2019-08-14 06:52:51 +02:00
Service node checkpointing: rebased on relaxed deregistration (#662) * core: do not commit half constructed batch db txn * Add defer macro * Revert dumb extra copy/move change * Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes * Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes * Update tests to compile with relaxed-registration changes * Get back to feature parity pre-relaxed registration changes * Remove debug changes noticed in code review and some small bugs
2019-06-26 06:00:05 +02:00
add_new_block(*b, bvc, checkpoint);
run clang format
2023-04-13 15:50:13 +02:00
if (update_miner_blocktemplate && bvc.m_added_to_main_chain)
m_miner.on_block_chain_update();
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 14:01:45 +02:00
CATCH_ENTRY_L0("core::handle_incoming_block()", false);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
// Used by the RPC server to check the size of an incoming
// block_blob
bool core::check_incoming_block_size(const std::string& block_blob) const {
cryptonote: rework block blob size sanity check Use the actual block weight limit, assuming that weight is always greater or equal to size
2019-04-04 02:15:57 +02:00
// note: we assume block weight is always >= block blob size, so we check incoming
// blob size against the block weight limit, which acts as a sanity check without
// having to parse/weigh first; in fact, since the block blob is the block header
// plus the tx hashes, the weight will typically be much larger than the blob size
run clang format
2023-04-13 15:50:13 +02:00
if (block_blob.size() > m_blockchain_storage.get_current_cumulative_block_weight_limit() +
BLOCK_SIZE_SANITY_LEEWAY) {
log::info(
logcat,
"WRONG BLOCK BLOB, sanity check failed on size {}, rejected",
block_blob.size());
return false;
Fix for orphan fix (check blocksize)
2014-06-11 23:32:53 +02:00
}
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
Update for LokiMQ 1.1.0's changes (#1122) With the 1.1.0 update, rather than LokiMQ getting a callback that it invokes on connection to get auth level for the duration of that connection, it now checks authentication each time a command is invoked. This improves message reliability (because on an invocation failure the remote doesn't have to reconnect to be reauthenticated). This helps storage server greatly (which has some tricky initialization issue that requires it to initialize lokimq before it has the SN list); it is less needed for quorumnet communications (which have been all using connection-time authenciation for a while now), but this change does some one "weirdness" that a remote client may be unable to issue a command that they *can* issue if they reconnect. The lokid update here piggypacks the call into quorumnet in quorum_cop's `add_block` callback to make the call; this would be far cleaner if we replaced the callbacks with stateful std::function's instead of inherited base class pointers, but I didn't want to go that far in this PR.
2020-04-15 01:22:18 +02:00
run clang format
2023-04-13 15:50:13 +02:00
void core::update_omq_sns() {
lokinet revert
2021-01-04 05:21:21 +01:00
// TODO: let callers (e.g. lokinet, ss) subscribe to callbacks when this fires
Rename lokimq -> oxenmq
2021-01-14 21:28:50 +01:00
oxenmq::pubkey_set active_sns;
RPC overhaul High-level details: This redesigns the RPC layer to make it much easier to work with, decouples it from an embedded HTTP server, and gets the vast majority of the RPC serialization and dispatch code out of a very commonly included header. There is unfortunately rather a lot of interconnected code here that cannot be easily separated out into separate commits. The full details of what happens here are as follows: Major details: - All of the RPC code is now in a `cryptonote::rpc` namespace; this renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME` becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already working in the `rpc` namespace). - `core_rpc_server` is now completely decoupled from providing any request protocol: it is now *just* the core RPC call handler. - The HTTP RPC interface now lives in a new rpc/http_server.h; this code handles listening for HTTP requests and dispatching them to core_rpc_server, then sending the results back to the caller. - There is similarly a rpc/lmq_server.h for LMQ RPC code; more details on this (and other LMQ specifics) below. - RPC implementing code now returns the response object and throws when things go wrong which simplifies much of the rpc error handling. They can throw anything; generic exceptions get logged and a generic "internal error" message gets returned to the caller, but there is also an `rpc_error` class to return an error code and message used by some json-rpc commands. - RPC implementing functions now overload `core_rpc_server::invoke` following the pattern: RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context); This overloading makes the code vastly simpler: all instantiations are now done with a small amount of generic instantiation code in a single .cpp rather than needing to go to hell and back with a nest of epee macros in a core header. - each RPC endpoint is now defined by the RPC types themselves, including its accessible names and permissions, in core_rpc_server_commands_defs.h: - every RPC structure now has a static `names()` function that returns the names by which the end point is accessible. (The first one is the primary, the others are for deprecated aliases). - RPC command wrappers define their permissions and type by inheriting from special tag classes: - rpc::RPC_COMMAND is a basic, admin-only, JSON command, available via JSON RPC. *All* JSON commands are now available via JSON RPC, instead of the previous mix of some being at /foo and others at /json_rpc. (Ones that were previously at /foo are still there for backwards compatibility; see `rpc::LEGACY` below). - rpc::PUBLIC specifies that the command should be available via a restricted RPC connection. - rpc::BINARY specifies that the command is not JSON, but rather is accessible as /name and takes and returns values in the magic epee binary "portable storage" (lol) data format. - rpc::LEGACY specifies that the command should be available via the non-json-rpc interface at `/name` for backwards compatibility (in addition to the JSON-RPC interface). - some epee serialization got unwrapped and de-templatized so that it can be moved into a .cpp file with just declarations in the .h. (This makes a *huge* difference for core_rpc_server_commands_defs.h and for every compilation unit that includes it which previously had to compile all the serialization code and then throw all by one copy away at link time). This required some new macros so as to not break a ton of places that will use the old way putting everything in the headers; The RPC code uses this as does a few other places; there are comments in contrib/epee/include/serialization/keyvalue_serialization.h as to how to use it. - Detemplatized a bunch of epee/storages code. Most of it should have have been using templates at all (because it can only ever be called with one type!), and now it isn't. This broke some things that didn't properly compile because of missing headers or (in one case) a messed up circular dependency. - Significantly simplified a bunch of over-templatized serialization code. - All RPC serialization definitions is now out of core_rpc_server_commands_defs.h and into a single .cpp file (core_rpc_server_commands_defs.cpp). - core RPC no longer uses the disgusting BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design that forced slamming tons of code into a common header that didn't need to be there. - epee::struct_init is gone. It was a horrible hack that instiated multiple templates just so the coder could be so lazy and write `some_type var;` instead of properly value initializing with `some_type var{};`. - Removed a bunch of useless crap from epee. In particular, forcing extra template instantiations all over the place in order to nest return objects inside JSON RPC values is no longer needed, as are a bunch of stuff related to the above de-macroization of the code. - get_all_service_nodes, get_service_nodes, and get_n_service_nodes are now combined into a single `get_service_nodes` (with deprecated aliases for the others), which eliminates a fair amount of duplication. The biggest obstacle here was getting the requested fields reference passed through: this is now done by a new ability to stash a context in the serialization object that can be retrieved by a sub-serialized type. LMQ-specifics: - The LokiMQ instance moves into `cryptonote::core` rather than being inside cryptonote_protocol. Currently the instance is used both for qnet and rpc calls (and so needs to be in a common place), but I also intend future PRs to use the batching code for job processing (replacing the current threaded job queue). - rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue. Unlike http_server it isn't technically running the whole LMQ stack from here, but the parallel name with http_server seemed appropriate. - All RPC endpoints are supported by LMQ under the same names as defined generically, but prefixed with `rpc.` for public commands and `admin.` for restricted ones. - service node keys are now always available, even when not running in `--service-node` mode: this is because we want the x25519 key for being able to offer CURVE encryption for lmq RPC end-points, and because it doesn't hurt to have them available all the time. In the RPC layer this is now called "get_service_keys" (with "get_service_node_key" as an alias) since they aren't strictly only for service nodes. This also means code needs to check m_service_node, and not m_service_node_keys, to tell if it is running as a service node. (This is also easier to notice because m_service_node_keys got renamed to `m_service_keys`). - Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and `sub.mempool` subscribes the connection for new block and new mempool TX notifications. The latter can notify on just blink txes, or all new mempool txes (but only new ones -- txes dumped from a block don't trigger it). The client gets pushed a [`notify.block`, `height`, `hash`] or [`notify.tx`, `txhash`, `blob`] message when something arrives. Minor details: - rpc::version_t is now a {major,minor} pair. Forcing everyone to pack and unpack a uint32_t was gross. - Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...). (This immediately revealed a couple of bugs in the RPC code that was assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because the macro allows implicit conversion to a char). - De-templatizing useless templates in epee (i.e. a bunch of templated types that were never invoked with different types) revealed a painful circular dependency between epee and non-epee code for tor_address and i2p_address. This crap is now handled in a suitably named `net/epee_network_address_hack.cpp` hack because it really isn't trivial to extricate this mess. - Removed `epee/include/serialization/serialize_base.h`. Amazingly the code somehow still all works perfectly with this previously vital header removed. - Removed bitrotted, unused epee "crypted_storage" and "gzipped_inmemstorage" code. - Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with LOKI_DEFERs. The epee version involves quite a bit more instantiation and is ugly as sin. Also made the `loki::defer` class invokable for some edge cases that need calling before destruction in particular conditions. - Moved the systemd code around; it makes much more sense to do the systemd started notification as in daemon.cpp as late as possible rather than in core (when we can still have startup failures, e.g. if the RPC layer can't start). - Made the systemd short status string available in the get_info RPC (and no longer require building with systemd). - during startup, print (only) the x25519 when not in SN mode, and continue to print all three when in SN mode. - DRYed out some RPC implementation code (such as set_limit) - Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
m_service_node_list.copy_active_x25519_pubkeys(std::inserter(active_sns, active_sns.end()));
lmq -> omq internal rename More rebrand updates to rename lmq (lokimq) internals with omq (oxenmq).
2021-04-08 18:38:51 +02:00
m_omq->set_active_sns(std::move(active_sns));
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
crypto::hash core::get_tail_id() const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.get_tail_id();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
difficulty_type core::get_block_cumulative_difficulty(uint64_t height) const {
protocol: pass blockchain cumulative difficulty when syncing Not used yet.
2017-08-12 16:24:39 +02:00
return m_blockchain_storage.get_db().get_block_cumulative_difficulty(height);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::have_block(const crypto::hash& id) const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.have_block(id);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
crypto::hash core::get_block_id_by_height(uint64_t height) const {
moved all stuff to github
2014-03-03 23:07:58 +01:00
return m_blockchain_storage.get_block_id_by_height(height);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_block_by_hash(const crypto::hash& h, block& blk, bool* orphan) const {
rpc: fix orphan_status when getting blocks It was always set to false, even for orphan blocks
2017-01-22 13:20:55 +01:00
return m_blockchain_storage.get_block_by_hash(h, blk, orphan);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::get_block_by_height(uint64_t height, block& blk) const {
More efficient get-block-by-height lookups Currently where we need to look up a block by height we do: 1. get block hash for the given height 2. look up block by hash This hits the lmdb layer and does: 3. look up height from hash in hashes-to-height table 4. look up block from height in blocks table which is pointless. This commit adds a `get_block_by_height()` that avoids the extra runaround, and converts code doing height lookups to use the new method.
2020-08-28 21:11:25 +02:00
return m_blockchain_storage.get_block_by_height(height, blk);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
static bool check_external_ping(
time_t last_ping, std::chrono::seconds lifetime, std::string_view what) {
Tweak uptime proof times; reduce times for testnet This makes uptime proof times network-dependent, and tweaks them a bit. Also converts the times to type-safe std::chrono types rather than macros. Mainnet/testnet/devnet: - Send the first proof 30s after startup rather than waiting 2 minutes. - Check for whether we need to send a proof every 30s rather than every 5mins. Mainnet: Other times unchanged. Testnet/devnet: - Send proofs every 10min instead of 1h, and consider nodes to be down after 21m instead of 2h5m. Fakechain: - Send 5s after startup, check every 5s, and send every 1min. - Expiry after 2min5s Also remove the cmake debug option for short proofs since the fakechain changes above basically incorporate what it did.
2021-03-24 23:20:16 +01:00
const std::chrono::seconds elapsed{std::time(nullptr) - last_ping};
run clang format
2023-04-13 15:50:13 +02:00
if (elapsed > lifetime) {
log::warning(
logcat,
"Have not heard from {} {}",
what,
(!last_ping ? "since starting"
: "since more than " + tools::get_human_readable_timespan(elapsed) +
" ago"));
return false;
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
}
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
void core::reset_proof_interval() {
Replace once_a_time_seconds; send proofs faster This replaces the horrible, horrible, badly misused templated once_a_time_seconds and once_a_time_milliseconds with a `periodic_task` that works the same way but takes parameters as constructor arguments instead of template parameters. It also makes various small improvements: - uses std::chrono::steady_clock instead of ifdef'ing platform dependent timer code. - takes a std::chrono duration rather than a template integer and scaling parameter. - timers can be reset to trigger on the next invocation, and this is thread-safe. - timer intervals can be changed at run-time. This all then gets used to reset the proof timer immediately upon receiving a ping (initially or after expiring) from storage server and lokinet so that we send proofs out faster.
2019-12-20 19:21:24 +01:00
m_check_uptime_proof_interval.reset();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::do_uptime_proof_call() {
std::vector<service_nodes::service_node_pubkey_info> const states =
get_service_node_list_state({m_service_keys.pub});
Service node checkpointing deregistration (#672) * Add deregistration of checkpoints by checking how many votes are missed Move uptime proofs and add checkpoint counts in the service_node_list because we typically prune uptime proofs by time, but it seems we want to switch to a model where we persist proof data until the node expires otherwise currently we would prune uptime entries and potentially our checkpoint vote counts which would cause premature deregistration as the expected vote counts start mismatching with the number of received votes. * Revise deregistration * Fix test breakages * uint16_t for port, remove debug false, min votes to 2 in integration mode * Fix integration build
2019-06-27 09:05:44 +02:00
Batching of service node rewards This updates the coinbase transactions to reward service nodes periodically rather than every block. If you recieve a service node reward this reward will be delayed x blocks, if you receive another reward to the same wallet before those blocks have been completed it will be added to your total and all will be paid out after those x blocks has passed. For example if our batching interval is 2 blocks: Block 1 - Address A receives reward of 10 oxen - added to batch Block 2 - Address A receives reward of 10 oxen - added to batch Block 3 - Address A is paid out 20 oxen. Batching accumulates a small reward for all nodes every block The batching of service node rewards allows us to drip feed rewards to service nodes. Rather than accruing each service node 16.5 oxen every time they are pulse block leader we now reward every node the 16.5 / num_service_nodes every block and pay each wallet the full amount that has been accrued after a period of time (Likely 3.5 days). To spread each payment evenly we now pay the rewards based on the address of the recipient. This modulus of their address determines which block the address should be paid and by setting the interval to our service_node_batching interval we can guarantee they will be paid out regularly and evenly distribute the payments for all wallets over this
2022-04-29 01:51:14 +02:00
// wait one block before starting uptime proofs (but not on testnet/devnet, where we sometimes
// have mass registrations/deregistrations where the waiting causes problems).
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
uint64_t delay_blocks = m_nettype == network_type::MAINNET ? 1 : 0;
run clang format
2023-04-13 15:50:13 +02:00
if (!states.empty() &&
(states[0].info->registration_height + delay_blocks) < get_current_blockchain_height()) {
m_check_uptime_proof_interval.do_call([this]() {
// This timer is not perfectly precise and can leak seconds slightly, so send the uptime
// proof if we are within half a tick of the target time. (Essentially our target proof
// window becomes the first time this triggers in the 59.75-60.25 minute window).
uint64_t next_proof_time = 0;
m_service_node_list.access_proof(
m_service_keys.pub, [&](auto& proof) { next_proof_time = proof.timestamp; });
auto& netconf = get_net_config();
next_proof_time +=
std::chrono::seconds{
netconf.UPTIME_PROOF_FREQUENCY -
netconf.UPTIME_PROOF_CHECK_INTERVAL / 2}
.count();
if ((uint64_t)std::time(nullptr) < next_proof_time)
return;
Warn and refuse to submit proofs if we detect duplicate ed/x25519 keys (#1121)
2020-04-14 02:53:40 +02:00
run clang format
2023-04-13 15:50:13 +02:00
auto pubkey = m_service_node_list.get_pubkey_from_x25519(m_service_keys.pub_x25519);
if (pubkey && pubkey != m_service_keys.pub &&
m_service_node_list.is_service_node(pubkey, false /*don't require active*/)) {
log::info(
logcat,
fg(fmt::terminal_color::red),
"Failed to submit uptime proof: another service node on the network is "
"using the same ed/x25519 keys as this service node. This typically means "
"both have the same 'key_ed25519' private key file.");
return;
}
Add a check for duplicate storage server broadcasts Some users have gotten deregistered because of a cloned configuration that had multiple nodes broadcasting the same SS IP/port. This adds a check and loud warning when sending out a proof if some other SN is broadcasting the same SS info.
2020-10-19 16:33:13 +02:00
run clang format
2023-04-13 15:50:13 +02:00
{
std::vector<crypto::public_key> sn_pks;
auto sns = m_service_node_list.get_service_node_list_state();
sn_pks.reserve(sns.size());
for (const auto& sni : sns)
sn_pks.push_back(sni.pubkey);
m_service_node_list.for_each_service_node_info_and_proof(
sn_pks.begin(), sn_pks.end(), [&](auto& pk, auto& sni, auto& proof) {
if (pk != m_service_keys.pub &&
proof.proof->public_ip == m_sn_public_ip &&
(proof.proof->qnet_port == m_quorumnet_port ||
(m_nettype != network_type::DEVNET &&
(proof.proof->storage_https_port == storage_https_port() ||
proof.proof->storage_omq_port == storage_omq_port()))))
log::info(
logcat,
fg(fmt::terminal_color::red),
"Another service node ({}) is broadcasting the same public "
"IP and ports as this service node ({}:{}[qnet], "
":{}[SS-HTTP], :{}[SS-OMQ]). This will lead to "
"deregistration of one or both service nodes if not "
"corrected. (Do both service nodes have the correct IP for "
"the service-node-public-ip setting?)",
pk,
epee::string_tools::get_ip_string_from_int32(
m_sn_public_ip),
proof.proof->qnet_port,
proof.proof->storage_https_port,
proof.proof->storage_omq_port);
});
}
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
run clang format
2023-04-13 15:50:13 +02:00
if (m_nettype != network_type::DEVNET) {
if (!check_external_ping(
m_last_storage_server_ping,
get_net_config().UPTIME_PROOF_FREQUENCY,
"the storage server")) {
log::info(
logcat,
fg(fmt::terminal_color::red),
"Failed to submit uptime proof: have not heard from the storage server "
"recently. Make sure that it is running! It is required to run "
"alongside the Loki daemon");
return;
}
if (!check_external_ping(
m_last_lokinet_ping,
get_net_config().UPTIME_PROOF_FREQUENCY,
"Lokinet")) {
log::info(
logcat,
fg(fmt::terminal_color::red),
"Failed to submit uptime proof: have not heard from lokinet recently. "
"Make sure that it is running! It is required to run alongside the "
"Loki daemon");
return;
}
}
submit_uptime_proof();
});
} else {
// reset the interval so that we're ready when we register, OR if we get deregistered this
// primes us up for re-registration in the same session
m_check_uptime_proof_interval.reset();
core: submit uptime proof immediately after registering
2018-08-22 12:47:32 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::on_idle() {
if (!m_starter_message_showed) {
std::string main_message;
if (m_offline)
main_message =
"The daemon is running offline and will not attempt to sync to the Loki "
"network.";
else
main_message =
"The daemon will start synchronizing with the network. This may take a long "
"time to complete.";
log::info(
logcat,
fg(fmt::terminal_color::yellow),
R"(
Unwrap fmt::format with a style argument oxen::logging handles a style argument now (it is also safer e.g. if the final formatted string itself intentionally contains {}).
2022-09-23 02:35:35 +02:00
**********************************************************************
{}
You can set the level of process detailization through "set_log <level|categories>" command,
where <level> is between 0 (no details) and 4 (very verbose), or custom category based levels (eg, *:WARNING).
Use the "help" command to see the list of available commands.
Use "help <command>" to see a command's documentation.
**********************************************************************
run clang format
2023-04-13 15:50:13 +02:00
)",
main_message);
m_starter_message_showed = true;
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Replace boost::bind with lambdas
2020-06-03 05:10:08 +02:00
m_txpool_auto_relayer.do_call([this] { return relay_txpool_transactions(); });
m_service_node_vote_relayer.do_call([this] { return relay_service_node_votes(); });
m_check_disk_space_interval.do_call([this] { return check_disk_space(); });
m_block_rate_interval.do_call([this] { return check_block_rate(); });
run clang format
2023-04-13 15:50:13 +02:00
m_sn_proof_cleanup_interval.do_call([&snl = m_service_node_list] {
snl.cleanup_proofs();
return true;
});
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
Tweak uptime proof times; reduce times for testnet This makes uptime proof times network-dependent, and tweaks them a bit. Also converts the times to type-safe std::chrono types rather than macros. Mainnet/testnet/devnet: - Send the first proof 30s after startup rather than waiting 2 minutes. - Check for whether we need to send a proof every 30s rather than every 5mins. Mainnet: Other times unchanged. Testnet/devnet: - Send proofs every 10min instead of 1h, and consider nodes to be down after 21m instead of 2h5m. Fakechain: - Send 5s after startup, check every 5s, and send every 1min. - Expiry after 2min5s Also remove the cmake debug option for short proofs since the fakechain changes above basically incorporate what it did.
2021-03-24 23:20:16 +01:00
std::chrono::seconds lifetime{time(nullptr) - get_start_time()};
run clang format
2023-04-13 15:50:13 +02:00
if (m_service_node &&
lifetime > get_net_config().UPTIME_PROOF_STARTUP_DELAY) // Give us some time to connect to
// peers before sending uptimes
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
{
run clang format
2023-04-13 15:50:13 +02:00
do_uptime_proof_call();
Service Nodes Improve Reliability & Feedback (#262) * If unable to receive own uptime proof, switch to more granular pinging * Uptime proof simpler solution to repeatedly ping on failure
2018-09-27 10:31:08 +02:00
}
Replace boost::bind with lambdas
2020-06-03 05:10:08 +02:00
m_blockchain_pruning_interval.do_call([this] { return update_blockchain_pruning(); });
moved all stuff to github
2014-03-03 23:07:58 +01:00
m_miner.on_idle();
proper tx_pool handling from CryptoZoidberg / BBR
2014-06-15 09:48:13 +02:00
m_mempool.on_idle();
Revamp to use semaphores for IPC Delay hooks into stdin/out until core is idle for daemons Disable logging for daemon (super problematic)
2019-01-15 10:09:17 +01:00
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
#ifdef ENABLE_SYSTEMD
run clang format
2023-04-13 15:50:13 +02:00
m_systemd_notify_interval.do_call(
[this] { sd_notify(0, ("WATCHDOG=1\nSTATUS=" + get_status_string()).c_str()); });
Add systemd notify & watchdog support (#1022) This enables optional support for systemd notification which allows lokid to be run via `Type=notify`, allowing it to better signal status to systemd and enables systemd watchdog handling to restart if something goes wrong. Enabled here are: - systemd watchdog ping every 10s - systemd status update every 10s, so that `systemctl status loki-node` gives you a status line such as: Status: "Height: 450085, SN: active, proof: 15m12s, storage: 3m7s, lokinet: 27s" - initialization notification so that systemd can wait for and report on initialization status rather than just that the process has launched. - shutdown notification All of these require changing the service type to `Type=notify` in the `[Service]` section of the systemd service file; enabling the watchdog also requires adding a `WatchdogSec=5min` line in the `[Service]` section. The systemd support is optional and requires the libsystemd-dev package to be built (and is probably not feasible at all for a static build).
2020-01-19 23:44:18 +01:00
#endif
moved all stuff to github
2014-03-03 23:07:58 +01:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::check_disk_space() {
core: warn when free disk space is low
2017-11-12 00:25:12 +01:00
uint64_t free_space = get_free_space();
run clang format
2023-04-13 15:50:13 +02:00
if (free_space < 1ull * 1024 * 1024 * 1024) // 1 GB
log::warning(
logcat,
fg(fmt::terminal_color::red),
"Free space is below 1 GB on {}",
m_config_folder);
core: warn when free disk space is low
2017-11-12 00:25:12 +01:00
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
double factorial(unsigned int n) {
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
if (n <= 1)
run clang format
2023-04-13 15:50:13 +02:00
return 1.0;
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
double f = n;
while (n-- > 1)
run clang format
2023-04-13 15:50:13 +02:00
f *= n;
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
return f;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
static double probability1(unsigned int blocks, unsigned int expected) {
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
// https://www.umass.edu/wsp/resources/poisson/#computing
return pow(expected, blocks) / (factorial(blocks) * exp(expected));
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
static double probability(unsigned int blocks, unsigned int expected) {
core: improve block rate monitor trigger probabilities The original intent of one false positive a week on average was not met, since what we really want is not the probability of having N blocks in T seconds, but either N blocks of fewer in T seconds, or N blocks or more in T seconds. Some of this could be cached since it calculates the same fairly complex floating point values, but it seems pretty fast already.
2019-03-19 17:30:17 +01:00
double p = 0.0;
run clang format
2023-04-13 15:50:13 +02:00
if (blocks <= expected) {
for (unsigned int b = 0; b <= blocks; ++b)
p += probability1(b, expected);
} else if (blocks > expected) {
for (unsigned int b = blocks; b <= expected * 3 /* close enough */; ++b)
p += probability1(b, expected);
core: improve block rate monitor trigger probabilities The original intent of one false positive a week on average was not met, since what we really want is not the probability of having N blocks in T seconds, but either N blocks of fewer in T seconds, or N blocks or more in T seconds. Some of this could be cached since it calculates the same fairly complex floating point values, but it seems pretty fast already.
2019-03-19 17:30:17 +01:00
}
return p;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::check_block_rate() {
if (m_offline || m_nettype == network_type::FAKECHAIN ||
m_target_blockchain_height > get_current_blockchain_height() ||
m_target_blockchain_height == 0) {
log::debug(logcat, "Not checking block rate, offline or syncing");
return true;
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
}
run clang format
2023-04-13 15:50:13 +02:00
static constexpr double threshold =
1. / ((24h * 10) / TARGET_BLOCK_TIME); // one false positive every 10 days
core: point out when we hit the block rate visibility limit
2019-10-28 17:04:43 +01:00
static constexpr unsigned int max_blocks_checked = 150;
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
const time_t now = time(NULL);
run clang format
2023-04-13 15:50:13 +02:00
const std::vector<time_t> timestamps =
m_blockchain_storage.get_last_block_timestamps(max_blocks_checked);
static const unsigned int seconds[] = {5400, 3600, 1800, 1200, 600};
for (size_t n = 0; n < sizeof(seconds) / sizeof(seconds[0]); ++n) {
unsigned int b = 0;
const time_t time_boundary = now - static_cast<time_t>(seconds[n]);
for (time_t ts : timestamps)
b += ts >= time_boundary;
const double p = probability(b, seconds[n] / tools::to_seconds(TARGET_BLOCK_TIME));
log::debug(
logcat,
"blocks in the last {} minutes: {} (probability {})",
seconds[n] / 60,
b,
p);
if (p < threshold) {
log::warning(
logcat,
"There were {}{} blocks in the last {} minutes, \
Logging Refactor This replaces the current epee logging system with our oxen::log library. It replaces the easylogging library with spdlog, removes the macros and replaces with functions and standardises how we call the logs.
2022-09-13 08:57:56 +02:00
there might be large hash rate changes, or we might be partitioned, \
cut off from the Loki network or under attack, or your computer's time is off. \
run clang format
2023-04-13 15:50:13 +02:00
Or it could be just sheer bad luck.",
b,
(b == max_blocks_checked ? " or more" : ""),
seconds[n] / 60);
break; // no need to look further
}
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 15:31:45 +02:00
}
return true;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::flush_bad_txs_cache() {
rpc: add a flush_cache RPC This allows flushing internal caches (for now, the bad tx cache, which will allow debugging a stuck monerod after it has failed to verify a transaction in a block, since it would otherwise not try again, making subsequent log changes pointless)
2019-10-22 17:28:08 +02:00
bad_semantics_txes_lock.lock();
for (int idx = 0; idx < 2; ++idx)
run clang format
2023-04-13 15:50:13 +02:00
bad_semantics_txes[idx].clear();
rpc: add a flush_cache RPC This allows flushing internal caches (for now, the bad tx cache, which will allow debugging a stuck monerod after it has failed to verify a transaction in a block, since it would otherwise not try again, making subsequent log changes pointless)
2019-10-22 17:28:08 +02:00
bad_semantics_txes_lock.unlock();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::flush_invalid_blocks() {
rpc: add bad-blocks to flush_cache RPC Flushes m_invalid_blocks in Blockchain.
2019-12-02 19:51:28 +01:00
m_blockchain_storage.flush_invalid_blocks();
run clang format
2023-04-13 15:50:13 +02:00
}
bool core::update_blockchain_pruning() {
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
return m_blockchain_storage.update_blockchain_pruning();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::check_blockchain_pruning() {
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
return m_blockchain_storage.check_blockchain_pruning();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::set_target_blockchain_height(uint64_t target_blockchain_height) {
Revert "Fix fake 'network synchronized, begin using' messages" This reverts commit 78035d2b6c9922f4cd730df0766aa74f4854ccb2. The patch doesn't work, and causes constant SYNCHRONIZED OK spam.
2016-09-25 20:08:31 +02:00
m_target_blockchain_height = target_blockchain_height;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
uint64_t core::get_target_blockchain_height() const {
'getinfo' daemon HTTP-RPC returns 'target_height' for progress estimations
2014-06-04 22:50:13 +02:00
return m_target_blockchain_height;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
uint64_t core::prevalidate_block_hashes(uint64_t height, const std::vector<crypto::hash>& hashes) {
precomputed block hashes are now in blocks of N (currently 256) This shaves a lot of space off binaries
2017-09-13 22:03:04 +02:00
return get_blockchain_storage().prevalidate_block_hashes(height, hashes);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
uint64_t core::get_free_space() const {
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
return fs::space(m_config_folder).available;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
std::shared_ptr<const service_nodes::quorum> core::get_quorum(
service_nodes::quorum_type type,
uint64_t height,
bool include_old,
std::vector<std::shared_ptr<const service_nodes::quorum>>* alt_states) const {
Rename `testing_quorum` class to just `quorum` testing_quorum is a bit of a misnomer: it only does testing when used for an obligations quorum, but it is also used for checkpointing and soon Blink.
2019-10-15 19:52:49 +02:00
return m_service_node_list.get_quorum(type, height, include_old, alt_states);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::is_service_node(const crypto::public_key& pubkey, bool require_active) const {
Relax deregistration rules The replaces the deregistration mechanism with a new state change mechanism (beginning at the v12 fork) which can change a service node's network status via three potential values (and is extensible in the future to handle more): - deregistered -- this is the same as the existing deregistration; the SN is instantly removed from the SN list. - decommissioned -- this is a sort of temporary deregistration: your SN remains in the service node list, but is removed from the rewards list and from any network duties. - recommissioned -- this tx is sent by a quorum if they observe a decommissioned SN sending uptime proofs again. Upon reception, the SN is reactivated and put on the end of the reward list. Since this is broadening the quorum use, this also renames the relevant quorum to a "obligations" quorum (since it validates SN obligations), while the transactions are "state_change" transactions (since they change the state of a registered SN). The new parameters added to service_node_rules.h control how this works: // Service node decommissioning: as service nodes stay up they earn "credits" (measured in blocks) // towards a future outage. A new service node starts out with INITIAL_CREDIT, and then builds up // CREDIT_PER_DAY for each day the service node remains active up to a maximum of // DECOMMISSION_MAX_CREDIT. // // If a service node stops sending uptime proofs, a quorum will consider whether the service node // has built up enough credits (at least MINIMUM): if so, instead of submitting a deregistration, // it instead submits a decommission. This removes the service node from the list of active // service nodes both for rewards and for any active network duties. If the service node comes // back online (i.e. starts sending the required performance proofs again) before the credits run // out then a quorum will reinstate the service node using a recommission transaction, which adds // the service node back to the bottom of the service node reward list, and resets its accumulated // credits to 0. If it does not come back online within the required number of blocks (i.e. the // accumulated credit at the point of decommissioning) then a quorum will send a permanent // deregistration transaction to the network, starting a 30-day deregistration count down. This commit currently includes values (which are not necessarily finalized): - 8 hours (240 blocks) of credit required for activation of a decommission (rather than a deregister) - 0 initial credits at registration - a maximum of 24 hours (720 blocks) of credits - credits accumulate at a rate that you hit 24 hours of credits after 30 days of operation. Miscellaneous other details of this PR: - a new TX extra tag is used for the state change (including deregistrations). The old extra tag has no version or type tag, so couldn't be reused. The data in the new tag is slightly more efficiently packed than the old deregistration transaction, so it gets used for deregistrations (starting at the v12 fork) as well. - Correct validator/worker selection required generalizing the shuffle function to be able to shuffle just part of a vector. This lets us stick any down service nodes at the end of the potential list, then select validators by only shuffling the part of the index vector that contains active service indices. Once the validators are selected, the remainder of the list (this time including decommissioned SN indices) is shuffled to select quorum workers to check, thus allowing decommisioned nodes to be randomly included in the nodes to check without being selected as a validator. - Swarm recalculation was not quite right: swarms were recalculated on SN registrations, even if those registrations were include shared node registrations, but *not* recalculated on stakes. Starting with the upgrade this behaviour is fixed (swarms aren't actually used currently and aren't consensus-relevant so recalculating early won't hurt anything). - Details on decomm/dereg are added to RPC info and print_sn/print_sn_status - Slightly improves the % of reward output in the print_sn output by rounding it to two digits, and reserves space in the output string to avoid excessive reallocations. - Adds various debugging at higher debug levels to quorum voting (into all of voting itself, vote transmission, and vote reception). - Reset service node list internal data structure version to 0. The SN list has to be rescanned anyway at upgrade (its size has changed), so we might as well reset the version and remove the version-dependent serialization code. (Note that the affected code here is for SN states in lmdb storage, not for SN-to-SN communication serialization).
2019-06-18 23:57:02 +02:00
return m_service_node_list.is_service_node(pubkey, require_active);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
const std::vector<service_nodes::key_image_blacklist_entry>&
core::get_service_node_blacklisted_key_images() const {
Optimizations and simplifications This commit makes various simplifications and optimizations, mainly in the service node list code. All in all, this shaves about 5% of the CPU time used for re-processing the entire service node list. In particular: - changed m_state_history from a std::vector to a std::set that sorts on height. This is responsible for the bulk of the CPU reduction by significant reducing the amount of work required for checkpoint culling, which has to shuffle a lot of `state_t`s around when removing from the midde of a vector. - the above also allows replacing the binary-search `std::lower_bound` complexity with a much simpler `find()`. - since the history is now always sorted, removed the error messages related to sorting that either can't happen (on store) or don't matter (on load). - Added some converting constructors to simplify code (for example, a `state_t` can now be very usefully constructed from an r-value `state_serialized`). - Many construct + moves (and a couple construct + copy) were replaced with in-place constructions. - removed some unused variables
2019-08-11 05:46:59 +02:00
return m_service_node_list.get_blacklisted_key_images();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
std::vector<service_nodes::service_node_pubkey_info> core::get_service_node_list_state(
const std::vector<crypto::public_key>& service_node_pubkeys) const {
Optimizations and simplifications This commit makes various simplifications and optimizations, mainly in the service node list code. All in all, this shaves about 5% of the CPU time used for re-processing the entire service node list. In particular: - changed m_state_history from a std::vector to a std::set that sorts on height. This is responsible for the bulk of the CPU reduction by significant reducing the amount of work required for checkpoint culling, which has to shuffle a lot of `state_t`s around when removing from the midde of a vector. - the above also allows replacing the binary-search `std::lower_bound` complexity with a much simpler `find()`. - since the history is now always sorted, removed the error messages related to sorting that either can't happen (on store) or don't matter (on load). - Added some converting constructors to simplify code (for example, a `state_t` can now be very usefully constructed from an r-value `state_serialized`). - Many construct + moves (and a couple construct + copy) were replaced with in-place constructions. - removed some unused variables
2019-08-11 05:46:59 +02:00
return m_service_node_list.get_service_node_list_state(service_node_pubkeys);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::add_service_node_vote(
const service_nodes::quorum_vote_t& vote, vote_verification_context& vvc) {
Optimizations and simplifications This commit makes various simplifications and optimizations, mainly in the service node list code. All in all, this shaves about 5% of the CPU time used for re-processing the entire service node list. In particular: - changed m_state_history from a std::vector to a std::set that sorts on height. This is responsible for the bulk of the CPU reduction by significant reducing the amount of work required for checkpoint culling, which has to shuffle a lot of `state_t`s around when removing from the midde of a vector. - the above also allows replacing the binary-search `std::lower_bound` complexity with a much simpler `find()`. - since the history is now always sorted, removed the error messages related to sorting that either can't happen (on store) or don't matter (on load). - Added some converting constructors to simplify code (for example, a `state_t` can now be very usefully constructed from an r-value `state_serialized`). - Many construct + moves (and a couple construct + copy) were replaced with in-place constructions. - removed some unused variables
2019-08-11 05:46:59 +02:00
return m_quorum_cop.handle_vote(vote, vvc);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
uint32_t core::get_blockchain_pruning_seed() const {
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
return get_blockchain_storage().get_blockchain_pruning_seed();
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
bool core::prune_blockchain(uint32_t pruning_seed) {
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-30 00:30:51 +02:00
return get_blockchain_storage().prune_blockchain(pruning_seed);
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
std::time_t core::get_start_time() const {
Add start_time to get_info methods and show uptime
2017-01-09 00:50:29 +01:00
return start_time;
run clang format
2023-04-13 15:50:13 +02:00
}
//-----------------------------------------------------------------------------------------------
void core::graceful_exit() {
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 21:24:43 +02:00
raise(SIGTERM);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
run clang format
2023-04-13 15:50:13 +02:00
} // namespace cryptonote
Reference in a new issue Copy permalink