oxen-core/src/daemon/main.cpp
NoodleDoodleNoodleDoodleNoodleDoodleNoo e5d2680094 ** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)
Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-15 23:20:16 -07:00

286 lines
11 KiB
C++

// Copyright (c) 2014-2015, The Monero Project
//
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without modification, are
// permitted provided that the following conditions are met:
//
// 1. Redistributions of source code must retain the above copyright notice, this list of
// conditions and the following disclaimer.
//
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
// of conditions and the following disclaimer in the documentation and/or other
// materials provided with the distribution.
//
// 3. Neither the name of the copyright holder nor the names of its contributors may be
// used to endorse or promote products derived from this software without specific
// prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//
// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
#include "common/command_line.h"
#include "common/scoped_message_writer.h"
#include "common/util.h"
#include "cryptonote_core/cryptonote_core.h"
#include "cryptonote_core/miner.h"
#include "daemon/command_server.h"
#include "daemon/daemon.h"
#include "daemon/executor.h"
#include "daemonizer/daemonizer.h"
#include "misc_log_ex.h"
#include "p2p/net_node.h"
#include "rpc/core_rpc_server.h"
#include <boost/program_options.hpp>
#include "daemon/command_line_args.h"
#include "blockchain_db/db_types.h"
namespace po = boost::program_options;
namespace bf = boost::filesystem;
int main(int argc, char const * argv[])
{
try {
_note_c("dbg/main", "Begin of main()");
// TODO parse the debug options like set log level right here at start
epee::string_tools::set_module_name_and_folder(argv[0]);
// Build argument description
po::options_description all_options("All");
po::options_description hidden_options("Hidden");
po::options_description visible_options("Options");
po::options_description core_settings("Settings");
po::positional_options_description positional_options;
{
bf::path default_data_dir = daemonizer::get_default_data_dir();
bf::path default_testnet_data_dir = {default_data_dir / "testnet"};
// Misc Options
command_line::add_arg(visible_options, command_line::arg_help);
command_line::add_arg(visible_options, command_line::arg_version);
command_line::add_arg(visible_options, daemon_args::arg_os_version);
command_line::add_arg(visible_options, command_line::arg_data_dir, default_data_dir.string());
command_line::add_arg(visible_options, command_line::arg_testnet_data_dir, default_testnet_data_dir.string());
bf::path default_conf = default_data_dir / std::string(CRYPTONOTE_NAME ".conf");
command_line::add_arg(visible_options, daemon_args::arg_config_file, default_conf.string());
command_line::add_arg(visible_options, command_line::arg_test_drop_download);
command_line::add_arg(visible_options, command_line::arg_test_dbg_lock_sleep);
command_line::add_arg(visible_options, command_line::arg_test_drop_download_height);
// Settings
bf::path default_log = default_data_dir / std::string(CRYPTONOTE_NAME ".log");
command_line::add_arg(core_settings, daemon_args::arg_log_file, default_log.string());
command_line::add_arg(core_settings, daemon_args::arg_log_level);
command_line::add_arg(core_settings, daemon_args::arg_testnet_on);
command_line::add_arg(core_settings, daemon_args::arg_dns_checkpoints);
command_line::add_arg(core_settings, daemon_args::arg_db_type);
command_line::add_arg(core_settings, daemon_args::arg_prep_blocks_threads);
command_line::add_arg(core_settings, daemon_args::arg_fast_block_sync);
command_line::add_arg(core_settings, daemon_args::arg_db_sync_mode);
command_line::add_arg(core_settings, daemon_args::arg_show_time_stats);
command_line::add_arg(core_settings, daemon_args::arg_db_auto_remove_logs);
daemonizer::init_options(hidden_options, visible_options);
daemonize::t_executor::init_options(core_settings);
// Hidden options
command_line::add_arg(hidden_options, daemon_args::arg_command);
visible_options.add(core_settings);
all_options.add(visible_options);
all_options.add(hidden_options);
// Positional
positional_options.add(daemon_args::arg_command.name, -1); // -1 for unlimited arguments
}
// Do command line parsing
po::variables_map vm;
bool ok = command_line::handle_error_helper(visible_options, [&]()
{
boost::program_options::store(
boost::program_options::command_line_parser(argc, argv)
.options(all_options).positional(positional_options).run()
, vm
);
return true;
});
if (!ok) return 1;
if (command_line::get_arg(vm, command_line::arg_help))
{
std::cout << CRYPTONOTE_NAME << " v" << MONERO_VERSION_FULL << ENDL << ENDL;
std::cout << "Usage: " + std::string{argv[0]} + " [options|settings] [daemon_command...]" << std::endl << std::endl;
std::cout << visible_options << std::endl;
return 0;
}
// Monero Version
if (command_line::get_arg(vm, command_line::arg_version))
{
std::cout << CRYPTONOTE_NAME << " v" << MONERO_VERSION_FULL << ENDL;
return 0;
}
// OS
if (command_line::get_arg(vm, daemon_args::arg_os_version))
{
std::cout << "OS: " << tools::get_os_version_string() << ENDL;
return 0;
}
epee::g_test_dbg_lock_sleep = command_line::get_arg(vm, command_line::arg_test_dbg_lock_sleep);
std::string db_type = command_line::get_arg(vm, daemon_args::arg_db_type);
// verify that blockchaindb type is valid
if(cryptonote::blockchain_db_types.count(db_type) == 0)
{
std::cout << "Invalid database type (" << db_type << "), available types are:" << std::endl;
for (const auto& type : cryptonote::blockchain_db_types)
{
std::cout << "\t" << type << std::endl;
}
return 0;
}
bool testnet_mode = command_line::get_arg(vm, daemon_args::arg_testnet_on);
auto data_dir_arg = testnet_mode ? command_line::arg_testnet_data_dir : command_line::arg_data_dir;
// data_dir
// default: e.g. ~/.bitmonero/ or ~/.bitmonero/testnet
// if data-dir argument given:
// absolute path
// relative path: relative to cwd
// Create data dir if it doesn't exist
boost::filesystem::path data_dir = boost::filesystem::absolute(
command_line::get_arg(vm, data_dir_arg));
tools::create_directories_if_necessary(data_dir.string());
// FIXME: not sure on windows implementation default, needs further review
//bf::path relative_path_base = daemonizer::get_relative_path_base(vm);
bf::path relative_path_base = data_dir;
std::string config = command_line::get_arg(vm, daemon_args::arg_config_file);
boost::filesystem::path data_dir_path(data_dir);
boost::filesystem::path config_path(config);
if (!config_path.has_parent_path())
{
config_path = data_dir / config_path;
}
boost::system::error_code ec;
if (bf::exists(config_path, ec))
{
po::store(po::parse_config_file<char>(config_path.string<std::string>().c_str(), core_settings), vm);
}
po::notify(vm);
// If there are positional options, we're running a daemon command
{
auto command = command_line::get_arg(vm, daemon_args::arg_command);
if (command.size())
{
auto rpc_ip_str = command_line::get_arg(vm, cryptonote::core_rpc_server::arg_rpc_bind_ip);
auto rpc_port_str = command_line::get_arg(vm, cryptonote::core_rpc_server::arg_rpc_bind_port);
if (testnet_mode)
{
rpc_port_str = command_line::get_arg(vm, cryptonote::core_rpc_server::arg_testnet_rpc_bind_port);
}
uint32_t rpc_ip;
uint16_t rpc_port;
if (!epee::string_tools::get_ip_int32_from_string(rpc_ip, rpc_ip_str))
{
std::cerr << "Invalid IP: " << rpc_ip_str << std::endl;
return 1;
}
if (!epee::string_tools::get_xtype_from_string(rpc_port, rpc_port_str))
{
std::cerr << "Invalid port: " << rpc_port_str << std::endl;
return 1;
}
daemonize::t_command_server rpc_commands{rpc_ip, rpc_port};
if (rpc_commands.process_command_vec(command))
{
return 0;
}
else
{
std::cerr << "Unknown command" << std::endl;
return 1;
}
}
}
// Start with log level 0
epee::log_space::get_set_log_detalisation_level(true, LOG_LEVEL_0);
// Set log level
{
int new_log_level = command_line::get_arg(vm, daemon_args::arg_log_level);
if(new_log_level < LOG_LEVEL_MIN || new_log_level > LOG_LEVEL_MAX)
{
LOG_PRINT_L0("Wrong log level value: " << new_log_level);
}
else if (epee::log_space::get_set_log_detalisation_level(false) != new_log_level)
{
epee::log_space::get_set_log_detalisation_level(true, new_log_level);
int otshell_utils_log_level = 100 - (new_log_level * 20);
gCurrentLogger.setDebugLevel(otshell_utils_log_level);
LOG_PRINT_L0("LOG_LEVEL set to " << new_log_level);
}
}
// log_file_path
// default: <data_dir>/<CRYPTONOTE_NAME>.log
// if log-file argument given:
// absolute path
// relative path: relative to data_dir
// Set log file
{
bf::path log_file_path {data_dir / std::string(CRYPTONOTE_NAME ".log")};
if (! vm["log-file"].defaulted())
log_file_path = command_line::get_arg(vm, daemon_args::arg_log_file);
log_file_path = bf::absolute(log_file_path, relative_path_base);
epee::log_space::log_singletone::add_logger(
LOGGER_FILE
, log_file_path.filename().string().c_str()
, log_file_path.parent_path().string().c_str()
);
}
_note_c("dbg/main", "Moving from main() into the daemonize now.");
return daemonizer::daemonize(argc, argv, daemonize::t_executor{}, vm);
}
catch (std::exception const & ex)
{
LOG_ERROR("Exception in main! " << ex.what());
}
catch (...)
{
LOG_ERROR("Exception in main!");
}
return 1;
}