2019-04-12 06:36:43 +02:00
// Copyright (c) 2014-2019, The Monero Project
2015-12-14 05:54:39 +01:00
//
2014-07-23 15:03:52 +02:00
// All rights reserved.
2015-12-14 05:54:39 +01:00
//
2014-07-23 15:03:52 +02:00
// Redistribution and use in source and binary forms, with or without modification, are
// permitted provided that the following conditions are met:
2015-12-14 05:54:39 +01:00
//
2014-07-23 15:03:52 +02:00
// 1. Redistributions of source code must retain the above copyright notice, this list of
// conditions and the following disclaimer.
2015-12-14 05:54:39 +01:00
//
2014-07-23 15:03:52 +02:00
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
// of conditions and the following disclaimer in the documentation and/or other
// materials provided with the distribution.
2015-12-14 05:54:39 +01:00
//
2014-07-23 15:03:52 +02:00
// 3. Neither the name of the copyright holder nor the names of its contributors may be
// used to endorse or promote products derived from this software without specific
// prior written permission.
2015-12-14 05:54:39 +01:00
//
2014-07-23 15:03:52 +02:00
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
2015-12-14 05:54:39 +01:00
//
2014-07-23 15:03:52 +02:00
// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
2014-03-03 23:07:58 +01:00
# pragma once
# include <set>
# include <unordered_map>
# include <unordered_set>
2015-04-30 07:02:12 +02:00
# include <queue>
2014-03-03 23:07:58 +01:00
# include <boost/serialization/version.hpp>
RPC overhaul
High-level details:
This redesigns the RPC layer to make it much easier to work with,
decouples it from an embedded HTTP server, and gets the vast majority of
the RPC serialization and dispatch code out of a very commonly included
header.
There is unfortunately rather a lot of interconnected code here that
cannot be easily separated out into separate commits. The full details
of what happens here are as follows:
Major details:
- All of the RPC code is now in a `cryptonote::rpc` namespace; this
renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK
becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME`
becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already
working in the `rpc` namespace).
- `core_rpc_server` is now completely decoupled from providing any
request protocol: it is now *just* the core RPC call handler.
- The HTTP RPC interface now lives in a new rpc/http_server.h; this code
handles listening for HTTP requests and dispatching them to
core_rpc_server, then sending the results back to the caller.
- There is similarly a rpc/lmq_server.h for LMQ RPC code; more details
on this (and other LMQ specifics) below.
- RPC implementing code now returns the response object and throws when
things go wrong which simplifies much of the rpc error handling. They
can throw anything; generic exceptions get logged and a generic
"internal error" message gets returned to the caller, but there is
also an `rpc_error` class to return an error code and message used by
some json-rpc commands.
- RPC implementing functions now overload `core_rpc_server::invoke`
following the pattern:
RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context);
This overloading makes the code vastly simpler: all instantiations are
now done with a small amount of generic instantiation code in a single
.cpp rather than needing to go to hell and back with a nest of epee
macros in a core header.
- each RPC endpoint is now defined by the RPC types themselves,
including its accessible names and permissions, in
core_rpc_server_commands_defs.h:
- every RPC structure now has a static `names()` function that returns
the names by which the end point is accessible. (The first one is
the primary, the others are for deprecated aliases).
- RPC command wrappers define their permissions and type by inheriting
from special tag classes:
- rpc::RPC_COMMAND is a basic, admin-only, JSON command, available
via JSON RPC. *All* JSON commands are now available via JSON RPC,
instead of the previous mix of some being at /foo and others at
/json_rpc. (Ones that were previously at /foo are still there for
backwards compatibility; see `rpc::LEGACY` below).
- rpc::PUBLIC specifies that the command should be available via a
restricted RPC connection.
- rpc::BINARY specifies that the command is not JSON, but rather is
accessible as /name and takes and returns values in the magic epee
binary "portable storage" (lol) data format.
- rpc::LEGACY specifies that the command should be available via the
non-json-rpc interface at `/name` for backwards compatibility (in
addition to the JSON-RPC interface).
- some epee serialization got unwrapped and de-templatized so that it
can be moved into a .cpp file with just declarations in the .h. (This
makes a *huge* difference for core_rpc_server_commands_defs.h and for
every compilation unit that includes it which previously had to
compile all the serialization code and then throw all by one copy away
at link time). This required some new macros so as to not break a ton
of places that will use the old way putting everything in the headers;
The RPC code uses this as does a few other places; there are comments
in contrib/epee/include/serialization/keyvalue_serialization.h as to
how to use it.
- Detemplatized a bunch of epee/storages code. Most of it should have
have been using templates at all (because it can only ever be called
with one type!), and now it isn't. This broke some things that didn't
properly compile because of missing headers or (in one case) a messed
up circular dependency.
- Significantly simplified a bunch of over-templatized serialization
code.
- All RPC serialization definitions is now out of
core_rpc_server_commands_defs.h and into a single .cpp file
(core_rpc_server_commands_defs.cpp).
- core RPC no longer uses the disgusting
BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design
that forced slamming tons of code into a common header that didn't
need to be there.
- epee::struct_init is gone. It was a horrible hack that instiated
multiple templates just so the coder could be so lazy and write
`some_type var;` instead of properly value initializing with
`some_type var{};`.
- Removed a bunch of useless crap from epee. In particular, forcing
extra template instantiations all over the place in order to nest
return objects inside JSON RPC values is no longer needed, as are a
bunch of stuff related to the above de-macroization of the code.
- get_all_service_nodes, get_service_nodes, and get_n_service_nodes are
now combined into a single `get_service_nodes` (with deprecated
aliases for the others), which eliminates a fair amount of
duplication. The biggest obstacle here was getting the requested
fields reference passed through: this is now done by a new ability to
stash a context in the serialization object that can be retrieved by a
sub-serialized type.
LMQ-specifics:
- The LokiMQ instance moves into `cryptonote::core` rather than being
inside cryptonote_protocol. Currently the instance is used both for
qnet and rpc calls (and so needs to be in a common place), but I also
intend future PRs to use the batching code for job processing
(replacing the current threaded job queue).
- rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue.
Unlike http_server it isn't technically running the whole LMQ stack
from here, but the parallel name with http_server seemed appropriate.
- All RPC endpoints are supported by LMQ under the same names as defined
generically, but prefixed with `rpc.` for public commands and `admin.`
for restricted ones.
- service node keys are now always available, even when not running in
`--service-node` mode: this is because we want the x25519 key for
being able to offer CURVE encryption for lmq RPC end-points, and
because it doesn't hurt to have them available all the time. In the
RPC layer this is now called "get_service_keys" (with
"get_service_node_key" as an alias) since they aren't strictly only
for service nodes. This also means code needs to check
m_service_node, and not m_service_node_keys, to tell if it is running
as a service node. (This is also easier to notice because
m_service_node_keys got renamed to `m_service_keys`).
- Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and
`sub.mempool` subscribes the connection for new block and new mempool
TX notifications. The latter can notify on just blink txes, or all
new mempool txes (but only new ones -- txes dumped from a block don't
trigger it). The client gets pushed a [`notify.block`, `height`,
`hash`] or [`notify.tx`, `txhash`, `blob`] message when something
arrives.
Minor details:
- rpc::version_t is now a {major,minor} pair. Forcing everyone to pack
and unpack a uint32_t was gross.
- Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...).
(This immediately revealed a couple of bugs in the RPC code that was
assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because
the macro allows implicit conversion to a char).
- De-templatizing useless templates in epee (i.e. a bunch of templated
types that were never invoked with different types) revealed a painful
circular dependency between epee and non-epee code for tor_address and
i2p_address. This crap is now handled in a suitably named
`net/epee_network_address_hack.cpp` hack because it really isn't
trivial to extricate this mess.
- Removed `epee/include/serialization/serialize_base.h`. Amazingly the
code somehow still all works perfectly with this previously vital
header removed.
- Removed bitrotted, unused epee "crypted_storage" and
"gzipped_inmemstorage" code.
- Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with
LOKI_DEFERs. The epee version involves quite a bit more instantiation
and is ugly as sin. Also made the `loki::defer` class invokable for
some edge cases that need calling before destruction in particular
conditions.
- Moved the systemd code around; it makes much more sense to do the
systemd started notification as in daemon.cpp as late as possible
rather than in core (when we can still have startup failures, e.g. if
the RPC layer can't start).
- Made the systemd short status string available in the get_info RPC
(and no longer require building with systemd).
- during startup, print (only) the x25519 when not in SN mode, and
continue to print all three when in SN mode.
- DRYed out some RPC implementation code (such as set_limit)
- Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
# include <functional>
2014-03-03 23:07:58 +01:00
2020-10-24 00:49:42 +02:00
# include "epee/string_tools.h"
2020-05-26 07:15:25 +02:00
# include "common/periodic_task.h"
2017-01-26 16:07:23 +01:00
# include "cryptonote_basic/cryptonote_basic_impl.h"
# include "cryptonote_basic/verification_context.h"
2017-05-14 15:06:55 +02:00
# include "blockchain_db/blockchain_db.h"
2014-03-03 23:07:58 +01:00
# include "crypto/hash.h"
2015-04-23 14:13:07 +02:00
# include "rpc/core_rpc_server_commands_defs.h"
2019-10-27 23:47:19 +01:00
# include "tx_blink.h"
2021-01-04 01:09:45 +01:00
# include "oxen_economy.h"
2019-07-04 09:25:02 +02:00
2014-03-03 23:07:58 +01:00
namespace cryptonote
{
2014-10-28 04:44:45 +01:00
class Blockchain ;
2014-03-03 23:07:58 +01:00
/************************************************************************/
/* */
/************************************************************************/
2018-08-21 09:01:19 +02:00
//! tuple of <deregister, transaction fee, receive time> for organization
typedef std : : pair < std : : tuple < bool , double , std : : time_t > , crypto : : hash > tx_by_fee_and_receive_time_entry ;
2016-03-25 01:03:02 +01:00
2015-05-14 02:27:06 +02:00
class txCompare
{
public :
2019-02-01 15:02:39 +01:00
bool operator ( ) ( const tx_by_fee_and_receive_time_entry & a , const tx_by_fee_and_receive_time_entry & b ) const
2015-05-14 02:27:06 +02:00
{
2019-10-30 22:23:47 +01:00
// Sort order: non-standard txes, fee (descending), arrival time, hash
2019-11-24 20:14:29 +01:00
return std : : make_tuple ( ! std : : get < 0 > ( a . first ) , - std : : get < 1 > ( a . first ) , std : : get < 2 > ( a . first ) , a . second )
< std : : make_tuple ( ! std : : get < 0 > ( b . first ) , - std : : get < 1 > ( b . first ) , std : : get < 2 > ( b . first ) , b . second ) ;
2015-05-14 02:27:06 +02:00
}
} ;
2016-03-25 01:03:02 +01:00
//! container for sorting transactions by fee per unit size
2017-01-13 16:08:37 +01:00
typedef std : : set < tx_by_fee_and_receive_time_entry , txCompare > sorted_tx_container ;
2015-05-14 02:27:06 +02:00
Generic burn fee checking + blink burn fee checking
This adds the ability for check_fee() to also check the burn amount.
This requires passing extra info through `add_tx()` (and the various
things that call it), so I took the:
bool keeped_by_block, bool relayed, bool do_not_relay
argument triplet, moved it into a struct in tx_pool.h, then added the other fee
options there (along with some static factory functions for generating the
typical sets of option).
The majority of this commit is chasing that change through the codebase and
test suite.
This is used by blink but should also help LNS and other future burn
transactions to verify a burn amount simply when adding the transation to the
mempool. It supports a fixed burn amount, a burn amount as a multiple of the
minimum tx fee, and also allows you to increase the minimum tx fee (so that,
for example, we could require blink txes to pay miners 250% of the usual
minimum (unimportant) priority tx fee.
- Removed a useless core::add_new_tx() overload that wasn't used anywhere.
Blink-specific changes:
(I'd normally separate these into a separate commit, but they got interwoven
fairly heavily with the above change).
- changed the way blink burning is specified so that we have three knobs for
fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee).
The fixed amount is currently 0, base fee is 400%, and require miner tx fee is
simply 100% (i.e. no different than a normal transaction). This is the same as
before this commit, but is changing how they are being specified in
cryptonote_config.h.
- blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked
before signing a blink tx. (These fee checks don't apply to anyone else --
when propagating over the network only the miner tx fee is checked).
- Added a couple of checks for blink quorums: 1) make sure they have reached
the blink hf; 2) make sure the submitted tx version conforms to the current hf
min/max tx version.
- print blink fee information in simplewallet's `fee` output
- add "typical" fee calculations in the `fee` output:
[wallet T6SCwL (has locked stakes)]: fee
Current fee is 0.000000850 loki per byte + 0.020000000 loki per output
No backlog at priority 1
No backlog at priority 2
No backlog at priority 3
No backlog at priority 4
Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output
Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink)
where "small" here is the same tx size (2500 bytes + 2 outputs) used to
estimate backlogs.
2019-11-09 04:14:15 +01:00
/// Argument passed into add_tx specifying different requires on the transaction
struct tx_pool_options {
bool kept_by_block = false ; ///< has this transaction been in a block?
bool relayed = false ; ///< was this transaction from the network or a local client?
bool do_not_relay = false ; ///< to avoid relaying the transaction to the network
2019-11-25 05:57:15 +01:00
bool approved_blink = false ; ///< signals that this is a blink tx and so should be accepted even if it conflicts with mempool or recent txes in non-immutable block; typically specified indirectly (via core.handle_incoming_txs())
Generic burn fee checking + blink burn fee checking
This adds the ability for check_fee() to also check the burn amount.
This requires passing extra info through `add_tx()` (and the various
things that call it), so I took the:
bool keeped_by_block, bool relayed, bool do_not_relay
argument triplet, moved it into a struct in tx_pool.h, then added the other fee
options there (along with some static factory functions for generating the
typical sets of option).
The majority of this commit is chasing that change through the codebase and
test suite.
This is used by blink but should also help LNS and other future burn
transactions to verify a burn amount simply when adding the transation to the
mempool. It supports a fixed burn amount, a burn amount as a multiple of the
minimum tx fee, and also allows you to increase the minimum tx fee (so that,
for example, we could require blink txes to pay miners 250% of the usual
minimum (unimportant) priority tx fee.
- Removed a useless core::add_new_tx() overload that wasn't used anywhere.
Blink-specific changes:
(I'd normally separate these into a separate commit, but they got interwoven
fairly heavily with the above change).
- changed the way blink burning is specified so that we have three knobs for
fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee).
The fixed amount is currently 0, base fee is 400%, and require miner tx fee is
simply 100% (i.e. no different than a normal transaction). This is the same as
before this commit, but is changing how they are being specified in
cryptonote_config.h.
- blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked
before signing a blink tx. (These fee checks don't apply to anyone else --
when propagating over the network only the miner tx fee is checked).
- Added a couple of checks for blink quorums: 1) make sure they have reached
the blink hf; 2) make sure the submitted tx version conforms to the current hf
min/max tx version.
- print blink fee information in simplewallet's `fee` output
- add "typical" fee calculations in the `fee` output:
[wallet T6SCwL (has locked stakes)]: fee
Current fee is 0.000000850 loki per byte + 0.020000000 loki per output
No backlog at priority 1
No backlog at priority 2
No backlog at priority 3
No backlog at priority 4
Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output
Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink)
where "small" here is the same tx size (2500 bytes + 2 outputs) used to
estimate backlogs.
2019-11-09 04:14:15 +01:00
uint64_t fee_percent = 100 ; ///< the required miner tx fee in percent relative to the base required miner tx fee; must be >= 100.
uint64_t burn_fixed = 0 ; ///< a required minimum amount that must be burned (in atomic currency)
uint64_t burn_percent = 0 ; ///< a required amount as a percentage of the base required miner tx fee that must be burned (additive with burn_fixed, if both > 0)
static tx_pool_options from_block ( ) { tx_pool_options o ; o . kept_by_block = true ; o . relayed = true ; return o ; }
static tx_pool_options from_peer ( ) { tx_pool_options o ; o . relayed = true ; return o ; }
static tx_pool_options new_tx ( bool do_not_relay = false ) { tx_pool_options o ; o . do_not_relay = do_not_relay ; return o ; }
2020-02-18 22:47:37 +01:00
static tx_pool_options new_blink ( bool approved , uint8_t hf_version ) {
Generic burn fee checking + blink burn fee checking
This adds the ability for check_fee() to also check the burn amount.
This requires passing extra info through `add_tx()` (and the various
things that call it), so I took the:
bool keeped_by_block, bool relayed, bool do_not_relay
argument triplet, moved it into a struct in tx_pool.h, then added the other fee
options there (along with some static factory functions for generating the
typical sets of option).
The majority of this commit is chasing that change through the codebase and
test suite.
This is used by blink but should also help LNS and other future burn
transactions to verify a burn amount simply when adding the transation to the
mempool. It supports a fixed burn amount, a burn amount as a multiple of the
minimum tx fee, and also allows you to increase the minimum tx fee (so that,
for example, we could require blink txes to pay miners 250% of the usual
minimum (unimportant) priority tx fee.
- Removed a useless core::add_new_tx() overload that wasn't used anywhere.
Blink-specific changes:
(I'd normally separate these into a separate commit, but they got interwoven
fairly heavily with the above change).
- changed the way blink burning is specified so that we have three knobs for
fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee).
The fixed amount is currently 0, base fee is 400%, and require miner tx fee is
simply 100% (i.e. no different than a normal transaction). This is the same as
before this commit, but is changing how they are being specified in
cryptonote_config.h.
- blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked
before signing a blink tx. (These fee checks don't apply to anyone else --
when propagating over the network only the miner tx fee is checked).
- Added a couple of checks for blink quorums: 1) make sure they have reached
the blink hf; 2) make sure the submitted tx version conforms to the current hf
min/max tx version.
- print blink fee information in simplewallet's `fee` output
- add "typical" fee calculations in the `fee` output:
[wallet T6SCwL (has locked stakes)]: fee
Current fee is 0.000000850 loki per byte + 0.020000000 loki per output
No backlog at priority 1
No backlog at priority 2
No backlog at priority 3
No backlog at priority 4
Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output
Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink)
where "small" here is the same tx size (2500 bytes + 2 outputs) used to
estimate backlogs.
2019-11-09 04:14:15 +01:00
tx_pool_options o ;
o . do_not_relay = ! approved ;
2019-12-26 07:38:00 +01:00
o . approved_blink = approved ;
Generic burn fee checking + blink burn fee checking
This adds the ability for check_fee() to also check the burn amount.
This requires passing extra info through `add_tx()` (and the various
things that call it), so I took the:
bool keeped_by_block, bool relayed, bool do_not_relay
argument triplet, moved it into a struct in tx_pool.h, then added the other fee
options there (along with some static factory functions for generating the
typical sets of option).
The majority of this commit is chasing that change through the codebase and
test suite.
This is used by blink but should also help LNS and other future burn
transactions to verify a burn amount simply when adding the transation to the
mempool. It supports a fixed burn amount, a burn amount as a multiple of the
minimum tx fee, and also allows you to increase the minimum tx fee (so that,
for example, we could require blink txes to pay miners 250% of the usual
minimum (unimportant) priority tx fee.
- Removed a useless core::add_new_tx() overload that wasn't used anywhere.
Blink-specific changes:
(I'd normally separate these into a separate commit, but they got interwoven
fairly heavily with the above change).
- changed the way blink burning is specified so that we have three knobs for
fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee).
The fixed amount is currently 0, base fee is 400%, and require miner tx fee is
simply 100% (i.e. no different than a normal transaction). This is the same as
before this commit, but is changing how they are being specified in
cryptonote_config.h.
- blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked
before signing a blink tx. (These fee checks don't apply to anyone else --
when propagating over the network only the miner tx fee is checked).
- Added a couple of checks for blink quorums: 1) make sure they have reached
the blink hf; 2) make sure the submitted tx version conforms to the current hf
min/max tx version.
- print blink fee information in simplewallet's `fee` output
- add "typical" fee calculations in the `fee` output:
[wallet T6SCwL (has locked stakes)]: fee
Current fee is 0.000000850 loki per byte + 0.020000000 loki per output
No backlog at priority 1
No backlog at priority 2
No backlog at priority 3
No backlog at priority 4
Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output
Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink)
where "small" here is the same tx size (2500 bytes + 2 outputs) used to
estimate backlogs.
2019-11-09 04:14:15 +01:00
o . fee_percent = BLINK_MINER_TX_FEE_PERCENT ;
2021-04-13 06:50:44 +02:00
2021-06-04 21:49:35 +02:00
o . burn_percent = BLINK_BURN_TX_FEE_PERCENT_V18 ;
Generic burn fee checking + blink burn fee checking
This adds the ability for check_fee() to also check the burn amount.
This requires passing extra info through `add_tx()` (and the various
things that call it), so I took the:
bool keeped_by_block, bool relayed, bool do_not_relay
argument triplet, moved it into a struct in tx_pool.h, then added the other fee
options there (along with some static factory functions for generating the
typical sets of option).
The majority of this commit is chasing that change through the codebase and
test suite.
This is used by blink but should also help LNS and other future burn
transactions to verify a burn amount simply when adding the transation to the
mempool. It supports a fixed burn amount, a burn amount as a multiple of the
minimum tx fee, and also allows you to increase the minimum tx fee (so that,
for example, we could require blink txes to pay miners 250% of the usual
minimum (unimportant) priority tx fee.
- Removed a useless core::add_new_tx() overload that wasn't used anywhere.
Blink-specific changes:
(I'd normally separate these into a separate commit, but they got interwoven
fairly heavily with the above change).
- changed the way blink burning is specified so that we have three knobs for
fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee).
The fixed amount is currently 0, base fee is 400%, and require miner tx fee is
simply 100% (i.e. no different than a normal transaction). This is the same as
before this commit, but is changing how they are being specified in
cryptonote_config.h.
- blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked
before signing a blink tx. (These fee checks don't apply to anyone else --
when propagating over the network only the miner tx fee is checked).
- Added a couple of checks for blink quorums: 1) make sure they have reached
the blink hf; 2) make sure the submitted tx version conforms to the current hf
min/max tx version.
- print blink fee information in simplewallet's `fee` output
- add "typical" fee calculations in the `fee` output:
[wallet T6SCwL (has locked stakes)]: fee
Current fee is 0.000000850 loki per byte + 0.020000000 loki per output
No backlog at priority 1
No backlog at priority 2
No backlog at priority 3
No backlog at priority 4
Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output
Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink)
where "small" here is the same tx size (2500 bytes + 2 outputs) used to
estimate backlogs.
2019-11-09 04:14:15 +01:00
o . burn_fixed = BLINK_BURN_FIXED ;
return o ;
}
} ;
2016-03-25 01:03:02 +01:00
/**
* @ brief Transaction pool , handles transactions which are not part of a block
*
* This class handles all transactions which have been received , but not as
* part of a block .
*
* This handling includes :
* storing the transactions
2018-07-18 23:24:53 +02:00
* organizing the transactions by fee per weight unit
2016-03-25 01:03:02 +01:00
* taking / giving transactions to and from various other components
* saving the transactions to disk on shutdown
* helping create a new block template by choosing transactions for it
*
*/
2019-10-27 23:47:19 +01:00
class tx_memory_pool
2014-03-03 23:07:58 +01:00
{
public :
2016-03-25 01:03:02 +01:00
/**
* @ brief Constructor
*
* @ param bchs a Blockchain class instance , for getting chain info
*/
2014-10-28 04:44:45 +01:00
tx_memory_pool ( Blockchain & bchs ) ;
2016-03-25 01:03:02 +01:00
2019-10-27 23:47:19 +01:00
// Non-copyable
tx_memory_pool ( const tx_memory_pool & ) = delete ;
tx_memory_pool & operator = ( const tx_memory_pool & ) = delete ;
2016-03-25 01:03:02 +01:00
/**
Generic burn fee checking + blink burn fee checking
This adds the ability for check_fee() to also check the burn amount.
This requires passing extra info through `add_tx()` (and the various
things that call it), so I took the:
bool keeped_by_block, bool relayed, bool do_not_relay
argument triplet, moved it into a struct in tx_pool.h, then added the other fee
options there (along with some static factory functions for generating the
typical sets of option).
The majority of this commit is chasing that change through the codebase and
test suite.
This is used by blink but should also help LNS and other future burn
transactions to verify a burn amount simply when adding the transation to the
mempool. It supports a fixed burn amount, a burn amount as a multiple of the
minimum tx fee, and also allows you to increase the minimum tx fee (so that,
for example, we could require blink txes to pay miners 250% of the usual
minimum (unimportant) priority tx fee.
- Removed a useless core::add_new_tx() overload that wasn't used anywhere.
Blink-specific changes:
(I'd normally separate these into a separate commit, but they got interwoven
fairly heavily with the above change).
- changed the way blink burning is specified so that we have three knobs for
fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee).
The fixed amount is currently 0, base fee is 400%, and require miner tx fee is
simply 100% (i.e. no different than a normal transaction). This is the same as
before this commit, but is changing how they are being specified in
cryptonote_config.h.
- blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked
before signing a blink tx. (These fee checks don't apply to anyone else --
when propagating over the network only the miner tx fee is checked).
- Added a couple of checks for blink quorums: 1) make sure they have reached
the blink hf; 2) make sure the submitted tx version conforms to the current hf
min/max tx version.
- print blink fee information in simplewallet's `fee` output
- add "typical" fee calculations in the `fee` output:
[wallet T6SCwL (has locked stakes)]: fee
Current fee is 0.000000850 loki per byte + 0.020000000 loki per output
No backlog at priority 1
No backlog at priority 2
No backlog at priority 3
No backlog at priority 4
Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output
Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink)
where "small" here is the same tx size (2500 bytes + 2 outputs) used to
estimate backlogs.
2019-11-09 04:14:15 +01:00
* @ copydoc add_tx ( transaction & , tx_verification_context & , const tx_pool_options & , uint8_t )
2016-03-25 01:03:02 +01:00
*
* @ param id the transaction ' s hash
2018-07-18 23:24:53 +02:00
* @ param tx_weight the transaction ' s weight
2019-11-25 05:57:15 +01:00
* @ param blink_rollback_height if tx is a blink that conflicts with a recent ( non - immutable )
2019-12-26 07:38:00 +01:00
* block tx then set this pointer to the required new height : that is , all blocks with height
* ` block_rollback_height ` and above must be removed .
2016-03-25 01:03:02 +01:00
*/
2019-11-25 05:57:15 +01:00
bool add_tx ( transaction & tx , const crypto : : hash & id , const cryptonote : : blobdata & blob , size_t tx_weight , tx_verification_context & tvc , const tx_pool_options & opts , uint8_t hf_version , uint64_t * blink_rollback_height = nullptr ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief add a transaction to the transaction pool
*
* Most likely the transaction will come from the network , but it is
* also possible for transactions to come from popped blocks during
* a reorg , or from local clients creating a transaction and
* submitting it to the network
*
* @ param tx the transaction to be added
* @ param tvc return - by - reference status about the transaction verification
Generic burn fee checking + blink burn fee checking
This adds the ability for check_fee() to also check the burn amount.
This requires passing extra info through `add_tx()` (and the various
things that call it), so I took the:
bool keeped_by_block, bool relayed, bool do_not_relay
argument triplet, moved it into a struct in tx_pool.h, then added the other fee
options there (along with some static factory functions for generating the
typical sets of option).
The majority of this commit is chasing that change through the codebase and
test suite.
This is used by blink but should also help LNS and other future burn
transactions to verify a burn amount simply when adding the transation to the
mempool. It supports a fixed burn amount, a burn amount as a multiple of the
minimum tx fee, and also allows you to increase the minimum tx fee (so that,
for example, we could require blink txes to pay miners 250% of the usual
minimum (unimportant) priority tx fee.
- Removed a useless core::add_new_tx() overload that wasn't used anywhere.
Blink-specific changes:
(I'd normally separate these into a separate commit, but they got interwoven
fairly heavily with the above change).
- changed the way blink burning is specified so that we have three knobs for
fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee).
The fixed amount is currently 0, base fee is 400%, and require miner tx fee is
simply 100% (i.e. no different than a normal transaction). This is the same as
before this commit, but is changing how they are being specified in
cryptonote_config.h.
- blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked
before signing a blink tx. (These fee checks don't apply to anyone else --
when propagating over the network only the miner tx fee is checked).
- Added a couple of checks for blink quorums: 1) make sure they have reached
the blink hf; 2) make sure the submitted tx version conforms to the current hf
min/max tx version.
- print blink fee information in simplewallet's `fee` output
- add "typical" fee calculations in the `fee` output:
[wallet T6SCwL (has locked stakes)]: fee
Current fee is 0.000000850 loki per byte + 0.020000000 loki per output
No backlog at priority 1
No backlog at priority 2
No backlog at priority 3
No backlog at priority 4
Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output
Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink)
where "small" here is the same tx size (2500 bytes + 2 outputs) used to
estimate backlogs.
2019-11-09 04:14:15 +01:00
* @ param opts the options controlling how this tx will be accepted / added
* @ param hf_version the hard fork version used to create the transaction
2016-03-25 01:03:02 +01:00
*
* @ return true if the transaction passes validations , otherwise false
*/
Generic burn fee checking + blink burn fee checking
This adds the ability for check_fee() to also check the burn amount.
This requires passing extra info through `add_tx()` (and the various
things that call it), so I took the:
bool keeped_by_block, bool relayed, bool do_not_relay
argument triplet, moved it into a struct in tx_pool.h, then added the other fee
options there (along with some static factory functions for generating the
typical sets of option).
The majority of this commit is chasing that change through the codebase and
test suite.
This is used by blink but should also help LNS and other future burn
transactions to verify a burn amount simply when adding the transation to the
mempool. It supports a fixed burn amount, a burn amount as a multiple of the
minimum tx fee, and also allows you to increase the minimum tx fee (so that,
for example, we could require blink txes to pay miners 250% of the usual
minimum (unimportant) priority tx fee.
- Removed a useless core::add_new_tx() overload that wasn't used anywhere.
Blink-specific changes:
(I'd normally separate these into a separate commit, but they got interwoven
fairly heavily with the above change).
- changed the way blink burning is specified so that we have three knobs for
fee adjustment (fixed burn fee; base fee multiple; and required miner tx fee).
The fixed amount is currently 0, base fee is 400%, and require miner tx fee is
simply 100% (i.e. no different than a normal transaction). This is the same as
before this commit, but is changing how they are being specified in
cryptonote_config.h.
- blink tx fee, burn amount, and miner tx fee (if > 100%) now get checked
before signing a blink tx. (These fee checks don't apply to anyone else --
when propagating over the network only the miner tx fee is checked).
- Added a couple of checks for blink quorums: 1) make sure they have reached
the blink hf; 2) make sure the submitted tx version conforms to the current hf
min/max tx version.
- print blink fee information in simplewallet's `fee` output
- add "typical" fee calculations in the `fee` output:
[wallet T6SCwL (has locked stakes)]: fee
Current fee is 0.000000850 loki per byte + 0.020000000 loki per output
No backlog at priority 1
No backlog at priority 2
No backlog at priority 3
No backlog at priority 4
Current blink fee is 0.000004250 loki per byte + 0.100000000 loki per output
Estimated typical small transaction fees: 0.042125000 (unimportant), 0.210625000 (normal), 1.053125000 (elevated), 5.265625000 (priority), 0.210625000 (blink)
where "small" here is the same tx size (2500 bytes + 2 outputs) used to
estimate backlogs.
2019-11-09 04:14:15 +01:00
bool add_tx ( transaction & tx , tx_verification_context & tvc , const tx_pool_options & opts , uint8_t hf_version ) ;
2016-03-25 01:03:02 +01:00
2019-10-27 23:47:19 +01:00
/**
2019-11-06 07:28:33 +01:00
* @ brief attempts to add a blink transaction to the transaction pool .
*
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
* This method must be called without a held blink lock .
*
2019-11-06 07:28:33 +01:00
* This is only for use for new transactions that should not exist yet on the chain or mempool
* ( and will fail if already does ) . See ` add_existing_blink ` instead to add blink data about a
* transaction that already exists . This is only meant to be called during the SN blink signing
* phase ( and requires that the ` tx ` transaction be properly set to a full transaction ) ;
* ordinary nodes receiving a blink tx from the network should be going through
* core . handle_incoming_blinks instead .
*
* Whether or not the transaction is added to the known blinks or marked for relaying depends on
* whether the passed - in transaction has an ` . approved ( ) ` status : if it does , the transaction is
* set for relaying and added to the active blinks immediately ; otherwise it is not added to the
* known blinks and will not be relayed .
*
* The transaction is * not * added to the known blinks or marked for relaying unless it is passed
* in with an ` . approved ( ) ` status .
2019-10-27 23:47:19 +01:00
*
* @ param blink - a shared_ptr to the blink details
* @ param tvc - the verification results
* @ param blink_exists - will be set to true if the addition fails because the blink tx already
* exists
*
2019-11-06 07:28:33 +01:00
* @ return true if the tx passes validations and has been added to the tx pool .
*/
bool add_new_blink ( const std : : shared_ptr < blink_tx > & blink , tx_verification_context & tvc , bool & blink_exists ) ;
/**
* @ brief attempts to add blink transaction information about an existing blink transaction
*
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
* You * must * already hold a blink_unique_lock ( ) .
*
2019-11-06 07:28:33 +01:00
* This method takes an approved blink_tx and records it in the known blinks data . No check is
* done that the transaction actually exists on the blockchain or mempool . It is assumed that
* the given shared_ptr is a new blink that is not yet shared between threads ( and thus doesn ' t
* need locking ) : sharing is expected only after it is added to the blinks via this method .
*
2019-12-25 05:13:02 +01:00
* NB : this function assumes that the given blink tx is valid and approved ( signed ) but does
* * not * check it ( except as an assert when compiling in debug mode ) .
2019-11-06 07:28:33 +01:00
*
* @ param blink the blink_tx shared_ptr
*
2019-11-25 05:57:15 +01:00
* @ return true if the blink data was recorded , false if the given blink was already known .
2019-11-06 07:28:33 +01:00
*/
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
bool add_existing_blink ( std : : shared_ptr < blink_tx > blink ) ;
2019-11-06 07:28:33 +01:00
/**
* @ brief accesses blink tx details if the given tx hash is a known , approved blink tx , nullptr
* otherwise .
*
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
* You * must * already hold a blink_shared_lock ( ) or blink_unique_lock ( ) .
*
2019-11-06 07:28:33 +01:00
* @ param tx_hash the hash of the tx to access
*/
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
std : : shared_ptr < blink_tx > get_blink ( const crypto : : hash & tx_hash ) const ;
2019-11-06 07:28:33 +01:00
/**
* Equivalent to ` ( bool ) get_blink ( . . . ) ` , but slightly more efficient when the blink information
* isn ' t actually needed beyond an existance test ( as it avoids copying the shared_ptr ) .
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
*
* You * must * already hold a blink_shared_lock ( ) or blink_unique_lock ( ) .
2019-10-27 23:47:19 +01:00
*/
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
bool has_blink ( const crypto : : hash & tx_hash ) const ;
2019-10-27 23:47:19 +01:00
/**
2019-11-25 05:57:15 +01:00
* @ brief modifies a vector of tx hashes to remove any that have known valid blink signatures
2019-11-18 23:18:41 +01:00
*
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
* Must not currently hold a blink lock .
*
2019-11-18 23:18:41 +01:00
* @ param txs the tx hashes to check
*/
2019-11-24 19:57:44 +01:00
void keep_missing_blinks ( std : : vector < crypto : : hash > & tx_hashes ) const ;
2019-11-18 23:18:41 +01:00
/**
* @ brief returns checksums of blink txes included in recently mined blocks and in the mempool
*
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
* Must not currently hold a blink lock .
*
2019-11-18 23:18:41 +01:00
* The returned map consists of height = > hashsum pairs where the height is the height in which
* the blink transactions were mined and the hashsum is a checksum of all the blink txes mined
* at that height . Unmined mempool blink txes are included at a height of 0. Only heights
* since the immutable checkpoint block are included . Any block height ( including the special
* " 0 " height ) that has no blink tx in it is not included .
*/
std : : map < uint64_t , crypto : : hash > get_blink_checksums ( ) const ;
/**
* @ brief returns the hashes of any non - immutable blink transactions mined in the given heights .
* A height of 0 is allowed : it indicates blinks in the mempool .
*
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
* Must not currently hold a blink lock .
*
2019-11-18 23:18:41 +01:00
* Note that this returned hashes by MINED HEIGHTS , not BLINK HEIGHTS where are a different
* concept .
*
* @ param set of heights
*
* @ return vector of hashes
2019-10-27 23:47:19 +01:00
*/
2019-11-18 23:18:41 +01:00
std : : vector < crypto : : hash > get_mined_blinks ( const std : : set < uint64_t > & heights ) const ;
2019-10-27 23:47:19 +01:00
2016-03-25 01:03:02 +01:00
/**
* @ brief takes a transaction with the given hash from the pool
*
* @ param id the hash of the transaction
* @ param tx return - by - reference the transaction taken
2018-11-11 15:51:03 +01:00
* @ param txblob return - by - reference the transaction as a blob
2018-07-18 23:24:53 +02:00
* @ param tx_weight return - by - reference the transaction ' s weight
2016-03-25 01:03:02 +01:00
* @ param fee the transaction fee
* @ param relayed return - by - reference was transaction relayed to us by the network ?
2017-01-14 14:01:21 +01:00
* @ param do_not_relay return - by - reference is transaction not to be relayed to the network ?
2017-09-22 14:57:20 +02:00
* @ param double_spend_seen return - by - reference was a double spend seen for that transaction ?
2016-03-25 01:03:02 +01:00
*
* @ return true unless the transaction cannot be found in the pool
*/
2018-11-11 15:51:03 +01:00
bool take_tx ( const crypto : : hash & id , transaction & tx , cryptonote : : blobdata & txblob , size_t & tx_weight , uint64_t & fee , bool & relayed , bool & do_not_relay , bool & double_spend_seen ) ;
2014-03-03 23:07:58 +01:00
2016-03-25 01:03:02 +01:00
/**
* @ brief checks if the pool has a transaction with the given hash
*
* @ param id the hash to look for
*
* @ return true if the transaction is in the pool , otherwise false
*/
2014-07-17 16:31:44 +02:00
bool have_tx ( const crypto : : hash & id ) const ;
2016-03-25 01:03:02 +01:00
2019-11-06 07:28:33 +01:00
/**
* @ brief determines whether the given tx hashes are in the mempool
*
* @ param hashes vector of tx hashes
*
* @ return vector of the same size as ` hashes ` of true ( 1 ) or false ( 0 ) values . ( Not using
* std : : vector < bool > because it is broken by design ) .
*/
std : : vector < uint8_t > have_txs ( const std : : vector < crypto : : hash > & hashes ) const ;
2016-03-25 01:03:02 +01:00
/**
* @ brief action to take when notified of a block added to the blockchain
*
* @ param new_block_height the height of the blockchain after the change
* @ param top_block_id the hash of the new top block
*
* @ return true
*/
2019-10-27 23:26:38 +01:00
bool on_blockchain_inc ( block const & blk ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief action to take when notified of a block removed from the blockchain
*
* @ param new_block_height the height of the blockchain after the change
* @ param top_block_id the hash of the new top block
*
* @ return true
*/
2019-08-12 01:48:47 +02:00
bool on_blockchain_dec ( ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief action to take periodically
*
* Currently checks transaction pool for stale ( " stuck " ) transactions
*/
2014-06-15 09:48:13 +02:00
void on_idle ( ) ;
2014-03-03 23:07:58 +01:00
RPC overhaul
High-level details:
This redesigns the RPC layer to make it much easier to work with,
decouples it from an embedded HTTP server, and gets the vast majority of
the RPC serialization and dispatch code out of a very commonly included
header.
There is unfortunately rather a lot of interconnected code here that
cannot be easily separated out into separate commits. The full details
of what happens here are as follows:
Major details:
- All of the RPC code is now in a `cryptonote::rpc` namespace; this
renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK
becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME`
becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already
working in the `rpc` namespace).
- `core_rpc_server` is now completely decoupled from providing any
request protocol: it is now *just* the core RPC call handler.
- The HTTP RPC interface now lives in a new rpc/http_server.h; this code
handles listening for HTTP requests and dispatching them to
core_rpc_server, then sending the results back to the caller.
- There is similarly a rpc/lmq_server.h for LMQ RPC code; more details
on this (and other LMQ specifics) below.
- RPC implementing code now returns the response object and throws when
things go wrong which simplifies much of the rpc error handling. They
can throw anything; generic exceptions get logged and a generic
"internal error" message gets returned to the caller, but there is
also an `rpc_error` class to return an error code and message used by
some json-rpc commands.
- RPC implementing functions now overload `core_rpc_server::invoke`
following the pattern:
RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context);
This overloading makes the code vastly simpler: all instantiations are
now done with a small amount of generic instantiation code in a single
.cpp rather than needing to go to hell and back with a nest of epee
macros in a core header.
- each RPC endpoint is now defined by the RPC types themselves,
including its accessible names and permissions, in
core_rpc_server_commands_defs.h:
- every RPC structure now has a static `names()` function that returns
the names by which the end point is accessible. (The first one is
the primary, the others are for deprecated aliases).
- RPC command wrappers define their permissions and type by inheriting
from special tag classes:
- rpc::RPC_COMMAND is a basic, admin-only, JSON command, available
via JSON RPC. *All* JSON commands are now available via JSON RPC,
instead of the previous mix of some being at /foo and others at
/json_rpc. (Ones that were previously at /foo are still there for
backwards compatibility; see `rpc::LEGACY` below).
- rpc::PUBLIC specifies that the command should be available via a
restricted RPC connection.
- rpc::BINARY specifies that the command is not JSON, but rather is
accessible as /name and takes and returns values in the magic epee
binary "portable storage" (lol) data format.
- rpc::LEGACY specifies that the command should be available via the
non-json-rpc interface at `/name` for backwards compatibility (in
addition to the JSON-RPC interface).
- some epee serialization got unwrapped and de-templatized so that it
can be moved into a .cpp file with just declarations in the .h. (This
makes a *huge* difference for core_rpc_server_commands_defs.h and for
every compilation unit that includes it which previously had to
compile all the serialization code and then throw all by one copy away
at link time). This required some new macros so as to not break a ton
of places that will use the old way putting everything in the headers;
The RPC code uses this as does a few other places; there are comments
in contrib/epee/include/serialization/keyvalue_serialization.h as to
how to use it.
- Detemplatized a bunch of epee/storages code. Most of it should have
have been using templates at all (because it can only ever be called
with one type!), and now it isn't. This broke some things that didn't
properly compile because of missing headers or (in one case) a messed
up circular dependency.
- Significantly simplified a bunch of over-templatized serialization
code.
- All RPC serialization definitions is now out of
core_rpc_server_commands_defs.h and into a single .cpp file
(core_rpc_server_commands_defs.cpp).
- core RPC no longer uses the disgusting
BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design
that forced slamming tons of code into a common header that didn't
need to be there.
- epee::struct_init is gone. It was a horrible hack that instiated
multiple templates just so the coder could be so lazy and write
`some_type var;` instead of properly value initializing with
`some_type var{};`.
- Removed a bunch of useless crap from epee. In particular, forcing
extra template instantiations all over the place in order to nest
return objects inside JSON RPC values is no longer needed, as are a
bunch of stuff related to the above de-macroization of the code.
- get_all_service_nodes, get_service_nodes, and get_n_service_nodes are
now combined into a single `get_service_nodes` (with deprecated
aliases for the others), which eliminates a fair amount of
duplication. The biggest obstacle here was getting the requested
fields reference passed through: this is now done by a new ability to
stash a context in the serialization object that can be retrieved by a
sub-serialized type.
LMQ-specifics:
- The LokiMQ instance moves into `cryptonote::core` rather than being
inside cryptonote_protocol. Currently the instance is used both for
qnet and rpc calls (and so needs to be in a common place), but I also
intend future PRs to use the batching code for job processing
(replacing the current threaded job queue).
- rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue.
Unlike http_server it isn't technically running the whole LMQ stack
from here, but the parallel name with http_server seemed appropriate.
- All RPC endpoints are supported by LMQ under the same names as defined
generically, but prefixed with `rpc.` for public commands and `admin.`
for restricted ones.
- service node keys are now always available, even when not running in
`--service-node` mode: this is because we want the x25519 key for
being able to offer CURVE encryption for lmq RPC end-points, and
because it doesn't hurt to have them available all the time. In the
RPC layer this is now called "get_service_keys" (with
"get_service_node_key" as an alias) since they aren't strictly only
for service nodes. This also means code needs to check
m_service_node, and not m_service_node_keys, to tell if it is running
as a service node. (This is also easier to notice because
m_service_node_keys got renamed to `m_service_keys`).
- Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and
`sub.mempool` subscribes the connection for new block and new mempool
TX notifications. The latter can notify on just blink txes, or all
new mempool txes (but only new ones -- txes dumped from a block don't
trigger it). The client gets pushed a [`notify.block`, `height`,
`hash`] or [`notify.tx`, `txhash`, `blob`] message when something
arrives.
Minor details:
- rpc::version_t is now a {major,minor} pair. Forcing everyone to pack
and unpack a uint32_t was gross.
- Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...).
(This immediately revealed a couple of bugs in the RPC code that was
assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because
the macro allows implicit conversion to a char).
- De-templatizing useless templates in epee (i.e. a bunch of templated
types that were never invoked with different types) revealed a painful
circular dependency between epee and non-epee code for tor_address and
i2p_address. This crap is now handled in a suitably named
`net/epee_network_address_hack.cpp` hack because it really isn't
trivial to extricate this mess.
- Removed `epee/include/serialization/serialize_base.h`. Amazingly the
code somehow still all works perfectly with this previously vital
header removed.
- Removed bitrotted, unused epee "crypted_storage" and
"gzipped_inmemstorage" code.
- Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with
LOKI_DEFERs. The epee version involves quite a bit more instantiation
and is ugly as sin. Also made the `loki::defer` class invokable for
some edge cases that need calling before destruction in particular
conditions.
- Moved the systemd code around; it makes much more sense to do the
systemd started notification as in daemon.cpp as late as possible
rather than in core (when we can still have startup failures, e.g. if
the RPC layer can't start).
- Made the systemd short status string available in the get_info RPC
(and no longer require building with systemd).
- during startup, print (only) the x25519 when not in SN mode, and
continue to print all three when in SN mode.
- DRYed out some RPC implementation code (such as set_limit)
- Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
/**
* Specifies a callback to invoke when one or more transactions is added to the mempool . Note
* that , because incoming blocks have their transactions added to the mempool , this * does *
* trigger for txes that arrive in new blocks .
*
* It does not , however , trigger for transactions that fail verification , that are flagged
* do - not - relay , or that are returned to the pool from a block ( i . e . when doing a reorg ) .
*/
void add_notify ( std : : function < void ( const crypto : : hash & , const transaction & , const std : : string & blob , const tx_pool_options & ) > notify ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief locks the transaction pool
*/
2019-11-24 19:45:09 +01:00
void lock ( ) const { m_transactions_lock . lock ( ) ; }
2016-03-25 01:03:02 +01:00
/**
* @ brief unlocks the transaction pool
*/
2019-11-24 19:45:09 +01:00
void unlock ( ) const { m_transactions_lock . unlock ( ) ; }
/**
* @ briefs does a non - blocking attempt to lock the transaction pool
*/
bool try_lock ( ) const { return m_transactions_lock . try_lock ( ) ; }
2014-03-03 23:07:58 +01:00
2019-11-06 07:28:33 +01:00
/**
* @ brief obtains a unique lock on the approved blink tx pool
*/
template < typename . . . Args >
2020-06-02 01:01:24 +02:00
auto blink_unique_lock ( Args & & . . . args ) const { return std : : unique_lock { m_blinks_mutex , std : : forward < Args > ( args ) . . . } ; }
2019-11-06 07:28:33 +01:00
/**
* @ brief obtains a shared lock on the approved blink tx pool
*/
template < typename . . . Args >
2020-06-02 01:01:24 +02:00
auto blink_shared_lock ( Args & & . . . args ) const { return std : : shared_lock { m_blinks_mutex , std : : forward < Args > ( args ) . . . } ; }
2019-11-06 07:28:33 +01:00
2014-03-03 23:07:58 +01:00
// load/store operations
2016-03-25 01:03:02 +01:00
/**
* @ brief loads pool state ( if any ) from disk , and initializes pool
*
2018-07-18 23:24:53 +02:00
* @ param max_txpool_weight the max weight in bytes
2016-03-25 01:03:02 +01:00
*
* @ return true
*/
2018-07-18 23:24:53 +02:00
bool init ( size_t max_txpool_weight = 0 ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief attempts to save the transaction pool state to disk
*
* Currently fails ( returns false ) if the data directory from init ( )
* does not exist and cannot be created , but returns true even if
* saving to disk is unsuccessful .
*
* @ return true in most cases ( see above )
*/
2014-03-03 23:07:58 +01:00
bool deinit ( ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief Chooses transactions for a block to include
*
* @ param bl return - by - reference the block to fill in with transactions
2018-07-18 23:24:53 +02:00
* @ param median_weight the current median block weight
2016-03-25 01:03:02 +01:00
* @ param already_generated_coins the current total number of coins " minted "
2018-07-18 23:24:53 +02:00
* @ param total_weight return - by - reference the total weight of the new block
2020-09-25 01:51:13 +02:00
* @ param raw_fee return - by - reference the total of fees from the included transactions . Note that this does not subtract any large block penalty fees ; this is just the raw sum of fees of included txes .
* @ param expected_reward return - by - reference the total reward awarded to the block producer finding this block , including transaction fees and , if applicable , a large block reward penalty .
2017-01-06 21:34:37 +01:00
* @ param version hard fork version to use for consensus rules
2016-03-25 01:03:02 +01:00
*
* @ return true
*/
2020-09-25 02:50:45 +02:00
bool fill_block_template ( block & bl , size_t median_weight , uint64_t already_generated_coins , size_t & total_weight , uint64_t & raw_fee , uint64_t & expected_reward , uint8_t version , uint64_t height ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief get a list of all transactions in the pool
*
* @ param txs return - by - reference the list of transactions
2017-11-08 13:06:41 +01:00
* @ param include_unrelayed_txes include unrelayed txes in the result
*
2016-03-25 01:03:02 +01:00
*/
2018-04-16 01:16:02 +02:00
void get_transactions ( std : : vector < transaction > & txs , bool include_unrelayed_txes = true ) const ;
2016-03-25 01:03:02 +01:00
2017-03-22 19:03:23 +01:00
/**
* @ brief get a list of all transaction hashes in the pool
*
* @ param txs return - by - reference the list of transactions
2017-11-08 13:06:41 +01:00
* @ param include_unrelayed_txes include unrelayed txes in the result
*
2017-03-22 19:03:23 +01:00
*/
2020-10-01 10:08:19 +02:00
void get_transaction_hashes ( std : : vector < crypto : : hash > & txs , bool include_unrelayed_txes = true , bool include_only_blinked = false ) const ;
2017-03-22 19:03:23 +01:00
2017-08-26 17:23:31 +02:00
/**
2018-07-18 23:24:53 +02:00
* @ brief get ( weight , fee , receive time ) for all transaction in the pool
2017-08-26 17:23:31 +02:00
*
* @ param txs return - by - reference that data
2017-11-08 13:06:41 +01:00
* @ param include_unrelayed_txes include unrelayed txes in the result
*
2017-08-26 17:23:31 +02:00
*/
RPC overhaul
High-level details:
This redesigns the RPC layer to make it much easier to work with,
decouples it from an embedded HTTP server, and gets the vast majority of
the RPC serialization and dispatch code out of a very commonly included
header.
There is unfortunately rather a lot of interconnected code here that
cannot be easily separated out into separate commits. The full details
of what happens here are as follows:
Major details:
- All of the RPC code is now in a `cryptonote::rpc` namespace; this
renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK
becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME`
becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already
working in the `rpc` namespace).
- `core_rpc_server` is now completely decoupled from providing any
request protocol: it is now *just* the core RPC call handler.
- The HTTP RPC interface now lives in a new rpc/http_server.h; this code
handles listening for HTTP requests and dispatching them to
core_rpc_server, then sending the results back to the caller.
- There is similarly a rpc/lmq_server.h for LMQ RPC code; more details
on this (and other LMQ specifics) below.
- RPC implementing code now returns the response object and throws when
things go wrong which simplifies much of the rpc error handling. They
can throw anything; generic exceptions get logged and a generic
"internal error" message gets returned to the caller, but there is
also an `rpc_error` class to return an error code and message used by
some json-rpc commands.
- RPC implementing functions now overload `core_rpc_server::invoke`
following the pattern:
RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context);
This overloading makes the code vastly simpler: all instantiations are
now done with a small amount of generic instantiation code in a single
.cpp rather than needing to go to hell and back with a nest of epee
macros in a core header.
- each RPC endpoint is now defined by the RPC types themselves,
including its accessible names and permissions, in
core_rpc_server_commands_defs.h:
- every RPC structure now has a static `names()` function that returns
the names by which the end point is accessible. (The first one is
the primary, the others are for deprecated aliases).
- RPC command wrappers define their permissions and type by inheriting
from special tag classes:
- rpc::RPC_COMMAND is a basic, admin-only, JSON command, available
via JSON RPC. *All* JSON commands are now available via JSON RPC,
instead of the previous mix of some being at /foo and others at
/json_rpc. (Ones that were previously at /foo are still there for
backwards compatibility; see `rpc::LEGACY` below).
- rpc::PUBLIC specifies that the command should be available via a
restricted RPC connection.
- rpc::BINARY specifies that the command is not JSON, but rather is
accessible as /name and takes and returns values in the magic epee
binary "portable storage" (lol) data format.
- rpc::LEGACY specifies that the command should be available via the
non-json-rpc interface at `/name` for backwards compatibility (in
addition to the JSON-RPC interface).
- some epee serialization got unwrapped and de-templatized so that it
can be moved into a .cpp file with just declarations in the .h. (This
makes a *huge* difference for core_rpc_server_commands_defs.h and for
every compilation unit that includes it which previously had to
compile all the serialization code and then throw all by one copy away
at link time). This required some new macros so as to not break a ton
of places that will use the old way putting everything in the headers;
The RPC code uses this as does a few other places; there are comments
in contrib/epee/include/serialization/keyvalue_serialization.h as to
how to use it.
- Detemplatized a bunch of epee/storages code. Most of it should have
have been using templates at all (because it can only ever be called
with one type!), and now it isn't. This broke some things that didn't
properly compile because of missing headers or (in one case) a messed
up circular dependency.
- Significantly simplified a bunch of over-templatized serialization
code.
- All RPC serialization definitions is now out of
core_rpc_server_commands_defs.h and into a single .cpp file
(core_rpc_server_commands_defs.cpp).
- core RPC no longer uses the disgusting
BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design
that forced slamming tons of code into a common header that didn't
need to be there.
- epee::struct_init is gone. It was a horrible hack that instiated
multiple templates just so the coder could be so lazy and write
`some_type var;` instead of properly value initializing with
`some_type var{};`.
- Removed a bunch of useless crap from epee. In particular, forcing
extra template instantiations all over the place in order to nest
return objects inside JSON RPC values is no longer needed, as are a
bunch of stuff related to the above de-macroization of the code.
- get_all_service_nodes, get_service_nodes, and get_n_service_nodes are
now combined into a single `get_service_nodes` (with deprecated
aliases for the others), which eliminates a fair amount of
duplication. The biggest obstacle here was getting the requested
fields reference passed through: this is now done by a new ability to
stash a context in the serialization object that can be retrieved by a
sub-serialized type.
LMQ-specifics:
- The LokiMQ instance moves into `cryptonote::core` rather than being
inside cryptonote_protocol. Currently the instance is used both for
qnet and rpc calls (and so needs to be in a common place), but I also
intend future PRs to use the batching code for job processing
(replacing the current threaded job queue).
- rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue.
Unlike http_server it isn't technically running the whole LMQ stack
from here, but the parallel name with http_server seemed appropriate.
- All RPC endpoints are supported by LMQ under the same names as defined
generically, but prefixed with `rpc.` for public commands and `admin.`
for restricted ones.
- service node keys are now always available, even when not running in
`--service-node` mode: this is because we want the x25519 key for
being able to offer CURVE encryption for lmq RPC end-points, and
because it doesn't hurt to have them available all the time. In the
RPC layer this is now called "get_service_keys" (with
"get_service_node_key" as an alias) since they aren't strictly only
for service nodes. This also means code needs to check
m_service_node, and not m_service_node_keys, to tell if it is running
as a service node. (This is also easier to notice because
m_service_node_keys got renamed to `m_service_keys`).
- Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and
`sub.mempool` subscribes the connection for new block and new mempool
TX notifications. The latter can notify on just blink txes, or all
new mempool txes (but only new ones -- txes dumped from a block don't
trigger it). The client gets pushed a [`notify.block`, `height`,
`hash`] or [`notify.tx`, `txhash`, `blob`] message when something
arrives.
Minor details:
- rpc::version_t is now a {major,minor} pair. Forcing everyone to pack
and unpack a uint32_t was gross.
- Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...).
(This immediately revealed a couple of bugs in the RPC code that was
assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because
the macro allows implicit conversion to a char).
- De-templatizing useless templates in epee (i.e. a bunch of templated
types that were never invoked with different types) revealed a painful
circular dependency between epee and non-epee code for tor_address and
i2p_address. This crap is now handled in a suitably named
`net/epee_network_address_hack.cpp` hack because it really isn't
trivial to extricate this mess.
- Removed `epee/include/serialization/serialize_base.h`. Amazingly the
code somehow still all works perfectly with this previously vital
header removed.
- Removed bitrotted, unused epee "crypted_storage" and
"gzipped_inmemstorage" code.
- Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with
LOKI_DEFERs. The epee version involves quite a bit more instantiation
and is ugly as sin. Also made the `loki::defer` class invokable for
some edge cases that need calling before destruction in particular
conditions.
- Moved the systemd code around; it makes much more sense to do the
systemd started notification as in daemon.cpp as late as possible
rather than in core (when we can still have startup failures, e.g. if
the RPC layer can't start).
- Made the systemd short status string available in the get_info RPC
(and no longer require building with systemd).
- during startup, print (only) the x25519 when not in SN mode, and
continue to print all three when in SN mode.
- DRYed out some RPC implementation code (such as set_limit)
- Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
void get_transaction_backlog ( std : : vector < rpc : : tx_backlog_entry > & backlog , bool include_unrelayed_txes = true ) const ;
2017-08-26 17:23:31 +02:00
2021-08-13 02:32:03 +02:00
/// Return type of get_transaction_stats()
struct tx_stats
{
uint64_t bytes_total ; ///< Total size of all transactions in pool.
uint32_t bytes_min ; ///< Min transaction size in pool.
uint32_t bytes_max ; ///< Max transaction size in pool.
uint32_t bytes_med ; ///< Median transaction size in pool.
uint64_t fee_total ; ///< Total fee's in pool in atomic units.
uint64_t oldest ; ///< Unix time of the oldest transaction in the pool.
uint32_t txs_total ; ///< Total number of transactions.
uint32_t num_failing ; ///< Bumber of failing transactions.
uint32_t num_10m ; ///< Number of transactions in pool for more than 10 minutes.
uint32_t num_not_relayed ; ///< Number of non-relayed transactions.
uint64_t histo_98pc ; ///< the time 98% of txes are "younger" than.
std : : vector < std : : pair < uint32_t , uint64_t > > histo ; ///< List of txpool histo [number of txes, size in bytes] pairs.
uint32_t num_double_spends ; ///< Number of double spend transactions.
} ;
2017-05-31 20:11:56 +02:00
/**
* @ brief get a summary statistics of all transaction hashes in the pool
*
2017-11-08 13:06:41 +01:00
* @ param include_unrelayed_txes include unrelayed txes in the result
*
2021-08-13 02:32:03 +02:00
* @ return txpool_stats struct of pool statistics
2017-05-31 20:11:56 +02:00
*/
2021-08-13 02:32:03 +02:00
tx_stats get_transaction_stats ( bool include_unrelayed_txes = true ) const ;
2017-05-31 20:11:56 +02:00
2016-03-25 01:03:02 +01:00
/**
* @ brief get information about all transactions and key images in the pool
*
* see documentation on tx_info and spent_key_image_info for more details
*
* @ param tx_infos return - by - reference the transactions ' information
* @ param key_image_infos return - by - reference the spent key images ' information
2020-08-20 06:16:37 +02:00
* @ param post_process optional function to call to do any extra tx_info processing from the transaction
2017-11-08 13:06:41 +01:00
* @ param include_sensitive_data include unrelayed txes and fields that are sensitive to the node privacy
2016-03-25 01:03:02 +01:00
*
* @ return true
*/
2020-08-20 06:16:37 +02:00
bool get_transactions_and_spent_keys_info ( std : : vector < rpc : : tx_info > & tx_infos , std : : vector < rpc : : spent_key_image_info > & key_image_infos , std : : function < void ( const transaction & tx , rpc : : tx_info & ) > post_process = nullptr , bool include_sensitive_data = true ) const ;
2016-03-25 01:03:02 +01:00
2017-09-05 18:20:27 +02:00
/**
* @ brief check for presence of key images in the pool
*
* @ param key_images [ in ] vector of key images to check
* @ param spent [ out ] vector of bool to return
*
* @ return true
*/
2019-10-06 05:28:30 +02:00
bool check_for_key_images ( const std : : vector < crypto : : key_image > & key_images , std : : vector < bool > & spent ) const ;
2017-09-05 18:20:27 +02:00
2016-03-25 01:03:02 +01:00
/**
* @ brief get a specific transaction from the pool
*
* @ param h the hash of the transaction to get
2017-05-14 15:06:55 +02:00
* @ param tx return - by - reference the transaction blob requested
2016-03-25 01:03:02 +01:00
*
* @ return true if the transaction is found , otherwise false
*/
2017-05-14 15:06:55 +02:00
bool get_transaction ( const crypto : : hash & h , cryptonote : : blobdata & txblob ) const ;
2016-03-25 01:03:02 +01:00
/**
2019-11-20 01:24:48 +01:00
* @ brief get specific transactions from the pool
*
* @ param hashes - tx hashes of desired transactions
* @ param txblobs - vector of blobdata ( i . e . std : : strings ) to which found blobs should be
* appended . The vector is * not * cleared of existing values .
*
* @ return number of transactions added to txblobs
*/
int find_transactions ( const std : : vector < crypto : : hash > & tx_hashes , std : : vector < cryptonote : : blobdata > & txblobs ) const ;
/**
2016-03-25 01:03:02 +01:00
* @ brief get a list of all relayable transactions and their hashes
*
* " relayable " in this case means :
2019-11-03 16:28:47 +01:00
* nonzero fee - or - a zero - fee SN state change tx
2016-03-25 01:03:02 +01:00
* hasn ' t been relayed too recently
* isn ' t old enough that relaying it is considered harmful
2019-11-03 16:28:47 +01:00
* doesn ' t have do_not_relay set
2016-03-25 01:03:02 +01:00
*
* @ param txs return - by - reference the transactions and their hashes
*
* @ return true
*/
2018-04-16 01:16:02 +02:00
bool get_relayable_transactions ( std : : vector < std : : pair < crypto : : hash , cryptonote : : blobdata > > & txs ) const ;
2016-03-25 01:03:02 +01:00
2019-11-06 07:28:33 +01:00
/**
* @ brief clear transactions ' ` do_not_relay ` flags ( if set ) so that they can start being
* relayed . ( Note that it still must satisfy the other conditions of
* ` get_relayable_transactions ` to actually be relayable ) .
*
* @ return the number of txes that were found with an active ` do_not_relay ` flag that was
* cleared .
*/
int set_relayable ( const std : : vector < crypto : : hash > & tx_hashes ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief tell the pool that certain transactions were just relayed
*
* @ param txs the list of transactions ( and their hashes )
*/
2018-04-16 01:16:02 +02:00
void set_relayed ( const std : : vector < std : : pair < crypto : : hash , cryptonote : : blobdata > > & txs ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief get the total number of transactions in the pool
*
* @ return the number of transactions in the pool
*/
2017-11-08 13:06:41 +01:00
size_t get_transactions_count ( bool include_unrelayed_txes = true ) const ;
2016-03-25 01:03:02 +01:00
/**
* @ brief remove transactions from the pool which are no longer valid
*
* With new versions of the currency , what conditions render a transaction
* invalid may change . This function clears those which were received
* before a version change and no longer conform to requirements .
*
* @ param version the version the transactions must conform to
*
* @ return the number of transactions removed
*/
2016-01-29 16:09:17 +01:00
size_t validate ( uint8_t version ) ;
2014-03-03 23:07:58 +01:00
2016-04-17 12:04:01 +02:00
/**
* @ brief return the cookie
*
* @ return the cookie
*/
uint64_t cookie ( ) const { return m_cookie ; }
2018-01-29 22:20:24 +01:00
/**
2018-07-18 23:24:53 +02:00
* @ brief get the cumulative txpool weight in bytes
2018-01-29 22:20:24 +01:00
*
2018-07-18 23:24:53 +02:00
* @ return the cumulative txpool weight in bytes
2018-01-29 22:20:24 +01:00
*/
2018-07-18 23:24:53 +02:00
size_t get_txpool_weight ( ) const ;
2018-01-29 22:20:24 +01:00
/**
2018-07-18 23:24:53 +02:00
* @ brief set the max cumulative txpool weight in bytes
2018-01-29 22:20:24 +01:00
*
2018-07-18 23:24:53 +02:00
* @ param bytes the max cumulative txpool weight in bytes
2018-01-29 22:20:24 +01:00
*/
2018-07-18 23:24:53 +02:00
void set_txpool_max_weight ( size_t bytes ) ;
2014-03-03 23:07:58 +01:00
private :
2016-03-25 01:03:02 +01:00
2017-05-14 15:06:55 +02:00
/**
* @ brief insert key images into m_spent_key_images
*
* @ return true on success , false on error
*/
2018-09-16 20:30:39 +02:00
bool insert_key_images ( const transaction_prefix & tx , const crypto : : hash & txid , bool kept_by_block ) ;
2017-05-14 15:06:55 +02:00
2016-03-25 01:03:02 +01:00
/**
* @ brief remove old transactions from the pool
*
* After a certain time , it is assumed that a transaction which has not
* yet been mined will likely not be mined . These transactions are removed
* from the pool to avoid buildup .
*
* @ return true
*/
2015-12-14 05:54:39 +01:00
bool remove_stuck_transactions ( ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief check if a transaction in the pool has a given spent key image
*
* @ param key_im the spent key image to look for
*
* @ return true if the spent key image is present , otherwise false
*/
2014-07-17 16:31:44 +02:00
bool have_tx_keyimg_as_spent ( const crypto : : key_image & key_im ) const ;
2016-03-25 01:03:02 +01:00
Service Node Deregister Part 5 (#89)
* Retrieve quorum list from height, reviewed
* Setup data structures for de/register TX
* Submit and validate partial/full deregisters
* Add P2P relaying of partial deregistration votes
* Code review adjustments for deregistration part 1
- Fix check_tx_semantic
- Remove signature_pod as votes are now stored as blobs. Serialization
overrides don't intefere with crypto::signature anymore.
* deregistration_vote_pool - changed sign/verify interface and removed repeated code
* Misc review, fix sign/verify api, vote threshold
* Deregister/tx edge case handling for combinatoric votes
* core, service_node_list: separated address from service node pubkey
* Retrieve quorum list from height, reviewed
* Setup data structures for de/register TX
* Submit and validate partial/full deregisters
* Add P2P relaying of partial deregistration votes
* Code review adjustments for deregistration part 1
- Fix check_tx_semantic
- Remove signature_pod as votes are now stored as blobs. Serialization
overrides don't intefere with crypto::signature anymore.
* deregistration_vote_pool - changed sign/verify interface and removed repeated code
* Misc review, fix sign/verify api, vote threshold
* Deregister/tx edge case handling for combinatoric votes
* Store service node lists for the duration of deregister lifetimes
* Quorum min/max bug, sort node list, fix node to test list
* Change quorum to store acc pub address, fix oob bug
* Code review for expiring votes, acc keys to pub_key, improve err msgs
* Add early out for is_deregistration_tx and protect against quorum changes
* Remove debug code, fix segfault
* Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states
Incorrect assumption that a transaction can be kept in the chain if it could
eventually become invalid, because if it were the chain would be split and
eventually these transaction would be dropped. But also that we should not
override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
/**
2019-01-25 04:15:52 +01:00
* @ brief check if a tx that does not have a key - image component has a duplicate in the pool
Service Node Deregister Part 5 (#89)
* Retrieve quorum list from height, reviewed
* Setup data structures for de/register TX
* Submit and validate partial/full deregisters
* Add P2P relaying of partial deregistration votes
* Code review adjustments for deregistration part 1
- Fix check_tx_semantic
- Remove signature_pod as votes are now stored as blobs. Serialization
overrides don't intefere with crypto::signature anymore.
* deregistration_vote_pool - changed sign/verify interface and removed repeated code
* Misc review, fix sign/verify api, vote threshold
* Deregister/tx edge case handling for combinatoric votes
* core, service_node_list: separated address from service node pubkey
* Retrieve quorum list from height, reviewed
* Setup data structures for de/register TX
* Submit and validate partial/full deregisters
* Add P2P relaying of partial deregistration votes
* Code review adjustments for deregistration part 1
- Fix check_tx_semantic
- Remove signature_pod as votes are now stored as blobs. Serialization
overrides don't intefere with crypto::signature anymore.
* deregistration_vote_pool - changed sign/verify interface and removed repeated code
* Misc review, fix sign/verify api, vote threshold
* Deregister/tx edge case handling for combinatoric votes
* Store service node lists for the duration of deregister lifetimes
* Quorum min/max bug, sort node list, fix node to test list
* Change quorum to store acc pub address, fix oob bug
* Code review for expiring votes, acc keys to pub_key, improve err msgs
* Add early out for is_deregistration_tx and protect against quorum changes
* Remove debug code, fix segfault
* Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states
Incorrect assumption that a transaction can be kept in the chain if it could
eventually become invalid, because if it were the chain would be split and
eventually these transaction would be dropped. But also that we should not
override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
* @ return true if it already exists
*
*/
2019-10-27 23:26:38 +01:00
bool have_duplicated_non_standard_tx ( transaction const & tx , uint8_t hard_fork_version ) const ;
Service Node Deregister Part 5 (#89)
* Retrieve quorum list from height, reviewed
* Setup data structures for de/register TX
* Submit and validate partial/full deregisters
* Add P2P relaying of partial deregistration votes
* Code review adjustments for deregistration part 1
- Fix check_tx_semantic
- Remove signature_pod as votes are now stored as blobs. Serialization
overrides don't intefere with crypto::signature anymore.
* deregistration_vote_pool - changed sign/verify interface and removed repeated code
* Misc review, fix sign/verify api, vote threshold
* Deregister/tx edge case handling for combinatoric votes
* core, service_node_list: separated address from service node pubkey
* Retrieve quorum list from height, reviewed
* Setup data structures for de/register TX
* Submit and validate partial/full deregisters
* Add P2P relaying of partial deregistration votes
* Code review adjustments for deregistration part 1
- Fix check_tx_semantic
- Remove signature_pod as votes are now stored as blobs. Serialization
overrides don't intefere with crypto::signature anymore.
* deregistration_vote_pool - changed sign/verify interface and removed repeated code
* Misc review, fix sign/verify api, vote threshold
* Deregister/tx edge case handling for combinatoric votes
* Store service node lists for the duration of deregister lifetimes
* Quorum min/max bug, sort node list, fix node to test list
* Change quorum to store acc pub address, fix oob bug
* Code review for expiring votes, acc keys to pub_key, improve err msgs
* Add early out for is_deregistration_tx and protect against quorum changes
* Remove debug code, fix segfault
* Remove irrelevant check for tx v3 in blockchain, fix >= height for pruning quorum states
Incorrect assumption that a transaction can be kept in the chain if it could
eventually become invalid, because if it were the chain would be split and
eventually these transaction would be dropped. But also that we should not
override the pre-existing logic which handles this case anyway.
2018-07-18 04:42:47 +02:00
2016-03-25 01:03:02 +01:00
/**
* @ brief check if any spent key image in a transaction is in the pool
*
* Checks if any of the spent key images in a given transaction are present
* in any of the transactions in the transaction pool .
*
* @ note see tx_pool : : have_tx_keyimg_as_spent
*
* @ param tx the transaction to check spent key images of
2019-11-25 05:57:15 +01:00
* @ param found if specified , append the hashes of all conflicting mempool txes here
2016-03-25 01:03:02 +01:00
*
* @ return true if any spent key images are present in the pool , otherwise false
*/
2019-11-25 05:57:15 +01:00
bool have_tx_keyimges_as_spent ( const transaction & tx , std : : vector < crypto : : hash > * conflicting = nullptr ) const ;
2016-03-25 01:03:02 +01:00
/**
* @ brief forget a transaction ' s spent key images
*
* Spent key images are stored separately from transactions for
* convenience / speed , so this is part of the process of removing
* a transaction from the pool .
*
* @ param tx the transaction
2018-09-16 20:30:39 +02:00
* @ param txid the transaction ' s hash
2016-03-25 01:03:02 +01:00
*
* @ return false if any key images to be removed cannot be found , otherwise true
*/
2018-09-16 20:30:39 +02:00
bool remove_transaction_keyimages ( const transaction_prefix & tx , const crypto : : hash & txid ) ;
2016-03-25 01:03:02 +01:00
/**
* @ brief check if a transaction is a valid candidate for inclusion in a block
*
* @ param txd the transaction to check ( and info about it )
2018-06-24 11:46:57 +02:00
* @ param txid the txid of the transaction to check
2018-06-01 22:50:46 +02:00
* @ param txblob the transaction blob to check
2018-10-31 13:14:30 +01:00
* @ param tx the parsed transaction , if successful
2016-03-25 01:03:02 +01:00
*
* @ return true if the transaction is good to go , otherwise false
*/
2018-09-16 20:30:39 +02:00
bool is_transaction_ready_to_go ( txpool_tx_meta_t & txd , const crypto : : hash & txid , const cryptonote : : blobdata & txblob , transaction & tx ) const ;
2016-03-25 01:03:02 +01:00
2017-09-22 14:57:20 +02:00
/**
* @ brief mark all transactions double spending the one passed
*/
void mark_double_spend ( const transaction & tx ) ;
2019-11-24 19:45:09 +01:00
/**
* @ brief remove a transaction from the mempool
*
* This is called when pruning the mempool to reduce its size , and when deleting transactions
* from the mempool because of a conflicting blink transaction arriving . Transactions lock and
* blockchain lock must be held by the caller .
*
* @ param txid the transaction id to remove
* @ param meta optional pointer to txpool_tx_meta_t ; will be looked up if omitted
* @ param stc_it an optional iterator to the tx ' s entry in m_txs_by_fee_and_receive_time to save
* a ( linear ) scan to find it when already available . The given iterator will be invalidated if
* removed .
*
* @ return true if the transaction was removed , false on failure .
*/
bool remove_tx ( const crypto : : hash & txid , const txpool_tx_meta_t * meta = nullptr , const sorted_tx_container : : iterator * stc_it = nullptr ) ;
2018-01-29 22:20:24 +01:00
/**
* @ brief prune lowest fee / byte txes till we ' re not above bytes
*
2019-11-25 05:57:15 +01:00
* @ param skip don ' t prune the given ID this time ( because it was just added )
2018-01-29 22:20:24 +01:00
*/
2019-11-25 05:57:15 +01:00
void prune ( const crypto : : hash & skip ) ;
/**
* @ brief Attempt to add a blink tx " by force " , removing conflicting non - blink txs
*
* The given transactions are removed from the mempool , if possible , to make way for this blink
* transactions . In order for any removal to happen , all the conflicting txes must be non - blink
* transactions , and must either :
* - be a mempool transaction
* - be a mined , non - blink transaction in the recent ( mutable ) section of the chain
*
* If all conflicting txs satisfy the above then conflicting mempool txs are removed and the
* blink_rollback_height pointer is updated to the required rollback height to eject any mined
* txs ( if not already at that height or lower ) . True is returned .
*
* If any txs are found that do not satisfy the above then nothing is removed and false is
* returned .
*
* @ param the id of the incoming blink tx
* @ param conflict_txs vector of conflicting transaction hashes that are preventing the blink tx
* @ param blink_rollback_height a pointer to update to the new required height if a chain
2019-12-26 07:38:00 +01:00
* rollback is needed for the blink tx . ( That is , all blocks with height > =
* blink_rollback_height need to be popped ) .
2019-11-25 05:57:15 +01:00
*
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
* This method is * not * called with a blink lock held .
*
2019-11-25 05:57:15 +01:00
* @ return true if the conflicting transactions have been removed ( and / or the rollback height
* set ) , false if tx removal and / or rollback are insufficient to eliminate conflicting txes .
*/
bool remove_blink_conflicts ( const crypto : : hash & id , const std : : vector < crypto : : hash > & conflict_txs , uint64_t * blink_rollback_height ) ;
2018-01-29 22:20:24 +01:00
2016-03-25 01:03:02 +01:00
//TODO: confirm the below comments and investigate whether or not this
// is the desired behavior
//! map key images to transactions which spent them
/*! this seems odd, but it seems that multiple transactions can exist
* in the pool which both have the same spent key . This would happen
* in the event of a reorg where someone creates a new / different
* transaction on the assumption that the original will not be in a
* block again .
*/
2014-03-03 23:07:58 +01:00
typedef std : : unordered_map < crypto : : key_image , std : : unordered_set < crypto : : hash > > key_images_container ;
2020-06-02 01:01:24 +02:00
mutable std : : recursive_mutex m_transactions_lock ; //!< mutex for the pool
2016-03-25 01:03:02 +01:00
//! container for spent key images from the transactions in the pool
key_images_container m_spent_key_images ;
//TODO: this time should be a named constant somewhere, not hard-coded
//! interval on which to check for stale/"stuck" transactions
2020-05-26 07:15:25 +02:00
tools : : periodic_task m_remove_stuck_tx_interval { 30 s } ;
2014-03-03 23:07:58 +01:00
2016-03-25 01:03:02 +01:00
//TODO: look into doing this better
2017-01-13 16:08:37 +01:00
//!< container for transactions organized by fee per size and receive time
sorted_tx_container m_txs_by_fee_and_receive_time ;
2015-05-14 02:27:06 +02:00
2016-04-17 12:04:01 +02:00
std : : atomic < uint64_t > m_cookie ; //!< incremented at each change
RPC overhaul
High-level details:
This redesigns the RPC layer to make it much easier to work with,
decouples it from an embedded HTTP server, and gets the vast majority of
the RPC serialization and dispatch code out of a very commonly included
header.
There is unfortunately rather a lot of interconnected code here that
cannot be easily separated out into separate commits. The full details
of what happens here are as follows:
Major details:
- All of the RPC code is now in a `cryptonote::rpc` namespace; this
renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK
becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME`
becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already
working in the `rpc` namespace).
- `core_rpc_server` is now completely decoupled from providing any
request protocol: it is now *just* the core RPC call handler.
- The HTTP RPC interface now lives in a new rpc/http_server.h; this code
handles listening for HTTP requests and dispatching them to
core_rpc_server, then sending the results back to the caller.
- There is similarly a rpc/lmq_server.h for LMQ RPC code; more details
on this (and other LMQ specifics) below.
- RPC implementing code now returns the response object and throws when
things go wrong which simplifies much of the rpc error handling. They
can throw anything; generic exceptions get logged and a generic
"internal error" message gets returned to the caller, but there is
also an `rpc_error` class to return an error code and message used by
some json-rpc commands.
- RPC implementing functions now overload `core_rpc_server::invoke`
following the pattern:
RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context);
This overloading makes the code vastly simpler: all instantiations are
now done with a small amount of generic instantiation code in a single
.cpp rather than needing to go to hell and back with a nest of epee
macros in a core header.
- each RPC endpoint is now defined by the RPC types themselves,
including its accessible names and permissions, in
core_rpc_server_commands_defs.h:
- every RPC structure now has a static `names()` function that returns
the names by which the end point is accessible. (The first one is
the primary, the others are for deprecated aliases).
- RPC command wrappers define their permissions and type by inheriting
from special tag classes:
- rpc::RPC_COMMAND is a basic, admin-only, JSON command, available
via JSON RPC. *All* JSON commands are now available via JSON RPC,
instead of the previous mix of some being at /foo and others at
/json_rpc. (Ones that were previously at /foo are still there for
backwards compatibility; see `rpc::LEGACY` below).
- rpc::PUBLIC specifies that the command should be available via a
restricted RPC connection.
- rpc::BINARY specifies that the command is not JSON, but rather is
accessible as /name and takes and returns values in the magic epee
binary "portable storage" (lol) data format.
- rpc::LEGACY specifies that the command should be available via the
non-json-rpc interface at `/name` for backwards compatibility (in
addition to the JSON-RPC interface).
- some epee serialization got unwrapped and de-templatized so that it
can be moved into a .cpp file with just declarations in the .h. (This
makes a *huge* difference for core_rpc_server_commands_defs.h and for
every compilation unit that includes it which previously had to
compile all the serialization code and then throw all by one copy away
at link time). This required some new macros so as to not break a ton
of places that will use the old way putting everything in the headers;
The RPC code uses this as does a few other places; there are comments
in contrib/epee/include/serialization/keyvalue_serialization.h as to
how to use it.
- Detemplatized a bunch of epee/storages code. Most of it should have
have been using templates at all (because it can only ever be called
with one type!), and now it isn't. This broke some things that didn't
properly compile because of missing headers or (in one case) a messed
up circular dependency.
- Significantly simplified a bunch of over-templatized serialization
code.
- All RPC serialization definitions is now out of
core_rpc_server_commands_defs.h and into a single .cpp file
(core_rpc_server_commands_defs.cpp).
- core RPC no longer uses the disgusting
BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design
that forced slamming tons of code into a common header that didn't
need to be there.
- epee::struct_init is gone. It was a horrible hack that instiated
multiple templates just so the coder could be so lazy and write
`some_type var;` instead of properly value initializing with
`some_type var{};`.
- Removed a bunch of useless crap from epee. In particular, forcing
extra template instantiations all over the place in order to nest
return objects inside JSON RPC values is no longer needed, as are a
bunch of stuff related to the above de-macroization of the code.
- get_all_service_nodes, get_service_nodes, and get_n_service_nodes are
now combined into a single `get_service_nodes` (with deprecated
aliases for the others), which eliminates a fair amount of
duplication. The biggest obstacle here was getting the requested
fields reference passed through: this is now done by a new ability to
stash a context in the serialization object that can be retrieved by a
sub-serialized type.
LMQ-specifics:
- The LokiMQ instance moves into `cryptonote::core` rather than being
inside cryptonote_protocol. Currently the instance is used both for
qnet and rpc calls (and so needs to be in a common place), but I also
intend future PRs to use the batching code for job processing
(replacing the current threaded job queue).
- rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue.
Unlike http_server it isn't technically running the whole LMQ stack
from here, but the parallel name with http_server seemed appropriate.
- All RPC endpoints are supported by LMQ under the same names as defined
generically, but prefixed with `rpc.` for public commands and `admin.`
for restricted ones.
- service node keys are now always available, even when not running in
`--service-node` mode: this is because we want the x25519 key for
being able to offer CURVE encryption for lmq RPC end-points, and
because it doesn't hurt to have them available all the time. In the
RPC layer this is now called "get_service_keys" (with
"get_service_node_key" as an alias) since they aren't strictly only
for service nodes. This also means code needs to check
m_service_node, and not m_service_node_keys, to tell if it is running
as a service node. (This is also easier to notice because
m_service_node_keys got renamed to `m_service_keys`).
- Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and
`sub.mempool` subscribes the connection for new block and new mempool
TX notifications. The latter can notify on just blink txes, or all
new mempool txes (but only new ones -- txes dumped from a block don't
trigger it). The client gets pushed a [`notify.block`, `height`,
`hash`] or [`notify.tx`, `txhash`, `blob`] message when something
arrives.
Minor details:
- rpc::version_t is now a {major,minor} pair. Forcing everyone to pack
and unpack a uint32_t was gross.
- Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...).
(This immediately revealed a couple of bugs in the RPC code that was
assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because
the macro allows implicit conversion to a char).
- De-templatizing useless templates in epee (i.e. a bunch of templated
types that were never invoked with different types) revealed a painful
circular dependency between epee and non-epee code for tor_address and
i2p_address. This crap is now handled in a suitably named
`net/epee_network_address_hack.cpp` hack because it really isn't
trivial to extricate this mess.
- Removed `epee/include/serialization/serialize_base.h`. Amazingly the
code somehow still all works perfectly with this previously vital
header removed.
- Removed bitrotted, unused epee "crypted_storage" and
"gzipped_inmemstorage" code.
- Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with
LOKI_DEFERs. The epee version involves quite a bit more instantiation
and is ugly as sin. Also made the `loki::defer` class invokable for
some edge cases that need calling before destruction in particular
conditions.
- Moved the systemd code around; it makes much more sense to do the
systemd started notification as in daemon.cpp as late as possible
rather than in core (when we can still have startup failures, e.g. if
the RPC layer can't start).
- Made the systemd short status string available in the get_info RPC
(and no longer require building with systemd).
- during startup, print (only) the x25519 when not in SN mode, and
continue to print all three when in SN mode.
- DRYed out some RPC implementation code (such as set_limit)
- Made wallet_rpc stop using a raw m_wallet pointer
2020-04-28 01:25:43 +02:00
/// Callbacks for new tx notifications
std : : vector < std : : function < void ( const crypto : : hash & , const transaction & , const std : : string & blob , const tx_pool_options & ) > > m_tx_notify ;
2016-03-25 01:03:02 +01:00
/**
* @ brief get an iterator to a transaction in the sorted container
*
* @ param id the hash of the transaction to look for
*
* @ return an iterator , possibly to the end of the container if not found
*/
2015-05-14 02:27:06 +02:00
sorted_tx_container : : iterator find_tx_in_sorted_container ( const crypto : : hash & id ) const ;
2015-04-30 07:02:12 +02:00
2018-06-24 11:46:57 +02:00
//! cache/call Blockchain::check_tx_inputs results
2019-12-26 07:38:00 +01:00
bool check_tx_inputs ( const std : : function < cryptonote : : transaction & ( ) > & get_tx , const crypto : : hash & txid , uint64_t & max_used_block_height , crypto : : hash & max_used_block_id , tx_verification_context & tvc , bool kept_by_block = false , uint64_t * blink_rollback_height = nullptr ) const ;
2018-06-24 11:46:57 +02:00
2016-03-25 01:03:02 +01:00
//! transactions which are unlikely to be included in blocks
/*! These transactions are kept in RAM in case they *are* included
* in a block eventually , but this container is not saved to disk .
*/
2016-01-29 18:15:09 +01:00
std : : unordered_set < crypto : : hash > m_timed_out_transactions ;
2016-03-25 01:03:02 +01:00
Blockchain & m_blockchain ; //!< reference to the Blockchain object
2018-01-29 22:20:24 +01:00
2018-07-18 23:24:53 +02:00
size_t m_txpool_max_weight ;
size_t m_txpool_weight ;
2018-06-24 11:46:57 +02:00
mutable std : : unordered_map < crypto : : hash , std : : tuple < bool , tx_verification_context , uint64_t , crypto : : hash > > m_input_cache ;
2018-10-31 14:52:16 +01:00
std : : unordered_map < crypto : : hash , transaction > m_parsed_tx_cache ;
2019-10-27 23:47:19 +01:00
2020-06-02 01:01:24 +02:00
mutable std : : shared_mutex m_blinks_mutex ;
2019-11-06 07:28:33 +01:00
// Contains blink metadata for approved blink transactions. { txhash => blink_tx, ... }.
2019-11-27 18:47:28 +01:00
mutable std : : unordered_map < crypto : : hash , std : : shared_ptr < cryptonote : : blink_tx > > m_blinks ;
2019-11-06 07:28:33 +01:00
2019-11-27 18:47:28 +01:00
// Helper method: retrieves hashes and mined heights of blink txes since the immutable block;
// mempool blinks are included with a height of 0. Also takes care of cleaning up any blinks
Fix rare blink pool deadlock
cryptonote_protocol_handler calls `pool.get_blink(hash)` while already
holding a blink shared lock, which should have been
`pool.get_blink(hash, true)` to avoid `get_blink` trying to take its own
lock.
That double lock is undefined behaviour and can cause a deadlock on the
mutex, although it appears rare that it actually does. If it does,
however, this eventually backs up into vote relaying during the idle
loop, which then stalls the idle loop so we stop sending out uptime
proofs (since that is also in the idle loop).
A simple fix here is to add the `true` argument, but on reconsideration
this extra argument to take or not take a lock is messy and error prone,
so this commit instead removes the second argument entirely and instead
documents which call must and must not hold a lock, getting rid of the
three methods (get_blink, has_blink, and add_existing_blink) that had
the `have_lock` argument. This ends up having only a small impact on
calling code - the vast majority of callers already hold a lock, and the
few that don't are easily adjusted.
2020-01-18 19:01:45 +01:00
// that have become immutable. Blink lock must not be already held.
2019-11-18 23:18:41 +01:00
std : : pair < std : : vector < crypto : : hash > , std : : vector < uint64_t > > get_blink_hashes_and_mined_heights ( ) const ;
2014-03-03 23:07:58 +01:00
} ;
}