This moves all the conditional HAVE_READLINE into once place rather than
scattering it everywhere we want to suspend readline. (Since the class
does nothing the compiler can trivially optimize it away when we don't
have readline).
Link readline directly into epee; having a separate epee_readline
library is not saving anything since we have it widely linked anyway.
Conditionally linking it to epee simplifies a bit of CMake code.
Also simplify how epee detects cmake to just look for a `readline`
target, which we now only set up if we find readline in the top-level
CMakeLists.txt
High-level details:
This redesigns the RPC layer to make it much easier to work with,
decouples it from an embedded HTTP server, and gets the vast majority of
the RPC serialization and dispatch code out of a very commonly included
header.
There is unfortunately rather a lot of interconnected code here that
cannot be easily separated out into separate commits. The full details
of what happens here are as follows:
Major details:
- All of the RPC code is now in a `cryptonote::rpc` namespace; this
renames quite a bit to be less verbose: e.g. CORE_RPC_STATUS_OK
becomes `rpc::STATUS_OK`, and `cryptonote::COMMAND_RPC_SOME_LONG_NAME`
becomes `rpc::SOME_LONG_NAME` (or just SOME_LONG_NAME for code already
working in the `rpc` namespace).
- `core_rpc_server` is now completely decoupled from providing any
request protocol: it is now *just* the core RPC call handler.
- The HTTP RPC interface now lives in a new rpc/http_server.h; this code
handles listening for HTTP requests and dispatching them to
core_rpc_server, then sending the results back to the caller.
- There is similarly a rpc/lmq_server.h for LMQ RPC code; more details
on this (and other LMQ specifics) below.
- RPC implementing code now returns the response object and throws when
things go wrong which simplifies much of the rpc error handling. They
can throw anything; generic exceptions get logged and a generic
"internal error" message gets returned to the caller, but there is
also an `rpc_error` class to return an error code and message used by
some json-rpc commands.
- RPC implementing functions now overload `core_rpc_server::invoke`
following the pattern:
RPC_BLAH_BLAH::response core_rpc_server::invoke(RPC_BLAH_BLAH::request&& req, rpc_context context);
This overloading makes the code vastly simpler: all instantiations are
now done with a small amount of generic instantiation code in a single
.cpp rather than needing to go to hell and back with a nest of epee
macros in a core header.
- each RPC endpoint is now defined by the RPC types themselves,
including its accessible names and permissions, in
core_rpc_server_commands_defs.h:
- every RPC structure now has a static `names()` function that returns
the names by which the end point is accessible. (The first one is
the primary, the others are for deprecated aliases).
- RPC command wrappers define their permissions and type by inheriting
from special tag classes:
- rpc::RPC_COMMAND is a basic, admin-only, JSON command, available
via JSON RPC. *All* JSON commands are now available via JSON RPC,
instead of the previous mix of some being at /foo and others at
/json_rpc. (Ones that were previously at /foo are still there for
backwards compatibility; see `rpc::LEGACY` below).
- rpc::PUBLIC specifies that the command should be available via a
restricted RPC connection.
- rpc::BINARY specifies that the command is not JSON, but rather is
accessible as /name and takes and returns values in the magic epee
binary "portable storage" (lol) data format.
- rpc::LEGACY specifies that the command should be available via the
non-json-rpc interface at `/name` for backwards compatibility (in
addition to the JSON-RPC interface).
- some epee serialization got unwrapped and de-templatized so that it
can be moved into a .cpp file with just declarations in the .h. (This
makes a *huge* difference for core_rpc_server_commands_defs.h and for
every compilation unit that includes it which previously had to
compile all the serialization code and then throw all by one copy away
at link time). This required some new macros so as to not break a ton
of places that will use the old way putting everything in the headers;
The RPC code uses this as does a few other places; there are comments
in contrib/epee/include/serialization/keyvalue_serialization.h as to
how to use it.
- Detemplatized a bunch of epee/storages code. Most of it should have
have been using templates at all (because it can only ever be called
with one type!), and now it isn't. This broke some things that didn't
properly compile because of missing headers or (in one case) a messed
up circular dependency.
- Significantly simplified a bunch of over-templatized serialization
code.
- All RPC serialization definitions is now out of
core_rpc_server_commands_defs.h and into a single .cpp file
(core_rpc_server_commands_defs.cpp).
- core RPC no longer uses the disgusting
BEGIN_URI_MAP2/MAP_URI_BLAH_BLAH macros. This was a terrible design
that forced slamming tons of code into a common header that didn't
need to be there.
- epee::struct_init is gone. It was a horrible hack that instiated
multiple templates just so the coder could be so lazy and write
`some_type var;` instead of properly value initializing with
`some_type var{};`.
- Removed a bunch of useless crap from epee. In particular, forcing
extra template instantiations all over the place in order to nest
return objects inside JSON RPC values is no longer needed, as are a
bunch of stuff related to the above de-macroization of the code.
- get_all_service_nodes, get_service_nodes, and get_n_service_nodes are
now combined into a single `get_service_nodes` (with deprecated
aliases for the others), which eliminates a fair amount of
duplication. The biggest obstacle here was getting the requested
fields reference passed through: this is now done by a new ability to
stash a context in the serialization object that can be retrieved by a
sub-serialized type.
LMQ-specifics:
- The LokiMQ instance moves into `cryptonote::core` rather than being
inside cryptonote_protocol. Currently the instance is used both for
qnet and rpc calls (and so needs to be in a common place), but I also
intend future PRs to use the batching code for job processing
(replacing the current threaded job queue).
- rpc/lmq_server.h handles the actual LMQ-request-to-core-RPC glue.
Unlike http_server it isn't technically running the whole LMQ stack
from here, but the parallel name with http_server seemed appropriate.
- All RPC endpoints are supported by LMQ under the same names as defined
generically, but prefixed with `rpc.` for public commands and `admin.`
for restricted ones.
- service node keys are now always available, even when not running in
`--service-node` mode: this is because we want the x25519 key for
being able to offer CURVE encryption for lmq RPC end-points, and
because it doesn't hurt to have them available all the time. In the
RPC layer this is now called "get_service_keys" (with
"get_service_node_key" as an alias) since they aren't strictly only
for service nodes. This also means code needs to check
m_service_node, and not m_service_node_keys, to tell if it is running
as a service node. (This is also easier to notice because
m_service_node_keys got renamed to `m_service_keys`).
- Added block and mempool monitoring LMQ RPC endpoints: `sub.block` and
`sub.mempool` subscribes the connection for new block and new mempool
TX notifications. The latter can notify on just blink txes, or all
new mempool txes (but only new ones -- txes dumped from a block don't
trigger it). The client gets pushed a [`notify.block`, `height`,
`hash`] or [`notify.tx`, `txhash`, `blob`] message when something
arrives.
Minor details:
- rpc::version_t is now a {major,minor} pair. Forcing everyone to pack
and unpack a uint32_t was gross.
- Changed some macros to constexprs (e.g. CORE_RPC_ERROR_CODE_...).
(This immediately revealed a couple of bugs in the RPC code that was
assigning CORE_RPC_ERROR_CODE_... to a string, and it worked because
the macro allows implicit conversion to a char).
- De-templatizing useless templates in epee (i.e. a bunch of templated
types that were never invoked with different types) revealed a painful
circular dependency between epee and non-epee code for tor_address and
i2p_address. This crap is now handled in a suitably named
`net/epee_network_address_hack.cpp` hack because it really isn't
trivial to extricate this mess.
- Removed `epee/include/serialization/serialize_base.h`. Amazingly the
code somehow still all works perfectly with this previously vital
header removed.
- Removed bitrotted, unused epee "crypted_storage" and
"gzipped_inmemstorage" code.
- Replaced a bunch of epee::misc_utils::auto_scope_leave_caller with
LOKI_DEFERs. The epee version involves quite a bit more instantiation
and is ugly as sin. Also made the `loki::defer` class invokable for
some edge cases that need calling before destruction in particular
conditions.
- Moved the systemd code around; it makes much more sense to do the
systemd started notification as in daemon.cpp as late as possible
rather than in core (when we can still have startup failures, e.g. if
the RPC layer can't start).
- Made the systemd short status string available in the get_info RPC
(and no longer require building with systemd).
- during startup, print (only) the x25519 when not in SN mode, and
continue to print all three when in SN mode.
- DRYed out some RPC implementation code (such as set_limit)
- Made wallet_rpc stop using a raw m_wallet pointer
Removes all "using namespace epee;" and "using namespace std;" from the
code and fixes up the various crappy places where unnamespaced types
were being used.
Also removes the ENDL macro (which was defined to be `std::endl`)
because it is retarded, and because even using std::endl instead of a
plain "\n" is usually a mistake (`<< std::endl` is equivalent to `<<
"\n" << std::flush`, and that explicit flush is rarely desirable).
This function is broken: if it encounters a \ it stays in "escape" more
forever. However the entire function is also moronic because it is only
used in the test suite (and thus doesn't belong out of the test suite),
but is also only used for the test `--filter` arguments which have this
help message:
Regular expression filter for which tests to run
Since a glob is not a regular expression, and because passing legitimate
regular expressions through this glob_to_regex function serious breaks
them, removing this stupid broken code is actually a bug fix.
This is a big chunk of code with little usefulness: It is not the job of
Loki to tell you what operating system you are using. If someone needs
to know what OS they are on they can run `uname` themselves on anything
other than Windows, and click the Start menu in Windows.
(Contrary to the option description, this actually just tells you the
current OS, not the compiled-for OS).
This uses templates to simplify the is_yes (etc.) functions, also
removing a boost dependency and making them more flexible where callers
sometimes want "is_yes or one of these other truthy strings".
760ecf2 console_handler: do not let exception past the dor (moneromooo-monero)
09c8111 threadpool: lock mutex in create (moneromooo-monero)
e377977 tx_pool: catch theoretical error in get_block_reward (moneromooo-monero)
This adds the loki-mq dependency and replaces SNNetwork with it (along
with some syntax updates for how loki-mq changed a bit from SNNetwork).
This also replaces common/hex.h and common/string_view.h with loki-mq's
faster (hex) and more complete and tested (string_view) implementations.
The archaic (i.e. decade old) cmake usage here really got in the way of
trying to properly use newer libraries (like lokimq), so this undertakes
overhauling it considerably to make it much more sane (and significantly
reduce the size).
I left more of the architecture-specific bits in the top-level
CMakeLists.txt intact; most of the efforts here are about properly
loading dependencies, specifying dependencies and avoiding a whole pile
of cmake antipatterns.
This bumps the required cmake version to 3.5, which is what xenial comes
with.
- extensive use of interface libraries to include libraries,
definitions, and include paths
- use Boost::whatever instead of ${Boost_WHATEVER_LIBRARY}. The
interface targets are (again) much better as they also give you any
needed include or linking flags without needing to worry about them.
- don't list header files when building things. This has *never* been
correct cmake usage (cmake has always known how to wallet_rpc_headers
the headers that .cpp files include to know about build changes).
- remove the loki_add_library monstrosity; it breaks target names and
makes compiling less efficient because the author couldn't figure out
how to link things together.
- make loki_add_executable take the output filename, and set the output
path to bin/ and install to bin because *every single usage* of
loki_add_executable was immediately followed by setting the output
filename and setting the output path to bin/ and installing to bin.
- move a bunch of crap that is only used in one particular
src/whatever/CMakeLists.txt into that particular CMakeLists.txt instead
of the top level CMakeLists.txt (or src/CMakeLists.txt).
- Remove a bunch of redundant dependencies; most of them look like they
were just copy-and-pasted in, and many more aren't needed (since they
are implied by the PUBLIC linking of other dependencies).
- Removed `die` since it just does a FATAL_ERROR, but adds color (which
is useless since CMake already makes FATAL_ERRORs perfectly visible).
- Change the way LOKI_DAEMON_AND_WALLET_ONLY works to just change the
make targets to daemon and simplewallet rather than changing the build
process (this should make it faster, too, since there are various other
things that will be excluded).
88b82bef simplewallet: point to "set help" in the lock screen message (moneromooo-monero)
f19c9f23 util: allow newlines in string to be split (moneromooo-monero)
- constexpr functions in common/loki.h for inlining
- move hex functions out from common/loki.h to common/hex.h
- use and apply prev_txid on LNS TX's to all LNS types (for updating in the future)
- add lns burn type, for custom burn amounts
- accept and validate lokinet addresses via base32z
- return lokinet addresses in RPC LNS calls via base32z
- updated Messenger references to Session
- update documentation to note that only Session LNS entries are allowed currently
- remove raw c-string interface from LNS db
- update multi-SQL queries into single SQL queries
- remove tx estimation backlog in anticipation for 2 priorities only, blink + unimportant
In some contrived case, it might theoretically be the case that
destroy is called from another thread, which would modify the
threads array from two threads.
Coverity 208372
`--regtest` didn't work in some edge cases, this fixes various things:
- the genesis block wasn't accepted because it needed to be v7, not
vMax
- reduce initial uptime proof delay to 5s in regtest mode
- add --regtest flag to the wallet so that it can talk to a daemon in
--regtest mode.
This also adds two new mining options, available via rpc:
- slow_mining - this avoids the RandomX initialization. It is much
slower, but for regtest with fixed difficulty of 1 that is perfectly
fine.
- `num_blocks` - instruct the miner to mine for the given number of
blocks, then stop. (This can overmine if mining with multiple
threads at a low difficulty, but that's fine).
The MacOSX 10.11 SDK we use is broken AF: it lies about supporting
C++14, but really only upgraded the headers but not the library itself,
so using std::shared_timed_mutex just results in a linking failure.
Upgrading the SDK is a huge pain (I tried, failed, and gave up), so for
now temporarily switch to boost::shared_mutex until we sort out the
macOS build disaster.
If a block adding fails (triggering the "Block added hook signalled
failure" error message) the service node list doesn't get reset, which
immediately leads to a bad service node winner (because the winner was
already incremented and not popped off).
This updates it to call the blockchain detached hooks to do the cleanup.
It also changes around loki::defer a little bit to rename the internal
class to `deferred` and make it cancellable (by calling `.cancel()`).
`loki::defer` is repurposed as a free function to get a named `deferred`
object given a lambda, which is needed to be able to call `cancel()` on
it. (The LOKI_DEFER macro still works as is).
macOS's std::lock() is broken in that it internally calls non-namespaced
function `try_lock` leading to an ADL conflict with boost::try_lock when
any of the arguments is a `boost::whatever`. `boost::lock` will do the
job for now.
auto locks = tools::unique_locks(mutex1, mutex2, ...);
gives you a tuple of unique_locks and obtains the locks atomically.
auto lock = tools::unique_lock(lock1);
is essentially the same as:
std::unique_lock<decltype(lock1)> lock{lock1};
but less ugly (and extends nicely to the plural version).
This reverts the parts of upstream commit
6cf56682bc related to doing rdtsc timing
on x86-64 because it is incredibly broken:
- rdtsc is not a reliable duration timer on x86-64 on all: among other
problems, CPU frequencies are not even close to constant, nor are
returned values reliable across threads.
- As if the unreliability wasn't bad enough, this code also spends a
**full second** in a busy loop to try to measure the number of rdtsc
ticks per wallclock nanosecond, and does this during static
initialization. Thus every invocation of every binary wastes a full
second a CPU to calibrate some timer ratio value that isn't even
remotely reliable in the first place.
The second one is particularly annoying: trying to run `--help` or
invoke an rpc command adds a full second of 100% CPU usage delay.
Before this commit:
$ time ./bin/lokid --help >/dev/null
real 0m1.018s
user 0m1.014s
sys 0m0.004s
and after:
$ time ./bin/lokid --help >/dev/null
real 0m0.013s
user 0m0.008s
sys 0m0.004s
It is sad that a prominent upstream monero contributor added `__thread`
to C++ code in 2017 when `thread_local` has been standardized since
C++11.
`__thread` isn't even supported by some compilers (like MSVC).
We unnecessarily vendor this much of the time because the existing
search code is primitive; use pkg-config instead which works much
better, and lets us properly depend on particular versions.
This lets us reenable the system miniupnpc for >=2.1, saving an
unnecessary compilation most of the time. From the git history it
appears that it is built from source always unconditionally in a fit of
rage.
This adds a thread-local, pre-seeded rng at `tools::rng` (to avoid the
multiple places we are creating + seeding such an RNG currently).
This also moves the portable uniform value and generic shuffle code
there as well as neither function is specific to service nodes and this
seems a logical place for them.
We don't impose any alignment on hashable types, but this means the
hashing function is doing invalid misaligned access when converting to a
size_t. This aligns all of the primitive data types (crypto::hash,
public keys, etc.) to the same alignment as size_t.
That cascades into a few places in epee which only allow byte spanning
types that have byte alignment when what it really requires is just that
the type has no padding. In C++17 this is exactly the purpose of
std::has_unique_object_representations, but that isn't available (or
even implementable) in C++14 so add specializations for the type that
need it to tell epee that we know those types are properly packed and
that it can safely use them as bytes.
Related to this, monero/epee also misuses `is_standard_layout` when the
purpose is actually `is_trivially_copyable`, so fixed that too. (You
need the latter but don't need the former for a type to be safely
memcpy'able; the only purpose of `is_standard_layout` is when you need
to be sure your structs are compatible with C structs which is
irrelevant here).
Unify the field we use to store the count as `_count` (using the leading
underscore to indicate a private value rather than an intended enum
value) and add/use a new `enum_count` template variable to extract the
_count enum value and cast it to the enum's underlying_type.
Avoids a DB error (leading to an assert) where a thread uses
a read txn previously created with an environment that was
since closed and reopened. While this usually works since
BlockchainLMDB renews txns if it detects the environment has
changed, this will not work if objects end up being allocated
at the same address as the previous instance, leading to stale
data usage.
Thanks hyc for the LMDB debugging.
Daemons intended for public use can be set up to require payment
in the form of hashes in exchange for RPC service. This enables
public daemons to receive payment for their work over a large
number of calls. This system behaves similarly to a pool, so
payment takes the form of valid blocks every so often, yielding
a large one off payment, rather than constant micropayments.
This system can also be used by third parties as a "paywall"
layer, where users of a service can pay for use by mining Monero
to the service provider's address. An example of this for web
site access is Primo, a Monero mining based website "paywall":
https://github.com/selene-kovri/primo
This has some advantages:
- incentive to run a node providing RPC services, thereby promoting the availability of third party nodes for those who can't run their own
- incentive to run your own node instead of using a third party's, thereby promoting decentralization
- decentralized: payment is done between a client and server, with no third party needed
- private: since the system is "pay as you go", you don't need to identify yourself to claim a long lived balance
- no payment occurs on the blockchain, so there is no extra transactional load
- one may mine with a beefy server, and use those credits from a phone, by reusing the client ID (at the cost of some privacy)
- no barrier to entry: anyone may run a RPC node, and your expected revenue depends on how much work you do
- Sybil resistant: if you run 1000 idle RPC nodes, you don't magically get more revenue
- no large credit balance maintained on servers, so they have no incentive to exit scam
- you can use any/many node(s), since there's little cost in switching servers
- market based prices: competition between servers to lower costs
- incentive for a distributed third party node system: if some public nodes are overused/slow, traffic can move to others
- increases network security
- helps counteract mining pools' share of the network hash rate
- zero incentive for a payer to "double spend" since a reorg does not give any money back to the miner
And some disadvantages:
- low power clients will have difficulty mining (but one can optionally mine in advance and/or with a faster machine)
- payment is "random", so a server might go a long time without a block before getting one
- a public node's overall expected payment may be small
Public nodes are expected to compete to find a suitable level for
cost of service.
The daemon can be set up this way to require payment for RPC services:
monerod --rpc-payment-address 4xxxxxx \
--rpc-payment-credits 250 --rpc-payment-difficulty 1000
These values are an example only.
The --rpc-payment-difficulty switch selects how hard each "share" should
be, similar to a mining pool. The higher the difficulty, the fewer
shares a client will find.
The --rpc-payment-credits switch selects how many credits are awarded
for each share a client finds.
Considering both options, clients will be awarded credits/difficulty
credits for every hash they calculate. For example, in the command line
above, 0.25 credits per hash. A client mining at 100 H/s will therefore
get an average of 25 credits per second.
For reference, in the current implementation, a credit is enough to
sync 20 blocks, so a 100 H/s client that's just starting to use Monero
and uses this daemon will be able to sync 500 blocks per second.
The wallet can be set to automatically mine if connected to a daemon
which requires payment for RPC usage. It will try to keep a balance
of 50000 credits, stopping mining when it's at this level, and starting
again as credits are spent. With the example above, a new client will
mine this much credits in about half an hour, and this target is enough
to sync 500000 blocks (currently about a third of the monero blockchain).
There are three new settings in the wallet:
- credits-target: this is the amount of credits a wallet will try to
reach before stopping mining. The default of 0 means 50000 credits.
- auto-mine-for-rpc-payment-threshold: this controls the minimum
credit rate which the wallet considers worth mining for. If the
daemon credits less than this ratio, the wallet will consider mining
to be not worth it. In the example above, the rate is 0.25
- persistent-rpc-client-id: if set, this allows the wallet to reuse
a client id across runs. This means a public node can tell a wallet
that's connecting is the same as one that connected previously, but
allows a wallet to keep their credit balance from one run to the
other. Since the wallet only mines to keep a small credit balance,
this is not normally worth doing. However, someone may want to mine
on a fast server, and use that credit balance on a low power device
such as a phone. If left unset, a new client ID is generated at
each wallet start, for privacy reasons.
To mine and use a credit balance on two different devices, you can
use the --rpc-client-secret-key switch. A wallet's client secret key
can be found using the new rpc_payments command in the wallet.
Note: anyone knowing your RPC client secret key is able to use your
credit balance.
The wallet has a few new commands too:
- start_mining_for_rpc: start mining to acquire more credits,
regardless of the auto mining settings
- stop_mining_for_rpc: stop mining to acquire more credits
- rpc_payments: display information about current credits with
the currently selected daemon
The node has an extra command:
- rpc_payments: display information about clients and their
balances
The node will forget about any balance for clients which have
been inactive for 6 months. Balances carry over on node restart.
Lets you write `memcpy_le(dest, foo, bar, baz)` and memcpy the contents
of foo, bar, and baz sequentially. Additionally, if any of those three
are integers, they are converted to little-endian order.
POD_CLASS is a retarded macro (it is always just `struct`).
The pack pragmas here do nothing because every type defined inside them
only has char array members which are already guaranteed by C++
alignment rules to have byte alignment.
When processing a quorum for a block, if you are not in the quorum to
validate other nodes, check if you're a node that is going to be tested.
If you are, check based on your current data if you're potentially
a candidate to be decommissioned/deregistered and if so report it to the
console log.
Note that this is only a heuristic and ultimately the decision lies on
what the the other Service Nodes perceive the current state of your node
is (i.e. if they're acting malicious then you will be deregistered
irrespectively).
0605406 daemon: sort alt chains by height (moneromooo-monero)
4228ee0 daemon: add optional arguments to alt_chain_info (moneromooo-monero)
880ebfd daemon: add more chain specific info in alt_chain_info (moneromooo-monero)
* Add deregistration of checkpoints by checking how many votes are missed
Move uptime proofs and add checkpoint counts in the service_node_list
because we typically prune uptime proofs by time, but it seems we want
to switch to a model where we persist proof data until the node expires
otherwise currently we would prune uptime entries and potentially our
checkpoint vote counts which would cause premature deregistration as the
expected vote counts start mismatching with the number of received
votes.
* Revise deregistration
* Fix test breakages
* uint16_t for port, remove debug false, min votes to 2 in integration mode
* Fix integration build
* core: do not commit half constructed batch db txn
* Add defer macro
* Revert dumb extra copy/move change
* Fix pop_blocks not calling hooks, fix BaseTestDB missing prototypes
* Merge ServiceNodeCheckpointing5 branch, syncing and integration fixes
* Update tests to compile with relaxed-registration changes
* Get back to feature parity pre-relaxed registration changes
* Remove debug changes noticed in code review and some small bugs
* Add functions for storing checkpoints to the DB
* Allocate the DB entry on the stack instead of heap
* Add virtual overrides for new checkpoint functions
* Clean up for pull request, simplify some logic
* Revise API to include height in checkpoint header
* Move log to top of function even if early exit
* Begin moving checkpoints to db
* Allow storing of checkpoints to DB
* Cleanup for code reviewer, fix unit tests
* Fix tests, fix casting issues
* Don't use DUPSORT, use height->checkpoint mapping in DB
* Remove if 0 disabling checkpoint vote, we already check HF12
* Fix unit test infinite loop
* Update db schemas to match blk_checkpoint_header
* Code review
