- Remove implicit `operator bool` from ec_point/public_key/etc. which
was causing all sorts of implicit conversion mess and bugs.
- Change ec_point/public_key/etc. to use a `std::array<unsigned char,
32>` (via a base type) rather than a C-array of char that has to be
reinterpret_cast<>'ed all over the place.
- Add methods to ec_point/public_key/etc. that make it work more like a
container of bytes (`.data()`, `.size()`, `operator[]`, `begin()`,
`end()`).
- Make a generic `crypto::null<T>` that is a constexpr all-0 `T`, rather
than the mishmash `crypto::null_hash`, crypto::null_pkey,
crypto:#️⃣:null(), and so on.
- Replace three metric tons of `crypto::hash blahblah =
crypto::null_hash;` with the much simpler `crypto::hash blahblah{};`,
because there's no need to make a copy of a null hash in all these
cases. (Likewise for a few other null_whatevers).
- Remove a whole bunch of `if (blahblah == crypto::null_hash)` and `if
(blahblah != crypto::null_hash)` with the more concise `if
(!blahblah)` and `if (blahblah)` (which are fine via the newly
*explicit* bool conversion operators).
- `crypto::signature` becomes a 64-byte container (as above) but with
`c()` and `r()` to get the c() and r() data pointers. (Previously
`.c` and `.r` were `ec_scalar`s).
- Delete with great prejudice CRYPTO_MAKE_COMPARABLE and
CRYPTO_MAKE_HASHABLE and all the other utter trash in
`crypto/generic-ops.h`.
- De-inline functions in very common crypto/*.h files so that they don't
have to get compiled 300 times.
- Remove the disgusting include-a-C-header-inside-a-C++-namespace
garbage from some crypto headers trying to be both a C and *different*
C++ header at once.
- Remove the toxic, disgusting, shameful `operator&` on ec_scalar, etc.
that replace `&x` with `reinterpret_cast x into an unsigned char*`.
This was pure toxic waste.
- changed some `<<` outputs to fmt
- Random other small changes encountered while fixing everything that
cascaded out of the above changes.
oxen::log::info(...), etc. are a bit too verbose; this simplifies them
to just `log::info(...)`, etc. by aliasing the `oxen::log` namespace
into most of the common namespaces we use in core.
This result is usage that is shorter but also reads better:
oxen::log::info(logcat, "blah: {}", 42);
log::info(logcat, "blah: {}", 42);
This replaces the current epee logging system with our oxen::log
library. It replaces the easylogging library with spdlog, removes the
macros and replaces with functions and standardises how we call the
logs.
- Replace all cryptonote_config macros with constexpr variables. Some
become integer types, some become chrono types.
- generally this involved removing a "CRYPTONOTE_" prefix since the
values are now in the `cryptonote` namespace
- some constants are grouped into sub-namespaces (e.g.
cryptonote::p2p)
- deprecated constants (i.e. for old HFs) are in the `cryptonote::old`
namespace.
- all the magic hash key domain separating strings are now in
cryptonote::hashkey::WHATEVER.
- Move some economy-related constants to oxen_economy.h instead
- Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more
straightforward `BLOCKS_PER_DAY` value (i.e. old
`BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`.
- Replaced `network_version` unscoped enum with a scoped enum
`cryptonote::hf`, replacing all the raw uint8_t values where it was
currently accepted with the new `hf` type.
- Made `network_type` a scoped enum so that it now has to be qualified
(network_type::TESTNET) and can't be arbitrarily/unintentionally
converted to/from an int.
- HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER
constexpr hf values.
- Add `revision` to rpc hard_fork_info response
- Don't build trezor code at all (previously we were pointlessly
building an empty dummy lib).
openssl is a miserable dependency to fight with, especially for
iOS/Android, and we use it for very little:
- it gets (mis)used to write base64 data to a file in wallet2 for things
like multisig data. However this mode is *not* enabled by default,
basically completely unknown, completely unused, only exists in the
cli wallet, and is just dumb. (Honestly the justification given in
the PR is that "Windows users might want it", presupposing that there
exists Windows users who are capable of generating a multisig wallet
in a CLI-only application and yet are incapable of dealing with binary
files).
- it's a dependency of unbound (in order to do dnssec, I believe).
Unbound itself is fairly useless for Oxen, so I've removed it too:
- it does OpenAlias lookups, which are a Monero thing that has never
been used outside Monero, doesn't work reliably (because it fails
if the result isn't DNSSEC validated) and is pointless when we
have ONS.
- it does DNS resolution on seed nodes, but we have never set seed
nodes by name and just use seed node IPs instead (which seems a
bit better anyway since the DNS lookup leaks some metadata).
- it *was* being used for sha256, but an earlier commit in this PR
already replaced that with libsodium (entirely coincidentally).
- for static deps, it enables HTTPS support for the wallet. However
only the CLI wallet actually supports this (the GUI and mobile wallets
don't), and since oxend hasn't support this for a while I have strong
doubts it is being used anywhere. (Plus wallet3 will do everything
encrypted using zmq/libsodium, so doesn't need this to be secure).
Note here that it is *only* removed by this commit for static builds:
if doing a system build that links to libcurl supporting HTTPS then
HTTPS support will still work.
Libexpat is also gone because it was only there for libunbound.
- Aside from converting code, this commit also:
- Cleans out a bunch of epee garbage that this code touches.
- Removes some of the wipeable_string usage from wallet2 added by
upstream Monero, because that usage was worse than useless: every
instance removed uses a wipeable_string but then *copies the value
into a std::string*, which makes the entire thing useless.
It is, however, *worse* than useless because it is deceptive: a casual
observer would think that the values are being wiped, when in
actuality it only added an extra intermediate buffer that gets wiped
while the final destination doesn't. This deception is worse than
nothing, and so I've removed it.
- Removes epee md5 code. It is entirely unused, and was depending on
the removed hex code.
- Removes a bunch of useless functions from
epee/storages/parserse_base_utils.h: in particular the
exception-to-bool wrappers were only being used in the test suite
(which was dumb: the test suite is perfectly capable of a "does not
throw" assertion).
- De-templatizes the parsing code in parserse_base_utils.h that was only
ever called with a `const char*` "It" (except for one place in the
test suite was which easily fixed), and moved the implementation into
a .cpp file.
All the encoding parts move to oxen-encoding recently; this updates to
the latest version of oxen-mq, adds oxen-encoding, and converts
everything to use oxenc headers rather than the oxenmq compatibility
shims.
The public rpc port code from Monero is not used on Oxen (we have no
ability to enable it because we didn't want it), but still carried
around and printed pointlessly. This removes it entirely.
Along the way I ran into some really nasty p2p code using templates for
absolutely no reason at all, so I fixed that crap, and moved some p2p
serialization out of headers into a .cpp file. (This only scratches the
surface, but as we're going to replace the p2p code entirely eventually
I don't want to waste time trying to polish a turd).
Converts all use of boost::filesystem to std::filesystem.
For macos and potentially other exotic systems where std::filesystem
isn't available, we use ghc::filesystem instead (which is a drop-in
replacement for std::filesystem, unlike boost::filesystem).
This also greatly changes how we handle filenames internally by holding
them in filesystem::path objects as soon as possible (using
fs::u8path()), rather than strings, which avoids a ton of issues around
unicode filenames. As a result this lets us drop the boost::locale
dependency on Windows along with a bunch of messy Windows ifdef code,
and avoids the need for doing gross boost locale codecvt calls.
When targetting macos <10.14 macos won't allow use of anything from
C++17 that throws, such as:
- std::get on a variant
- std::visit
- std::optional::value()
- std::any_cast
This avoids all of these.
For std::get, we either replace with std::get_if (where appropriate), or
else use a `var::get` implementation of std::get added to lokimq (also
updated here). (This `var` namespace is just an `std` alias everywhere
*except* old target macos).
For std::visit, likewise lokimq adds an var::visit implementation for
old macos that we use.
std::optional::value() uses weren't useful anyway as everywhere it calls
them we've already checked that the option has a value, in which case we
can use `*opt` (which doesn't check for contents and throw).
std::any just has to be avoided as far as I can tell, but the one place
we used it is only ever a block, so I just replaced it with a `const
block*`.
Revamps how .loki LNS registrations work:
- Enable lokinet registrations beginning at HF16.
- rework renewal so that you can renew at any time and it simply adds to the end of the current
expiry. Previously there was only a window in which you could renew.
- Renewals are a new type of LNS transaction, distinct from buys and updates. (Internally it is an
update with no fields, which cannot be produced in the existing code).
- Add optional "type=" parameter to lns commands. Commands default to trying to auto-detect (i.e.
if the name ends with .loki it is lokinet), but the type allows you to be explicit *and* allows
select non-default registration lengths for lokinet buys/renewals.
- change .loki naming requirements: we now require <= 32 chars if it doesn't contain a -, and 63 if
it does. We also reserve names starting "??--" for any ?? other than xn-- (punycode), as this is
a DNS restriction. "loki.loki" and "snode.loki" are also now reserved (just in case someone
sticks .loki as a DNS search domain).
- Tweak LNS registration times to consider "a year" to be 368 days worth of blocks (to allow for
leap years and some minor block time drift).
- Overhaul how LNS registrations are displayed in the cli wallet. For example:
[wallet L6QPcD]: lns_print_name_to_owners jasonv.loki jason.loki jasonz.loki
Error: jasonv.loki not found
Name: jason.loki
Type: lokinet
Value: azfoj73snr9f3neh5c6sf7rtbaeabyxhr1m4un5aydsmsrxo964o.loki
Owner: L6QPcDVp6Fu7HwtXrXjtfvWvgBPvvMQ9FiyquMWn2BBEDsk2vydwu1A3BrK2uQcCo94G7HA5xiKvpZ4CMQva6pxW2GXkCG9
Last updated height: 46
Expiration height: 75
Encrypted value: 870e42cd172a(snip)
Error: jasonz.loki not found
- Add an RPC end-point to do simple LNS resolution; you can get the same info out of
names-to-owners, but the new lns_resolve end-point is considerably simpler for doing simple
lookups (e.g. for lokinet), and allows for a simpler SQL query + processing.
Code changes:
- Rename mapping_type::lokinet_1year to mapping_type::lokinet (see next point).
- Don't store lokinet_2y, etc. in the database, but instead always store as type=2/::lokinet. The
LNS extra data can still specify different lengths, but this now just affects the
expiration_height value that we set.
- Reworked some binding code to use std::variant's and add a bind_container() to simplify passing in
a variable list of bind parameters of different types.
- Accept both base64 and hex inputs for binary LNS parameters in the RPC interface.
- This commit adds some (incomplete) expiry adjustment code, but ignore it as it all gets replaced
with the following commit to overhaul record updating.
- Updated a bunch of test suite code, mainly related to lokinet.
- Some random C++17 niceties (string_view, variant, structured binding returns) in the related code.
- Tweaked the test suite to generate a bit fewer blocks in some cases where we just need to
confirm/unlock a transfers rather than a coinbase tx.
- Replaces the wallet RPC classes with ones like the core RPC server
(with serialization code moved into a new .cpp file).
- Restricted commands are now carried through the RPC serialization
types (by inheriting from RESTRICTED instead of RPC_COMMAND) and
restrictions are handled in one place rather than being handled in
each of the 49 restricted endpoints. This differs a little from how
the core http server works (which has a PUBLIC base class) because
for the wallet rpc server unrestricted really doesn't mean "public",
it means something closer to view-only.
- GET_TRANSFERS_CSV is now restricted (it looks like an oversight that
it wasn't before since GET_TRANSFERS is restricted)
- GET_ADDRESS_BOOK_ENTRY is now restricted. Since restricted mode is
meant to provide something like view-only access, it doesn't make
much sense that address book entries were available.
- Use uWebSockets to provide the wallet RPC server HTTP functionality.
This version is quite a bit simpler than the core RPC version since it
doesn't support multithreaded (parallel) requests, and so we don't
have to worry about queuing jobs.
- Converted all the numeric wallet rpc error codes defines to constexprs
- Changed how endpoints get called; previous this was called:
bool on_some_endpoint(const wallet_rpc::COMMAND_RPC_SOME_ENDPOINT::request& req, wallet_rpc::COMMAND_RPC_SOME_ENDPOINT::response& res, epee::json_rpc::error& er, const connection_context *ctx = NULL)
This PR changes it similarly to how core_rpc_server's endpoints work:
wallet_rpc::SOME_ENDPOINT invoke(wallet_rpc::COMMAND_RPC_SOME_ENDPOINT::request&& req);
That is:
- the response is now returned
- the request is provided by mutable rvalue reference
- the error object is gone (just throw instead)
- the connection_context is gone (it was not used at all by any wallet
rpc endpoint).
- Consolidated most of the (identical) exception handling to the RPC
method invocation point rather than needing to repeat it in each
individual endpoint. This means each endpoint's `invoke` method can
now just throw (or not catch) exceptions. Some try/catches are still
there because they are converting one type of exception into another,
but the generic ones that return a generic error are gone.
- Removed all remaining epee http code.
- DRYed out some wallet rpc code.
In short: epee's http client is garbage, standard violating, and
unreliable.
This completely removes the epee http client support and replaces it
with cpr, a curl-based C++ wrapper. rpc/http_client.h wraps cpr for RPC
requests specifically, but it is also usable directly.
This replacement has a number of advantages:
- requests are considerably more reliable. The epee http client code
assumes that a connection will be kept alive forever, and returns a
failure if a connection is ever closed. This results in some very
annoying things: for example, preparing a transaction and then waiting
a long tim before confirming it will usually result in an error
communication with the daemon. This is just terribly behaviour: the
right thing to do on a connection failure is to resubmit the request.
- epee's http client is broken in lots of other ways: for example, it
tries throwing SSL at the port to see if it is HTTPS, but this is
protocol violating and just breaks (with a several second timeout) on
anything that *isn't* epee http server (for example, when lokid is
behind a proxying server).
- even when it isn't doing the above, the client breaks in other ways:
for example, there is a comment (replaced in this PR) in the Trezor PR
code that forces a connection close after every request because epee's
http client doesn't do proper keep-alive request handling.
- it seems noticeably faster to me in practical use in this PR; both
simple requests (for example, when running `lokid status`) and
wallet<->daemon connections are faster, probably because of crappy
code in epee. (I think this is also related to the throw-ssl-at-it
junk above: the epee client always generates an ssl certificate during
static initialization because it might need one at some point).
- significantly reduces the amount of code we have to maintain.
- removes all the epee ssl option code: curl can handle all of that just
fine.
- removes the epee socks proxy code; curl can handle that just fine.
(And can do more: it also supports using HTTP/HTTPS proxies).
- When a cli wallet connection fails we know show why it failed (which
now is an error message from curl), which could have all sorts of
reasons like hostname resolution failure, bad ssl certificate, etc.
Previously you just got a useless generic error that tells you
nothing.
Other related changes in this PR:
- Drops the check-for-update and download-update code. To the best of
my knowledge these have never been supported in loki-core and so it
didn't seem worth the trouble to convert them to use cpr for the
requests.
- Cleaned up node_rpc_proxy return values: there was an inconsistent mix
of ways to return errors and how the returned strings were handled.
Instead this cleans it up to return a pair<bool, val>, which (with
C++17) can be transparently captured as:
auto [success, val] = node.whatever(req);
This drops the failure message string, but it was almost always set to
something fairly useless (if we want to resurrect it we could easily
change the first element to be a custom type with a bool operator for
success, and a `.error` attribute containing some error string, but
for the most part the current code wasn't doing much useful with the
failure string).
- changed local detection (for automatic trusted daemon determination)
to just look for localhost, and to not try to resolve anything.
Trusting non-public IPs does not work well (e.g. with lokinet where
all .loki addresses resolve to a local IP).
- ssl fingerprint option is removed; this isn't supported by curl
(because it is essentially just duplicating what a custom cainfo
bundle does)
- --daemon-ssl-allow-chained is removed; it wasn't a useful option (if
you don't want chaining, don't specify a cainfo chain).
- --daemon-address is now a URL instead of just host:port. (If you omit
the protocol, http:// is prepended).
- --daemon-host and --daemon-port are now deprecated and produce a
warning (in simplewallet) if used; the replacement is to use
--daemon-address.
- --daemon-ssl is deprecated; specify --daemon-address=https://whatever
instead.
- the above three are now hidden from --help
- reordered the wallet connection options to make more logical sense.
This moves the wallet structs that are needed elsewhere (for instance,
in the device drivers) from `tools::wallet2::whatever` to
`wallet::whatever`, allowing them to be included individually via
wallet/whatever.h without needing to include the entire wallet2.h.
What was particularly problematic here is that you couldn't even forward
declare them because they were nested inside the wallet2 class, but
instead had to include the full wallet2.h.
Because:
boost::join(v | boost::adaptors::transformed([](uint64_t out){return std::to_string(out);}), " ")
is ugly as sin, while:
tools::join(" ", v)
is nice and simple.
Also removes a few unnecessary boost iterator adaptor includes and uses.
A huge amount of this is repetitive:
- `boost::get<T>(variant)` becomes `std::get<T>(variant)`
- `boost::get<T>(variant_ptr)` becomes `std::get_if<T>(variant_ptr)`
- `variant.type() == typeid(T)` becomes `std::holds_alternative<T>(variant)`
There are also some simplifications to visitors using simpler stl
visitors, or (simpler still) generic lambdas as visitors.
Also adds boost serialization serializers for std::variant and
std::optional.
Removes all "using namespace epee;" and "using namespace std;" from the
code and fixes up the various crappy places where unnamespaced types
were being used.
Also removes the ENDL macro (which was defined to be `std::endl`)
because it is retarded, and because even using std::endl instead of a
plain "\n" is usually a mistake (`<< std::endl` is equivalent to `<<
"\n" << std::flush`, and that explicit flush is rarely desirable).
The archaic (i.e. decade old) cmake usage here really got in the way of
trying to properly use newer libraries (like lokimq), so this undertakes
overhauling it considerably to make it much more sane (and significantly
reduce the size).
I left more of the architecture-specific bits in the top-level
CMakeLists.txt intact; most of the efforts here are about properly
loading dependencies, specifying dependencies and avoiding a whole pile
of cmake antipatterns.
This bumps the required cmake version to 3.5, which is what xenial comes
with.
- extensive use of interface libraries to include libraries,
definitions, and include paths
- use Boost::whatever instead of ${Boost_WHATEVER_LIBRARY}. The
interface targets are (again) much better as they also give you any
needed include or linking flags without needing to worry about them.
- don't list header files when building things. This has *never* been
correct cmake usage (cmake has always known how to wallet_rpc_headers
the headers that .cpp files include to know about build changes).
- remove the loki_add_library monstrosity; it breaks target names and
makes compiling less efficient because the author couldn't figure out
how to link things together.
- make loki_add_executable take the output filename, and set the output
path to bin/ and install to bin because *every single usage* of
loki_add_executable was immediately followed by setting the output
filename and setting the output path to bin/ and installing to bin.
- move a bunch of crap that is only used in one particular
src/whatever/CMakeLists.txt into that particular CMakeLists.txt instead
of the top level CMakeLists.txt (or src/CMakeLists.txt).
- Remove a bunch of redundant dependencies; most of them look like they
were just copy-and-pasted in, and many more aren't needed (since they
are implied by the PUBLIC linking of other dependencies).
- Removed `die` since it just does a FATAL_ERROR, but adds color (which
is useless since CMake already makes FATAL_ERRORs perfectly visible).
- Change the way LOKI_DAEMON_AND_WALLET_ONLY works to just change the
make targets to daemon and simplewallet rather than changing the build
process (this should make it faster, too, since there are various other
things that will be excluded).
The blockchain prunes seven eighths of prunable tx data.
This saves about two thirds of the blockchain size, while
keeping the node useful as a sync source for an eighth
of the blockchain.
No other data is currently pruned.
There are three ways to prune a blockchain:
- run monerod with --prune-blockchain
- run "prune_blockchain" in the monerod console
- run the monero-blockchain-prune utility
The first two will prune in place. Due to how LMDB works, this
will not reduce the blockchain size on disk. Instead, it will
mark parts of the file as free, so that future data will use
that free space, causing the file to not grow until free space
grows scarce.
The third way will create a second database, a pruned copy of
the original one. Since this is a new file, this one will be
smaller than the original one.
Once the database is pruned, it will stay pruned as it syncs.
That is, there is no need to use --prune-blockchain again, etc.
Jason Rhinelander
Sean Darcy
Thomas Winget
Sean Darcy
Doyle
moneromooo-monero
Riccardo Spagni
binaryFate