Commit graph

112 commits

Author SHA1 Message Date
Jason Rhinelander b51972f296
Strip project build dir from logs 2022-12-16 13:49:17 -04:00
Sean Darcy bd6b4f1e7b option to exclude making pybind on android and windows by default 2022-12-06 11:25:19 +11:00
Sean Darcy 172093fbbf wallet3 cli
This introduces wallet3 cli to the codebase, build using python and
wraps the c++ wallet3 core using pybind.
2022-12-06 08:10:31 +11:00
Jason Rhinelander 7eb357971b
Make sure we are using fmt/spdlog submodules for static builds
Otherwise we can end up linking against libfmt and/or libspdlog
2022-10-17 13:45:23 -03:00
Jason Rhinelander eb4c9ca52d
Dep updates
- oxenmq requires >= .13 for the fmt header
- update oxen-logging with cmake build fix and text style support
2022-10-17 13:41:12 -03:00
Sean Darcy d7992b5940
Logging Refactor
This replaces the current epee logging system with our oxen::log
library. It replaces the easylogging library with spdlog, removes the
macros and replaces with functions and standardises how we call the
logs.
2022-10-17 13:41:10 -03:00
Thomas Winget b3a3461f92 submodule fix after merge 2022-07-25 22:18:14 -04:00
Thomas Winget 1311a20e9f merge dev branch with RPC/wallet3 changes
Incomplete, many things to fix, some annotated with
a comment MERGEFIX
2022-07-11 20:40:50 -04:00
Jason Rhinelander 6b785958b4
Clean up sys-or-submodule code, support system fmt, rename loki-mq
Synced up with oxen-storage-server.

Also rename loki-mq submodule to oxen-mq.
2022-05-30 15:59:03 -03:00
Jason Rhinelander 636ee3f622
Replace boost::endian conversion with oxenc 1.0.3
1.0.3 got some endian fixes that lets us avoid boost::endian everywhere.
2022-05-24 17:35:59 -03:00
Jason Rhinelander 47ff596533
Remove UPnP support
We have disabled uPnP by default for nearly a year now (#1423) with no
adverse affects; this commit finishes it off entirely, removing it from
the codebase.

Fixes #1419 -- see discussion in that issue for why UPnP really doesn't
add much utility to Oxen.
2022-04-14 14:34:49 -03:00
Jason Rhinelander cf1ee25220
cpr header build fix 2022-04-14 14:34:49 -03:00
Jason Rhinelander 6fcfd0b8ba
Update oxenmq to latest oxen-mq+oxen-encoding
All the encoding parts move to oxen-encoding recently; this updates to
the latest version of oxen-mq, adds oxen-encoding, and converts
everything to use oxenc headers rather than the oxenmq compatibility
shims.
2022-04-14 14:32:01 -03:00
Thomas Winget ecb62e8fab Wallet3 squashed initial commit
tx scanning for basic transactions working
  - TODO: subadresses.  The scanning code is there, but it does not
  currently know/care about any subaddresses.

daemon comms for basic syncing working

(multi-)wallet sync more or less working properly
  - seem to have a dangling shared_ptr somewhere when removing a wallet from
  daemon comms, so not working perfectly yet.

Lots of TODOs and cleanup needed, as well as further features of course.
2021-11-30 16:31:54 -05:00
Thomas Winget 18016ded02 rebase fixups 2021-11-01 17:27:44 -04:00
Jason Rhinelander 7b871e97fc Add fmtlib::fmt backport of C++20 std::format
Because I'm tired of waiting for std::format.
2021-11-01 17:03:47 -04:00
Jason Rhinelander cb6a7d0b60 Fix nlohmann submodule 2021-11-01 14:54:21 -04:00
Jason Rhinelander 1f16daf109 Replace epee C date crap with C++20 date backport 2021-11-01 14:54:18 -04:00
Jason Rhinelander 02f1b5b982 Add nlohmann-json submodule 2021-11-01 14:53:52 -04:00
Jason Rhinelander f66ff340aa Update oxenmq to latest dev (needed for inproc rpc support) 2021-11-01 14:19:27 -04:00
Thomas Winget 73e787a9ec sqlitecpp cmake nonsense 2021-10-06 19:36:46 -04:00
Thomas Winget 435c189a9b sqlitecpp 2021-10-04 20:33:24 -04:00
Jason Rhinelander eefd42f93f Add fmtlib::fmt backport of C++20 std::format
Because I'm tired of waiting for std::format.
2021-09-02 17:25:13 -03:00
Jason Rhinelander e5bc4e31c4 Clean up pkg-config use
- Convert pkg-config library finding to use IMPORTED_TARGET rather than
the old hacky way of using multiple variables.

- Remove libunbound finding from external (it gets set up already in the
root CMakeLists.txt)

- Remove libzmq searching since we no longer directly depend on it
(except through oxenmq).
2021-05-12 11:59:22 -03:00
Jason Rhinelander c789fd5d77 Remove dead DEPENDS code
We dropped the contrib/depends build system quite a while ago (because
it was nasty), but there are still various DEPENDS checks scattered
through cmake that are just dead code now.  This removes them.
2021-05-12 10:46:57 -03:00
Jason Rhinelander 6adb9f8ac2 Don't build libuv on ios 2021-04-21 22:03:19 -03:00
Jason Rhinelander 466a1317d2 Rename lokimq -> oxenmq 2021-01-14 19:35:00 -04:00
Jason Rhinelander daaa8dbefd Lots of build hacks for iOS 2020-11-04 15:06:59 -04:00
Jason Rhinelander e0aca73325 Add more crap into merged library 2020-11-04 12:11:20 -04:00
Jason Rhinelander bb40433612 Android build infrastructure updates
- Various static deps updated needed for properly cross-compiling
libraries for android.
- disable LTO because it doesn't work reliably with the android NDK.
- allow building without miniupnpc, and default to not building it under
android.
- make sure we build translation tools on the native arch when
cross-compiling for android.
- don't build ncurses, libusb, hidapi when doing an android build
2020-11-04 12:11:20 -04:00
Jason Rhinelander 2a88f593db macos: switch to libuv for uSockets loop
The native kqueue support breaks on macos 10.12 (and possibly other
versions): in particular our deferred tasks never fire, which breaks
refreshing.  libuv seems to work more reliably, so use it instead on
macos (as we already do on Windows).
2020-10-20 17:02:45 -03:00
Jason Rhinelander 17a6697a04 More cmake <3.12 compatibility fixes for libcurl 2020-08-10 14:47:09 -03:00
Jason Rhinelander e1d3a288eb cmake <3.12 compat 2020-08-07 21:17:45 -03:00
Jason Rhinelander b7a7dfc080 Add zlib + curl to static deps 2020-08-07 17:14:03 -03:00
Jason Rhinelander f96350075c Update cpr submodule to upstream
Upstream master now has the patches we need merged.
2020-08-07 17:14:03 -03:00
Jason Rhinelander fb0aff57f6 Replace epee http client with curl-based client
In short: epee's http client is garbage, standard violating, and
unreliable.

This completely removes the epee http client support and replaces it
with cpr, a curl-based C++ wrapper.  rpc/http_client.h wraps cpr for RPC
requests specifically, but it is also usable directly.

This replacement has a number of advantages:

- requests are considerably more reliable.  The epee http client code
  assumes that a connection will be kept alive forever, and returns a
  failure if a connection is ever closed.  This results in some very
  annoying things: for example, preparing a transaction and then waiting
  a long tim before confirming it will usually result in an error
  communication with the daemon.  This is just terribly behaviour: the
  right thing to do on a connection failure is to resubmit the request.

- epee's http client is broken in lots of other ways: for example, it
  tries throwing SSL at the port to see if it is HTTPS, but this is
  protocol violating and just breaks (with a several second timeout) on
  anything that *isn't* epee http server (for example, when lokid is
  behind a proxying server).

- even when it isn't doing the above, the client breaks in other ways:
  for example, there is a comment (replaced in this PR) in the Trezor PR
  code that forces a connection close after every request because epee's
  http client doesn't do proper keep-alive request handling.

- it seems noticeably faster to me in practical use in this PR; both
  simple requests (for example, when running `lokid status`) and
  wallet<->daemon connections are faster, probably because of crappy
  code in epee.  (I think this is also related to the throw-ssl-at-it
  junk above: the epee client always generates an ssl certificate during
  static initialization because it might need one at some point).

- significantly reduces the amount of code we have to maintain.

- removes all the epee ssl option code: curl can handle all of that just
  fine.

- removes the epee socks proxy code; curl can handle that just fine.
  (And can do more: it also supports using HTTP/HTTPS proxies).

- When a cli wallet connection fails we know show why it failed (which
  now is an error message from curl), which could have all sorts of
  reasons like hostname resolution failure, bad ssl certificate, etc.
  Previously you just got a useless generic error that tells you
  nothing.

Other related changes in this PR:

- Drops the check-for-update and download-update code.  To the best of
my knowledge these have never been supported in loki-core and so it
didn't seem worth the trouble to convert them to use cpr for the
requests.

- Cleaned up node_rpc_proxy return values: there was an inconsistent mix
  of ways to return errors and how the returned strings were handled.
  Instead this cleans it up to return a pair<bool, val>, which (with
  C++17) can be transparently captured as:

    auto [success, val] = node.whatever(req);

  This drops the failure message string, but it was almost always set to
  something fairly useless (if we want to resurrect it we could easily
  change the first element to be a custom type with a bool operator for
  success, and a `.error` attribute containing some error string, but
  for the most part the current code wasn't doing much useful with the
  failure string).

- changed local detection (for automatic trusted daemon determination)
  to just look for localhost, and to not try to resolve anything.
  Trusting non-public IPs does not work well (e.g. with lokinet where
  all .loki addresses resolve to a local IP).

- ssl fingerprint option is removed; this isn't supported by curl
  (because it is essentially just duplicating what a custom cainfo
  bundle does)

- --daemon-ssl-allow-chained is removed; it wasn't a useful option (if
  you don't want chaining, don't specify a cainfo chain).

- --daemon-address is now a URL instead of just host:port.  (If you omit
  the protocol, http:// is prepended).

- --daemon-host and --daemon-port are now deprecated and produce a
  warning (in simplewallet) if used; the replacement is to use
  --daemon-address.

- --daemon-ssl is deprecated; specify --daemon-address=https://whatever
  instead.

- the above three are now hidden from --help

- reordered the wallet connection options to make more logical sense.
2020-08-07 17:14:03 -03:00
Jason Rhinelander 43f2dfba50 Windows fixes
- Add libuv submodule (only needed and built on Windows)

- Temporarily switch uWebSockets to my github repo: I submitted some
mingw compilation fixes upstream; as soon as a new upstream release
comes out we will switch this back to the upstream repo.

- Switch BOOST_SCOPE_EXIT to LOKI_DEFER

- Don't compile `closefrom()` on Windows (it isn't used, and generates
an unused function warning).
2020-08-07 17:14:02 -03:00
Jason Rhinelander 42a7e83c33 Replace epee http rpc server with uWebSockets
This replaces the NIH epee http server which does not work all that well
with an external C++ library called uWebSockets.  Fundamentally this
gives the following advantages:

- Much less code to maintain
- Just one thread for handling HTTP connections versus epee's pool of
threads
- Uses existing LokiMQ job server and existing thread pool for handling
the actual tasks; they are processed/scheduled in the same "rpc" or
"admin" queues as lokimq rpc calls.  One notable benefit is that "admin"
rpc commands get their own queue (and thus cannot be delayed by long rpc
commands).  Currently the lokimq threads and the http rpc thread pool
and the p2p thread pool and the job queue thread pool and the dns lookup
thread pool and... are *all* different thread pools; this is a step
towards consolidating them.
- Very little mutex contention (which has been a major problem with epee
RPC in the past): there is one mutex (inside uWebSockets) for putting
responses back into the thread managing the connection; everything
internally gets handled through (lock-free) lokimq inproc sockets.
- Faster RPC performance on average, and much better worst case
performance.  Epee's http interface seems to have some race condition
that ocassionally stalls a request (even a very simple one) for a dozen
or more seconds for no good reason.
- Long polling gets redone here to no longer need threads; instead we
just store the request and respond when the thread pool, or else in a
timer (that runs once/second) for timing out long polls.

---

The basic idea of how this works from a high level:

We launch a single thread to handle HTTP RPC requests and response data.
This uWebSockets thread is essentially running an event loop: it never
actually handles any logic; it only serves to shuttle data that arrives
in a request to some other thread, and then, at some later point, to
send some reply back to that waiting connection.  Everything is
asynchronous and non-blocking here: the basic uWebSockets event loop
just operates as things arrive, passes it off immediately, and goes back
to waiting for the next thing to arrive.

The basic flow is like this:

    0. uWS thread -- listens on localhost:22023
    1. uWS thread -- incoming request on localhost:22023
    2. uWS thread -- fires callback, which injects the task into the LokiMQ job queue
    3. LMQ main loop -- schedules it as an RPC job
    4. LMQ rpc thread -- Some LokiMQ thread runs it, gets the result
    5. LMQ rpc thread -- Result gets queued up for the uWS thread
    6. uWS thread -- takes the request and starts sending it
       (asynchronously) back to the requestor.

In more detail:

uWebSockets has registered has registered handlers for non-jsonrpc
requests (legacy JSON or binary).  If the port is restricted then admin
commands get mapped to a "Access denied" response handler, otherwise
public commands (and admin commands on an unrestricted port) go to the
rpc command handler.

POST requests to /json_rpc have their own handler; this is a little
different than the above because it has to parse the request before it
can determine whether it is allowed or not, but once this is done it
continues roughly the same as legacy/binary requests.

uWebSockets then listens on the given IP/port for new incoming requests,
and starts listening for requests in a thread (we own this thread).
When a request arrives, it fires the event handler for that request.
(This may happen multiple times, if the client is sending a bunch of
data in a POST request).  Once we have the full request, we then queue
the job in LokiMQ, putting it in the "rpc" or "admin" command
categories.  (The one practical different here is that "admin" is
configured to be allowed to start up its own thread if all other threads
are busy, while "rpc" commands are prioritized along with everything
else.)  LokiMQ then schedules this, along with native LokiMQ "rpc." or
"admin." requests.

When a LMQ worker thread becomes available, the RPC command gets called
in it and runs.  Whatever output it produces (or error message, if it
throws) then gets wrapped up in jsonrpc boilerplate (if necessary), and
delivered to the uWebSockets thread to be sent in reply to that request.

uWebSockets picks up the data and sends whatever it can without
blocking, then buffers whatever it couldn't send to be sent again in a
later event loop iteration once the requestor can accept more data.
(This part is outside lokid; we only have to give uWS the data and let
it worry about delivery).

---

PR specifics:

Things removed from this PR:

1. ssl settings; with this PR the HTTP RPC interface is plain-text.  The
previous default generated a self-signed certificate for the server on
startup and then the client accepted any certificate.  This is actually
*worse* than unencrypted because it is entirely MITM-readable and yet
might make people think that their RPC communication is encrypted, and
setting up actual certificates is difficult enough that I think most
people don't bother.

uWebSockets *does* support HTTPS, and we could glue the existing options
into it, but I'm not convinced it's worthwhile: it works much better to
put HTTPS in a front-end proxy holding the certificate that proxies
requests to the backend (which can then listen in restricted mode on
some localhost port).  One reason this is better is that it is much
easier to reload and/or restart such a front-end server, while
certificate updates with lokid require a full restart.  Another reason
is that you get an error page instead of a timeout if something is wrong
with the backend.  Finally we also save having to generate a temporary
certificate on *every* lokid invocation.

2. HTTP Digest authentication.  Digest authentication is obsolete (and
was already obsolete when it got added to Monero).  HTTP-Digest was
originally an attempt to provide a password authentication mechanism
that does not leak the password in transit, but still required that the
server know the password.  It only has marginal value against replay
attacks, and is made entirely obsolete by sending traffic over HTTPS
instead.  No client out there supports Digest but *not* Basic auth, and
so given the limited usefulness it seems pointless to support more than
Basic auth for HTTP RPC login.

What's worse is that epee's HTTP Digest authentication is a terrible
implementation: it uses boost::spirit -- a recursive descent parser
meant for building complex language grammars -- just to parse a single
HTTP header for Digest auth.  This is a big load of crap that should
never have been accepted upstream, and that we should get rid of (even
if we wanted to support Digest auth it takes less than 100 lines of code
to do it when *not* using a recursive descent parser).
2020-08-07 17:14:02 -03:00
Jason Rhinelander 83dd656e74 C++17
Switch loki dev branch to C++17 compilation, and update the code with
various C++17 niceties.

- stop including the (deprecated) lokimq/string_view.h header and
instead switch everything to use std::string_view and `""sv` instead of
`""_sv`.

- std::string_view is much nicer than epee::span, so updated various
loki-specific code to use it instead.

- made epee "portable storage" serialization accept a std::string_view
instead of const lvalue std::string so that we can avoid copying.

- switched from mapbox::variant to std::variant

- use `auto [a, b] = whatever()` instead of `T1 a; T2 b; std::tie(a, b)
= whatever()` in a couple places (in the wallet code).

- switch to std::lock(...) instead of boost::lock(...) for simultaneous
lock acquisition.  boost::lock() won't compile in C++17 mode when given
locks of different types.

- removed various pre-C++17 workarounds, e.g. for fold expressions,
unused argument attributes, and byte-spannable object detection.

- class template deduction means lock types no longer have to specify
the mutex, so `std::unique_lock<std::mutex> lock{mutex}` can become
`std::unique_lock lock{mutex}`.  This will make switching any mutex
types (e.g. from boost to std mutexes) far easier as you just have to
update the type in the header and everything should work.  This also
makes the tools::unique_lock and tools::shared_lock methods redundant
(which were a sort of poor-mans-pre-C++17 way to eliminate the
redundancy) so they are now gone and replaced with direct unique_lock or
shared_lock constructions.

- Redid the LNS validation using a string_view; instead of using raw
char pointers the code now uses a string view and chops off parts of the
view as it validates.  So, for instance, it starts with "abcd.loki",
validates the ".loki" and chops the view to "abcd", then validates the
first character and chops to "bcd", validates the last and chops to
"bc", then can just check everything remaining for is-valid-middle-char.

- LNS validation gained a couple minor validation checks in the process:
  - slightly tightened the requirement on lokinet addresses to require
    that the last character of the mapped address is 'y' or 'o' (the
    last base32z char holds only one significant bit).
  - In parse_owner_to_generic_owner made sure that the owner value has
    the correct size (otherwise we could up end not filling or
    overfilling the pubkey buffer).

- Replaced base32z/base64/hex conversions with lokimq's versions which
have a nicer interface, are better optimized, and don't depend on epee.
2020-07-02 12:52:12 -03:00
Jason Rhinelander 70b9fed4fd Static builds: make usable binaries from cmake
This adds a static dependency script for libraries like boost, unbound,
etc. to cmake, invokable with:

    cmake .. -DBUILD_STATIC_DEPS=ON

which downloads and builds static versions of all our required
dependencies (boost, unbound, openssl, ncurses, etc.).  It also implies
-DSTATIC=ON to build other vendored deps (like miniupnpc, lokimq) as
static as well.

Unlike the contrib/depends system, this is easier to maintain (one
script using nicer cmake with functions instead of raw Makefile
spaghetti code), and isn't concerned with reproducible builds -- this
doesn't rebuild the compiler, for instance.  It also works with the
existing build system so that it is simply another way to invoke the
cmake build scripts but doesn't require any external tooling.

This works on Linux, Mac, and Windows.

Some random comments on this commit (for preserving history):

- Don't use target_link_libraries on imported targets.  Newer cmake is
fine with it, but Bionic's cmake doesn't like it but seems okay with
setting the properties directly.

- This rebuilds libzmq and libsodium, even though there is some
provision already within loki-core to do so: however, the existing
embedded libzmq fails with the static deps because it uses libzmq's
cmake build script, which relies on pkg-config to find libsodium which
ends up finding the system one (or not finding any), rather than the one
we build with DownloadLibSodium.  Since both libsodium and libzmq are
faily simple builds it seemed easiest to just add them to the cmake
static build rather than trying to shoehorn the current code into the
static build script.

- Half of the protobuf build system ignores CC/CXX just because Google,
and there's no documentation anywhere except for a random closed bug
report about needing to set these other variables (CC_FOR_BUILD,
CXX_FOR_BUILD) instead, but you need to.  Thanks Google.

- The boost build is set to output very little because even the minimum
-d1 output level spams ~15k lines of output just for the headers it
installs.
2020-06-15 12:49:33 -03:00
Jason Rhinelander 701732cfae Don't depend unnecessarily on PkgConfig 2020-06-14 22:10:36 -03:00
Jason Rhinelander f4e9ea7d40 Remove unbound submodule
There's really no reason to submodule it - we work with pretty much any
libunbound version, and it's a very commonly available library
(comparable to sqlite3 or boost, which we don't submodule).

This removes the submodule and switches it to a hard dependency.
2020-06-11 15:02:21 -03:00
Jason Rhinelander 62c77fe4c7 Deal with negative integer encodings
Change the blink quorum checksum value on the wire to a positive
uint64_t rather than the 2s complement int64_t encoded value that
bt_value produces by default.  Also updates the bt_dict_consumer code
to handle the needed int64_t -> uint64_t conversion if a negative
int64_t (from a previous lokid version) arrives on the wire.
2020-04-30 15:13:45 -03:00
Jason Rhinelander 3874e7478e
Fix blink submission replies (#1142)
LokiMQ's ConnectionID wasn't working properly for routing replies to
incoming connections from non-service nodes; I already noticed and fixed
this in the lokimq update in the LMQ RPC overhaul branch, but we need to
bring it back into master.

This also changes the SN selection for blink submissions so that we
always prefer submitting to higher version service nodes (and then
randomize among SNs with the same version).  This should make blinks
work robustly again: as long as the daemon the wallet talks to and at
least *one* of the 20 quorum members is upgraded we should get a reply
properly.
2020-04-29 10:34:32 +10:00
Jason Rhinelander 5d39a6ae68
Update for LokiMQ 1.1.0's changes (#1122)
With the 1.1.0 update, rather than LokiMQ getting a callback that it
invokes on connection to get auth level for the duration of that
connection, it now checks authentication each time a command is invoked.
This improves message reliability (because on an invocation failure the
remote doesn't have to reconnect to be reauthenticated).

This helps storage server greatly (which has some tricky initialization
issue that requires it to initialize lokimq before it has the SN list);
it is less needed for quorumnet communications (which have been all
using connection-time authenciation for a while now), but this change
does some one "weirdness" that a remote client may be unable to issue a
command that they *can* issue if they reconnect.

The lokid update here piggypacks the call into quorumnet in quorum_cop's
`add_block` callback to make the call; this would be far cleaner if we
replaced the callbacks with stateful std::function's instead of
inherited base class pointers, but I didn't want to go that far in this
PR.
2020-04-15 09:22:18 +10:00
Jason Rhinelander 3961a0c662
Update bundled & required lokimq to 1.0.4 (#1104) 2020-03-30 16:19:44 +11:00
Jason Rhinelander e84a111781 Various contrib/depends fixes
- set WITH_SYSTEMD=OFF in contrib/depends so that we don't try to go
look for it on the host system.
- link icu libs properly
- don't build embedded zmq but rather set up a target so that loki-mq
uses the externally built one
- fix OpenSSL::Crypto not properly depending on ws2_32
- build boost atomic in the built boost because boost::thread depends on
it (and we were just getting lucky before by not happening to touch
anything that needed it)
- make bundled unbound link against the `extra` interface for various
required windows crap (ws2_32 and other stuff).
2020-03-16 22:01:54 -03:00
Jason Rhinelander 743d4e60ce Various linking and build fixes
- updating to latest loki-mq (1.0.0 + various linking fixes)
- BUILD_SHARED_LIBS was being handled very strangely; make it a full
option instead (defaulting to off) that a cmake invoker can specify, as
per cmake recommendations.
- travis ci tweaks/changes:
  - Add a static bionic build
  - Simplify cmake argument code
  - Add `--version` invocation for lokid and loki-wallet-cli to test
    that the binaries were linked properly.
- always build an embedded sodium statically; if we do it dynamically
and an older system one exists we are going to have trouble.
- don't force epee and blocks to be static; rather they get controlled
by the above BUILD_SHARED_LIBS, just like all the other internal
libraries.
- use some PkgConfig:: imported targets rather than bunch-of-variables.
2020-03-15 14:29:47 -03:00
Jason Rhinelander a53b9fff22 Allow linking to a system-installed lokimq
It makes the debs easier (especially on the older distributions) to
package it separately.
2020-03-15 13:27:15 -03:00
Jason Rhinelander dc29d66ed7 Set OPENSSL_LIBRARIES hack for bundled libunbound 2020-03-15 13:27:15 -03:00