The public rpc port code from Monero is not used on Oxen (we have no
ability to enable it because we didn't want it), but still carried
around and printed pointlessly. This removes it entirely.
Along the way I ran into some really nasty p2p code using templates for
absolutely no reason at all, so I fixed that crap, and moved some p2p
serialization out of headers into a .cpp file. (This only scratches the
surface, but as we're going to replace the p2p code entirely eventually
I don't want to waste time trying to polish a turd).
Updates GET_PEER_LIST to new RPC.
Removes GET_PUBLIC_NODES because we don't use this option at all in
oxend. (At the point where we need a decentralized public node
repository we'll almost certainly want to use the service node network
to do it, not the p2p layer).
Updates to new RPC interface.
Removes distinct `limit`, `limit_up`, `limit_down` command and makes
them all go through `limit` by allowing limit to take the down and up
parameters.
limit_up 123
limit_down 456
becomes
limit 0 123
limit 456 0
or to set both at once:
limit 456 123
Also improves documentation, and allows "default" to be specified for
either limit to return the limit to its default. (-1 is also accepted
for this, but is a pain because on the command line it produces an
invalid option error unless you also prefix the command with `--` to
stop further arg parsing).
Also makes the command actually print an error (instead of nothing at
all) when given invalid arguments. Whoa such innovation!
This drops the dedicated GET_TRANSACTION_POOL endpoint entirely and
folds the txpool fetching into GET_TRANSACTIONS via a new `memory_pool`
parameter. (This is backwards incompatible, of course, but the results
may not be too harmful: the wallet doesn't actually use this endpoint,
so it mainly affects internal oxend commands (fixed here) and the block
explorer (which is fairly easily fixed).
This also removes some of the useless GET_TRANSACTIONS bits such as
decode_as_json.
`fee` and `burned` are now always provided in the return (for both
specific requested transactions and mempool transactions).
As a side effect, this code ventured deep into core/blockchain in terms
of dealing with missed transactions: previously they were always
returned (via lvalue ref) as a vector, and always had to be specified.
This changes to take via pointer to an unordered_set, which can be null
if you don't care about the missed ones. This reduces several calls
that indeed didn't care about collecting the missed values, and improved
virtually all the places that *did* care which need to query which ones
were missed rather than needing a specific order.
- Take the "include_unrelayed" as a parameter instead of switching on on
the admin context; there isn't likely to be anything sensitive there,
and it makes little sense for a wallet to get *different* results just
because it happens to be talking to a localhost oxend.
- Change the histogram code so that it always returns a fixed size
- Various improvements to histogram output; most notably printing a more
useful indication of what the histogram outputs are (e.g. instead of:
Age Txes Bytes
now 0 0
8.0 minutes ago 0 0
16.1 minutes ago 0 0
24.1 minutes ago 0 0
32.2 minutes ago 0 0
40.2 minutes ago 0 0
48.3 minutes ago 0 0
56.3 minutes ago 0 0
64.3 minutes ago 0 0
72.4 minutes ago 2 4559
it now displays the ranges:
Age Txes Bytes
now - 10.6min ago 0 0
10.6min - 21.3min ago 0 0
21.3min - 32.0min ago 0 0
32.0min - 42.6min ago 0 0
42.6min - 53.2min ago 0 0
53.2min - 63.9min ago 0 0
63.9min - 74.6min ago 0 0
74.6min - 85.2min ago 0 0
85.2min - 1.6hr ago 0 0
1.6hr - 1.8hr ago 2 4559
- add documentation
- change download/upload speed values to B/s instead of KiB/s
- return times as milliseconds instead of seconds, and rename those
fields from ..._time to ..._ms.
- Code cleanup of state strings
- remove completely unused peer state "idle"
Updates the internal callers of get_service_nodes to properly deal with
the new request format/fields.
Note that this also introduces some significant changes to the
`print_sn` output:
- contributor formats are more compact (one line per contributor), show
reserved amounts (if != contributed amouont), and show the stake %.
- reworded "Operator Cut (% of Fee Reward)" to "Operator Fee"
- checkpoint/pulse vote format is completely overhauled to list voted
and missed heights separately.
- pulse votes now show the pulse round for non-zero round results.
- timestamp checks renamed to Quorumnet tests
- qnet & timesync results now just print the number of successes/fails
rather than a list of true/false values.
The bt-encoding, in particular, is aimed at more efficient wallet3 rpc
interactions with a daemon.
This removes the rigidity from RPC request responses, making it easier
to return useful constructs that the current epee code doesn't support
(like lists of lists), and also adds support for bt-encoded RPC requests
(intended to eventually replace the ".bin" endpoints with Monero NIH
binary encoding).
Instead endpoints now set a nlohmann::json response object with whatever
arbitrary values they want without having to add a bunch of limited epee
serialization macro hell.
So, for example, to set the value "foo" to 20, you use:
rpc.response["height"] = 20;
Binary values (for things like hashes) are handled by going through a proxy object:
rpc.response_hex["hash"] = hash; // hash is a crypto::hash
which, for json, is equivalent to:
rpc.response["hash"] = tools::type_to_hex(hash);
but when the response is to be bt-encoded it leaves it as binary for
more efficient transfers.
There is also a `response_b64` that converts json-destined values to
base64 instead of hex (and again leaves bt-destined values as binary).
Parsing of incoming requests now moves to a new
core_rpc_server_command_parser file, which does free-form parsing from a
nlohmann::json or bt_dict_consumer, and has various templated helper
functions to make this easier.
This is preliminary: this builds the basic infrastucture for handling
requests, and converts three endpoints:
- get_info
- get_height
- ons_resolve
Currently only `make daemon` builds (there is code in the wallet that
hasn't been updated yet), and the other RPC interfaces are disabled by
this commit (until they get converted to the new style).
This code is bitrotting, doesn't compile, and isn't being maintained
anymore.
The integration test suite was an interesting idea, in early Loki days,
but is no longer being maintained and is quite cumbersome to run (for
instance, it is not possible to run it via CI because it depends on
xterm to actually run). The code to actually run it (in doy-lee's
loki-integration-testing repository) is also a large burden of "janky"
code that isn't worth maintaining.
Remove this from the code; if someone wants to pick it back up in the
future reverting this commit shouldn't be too difficult (though I'd
suggest that a much better approach to integration testing would be to
run different daemons/wallets via rpc commands, as the network-tests do,
rather than trying to feed stdin and parse stdout from running
individual oxends/wallets).
Snode revisions are a secondary version that let us put out a mandatory
update for snodes that isn't a hardfork (and so isn't mandatory for
wallets/exchanges/etc.).
The main point of this is to let us make a 9.2.0 release that includes
new mandatory minimums of future versions of storage server (2.2.0) and
lokinet (0.9.4) to bring upgrades to the network.
This slightly changes the HF7 blocks to 0 (instead of 1) because,
apparently, we weren't properly checking the HF value of the
pre-first-hf genesis block at all before. (In practice this changes
nothing because genesis blocks are v7 anyway).
This also changes (slightly) how we check for hard forks: now if we skip
some hard forks then we still want to know the height when a hard fork
triggers. For example, if the hf tables contains {7,14} then we still
need to know that the HF14 block height also is the height that
activates HF9, 10, etc.
It works just like storage server testing.
Renames the report_peer_storage_server_status to report_peer_status, and
repurposes the code to handle both SS and lokinet.
This *doesn't* need a HF by design because the reason bit field was
deliberately designed so that we can add reason fields (older clients
will just ignore unknown bits).
* Make preparing registration fail if lokinet or the storage server have not received a ping yet
* Add prepare_registration boolean default
* Add signature override for get_human_time_ago to accept an optional uint64. Add stricter comparisons to ping checks
* Remove duplicate logic
* Remove method override and assign/cast last_lokinet_ping/last_storage_server_ping
* Use auto definition and static_cast res.last_lokinet_ping
This option is incredibly misguided: exceptions are a normal part of C++
error handling that are used *as intended* in lots of places in the
code. Spewing massive amounts of output every time any exception is
thrown anywhere (even when caught!) is terrible.
More than that, we don't ever build with it enabled (for the above
reasons) so this is all just unused code.
This moves all the responsibility of ping testing (deciding when it's
unreachable, etc.) into oxend, allowing for better reporting on SS ping
results and eliminating some edge cases that can lead to oxend and
storage server getting "stuck" thinking each is in a different state.
Improves the oxend<->storage server communications protocol:
- pass storage server HTTPS port as part of the storage server ping
(which already carries the also-required OMQ port) rather than needing
to provide it when starting up oxend. --storage-server-port is now
obsolete (and ignored, if specified).
- Fix up the internal API to use `storage_https_port` and
`storage_omq_port` rather than `storage_port` and `storage_lmq_port`.
- Redo and the SS ping RPC endpoint so that it is less verbose and more
closely matches the lokinet endpoint; instead of:
{ "version_major": 2, "version_minor": 0, "version_patch": 9, "storage_lmq_port": 22222 }
we now expect:
{ "version": [2,0,9], "https_port": 11111, "omq_port": 22222 }
- Tweaks the (not-yet-released) SS proof key names: "s"->"shp" and "slp"->"sop"
The status doesn't have a nice way to tell it is active or awaiting
contributions, so add it.
If "all" reasons don't give any results then try showing the "any"
reasons, and if that doesn't work, show a "reason(s) not available"
message.
- Put SS/lokinet version on same line
- make checkpoint/pulse/timestamp/timesync each take one line instead of
three
- instead of [height,round,vote] for pulse just print [height,vote] and
add a '+R' on height if for a pulse round > 0.
- remove the space after the , in the checkpoint/pulse/etc. lines to
save a little
If `main` throws an exception before the log system is initialized then
the error message just got lost; this fixes it to print to stderr if
that happens.
The loki.conf -> oxen.conf migration wasn't working right when there is
also a ~/.loki -> ~/.oxen migration happening, so this rewrites it to
work properly:
- Make loki.conf -> oxen.conf migration leave behind a symlink
- Fix config file migration to also look for ~/.loki/loki.conf, and also
consider ~/.loki/oxen.conf as a valid load source. (The ~/.loki
consideration only happens when data-dir is default *and* neither
oxen.conf nor loki.conf are found in ~/.oxen).
- *Don't* look for ~/.loki/{loki,oxen}.conf if the default data dir
(~/.oxen) exists.
Other changes:
- remove the default handling for the config file/log file and put it in
main instead. This is non-trivial, and the existing default is broken
in that if you specify `--data-dir=blah` it still tries to load
`~/.oxen/oxen.conf` rather than `blah/oxen.conf`. With this commit it
now does the expected thing when a data-dir is specified.
- Append /regtest to data-dir when running in --regtest mode. The
existing behaviour of clobbering the mainnet data dir is nasty.
We shouldn't actually use it in `main.cpp` because it is called before
the log system is initialized, and it is a wrapper that saves basically
nothing, so just replace it everywhere with direct calls to
fs::create_directories and delete it.
random sampling of service nodes, call timestamp lmq message
checks timestamp of 5 service nodes, if local time is 30 seconds different from 80% of the nodes tested then warn user
tracks external timesync status and timestamp participation of service nodes
clean up includes
new template struct for participation history, individual types for participation entry
refactor checking participation
update select_randomly, move the testing for variance overflow
version locks, bump to 8.1.5
explicit casting for mac & clang
note to remove after hard fork
timestamp debugging log messages
debugging messages for before timesync - before message sent
logging errord with compiling
print version and change add_command to add_request_command
log if statement test
std::to_string replaced with tools::view_guts for x25519 key
check if my sn is active before sending timestamp requests
logging the failures
checking if statement for success of message
more logging, if guards arn't passing
more logging, successfully tests if service node might be out of sync
more tests before we decide we are out of sync
logging output if sn isn't passing tests
if check_participation fails then disconnect
print timestamp status
remove saving variance from the participation history
reduce MIN_TIME_IN_S_BEFORE_VOTING
reset participation history on recommission
undo reduction in startup time
reduce log levels
Set hardfork time in testnet
This json serialization layer was only used in the old Monero ZMQ
interface, which no longer exists, and so this is just dead code.
On top of that, it doesn't work properly for serializing CLSAG
transactions, so just delete it.
The RPC listening options inherited from Monero are a mess: this
overhauls them to support just two options:
--rpc-admin - takes an IP:PORT value to listen on (and can be
specified multiple times to listen on multiple IPs/ports)
--rpc-public - same as above, but the RPC will be restricted.
which allows you to listen on whatever IP/port you want in a much
simpler way.
This replaces all of:
--rpc-bind-ip
--rpc-bind-ipv6
--restricted-rpc
--rpc-use-ipv6
--rpc-ignore-ipv4
--rpc-bind-port
--rpc-restricted-bind-port
--confirm-external-bind
and preemptively also replaces Monero upstream's proposed future
argument:
--rpc-restricted-bind-ip
while being more flexible in every way.
For now the existing Monero options are still supported, but will
produce log warnings on startup when used. They will be removed in a
future major version upgrade (probably loki 9.x).
- Remove some useless epee functions, and add deprecated markers to ones
that have good replacements already.
- Don't use boost::lexical_cast when std::to_string or direct stream
output can be used just as well.
- Get rid of dumb epee "include_base_utils.h" header
Converts all use of boost::filesystem to std::filesystem.
For macos and potentially other exotic systems where std::filesystem
isn't available, we use ghc::filesystem instead (which is a drop-in
replacement for std::filesystem, unlike boost::filesystem).
This also greatly changes how we handle filenames internally by holding
them in filesystem::path objects as soon as possible (using
fs::u8path()), rather than strings, which avoids a ton of issues around
unicode filenames. As a result this lets us drop the boost::locale
dependency on Windows along with a bunch of messy Windows ifdef code,
and avoids the need for doing gross boost locale codecvt calls.
The old deprecated zmq-rpc-bind-port and other hidden deprecated options
weren't being properly recognized when in a config file because only the
visible settings were passed in. This fixes it to also pass in hidden
settings.
The existing boost::format was a monstrosity, so rewrote it to build a
stringstream instead.
- No longer print anything about mining on HF16 unless we are actually
mining.
- Don't print net hash for HF16+ since it is not valid.
- don't show incoming/outgoing connections count if response looks
restricted.
- Print "next fork in x.x hours" starting at 24.0 hours instead of 12.0
hours.
- Replace ridiculous integer-math-done-on-floats with actual integer
math.
- Instead of the Pulse quorums validators recording participation
between each other- so failures may not manifest in a decommission until
the several common nodes align and agree to vote off the node.
Voting now occurs when blocks arrives, validators participating in the
generation of the block are marked. This is shared information between
all nodes syncing the chain so decommissions are more readily agreeable
and acted upon in the Obligation quorums immediately.
- Allow retrieving multiple blocks headers by height in the same request
(matching the API already available to do that for by-hash lookups).
- make pow_hash an optional so that it gets omitted entirely if not
requested
- Likewise for block header responses when using the array arguments
instead of the single value argument.
Otherwise RPCing into the wallet and triggering a refresh doesn't really
work, i.e.
loki-wallet-cli --wallet-file file --password '' refresh
loki-wallet-cli --wallet-file file --password '' balance <-- this returns 0
