Replaces reintroduced boost::chronos and posix_times with std::chrono.
Also rewrote the time printing parts of the performance_tests code to do
it better use std::chrono types instead of uint64_ts.
Replaces some boost::functions, and removes some C-style ancient
`std::function<Return(void)>` with C++-style `std::function<Return()>`.
Edit: There's also a map -> unordered_map change hidden in here because
it was in the middle of the other changes and is minor.
REGISTER_CALLBACK and REGISTER_CALLBACK_METHOD do *exactly the same
thing*, and both take a different and unnecessary argument. DRY it out
and make it use a lambda instead of boost::bind.
This removes some boost thread interruption code, but that code is
highly unlikely to have done anything: you cannot preemptively interrupt
a thread, and pretty much anywhere a thread would get stuck is in I/O,
where boost's (voluntary) thread interruption cannot follow.
It also removes thread attributes because they have really limited
usefulness. There *was* a reason to introduce them many years ago (when
musl libc ran into a stack limit problem with libunbound) but musl
increased the default stack size long since.
There's a bunch of cruft in here that isn't used anywhere; delete it,
and move the two remaining functions to a .cpp file since this header
gets included almost everywhere.
Also simplify the duration printer to avoid needing boost.
The histogram data, in particular, is not blob serializable: it is a 16
byte struct on most 64-bit architectures, and a 12 byte struct on most
32-bit architectures.
Some of the invalid bulletproof tests were relying on the broken
behaviour of tx serialization failure still producing a tx hash based on
a partially filled serialized blob. The code now checks for
serialization failures, and throws when trying to get a tx hash from a
tx that fails to serialize. (This isn't new behaviour: it's just that
some serialization failures didn't actually signal a failure properly).
Fix up the tests to use a random tx hash if such a serialization
exception occurs, so that we can still add the invalid txes to blocks
for the purposes of testing block/tx failures.
loki_tx_builder fails for this test: it creates unserializable txes
because `vin` is too large. This was previously "passing" the test for
completely wrong reasons: the test produces transactions that fail to
serialize (triggering a "invalid mg_ss2 size: have 2, expected 3"
failure) while the point of the test was to produce serializable, but
invalid, transactions.
The failure wasn't being detected because the `tx_to_blob(t)` call in
`cryptonote_format_utils.cpp` would return a *partially filled*
serialization blob in case of serialization failure, which was
incredibly broken. `calculate_transaction_hash()` then was perfectly
happy to proceed with hashing this broken blob.
`tx_to_blob(t)` now returns an empty string on such a serialization
failure, and this made the test start failing because
`calculate_transaction_hash()` now fails.
This improves `calculate_transaction_hash()` a little more to add
earlier and more direct detection of such a failure, and fixes the test
by reverting it to the original Monero tx builder test code, which
creates the intended serializable-but-invalid transactions.
common/util.h has become something of a dumping ground of random
functions. This splits them up a little by moving the filesystem bits
to common/file.h, the sha256sum functions to common/sha256sum.h, and the
(singleton) signal handler to common/signal_handler.h.
The tx extra getting/setting API is a little non-DRY, so this dries it
out with some small changes:
- Added get_field_from_tx_extra that takes a specific tx_extra lvalue
reference and fetches it. This eliminates a bunch of nearly duplicate
code that calls parse - find - return.
- Add three new functions to add tagged data:
- add_tagged_data_to_tx_extra is a raw function that takes a
tag and string_view and appends it to tx_extra
- add_tx_extra<T>(tx_extra, val) lets you copy `val` directly as a
`T`. `val` does *not* have to be the same as `T` (`T` is not
deduced) but rather just has to have the same size. This makes it
very easy to add simple tx_extra values such as `tx_extra_pub_key`
from just a pub key without needing to wrap it first:
add_tx_extra<tx_extra_pub_key>(tx.extra, regular_old_pubkey);
- add_tx_extra<T>(tx, val) - very light wrapper around the above that
takes a transaction lvalue directly (rather than the extra vector)
for slight convenience.
These additions allow removing several now-unneeded methods and simplify
some code around the tx extra handling.
This commit also converts the TX_EXTRA_* macros into constexprs.
Add number parsing and basic string splitting to common/string_util.h
and use it (replacing some regexes and boost string utilities).
Includes unit tests.
A huge amount of this is repetitive:
- `boost::get<T>(variant)` becomes `std::get<T>(variant)`
- `boost::get<T>(variant_ptr)` becomes `std::get_if<T>(variant_ptr)`
- `variant.type() == typeid(T)` becomes `std::holds_alternative<T>(variant)`
There are also some simplifications to visitors using simpler stl
visitors, or (simpler still) generic lambdas as visitors.
Also adds boost serialization serializers for std::variant and
std::optional.
The serialization interface here had a lot going wrong with it. This
overhauls it drastically and makes it a lot nicer to deal with.
(This commit is in two parts: this first one updating the base
serialization code, the subsequent one updating various places using
it. There's a third part, depending on how you want to count,
converting boost::variant to std::variant which relies on the
serialization changes made here).
- everything is now in the `serialization` namespace instead of having
some things there and other things in the root namespace.
- serialization failures now throw exceptions with reasons for the
failure rather than needing to snake a bool back through the call stack
(without any message).
- the `template <bool W, template <bool> class Archive>` monstrosity is
gone. Instead an Archive class has a Archive::is_serializer or
Archive::is_deserializer constexpr bool that can be checked (there was
something sort of similar before, but required messing around with
boost::mpl crap in both the generic and specific serialization code).
- the serialization code is significantly more flexible: instead of
having to slam everything into a class itself, you can also serialize
using with a free function in the same namespace as the class.
- serialization macros are still provided, but now considered
deprecated, replaced with (ADL callable) function names that don't hide
the action from the caller. So:
FIELD(a);
VARINT_FIELD(b);
FIELD_N("c", some_c);
VARINT_FIELD_N("d", some_d);
FIELDS(x); // (this is like the above, but doesn't write a tag)
ENUM_FIELD(e, e < myenum::_count);
FIELD(f);
if (!W && f != 42) return false;
becomes (all of this is documented in serialization.h):
field(ar, "a", a);
field_varint(ar, "b", b);
field(ar, "c", some_c);
field_varint(ar, "d", some_d);
value(x);
// enums just get passed to field_varint. It takes an optional
// lambda to verify on deserialization:
field_varint(ar, "e", f, [&] { return e < myenum::_count; });
// But the verifier isn't limited to enums:
field(ar, "f", f, [&] { return f == 42; });
and all of this works without needing to `serialization::` qualify the
beginning. In the rare case where you have a conflicting function
defined (e.g. a local `field()`) you can qualify to disambiguate.
- The messy eof hacks where you call `serialize_noeof` is cleaned up.
You now call `serialization::serialize(ar, val)` to deserialize an
*entire* value (which requires that the whole strem is consumed), and
`serialization::value(ar, val)` to append a serialization.
- Container serialization is significantly simplified; the various
serialization/vector.h (and similar) are now extremely thin wrappers
around the generic main container serialization code.
- INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and
OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly
identical yet more flexible (for both the caller and the compiler)
relatively simple templated functions.
- Drastically simplified the ability to serialize to/from a string via
new `serialization::binary_string_archiver` and
`serialization::binary_string_unarchiver` serializers. The former takes
no arguments; the latter takes a string_view. This means you can
serialize to binary using:
serialization::binary_string_archiver ar;
serialization::serialize(ar, myvalue);
std::string serialized = ar.str();
and can deserialize using:
MyType myvalue;
serialization::binary_string_unarchiver ar{serialized};
serialization::serialize(ar, myvalue);
(though really this interface is for slightly more complicated cases
than these; see the next point)
- the existing dump_binary() and parse_binary() are tweaked a bit: both
now throw, and dump_binary() returns the result string instead of taking
it as an output parameter. (parse_binary() is now void, rather than
returning a bool, because there are many places where in-place
deserialization is desirable).
- make one_shot_read_buffer internal to binary_string_unarchiver, and
use it there. The interfaces here means we no longer need to rely on
the seeking behaviour, so the serialization issues on mac shouldn't
happen now.
- begin_array()/begin_object() now use an RAII interface to make
serializing arrays much easier. Where previously we had a lot of code
that did something like this:
ar.begin_array();
for (auto it = whatever.begin(); it != whatever.end(); it++)
{
FIELDS(val);
if (*(it+1) != whatever.end())
{
ar.delimit_array();
}
}
ar.end_array();
Now an array serialization looks like this:
auto arr = ar.begin_array();
for (auto& val : whatever)
value(arr.element(), val);
- serialized non-varint integer values weren't endian safe, now they
are.
- variant serialization converted to use std::variant instead of
boost::variant and significantly cleaned up using C++17 features.
- varint (no "a", i.e. variable length integers) was not easy to follow
and badly documented (the given examples in the description were flat
out wrong). Rewrote it, with substantially better documentation and
unit tests, and taking advantage of C++14 `0b` and `'` in integer
literals to make it far easier to follow.
This overhauls the (unpleasant) epee "portable storage" interface to be
a bit easier to work with. The diff is fairly large because of the
amount of type indirection that was being used, but at its core does
this simplifications:
- `hsection` was just a typedef for a `section*`, so just use `section*`
instead to not pretend it the value is something it isn't.
- use std::variant instead of boost::variant to store the portable
storage items.
- don't make the variant list recursive with itself since the
serialization doesn't support that anyway (i.e. it doesn't support a
list of lists).
- greatly simplified the variant visiting by using generic lambdas.
- the array traversal interface was a horrible mess. Replaced it with
a simpler custom iterator approach.
- replaced the `selector<bool>` templated class with templated function.
So for example,
epee::serialization::selector<is_store>::serialize_t_val_as_blob(...)
becomes:
epee::serialization::perform_serialize_blob<is_store>(bytes, stg, parent_section, "addr");
and similar for the other types of serializing.
