oxen-io/oxen-core
2
1
Fork 0
mirror of https://github.com/oxen-io/oxen-core.git synced 2023-12-14 02:22:56 +01:00
Code Issues Projects Releases Wiki Activity
oxen-core/tests/unit_tests/serialization.cpp

1257 lines
49 KiB
C++
Raw Normal View History

Update 2018 copyright
2018-01-07 06:05:16 +01:00
// Copyright (c) 2014-2018, The Monero Project
License updated to BSD 3-clause
2014-07-23 15:03:52 +02:00
//
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without modification, are
// permitted provided that the following conditions are met:
//
// 1. Redistributions of source code must retain the above copyright notice, this list of
// conditions and the following disclaimer.
//
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
// of conditions and the following disclaimer in the documentation and/or other
// materials provided with the distribution.
//
// 3. Neither the name of the copyright holder nor the names of its contributors may be
// used to endorse or promote products derived from this software without specific
// prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//
// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
moved all stuff to github
2014-03-03 23:07:58 +01:00
#include <cstring>
#include <cstdint>
#include <cstdio>
#include <iostream>
#include <vector>
portable serializer: tests added
2017-01-02 02:43:20 +01:00
#include <boost/archive/portable_binary_iarchive.hpp>
extract some basic code from libcryptonote_core into libcryptonote_basic
2017-01-26 16:07:23 +01:00
#include "cryptonote_basic/cryptonote_basic.h"
#include "cryptonote_basic/cryptonote_basic_impl.h"
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
#include "ringct/rctSigs.h"
moved all stuff to github
2014-03-03 23:07:58 +01:00
#include "serialization/binary_archive.h"
#include "serialization/json_archive.h"
#include "serialization/variant.h"
#include "serialization/vector.h"
#include "serialization/binary_utils.h"
portable serializer: tests added
2017-01-02 02:43:20 +01:00
#include "wallet/wallet2.h"
moved all stuff to github
2014-03-03 23:07:58 +01:00
#include "gtest/gtest.h"
tests: pass data dir as arg This fixes test failure on builds that happen to be built in 'build/' instead of 'build/release'. Use boost filesystem path type.
2017-09-25 04:46:14 +02:00
#include "unit_tests_utils.h"
Code modifications to integrate Ledger HW device into monero-wallet-cli. The basic approach it to delegate all sensitive data (master key, secret ephemeral key, key derivation, ....) and related operations to the device. As device has low memory, it does not keep itself the values (except for view/spend keys) but once computed there are encrypted (with AES are equivalent) and return back to monero-wallet-cli. When they need to be manipulated by the device, they are decrypted on receive. Moreover, using the client for storing the value in encrypted form limits the modification in the client code. Those values are transfered from one C-structure to another one as previously. The code modification has been done with the wishes to be open to any other hardware wallet. To achieve that a C++ class hw::Device has been introduced. Two initial implementations are provided: the "default", which remaps all calls to initial Monero code, and the "Ledger", which delegates all calls to Ledger device.
2018-02-20 17:01:27 +01:00
#include "device/device.hpp"
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
using namespace std::literals;
remove "using namespace std" from headers It's nasty, and actually breaks on Solaris, where if.h fails to build due to: struct map *if_memmap;
2017-10-10 16:47:08 +02:00
using namespace crypto;
moved all stuff to github
2014-03-03 23:07:58 +01:00
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
namespace whatever
{
moved all stuff to github
2014-03-03 23:07:58 +01:00
struct Struct
{
int32_t a;
int32_t b;
char blob[8];
};
template <class Archive>
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
void serialize_value(Archive& ar, Struct& s)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
auto obj = ar.begin_object();
ar.tag("a");
ar.serialize_int(s.a);
ar.tag("b");
ar.serialize_int(s.b);
ar.tag("blob");
ar.serialize_blob(s.blob, sizeof(s.blob));
}
}
moved all stuff to github
2014-03-03 23:07:58 +01:00
struct Struct1
{
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::vector<std::variant<whatever::Struct, int32_t>> si;
std::vector<int16_t> vi;
moved all stuff to github
2014-03-03 23:07:58 +01:00
BEGIN_SERIALIZE_OBJECT()
FIELD(si)
FIELD(vi)
END_SERIALIZE()
};
struct Blob
{
uint64_t a;
uint32_t b;
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
uint8_t c;
uint8_t d;
uint16_t e;
moved all stuff to github
2014-03-03 23:07:58 +01:00
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
constexpr bool operator==(const Blob& r) const
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
return std::tie(a, b, c, d, e) == std::tie(r.a, r.b, r.c, r.d, r.d);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
};
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
// If the type has padding then it isn't binary serializable:
static_assert(sizeof(Blob) == 16);
moved all stuff to github
2014-03-03 23:07:58 +01:00
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
VARIANT_TAG(whatever::Struct, "struct", 0xe0);
VARIANT_TAG(int, "int", 0xe1);
JSON_VARIANT_TAG(Struct1, "struct1");
moved all stuff to github
2014-03-03 23:07:58 +01:00
BLOB_SERIALIZER(Blob);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
void try_parse(std::string_view blob)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
Struct1 s1;
return serialization::parse_binary(blob, s1);
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, binary_archive_integers_fixed) {
moved all stuff to github
2014-03-03 23:07:58 +01:00
uint64_t x = 0xff00000000, x1;
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
serialization::binary_string_archiver oar;
ASSERT_NO_THROW(oar.serialize_int(x));
ASSERT_EQ(8, oar.str().size());
ASSERT_EQ("\0\0\0\0\xff\0\0\0"sv, oar.str());
moved all stuff to github
2014-03-03 23:07:58 +01:00
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
auto data = oar.str();
serialization::binary_string_unarchiver iar{data};
ASSERT_EQ(8, iar.remaining_bytes());
ASSERT_NO_THROW(iar.serialize_int(x1));
ASSERT_EQ(0, iar.remaining_bytes());
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(x, x1);
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, binary_archive_integers_variable) {
moved all stuff to github
2014-03-03 23:07:58 +01:00
uint64_t x = 0xff00000000, x1;
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
serialization::binary_string_archiver oar;
ASSERT_NO_THROW(oar.serialize_varint(x));
ASSERT_EQ(6, oar.str().size());
ASSERT_EQ("\x80\x80\x80\x80\xF0\x1F"sv, oar.str());
auto data = oar.str();
serialization::binary_string_unarchiver iar{data};
ASSERT_EQ(6, iar.remaining_bytes());
ASSERT_NO_THROW(varint(iar, x1));
ASSERT_EQ(0, iar.remaining_bytes());
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(x, x1);
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, custom_type_serialization) {
moved all stuff to github
2014-03-03 23:07:58 +01:00
Struct1 s1;
s1.si.push_back(0);
{
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
whatever::Struct s;
moved all stuff to github
2014-03-03 23:07:58 +01:00
s.a = 5;
s.b = 65539;
std::memcpy(s.blob, "12345678", 8);
s1.si.push_back(s);
}
s1.si.push_back(1);
s1.vi.push_back(10);
s1.vi.push_back(22);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::string blob;
ASSERT_NO_THROW(blob = serialization::dump_binary(s1));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob), "03e100000000e005000000030001003132333435363738e101000000020a001600");
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(try_parse(blob));
moved all stuff to github
2014-03-03 23:07:58 +01:00
blob[6] = '\xE1';
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(try_parse(blob), std::runtime_error);
moved all stuff to github
2014-03-03 23:07:58 +01:00
blob[6] = '\xE2';
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(try_parse(blob), std::runtime_error);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, overflow) {
moved all stuff to github
2014-03-03 23:07:58 +01:00
Blob x = { 0xff00000000 };
Blob x1;
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::string blob;
ASSERT_NO_THROW(blob = serialization::dump_binary(x));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(sizeof(Blob), blob.size());
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(serialization::parse_binary(blob, x1));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(x, x1);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::vector<Blob> bigvector;
ASSERT_THROW(serialization::parse_binary(blob, bigvector), std::runtime_error);
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(0, bigvector.size());
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serializes_vector_uint64_as_varint)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
std::vector<uint64_t> v;
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::string blob;
moved all stuff to github
2014-03-03 23:07:58 +01:00
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob), "00");
moved all stuff to github
2014-03-03 23:07:58 +01:00
// +1 byte
v.push_back(0);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob), "0100");
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
// ^^
moved all stuff to github
2014-03-03 23:07:58 +01:00
// +1 byte
v.push_back(1);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob), "020001");
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
// ^^
moved all stuff to github
2014-03-03 23:07:58 +01:00
// +2 bytes
v.push_back(0x80);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob), "0300018001");
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
// ^^^^
moved all stuff to github
2014-03-03 23:07:58 +01:00
// +2 bytes
v.push_back(0xFF);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob), "0400018001ff01");
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
// ^^^^
moved all stuff to github
2014-03-03 23:07:58 +01:00
// +2 bytes
v.push_back(0x3FFF);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob), "0500018001ff01ff7f");
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
// ^^^^
moved all stuff to github
2014-03-03 23:07:58 +01:00
// +3 bytes
v.push_back(0x40FF);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob), "0600018001ff01ff7fff8101");
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
// ^^^^^^
moved all stuff to github
2014-03-03 23:07:58 +01:00
// +10 bytes
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
v.push_back(0xFFFF'FFFF'FFFF'FFFF);
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob), "0700018001ff01ff7fff8101ffffffffffffffffff01");
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
// ^^^^^^^^^^^^^^^^^^^^
v = {0x64, 0xcc, 0xbf04};
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob), "0364cc0184fe02");
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serializes_vector_int64_as_fixed_int)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
std::vector<int64_t> v;
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::string blob;
moved all stuff to github
2014-03-03 23:07:58 +01:00
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(1, blob.size());
// +8 bytes
v.push_back(0);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(9, blob.size());
// +8 bytes
v.push_back(1);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(17, blob.size());
// +8 bytes
v.push_back(0x80);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(25, blob.size());
// +8 bytes
v.push_back(0xFF);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(33, blob.size());
// +8 bytes
v.push_back(0x3FFF);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(41, blob.size());
// +8 bytes
v.push_back(0x40FF);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(49, blob.size());
// +8 bytes
v.push_back(0xFFFFFFFFFFFFFFFF);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(v));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(57, blob.size());
}
namespace
{
template<typename T>
std::vector<T> linearize_vector2(const std::vector< std::vector<T> >& vec_vec)
{
std::vector<T> res;
Remove boost::value_initialized and BOOST_FOREACH (#921) Neither of these have a place in modern C++11; boost::value_initialized is entirely superseded by `Type var{};` which does value initialization (or default construction if a default constructor is defined). More problematically, each `boost::value_initialized<T>` requires instantiation of another wrapping templated type which is a pointless price to pay the compiler in C++11 or newer. Also removed is the AUTO_VAL_INIT macro (which is just a simple macro around constructing a boost::value_initialized<T>). BOOST_FOREACH is a similarly massive pile of code to implement C++11-style for-each loops. (And bizarrely it *doesn't* appear to fall back to C++ for-each loops even when under a C++11 compiler!) This removes both entirely from the codebase.
2019-10-31 23:26:58 +01:00
for (const auto& vec : vec_vec)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
res.insert(res.end(), vec.begin(), vec.end());
}
return res;
}
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serializes_transaction_signatures_correctly)
moved all stuff to github
2014-03-03 23:07:58 +01:00
{
using namespace cryptonote;
transaction tx;
transaction tx1;
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::string blob;
moved all stuff to github
2014-03-03 23:07:58 +01:00
// Empty tx
tx.set_null();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(5, blob.size()); // 5 bytes + 0 bytes extra + 0 bytes signatures
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(serialization::parse_binary(blob, tx1));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(tx, tx1);
ASSERT_EQ(linearize_vector2(tx.signatures), linearize_vector2(tx1.signatures));
// Miner tx without signatures
txin_gen txin_gen1;
txin_gen1.height = 0;
tx.set_null();
tx.vin.push_back(txin_gen1);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(7, blob.size()); // 5 bytes + 2 bytes vin[0] + 0 bytes extra + 0 bytes signatures
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(serialization::parse_binary(blob, tx1));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(tx, tx1);
ASSERT_EQ(linearize_vector2(tx.signatures), linearize_vector2(tx1.signatures));
// Miner tx with empty signatures 2nd vector
tx.signatures.resize(1);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(7, blob.size()); // 5 bytes + 2 bytes vin[0] + 0 bytes extra + 0 bytes signatures
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(serialization::parse_binary(blob, tx1));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(tx, tx1);
ASSERT_EQ(linearize_vector2(tx.signatures), linearize_vector2(tx1.signatures));
// Miner tx with one signature
tx.signatures[0].resize(1);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(blob = serialization::dump_binary(tx), std::invalid_argument);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// Miner tx with 2 empty vectors
tx.signatures.resize(2);
tx.signatures[0].resize(0);
tx.signatures[1].resize(0);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(blob = serialization::dump_binary(tx), std::invalid_argument);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// Miner tx with 2 signatures
tx.signatures[0].resize(1);
tx.signatures[1].resize(1);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(blob = serialization::dump_binary(tx), std::invalid_argument);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// Two txin_gen, no signatures
tx.vin.push_back(txin_gen1);
tx.signatures.resize(0);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(9, blob.size()); // 5 bytes + 2 * 2 bytes vins + 0 bytes extra + 0 bytes signatures
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(serialization::parse_binary(blob, tx1));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(tx, tx1);
ASSERT_EQ(linearize_vector2(tx.signatures), linearize_vector2(tx1.signatures));
// Two txin_gen, signatures vector contains only one empty element
tx.signatures.resize(1);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(blob = serialization::dump_binary(tx), std::invalid_argument);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// Two txin_gen, signatures vector contains two empty elements
tx.signatures.resize(2);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(9, blob.size()); // 5 bytes + 2 * 2 bytes vins + 0 bytes extra + 0 bytes signatures
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(serialization::parse_binary(blob, tx1));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(tx, tx1);
ASSERT_EQ(linearize_vector2(tx.signatures), linearize_vector2(tx1.signatures));
// Two txin_gen, signatures vector contains three empty elements
tx.signatures.resize(3);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(blob = serialization::dump_binary(tx), std::invalid_argument);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// Two txin_gen, signatures vector contains two non empty elements
tx.signatures.resize(2);
tx.signatures[0].resize(1);
tx.signatures[1].resize(1);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(blob = serialization::dump_binary(tx), std::invalid_argument);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// A few bytes instead of signature
tx.vin.clear();
tx.vin.push_back(txin_gen1);
tx.signatures.clear();
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(tx));
moved all stuff to github
2014-03-03 23:07:58 +01:00
blob.append(std::string(sizeof(crypto::signature) / 2, 'x'));
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(serialization::parse_binary(blob, tx1), std::runtime_error);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// blob contains one signature
blob.append(std::string(sizeof(crypto::signature) / 2, 'y'));
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(serialization::parse_binary(blob, tx1), std::runtime_error);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// Not enough signature vectors for all inputs
txin_to_key txin_to_key1;
tests: fix uninitialized data valgrind reports in serialization tests
2016-11-20 18:25:21 +01:00
txin_to_key1.amount = 1;
memset(&txin_to_key1.k_image, 0x42, sizeof(crypto::key_image));
txin_to_key1.key_offsets.push_back(12);
txin_to_key1.key_offsets.push_back(3453);
moved all stuff to github
2014-03-03 23:07:58 +01:00
tx.vin.clear();
tx.vin.push_back(txin_to_key1);
tx.vin.push_back(txin_to_key1);
tx.signatures.resize(1);
tx.signatures[0].resize(2);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(blob = serialization::dump_binary(tx), std::invalid_argument);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// Too much signatures for two inputs
tx.signatures.resize(3);
tx.signatures[0].resize(2);
tx.signatures[1].resize(2);
tx.signatures[2].resize(2);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(blob = serialization::dump_binary(tx), std::invalid_argument);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// First signatures vector contains too little elements
tx.signatures.resize(2);
tx.signatures[0].resize(1);
tx.signatures[1].resize(2);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(blob = serialization::dump_binary(tx), std::invalid_argument);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// First signatures vector contains too much elements
tx.signatures.resize(2);
tx.signatures[0].resize(3);
tx.signatures[1].resize(2);
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(blob = serialization::dump_binary(tx), std::invalid_argument);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// There are signatures for each input
tx.signatures.resize(2);
tx.signatures[0].resize(2);
tx.signatures[1].resize(2);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
for (char i : {0, 1})
for (char j : {0, 1})
Overhaul and fix crypto::{public_key,ec_point,etc.} types - Remove implicit `operator bool` from ec_point/public_key/etc. which was causing all sorts of implicit conversion mess and bugs. - Change ec_point/public_key/etc. to use a `std::array<unsigned char, 32>` (via a base type) rather than a C-array of char that has to be reinterpret_cast<>'ed all over the place. - Add methods to ec_point/public_key/etc. that make it work more like a container of bytes (`.data()`, `.size()`, `operator[]`, `begin()`, `end()`). - Make a generic `crypto::null<T>` that is a constexpr all-0 `T`, rather than the mishmash `crypto::null_hash`, crypto::null_pkey, crypto::hash::null(), and so on. - Replace three metric tons of `crypto::hash blahblah = crypto::null_hash;` with the much simpler `crypto::hash blahblah{};`, because there's no need to make a copy of a null hash in all these cases. (Likewise for a few other null_whatevers). - Remove a whole bunch of `if (blahblah == crypto::null_hash)` and `if (blahblah != crypto::null_hash)` with the more concise `if (!blahblah)` and `if (blahblah)` (which are fine via the newly *explicit* bool conversion operators). - `crypto::signature` becomes a 64-byte container (as above) but with `c()` and `r()` to get the c() and r() data pointers. (Previously `.c` and `.r` were `ec_scalar`s). - Delete with great prejudice CRYPTO_MAKE_COMPARABLE and CRYPTO_MAKE_HASHABLE and all the other utter trash in `crypto/generic-ops.h`. - De-inline functions in very common crypto/*.h files so that they don't have to get compiled 300 times. - Remove the disgusting include-a-C-header-inside-a-C++-namespace garbage from some crypto headers trying to be both a C and *different* C++ header at once. - Remove the toxic, disgusting, shameful `operator&` on ec_scalar, etc. that replace `&x` with `reinterpret_cast x into an unsigned char*`. This was pure toxic waste. - changed some `<<` outputs to fmt - Random other small changes encountered while fixing everything that cascaded out of the above changes.
2022-10-15 03:22:44 +02:00
tx.signatures[i][j].c()[2*i + j] = ((i+1) << 4) + 2*i + j + 1;
core: cache tx and block hashes in the respective classes An idea from smooth
2017-03-22 19:01:09 +01:00
tx.invalidate_hashes();
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_NO_THROW(blob = serialization::dump_binary(tx));
Update oxenmq to latest oxen-mq+oxen-encoding All the encoding parts move to oxen-encoding recently; this updates to the latest version of oxen-mq, adds oxen-encoding, and converts everything to use oxenc headers rather than the oxenmq compatibility shims.
2022-02-10 19:25:55 +01:00
ASSERT_EQ(oxenc::to_hex(blob),
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
"0100020201020cfd1a42424242424242424242424242424242424242424242424242424242424242420201020cfd1a42424242424242424242424242424242424242424242424242424242424242420000"
"11000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"00120000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"00002300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"00000024000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000");
ASSERT_NO_THROW(serialization::parse_binary(blob, tx1));
moved all stuff to github
2014-03-03 23:07:58 +01:00
ASSERT_EQ(tx, tx1);
ASSERT_EQ(linearize_vector2(tx.signatures), linearize_vector2(tx1.signatures));
// Blob doesn't contain enough data
blob.resize(blob.size() - sizeof(crypto::signature) / 2);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(serialization::parse_binary(blob, tx1), std::runtime_error);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// Blob contains too much data
blob.resize(blob.size() + sizeof(crypto::signature));
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(serialization::parse_binary(blob, tx1), std::runtime_error);
moved all stuff to github
2014-03-03 23:07:58 +01:00
// Blob contains one excess signature
blob.resize(blob.size() + sizeof(crypto::signature) / 2);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
ASSERT_THROW(serialization::parse_binary(blob, tx1), std::runtime_error);
moved all stuff to github
2014-03-03 23:07:58 +01:00
}
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
template <typename T>
T round_trip(T& x)
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
{
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
static_assert(!std::is_const_v<T>);
std::string blob = serialization::dump_binary(x);
T y;
serialization::parse_binary(blob, y);
return y;
}
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serialize_rct_key) {
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
auto key = rct::skGen();
ASSERT_EQ(key, round_trip(key));
}
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serialize_rct_key_vector) {
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
auto keyv = rct::skvGen(30);
for (auto& key : keyv)
key = rct::skGen();
ASSERT_EQ(keyv, round_trip(keyv));
}
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serialize_rct_key_matrix) {
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
auto keym = rct::keyMInit(9, 12);
for (auto& col : keym)
for (auto& key : col)
key = rct::skGen();
ASSERT_EQ(keym, round_trip(keym));
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serialize_rct_ctkey) {
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
rct::ctkey key;
rct::skpkGen(key.dest, key.mask);
rct::ctkey key2 = round_trip(key);
ASSERT_EQ(tools::view_guts(key), tools::view_guts(key2));
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serialize_rct_ctkey_vector) {
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
rct::ctkeyV keyv(14);
for (auto& key : keyv)
rct::skpkGen(key.dest, key.mask);
auto keyv2 = round_trip(keyv);
ASSERT_EQ(keyv.size(), keyv2.size());
for (size_t i = 0; i < keyv.size(); i++)
ASSERT_EQ(tools::view_guts(keyv[i]), tools::view_guts(keyv2[i]));
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serialize_rct_ctkey_matrix) {
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
rct::ctkeyM keym(9);
for (auto& col : keym) {
col.resize(11);
for (auto& key : col)
rct::skpkGen(key.dest, key.mask);
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
}
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
auto keym2 = round_trip(keym);
ASSERT_EQ(keym.size(), keym2.size());
for (size_t c = 0; c < keym.size(); c++) {
ASSERT_EQ(keym[c].size(), keym2[c].size());
for (size_t r = 0; r < keym[c].size(); r++)
ASSERT_EQ(tools::view_guts(keym[c][r]), tools::view_guts(keym2[c][r]));
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
}
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
}
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serialize_rct_ecdh) {
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
rct::ecdhTuple ecdh;
ecdh.mask = rct::skGen();
ecdh.amount = rct::skGen();
auto ecdh2 = round_trip(ecdh);
ASSERT_EQ(tools::view_guts(ecdh.mask), tools::view_guts(ecdh2.mask));
ASSERT_EQ(tools::view_guts(ecdh.amount), tools::view_guts(ecdh2.amount));
}
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serialize_boro_sig) {
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
rct::boroSig boro;
for (auto& s : boro.s0)
s = rct::skGen();
for (auto& s : boro.s1)
s = rct::skGen();
boro.ee = rct::skGen();
auto boro2 = round_trip(boro);
ASSERT_EQ(tools::view_guts(boro), tools::view_guts(boro2));
}
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, serializes_ringct)
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
{
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
// create a full rct signature to use its innards
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::vector<uint64_t> inamounts;
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
rct::ctkeyV sc, pc;
rct::ctkey sctmp, pctmp;
ringct: prevent use of full ringct signatures for more than one input
2019-04-11 20:41:41 +02:00
inamounts.push_back(6000);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::tie(sctmp, pctmp) = rct::ctskpkGen(inamounts.back());
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
sc.push_back(sctmp);
pc.push_back(pctmp);
ringct: prevent use of full ringct signatures for more than one input
2019-04-11 20:41:41 +02:00
inamounts.push_back(7000);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::tie(sctmp, pctmp) = rct::ctskpkGen(inamounts.back());
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
sc.push_back(sctmp);
pc.push_back(pctmp);
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::vector<uint64_t> amounts;
rct: make the amount key derivable by a third party with the tx key Scheme design from luigi1114.
2016-07-24 18:53:34 +02:00
rct::keyV amount_keys;
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
//add output 500
amounts.push_back(500);
ringct: check the size of amount_keys is the same as destinations
2016-10-29 14:33:08 +02:00
amount_keys.push_back(rct::hash_to_scalar(rct::zero()));
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
rct::keyV destinations;
rct::key Sk, Pk;
rct::skpkGen(Sk, Pk);
destinations.push_back(Pk);
//add output for 12500
amounts.push_back(12500);
rct: make the amount key derivable by a third party with the tx key Scheme design from luigi1114.
2016-07-24 18:53:34 +02:00
amount_keys.push_back(rct::hash_to_scalar(rct::zero()));
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
rct::skpkGen(Sk, Pk);
destinations.push_back(Pk);
Integrate CLSAGs into loki They are allowed from v16, and MLSAGs are rejected from v16 + 10 blocks.
2019-06-09 15:02:16 +02:00
Make RingCT and RangeProof types type-safe enum classes
2020-11-21 05:20:15 +01:00
const rct::RCTConfig rct_config_clsag{ rct::RangeProofType::PaddedBulletproof, 3 };
Remove MLSAG generation The blockchain only accepts CLSAG txes now, so no need to keep the MLSAG generation code around. (MLSAG verification stays, of course).
2020-11-23 23:25:01 +01:00
auto s0 = rct::genRctSimple(rct::zero(), sc, pc, destinations, inamounts, amounts, amount_keys, NULL, NULL, 0, 3, rct_config_clsag, hw::get_device("default"));
Integrate CLSAGs into loki They are allowed from v16, and MLSAGs are rejected from v16 + 10 blocks.
2019-06-09 15:02:16 +02:00
ASSERT_FALSE(s0.p.CLSAGs.empty());
ASSERT_TRUE(s0.p.MGs.empty());
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
auto& clsag = s0.p.CLSAGs[0];
auto clsag1 = round_trip(clsag);
ASSERT_EQ(clsag.s, clsag1.s);
ASSERT_EQ(clsag.c1, clsag1.c1);
Integrate CLSAGs into loki They are allowed from v16, and MLSAGs are rejected from v16 + 10 blocks.
2019-06-09 15:02:16 +02:00
// I is not serialized, they are meant to be reconstructed
Split up and DRY Modernize and DRY ringct serialization tests One big blob with 50 tests in it was stupid because you have no clue what failed when something fails.
2020-11-24 17:55:58 +01:00
ASSERT_EQ(clsag.D, clsag1.D);
make rct tx serialization work It may be suboptimal, but it's a pain to have to rebuild everything when some of this changes. Also, no clue why there seems to be two different code paths for serializing a tx...
2016-06-14 18:25:00 +02:00
}
portable serializer: tests added
2017-01-02 02:43:20 +01:00
compiles
2021-01-04 03:09:59 +01:00
// TODO(oxen): These tests are broken because they rely on testnet which has
Don't build i686 targets, disable portability wallet unit test This has broken multiple times, should be fixed when we can since the testnet genesis block has been updated.
2019-02-25 08:09:17 +01:00
// since been restarted, and so the genesis block of these predefined wallets
// are broken
// - 2019-02-25 Doyle
#if 0
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, portability_wallet)
portable serializer: tests added
2017-01-02 02:43:20 +01:00
{
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
const cryptonote::network_type nettype = cryptonote::network_type::TESTNET;
wallet2: make --restricted-rpc available for wallet RPC only
2018-07-30 07:38:55 +02:00
tools::wallet2 w(nettype);
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
const fs::path wallet_file = unit_test::data_dir / "wallet_testnet";
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
std::string password = "test";
portable serializer: tests added
2017-01-02 02:43:20 +01:00
bool r = false;
try
{
unit_tests: fix compiling on Windows
2017-10-08 23:59:08 +02:00
w.load(wallet_file.string(), password);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
r = true;
}
catch (const exception& e)
{}
ASSERT_TRUE(r);
/*
fields of tools::wallet2 to be checked:
std::vector<crypto::hash> m_blockchain
std::vector<transfer_details> m_transfers // TODO
cryptonote::account_public_address m_account_public_address
std::unordered_map<crypto::key_image, size_t> m_key_images
std::unordered_map<crypto::hash, unconfirmed_transfer_details> m_unconfirmed_txs
std::unordered_multimap<crypto::hash, payment_details> m_payments
std::unordered_map<crypto::hash, crypto::secret_key> m_tx_keys
std::unordered_map<crypto::hash, confirmed_transfer_details> m_confirmed_txs
std::unordered_map<crypto::hash, std::string> m_tx_notes
std::unordered_map<crypto::hash, payment_details> m_unconfirmed_payments
std::unordered_map<crypto::public_key, size_t> m_pub_keys
std::vector<tools::wallet2::address_book_row> m_address_book
*/
// blockchain
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
ASSERT_TRUE(w.m_blockchain.size() == 11652);
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(w.m_blockchain[0]) == "da0d7c7824d82c15fc1e046259ce16a7d216e0c0f4406122e207127cf9a77bec");
ASSERT_TRUE(tools::type_to_hex(w.m_blockchain[1]) == "ca9fcde568ad56e01342fa951db7cd0b8f8bdb8a7bf64eb3aa16af79fd44cfab");
ASSERT_TRUE(tools::type_to_hex(w.m_blockchain[11635]) == "84adf6f5a518fcebfea31c304334f7f9358fed610b40034576c46a8c4870b310");
ASSERT_TRUE(tools::type_to_hex(w.m_blockchain[11636]) == "e991be0387c636496ebd224fa9d135dfa917c4b6129bfcfed1458b77666d8b5c");
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// transfers (TODO)
ASSERT_TRUE(w.m_transfers.size() == 3);
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// account public address
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(w.m_account_public_address.m_view_public_key) == "fa871f6af764d2b3fc43eddb9f311b73b387652809992afa5ca3acacc6ff44b0");
ASSERT_TRUE(tools::type_to_hex(w.m_account_public_address.m_spend_public_key) == "efb5d51e3ab4a7c09c207f85650c970eeb671f03d98defcb70265930c0d86240");
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// key images
ASSERT_TRUE(w.m_key_images.size() == 3);
{
crypto::key_image ki[3];
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
tools::hex_to_type("05e1050df8262068682951b459a722495bfd5d070300e96a8d52c6255e300f11", ki[0]);
tools::hex_to_type("21dfe89b3dbde221eccd9b71e7f6383c81f9ada224a670956c895b230749a8d8", ki[1]);
tools::hex_to_type("92194cadfbb4f1317d25d39d6216cbf1030a2170a3edb47b5f008345a879150d", ki[2]);
unit_tests.serialization: refactored with ASSERT_EQ_MAP
2018-02-09 14:33:47 +01:00
ASSERT_EQ_MAP(0, w.m_key_images, ki[0]);
ASSERT_EQ_MAP(1, w.m_key_images, ki[1]);
ASSERT_EQ_MAP(2, w.m_key_images, ki[2]);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
}
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// unconfirmed txs
ASSERT_TRUE(w.m_unconfirmed_txs.size() == 0);
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// payments
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
ASSERT_TRUE(w.m_payments.size() == 1);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
{
auto pd0 = w.m_payments.begin();
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(pd0->first) == "0000000000000000000000000000000000000000000000000000000000000000");
ASSERT_TRUE(tools::type_to_hex(pd0->second.m_tx_hash) == "b77633fe663a07283b071d16c3b783fe838389273fde373a569ad08cb214ab1b");
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
ASSERT_TRUE(pd0->second.m_amount == 100000000000);
ASSERT_TRUE(pd0->second.m_block_height == 8478);
ASSERT_TRUE(pd0->second.m_unlock_time == 0);
ASSERT_TRUE(pd0->second.m_timestamp == 1524445103);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
}
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tx keys
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
ASSERT_TRUE(w.m_tx_keys.size() == 1);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
{
unit_tests.serialization: refactored with ASSERT_EQ_MAP
2018-02-09 14:33:47 +01:00
const std::vector<std::pair<std::string, std::string>> txid_txkey =
{
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
{"d986bc2e49ed83a990424ac42b2db9be0264be54c7ce13f7a8dca5177aa4781c", "b99125ba84a13ed3ee74a3327fd4f34ac11cd580f05e8560b49e755f2586a30b"},
unit_tests.serialization: refactored with ASSERT_EQ_MAP
2018-02-09 14:33:47 +01:00
};
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
unit_tests.serialization: refactored with ASSERT_EQ_MAP
2018-02-09 14:33:47 +01:00
for (size_t i = 0; i < txid_txkey.size(); ++i)
{
crypto::hash txid;
crypto::secret_key txkey;
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
tools::hex_to_type(txid_txkey[i].first, txid);
tools::hex_to_type(txid_txkey[i].second, txkey);
unit_tests.serialization: refactored with ASSERT_EQ_MAP
2018-02-09 14:33:47 +01:00
ASSERT_EQ_MAP(txkey, w.m_tx_keys, txid);
}
portable serializer: tests added
2017-01-02 02:43:20 +01:00
}
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// confirmed txs
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
ASSERT_TRUE(w.m_confirmed_txs.size() == 2);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tx notes
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
ASSERT_TRUE(w.m_tx_notes.size() == 1);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
{
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
crypto::hash h[1];
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
tools::hex_to_type("d986bc2e49ed83a990424ac42b2db9be0264be54c7ce13f7a8dca5177aa4781c", h[0]);
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
ASSERT_EQ_MAP("Unconfirmed transaction test", w.m_tx_notes, h[0]);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
}
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// unconfirmed payments
ASSERT_TRUE(w.m_unconfirmed_payments.size() == 0);
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// pub keys
ASSERT_TRUE(w.m_pub_keys.size() == 3);
{
crypto::public_key pubkey[3];
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
tools::hex_to_type("cc6ac78ac21c034210dcce72a96909b8ba7abd1b3d3917b5ee0c5bc0fe1f6a55", pubkey[0]);
tools::hex_to_type("4cea6373d27bdde002a745ef025375e36ca4b1042c4defdaf2fc56a48ef67230", pubkey[1]);
tools::hex_to_type("b143a6f53cf20f986cbfe87ace7d33143275457dfaa5ea6f14cb78861302dbff", pubkey[2]);
unit_tests.serialization: refactored with ASSERT_EQ_MAP
2018-02-09 14:33:47 +01:00
ASSERT_EQ_MAP(0, w.m_pub_keys, pubkey[0]);
ASSERT_EQ_MAP(1, w.m_pub_keys, pubkey[1]);
ASSERT_EQ_MAP(2, w.m_pub_keys, pubkey[2]);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
}
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// address book
ASSERT_TRUE(w.m_address_book.size() == 1);
{
auto address_book_row = w.m_address_book.begin();
Merge commit 'a26e5b3' into MergeUpstream3
2020-05-27 07:24:18 +02:00
<<<<<<< HEAD
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(address_book_row->m_address.m_spend_public_key) == "938fc84cbacb271fdbc9bfc34e9d887f4bdb89f20a9d4e2c05916d6b9f6a7cb8");
ASSERT_TRUE(tools::type_to_hex(address_book_row->m_address.m_view_public_key) == "9eec0bbb1728bce79209e1ae995cbae8e3f6cf78f7262b5db049594e4907bb33");
ASSERT_TRUE(tools::type_to_hex(address_book_row->m_payment_id) == "e0470453783dd65dc16bb740f82902b9a26a48216e4c89278586637011c858a3");
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
ASSERT_TRUE(address_book_row->m_description == "A test address");
Merge commit 'a26e5b3' into MergeUpstream3
2020-05-27 07:24:18 +02:00
=======
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(address_book_row->m_address.m_spend_public_key) == "9bc53a6ff7b0831c9470f71b6b972dbe5ad1e8606f72682868b1dda64e119fb3");
ASSERT_TRUE(tools::type_to_hex(address_book_row->m_address.m_view_public_key) == "49fece1ef97dc0c0f7a5e2106e75e96edd910f7e86b56e1e308cd0cf734df191");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(address_book_row->m_description == "testnet wallet 9y52S6");
Merge commit 'a26e5b3' into MergeUpstream3
2020-05-27 07:24:18 +02:00
>>>>>>> a26e5b3
portable serializer: tests added
2017-01-02 02:43:20 +01:00
}
}
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
#define OUTPUT_EXPORT_FILE_MAGIC "Loki output export\003"
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, portability_outputs)
portable serializer: tests added
2017-01-02 02:43:20 +01:00
{
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
const bool restricted = false;
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
tools::wallet2 w(cryptonote::network_type::TESTNET, restricted);
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
const fs::path wallet_file = unit_test::data_dir / "wallet_testnet";
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
const std::string password = "test";
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
w.load(wallet_file.string(), password);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// read file
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
const fs::path filename = unit_test::data_dir / "outputs";
portable serializer: tests added
2017-01-02 02:43:20 +01:00
std::string data;
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
bool r = tools::slurp_file(filename.string(), data);
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(r);
const size_t magiclen = strlen(OUTPUT_EXPORT_FILE_MAGIC);
ASSERT_FALSE(data.size() < magiclen || memcmp(data.data(), OUTPUT_EXPORT_FILE_MAGIC, magiclen));
// decrypt (copied from wallet2::decrypt)
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
auto decrypt = [] (const std::string &ciphertext, const crypto::secret_key &skey, bool authenticated) -> std::string
portable serializer: tests added
2017-01-02 02:43:20 +01:00
{
Add a chacha20 variant to go with chacha8
2017-12-07 14:27:11 +01:00
const size_t prefix_size = sizeof(chacha_iv) + (authenticated ? sizeof(crypto::signature) : 0);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
if(ciphertext.size() < prefix_size)
return {};
Add a chacha20 variant to go with chacha8
2017-12-07 14:27:11 +01:00
crypto::chacha_key key;
wallet: allow adjusting number of rounds for the key derivation function
2018-07-06 08:42:08 +02:00
crypto::generate_chacha_key(&skey, sizeof(skey), key, 1);
Add a chacha20 variant to go with chacha8
2017-12-07 14:27:11 +01:00
const crypto::chacha_iv &iv = *(const crypto::chacha_iv*)&ciphertext[0];
portable serializer: tests added
2017-01-02 02:43:20 +01:00
std::string plaintext;
plaintext.resize(ciphertext.size() - prefix_size);
if (authenticated)
{
crypto::hash hash;
crypto::cn_fast_hash(ciphertext.data(), ciphertext.size() - sizeof(signature), hash);
crypto::public_key pkey;
crypto::secret_key_to_public_key(skey, pkey);
const crypto::signature &signature = *(const crypto::signature*)&ciphertext[ciphertext.size() - sizeof(crypto::signature)];
if(!crypto::check_signature(hash, pkey, signature))
return {};
}
crypto::chacha8(ciphertext.data() + sizeof(iv), ciphertext.size() - prefix_size, key, iv, &plaintext[0]);
unit_tests: remove std::move in return statement This actually prevents copy elision
2018-08-15 15:09:14 +02:00
return plaintext;
portable serializer: tests added
2017-01-02 02:43:20 +01:00
};
crypto::secret_key view_secret_key;
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
tools::hex_to_type("cb979d21cde0fbcafb9ff083791a6771b750534948ede6d66058609884b27604", view_secret_key);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
bool authenticated = true;
data = decrypt(std::string(data, magiclen), view_secret_key, authenticated);
ASSERT_FALSE(data.empty());
// check public view/spend keys
const size_t headerlen = 2 * sizeof(crypto::public_key);
ASSERT_FALSE(data.size() < headerlen);
const crypto::public_key &public_spend_key = *(const crypto::public_key*)&data[0];
const crypto::public_key &public_view_key = *(const crypto::public_key*)&data[sizeof(crypto::public_key)];
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(public_spend_key) == "efb5d51e3ab4a7c09c207f85650c970eeb671f03d98defcb70265930c0d86240");
ASSERT_TRUE(tools::type_to_hex(public_view_key) == "fa871f6af764d2b3fc43eddb9f311b73b387652809992afa5ca3acacc6ff44b0");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
r = false;
std::vector<tools::wallet2::transfer_details> outputs;
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
try
{
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
std::string body(data, headerlen);
std::stringstream iss;
iss << body;
try
{
boost::archive::portable_binary_iarchive ar(iss);
ar >> outputs;
r = true;
}
catch (...)
{
}
portable serializer: tests added
2017-01-02 02:43:20 +01:00
}
catch (...)
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
{ }
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(r);
/*
fields of tools::wallet2::transfer_details to be checked:
uint64_t m_block_height
cryptonote::transaction_prefix m_tx // TODO
crypto::hash m_txid
size_t m_internal_output_index
uint64_t m_global_output_index
bool m_spent
uint64_t m_spent_height
crypto::key_image m_key_image
rct::key m_mask
uint64_t m_amount
bool m_rct
bool m_key_image_known
size_t m_pk_index
*/
ASSERT_TRUE(outputs.size() == 3);
auto& td0 = outputs[0];
auto& td1 = outputs[1];
auto& td2 = outputs[2];
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
ASSERT_TRUE(td0.m_block_height == 8478);
ASSERT_TRUE(td1.m_block_height == 11034);
ASSERT_TRUE(td2.m_block_height == 11651);
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(td0.m_txid) == "b77633fe663a07283b071d16c3b783fe838389273fde373a569ad08cb214ab1b");
ASSERT_TRUE(tools::type_to_hex(td1.m_txid) == "4baf027b724623c524c539c5aec441a41c5c76730e4074280189005797a7329d");
ASSERT_TRUE(tools::type_to_hex(td2.m_txid) == "d986bc2e49ed83a990424ac42b2db9be0264be54c7ce13f7a8dca5177aa4781c");
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
ASSERT_TRUE(td0.m_internal_output_index == 1);
ASSERT_TRUE(td1.m_internal_output_index == 1);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(td2.m_internal_output_index == 1);
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
ASSERT_TRUE(td0.m_global_output_index == 17104);
ASSERT_TRUE(td1.m_global_output_index == 22324);
ASSERT_TRUE(td2.m_global_output_index == 23560);
ASSERT_TRUE(td0.m_spent);
ASSERT_TRUE(td1.m_spent);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_FALSE(td2.m_spent);
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
ASSERT_TRUE(td0.m_spent_height == 11034);
ASSERT_TRUE(td1.m_spent_height == 11651);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(td2.m_spent_height == 0);
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(td0.m_key_image) == "05e1050df8262068682951b459a722495bfd5d070300e96a8d52c6255e300f11");
ASSERT_TRUE(tools::type_to_hex(td1.m_key_image) == "21dfe89b3dbde221eccd9b71e7f6383c81f9ada224a670956c895b230749a8d8");
ASSERT_TRUE(tools::type_to_hex(td2.m_key_image) == "92194cadfbb4f1317d25d39d6216cbf1030a2170a3edb47b5f008345a879150d");
ASSERT_TRUE(tools::type_to_hex(td0.m_mask) == "e87548646fdca2caf508c7036e975593063beb38ce6345dcebf6a4f78ac6690a");
ASSERT_TRUE(tools::type_to_hex(td1.m_mask) == "270fbc097ac0ce6d46f7d731ef8f6c28e7d29091106d50d8db5a96c2b43b0009");
ASSERT_TRUE(tools::type_to_hex(td2.m_mask) == "fedf66717b339fdcdd70809a20af7b4314645c859f3c71738567c0c0372f3509");
Change outputs unit test to use a loki generated output
2018-04-29 10:55:52 +02:00
ASSERT_TRUE(td0.m_amount == 100000000000);
ASSERT_TRUE(td1.m_amount == 47531982120);
ASSERT_TRUE(td2.m_amount == 35464004140);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(td0.m_rct);
ASSERT_TRUE(td1.m_rct);
ASSERT_TRUE(td2.m_rct);
ASSERT_TRUE(td0.m_key_image_known);
ASSERT_TRUE(td1.m_key_image_known);
ASSERT_TRUE(td2.m_key_image_known);
ASSERT_TRUE(td0.m_pk_index == 0);
ASSERT_TRUE(td1.m_pk_index == 0);
ASSERT_TRUE(td2.m_pk_index == 0);
}
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
#define UNSIGNED_TX_PREFIX "Loki unsigned tx set\004"
wallet2: only export necessary outputs and key images and disable annoying test that requires ridiculous amounts of skullduggery every time some format changes
2018-10-24 20:24:11 +02:00
struct unsigned_tx_set
{
std::vector<tools::wallet2::tx_construction_data> txes;
tools::wallet2::transfer_container transfers;
};
template <class Archive>
inline void serialize(Archive &a, unsigned_tx_set &x, const boost::serialization::version_type ver)
{
a & x.txes;
a & x.transfers;
}
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, portability_unsigned_tx)
portable serializer: tests added
2017-01-02 02:43:20 +01:00
{
compiles
2021-01-04 03:09:59 +01:00
// TODO(oxen): We updated testnet genesis, is broken
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
const bool restricted = false;
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
tools::wallet2 w(cryptonote::network_type::TESTNET, restricted);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
compiles
2021-01-04 03:09:59 +01:00
const fs::path filename = unit_test::data_dir / "unsigned_oxen_tx";
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
const fs::path wallet_file = unit_test::data_dir / "wallet_testnet";
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
const std::string password = "test";
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
w.load(wallet_file.string(), password);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
std::string s;
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
const cryptonote::network_type nettype = cryptonote::network_type::TESTNET;
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
bool r = tools::slurp_file(filename.string(), s);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(r);
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
size_t const magiclen = strlen(UNSIGNED_TX_PREFIX);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_FALSE(strncmp(s.c_str(), UNSIGNED_TX_PREFIX, magiclen));
wallet2: only export necessary outputs and key images and disable annoying test that requires ridiculous amounts of skullduggery every time some format changes
2018-10-24 20:24:11 +02:00
unsigned_tx_set exported_txs;
portable serializer: tests added
2017-01-02 02:43:20 +01:00
s = s.substr(magiclen);
r = false;
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
try
{
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
s = w.decrypt_with_view_secret_key(s);
try
{
std::istringstream iss(s);
boost::archive::portable_binary_iarchive ar(iss);
ar >> exported_txs;
r = true;
}
catch (...)
{
}
portable serializer: tests added
2017-01-02 02:43:20 +01:00
}
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
catch (const std::exception &e)
{
}
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(r);
/*
fields of tools::wallet2::unsigned_tx_set to be checked:
std::vector<tx_construction_data> txes
std::vector<wallet2::transfer_details> m_transfers
fields of toolw::wallet2::tx_construction_data to be checked:
std::vector<cryptonote::tx_source_entry> sources
cryptonote::tx_destination_entry change_dts
std::vector<cryptonote::tx_destination_entry> splitted_dsts
std::list<size_t> selected_transfers
std::vector<uint8_t> extra
uint64_t unlock_time
bool use_rct
std::vector<cryptonote::tx_destination_entry> dests
fields of cryptonote::tx_source_entry to be checked:
std::vector<std::pair<uint64_t, rct::ctkey>> outputs
size_t real_output
crypto::public_key real_out_tx_key
size_t real_output_in_tx_index
uint64_t amount
bool rct
rct::key mask
fields of cryptonote::tx_destination_entry to be checked:
uint64_t amount
account_public_address addr
*/
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// txes
ASSERT_TRUE(exported_txs.txes.size() == 1);
auto& tcd = exported_txs.txes[0];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tcd.sources
ASSERT_TRUE(tcd.sources.size() == 1);
auto& tse = tcd.sources[0];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tcd.sources[0].outputs
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(tse.outputs.size() == 10);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
auto& out0 = tse.outputs[0];
auto& out1 = tse.outputs[1];
auto& out2 = tse.outputs[2];
auto& out3 = tse.outputs[3];
auto& out4 = tse.outputs[4];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
auto& out5 = tse.outputs[5];
auto& out6 = tse.outputs[6];
auto& out7 = tse.outputs[7];
auto& out8 = tse.outputs[8];
auto& out9 = tse.outputs[9];
ASSERT_TRUE(out0.first == 4905);
ASSERT_TRUE(out1.first == 7366);
ASSERT_TRUE(out2.first == 14589);
ASSERT_TRUE(out3.first == 16559);
ASSERT_TRUE(out4.first == 19748);
ASSERT_TRUE(out5.first == 21209);
ASSERT_TRUE(out6.first == 21695);
ASSERT_TRUE(out7.first == 23200);
ASSERT_TRUE(out8.first == 23236);
ASSERT_TRUE(out9.first == 23560);
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(out0.second) == "b92dd8689d86f3ddf4a21998f9f206dfa478b2944a5a272be6ac0896a406b0186b94ba5f17c6aa64a4cc1332825e1176952417f671ea04342fe6f156c951417d");
ASSERT_TRUE(tools::type_to_hex(out1.second) == "50cb891b4cda04360647b4e0c764de48c7ec82807f844f2fe5446d2684b8df403cba128103c46f40f1564e78b640d8fa11bb2a9d40579e792de30f4febae5ac0");
ASSERT_TRUE(tools::type_to_hex(out2.second) == "fb046f73105eca35aac2ea9befb1be75649456e8e69b78a7104c925f0546e890ea03efb93dddd97a9527248f423d3c1a7afffdf0efff8c844feecf0fe72449fe");
ASSERT_TRUE(tools::type_to_hex(out3.second) == "49c0d412f994b69d09458a8af7602f2ed2fd41d8e430a694d6c0fa5e17bc507ab096d83facce8e3f6381244fe97070e344d71a9d7380f74b9bef209c20308549");
ASSERT_TRUE(tools::type_to_hex(out4.second) == "f4c96b02d97e480fd20d9ad063f10787114ddda3d7a200b532283c4f25707d7ae49bc555dbd83d17a089c1c3acde66dce6a163f75e19835d58f15c2d2cb90c42");
ASSERT_TRUE(tools::type_to_hex(out5.second) == "0e7ca50fec28b7a6f47ee293b7ef9c9522f85fd5bb60ac3105edb9ef8d1e3c079ded4d5d3a45cf6a67200b0e434e7b057230274fed40305e96cab710319bf5cc");
ASSERT_TRUE(tools::type_to_hex(out6.second) == "806d57c9a2ab3402c171332c3fad13838dd125df846f076ca4545e0304b121525525ce94d662ab1eff88cbce06d1bcec37bf1042c3d9b20d04f743bd7392c05e");
ASSERT_TRUE(tools::type_to_hex(out7.second) == "92306e8714fb9c958e3e1df44e798b8c64bb09264d2c97e994b18af4c2fc89e4eab67da527dd194e087a811ae419e5012e32eea80d0c54ef6e39e389bad14edb");
ASSERT_TRUE(tools::type_to_hex(out8.second) == "e29aeceb86042442cbaf15e3907e8bcd254a8740810a75b5583f853a9fdc2228bc74f6a7198c89f7cf770f6c76755f7285fdb13761abaa72d5c79be33d0bd199");
ASSERT_TRUE(tools::type_to_hex(out9.second) == "b143a6f53cf20f986cbfe87ace7d33143275457dfaa5ea6f14cb78861302dbffd013ecf4ffecb0bc694eb4e12cf36b55c80b150acae7a3da42a99b9932dcdd22");
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tcd.sources[0].{real_output, real_out_tx_key, real_output_in_tx_index, amount, rct, mask}
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(tse.real_output == 9);
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(tse.real_out_tx_key) == "1e2092d2e8a72eaec814d8e7ecab9276f2779a5c58324a0fb057e24ffa11a14e");
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(tse.real_output_in_tx_index == 1);
ASSERT_TRUE(tse.amount == 35464004140);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(tse.rct);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(tse.mask) == "fedf66717b339fdcdd70809a20af7b4314645c859f3c71738567c0c0372f3509");
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tcd.change_dts
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(tcd.change_dts.amount == 25396028820);
ASSERT_TRUE(cryptonote::get_account_address_as_str(nettype, false, tcd.change_dts.addr) == "T6UC2BbT5qMV4QgLDd92aoSGTN43yye3eh7JuikVhGWBHSBjsdauotocF5rZK8E8cG5bKU36Mv75r8BwXr8M26ri1bCtF165e");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tcd.splitted_dsts
ASSERT_TRUE(tcd.splitted_dsts.size() == 2);
auto& splitted_dst0 = tcd.splitted_dsts[0];
auto& splitted_dst1 = tcd.splitted_dsts[1];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(splitted_dst0.amount == 10000000000);
ASSERT_TRUE(splitted_dst1.amount == 25396028820);
ASSERT_TRUE(cryptonote::get_account_address_as_str(nettype, false, splitted_dst0.addr) == "T6TQ8id855U7YZFtT2wvCkPqCGhFujMfaE5NE15UWNKjMrxCHGsFLQPYbcSLjoF9xwYGFzbC6LdDw8Fhr5DNsjJe2cDkK1fSM");
ASSERT_TRUE(cryptonote::get_account_address_as_str(nettype, false, splitted_dst1.addr) == "T6UC2BbT5qMV4QgLDd92aoSGTN43yye3eh7JuikVhGWBHSBjsdauotocF5rZK8E8cG5bKU36Mv75r8BwXr8M26ri1bCtF165e");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tcd.selected_transfers
ASSERT_TRUE(tcd.selected_transfers.size() == 1);
ASSERT_TRUE(tcd.selected_transfers.front() == 2);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tcd.extra
ASSERT_TRUE(tcd.extra.size() == 68);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tcd.{unlock_time, use_rct}
ASSERT_TRUE(tcd.unlock_time == 0);
Infinite Staking Part 1 (#387) * Remove dead branches in hot-path check_tx_inputs Also renames #define for mixins to better match naming convention * Shuffle around some more code into common branches * Fix min/max tx version rules, since there 1 tx v2 on v9 fork * First draft infinite staking implementation * Actually generate the right key image and expire appropriately * Add framework to lock key images after expiry * Return locked key images for nodes, add request unlock option * Introduce transaction types for key image unlock * Update validation steps to accept tx types, key_image_unlock * Add mapping for lockable key images to amounts * Change inconsistent naming scheme of contributors * Create key image unlock transaction type and process it * Update tx params to allow v4 types and as a result construct_tx* * Fix some serialisation issues not sending all the information * Fix dupe tx extra tag causing incorrect deserialisation * Add warning comments * Fix key image unlocks parsing error * Simplify key image proof checks * Fix rebase errors * Correctly calculate the key image unlock times * Blacklist key image on deregistration * Serialise key image blacklist * Rollback blacklisted key images * Fix expiry logic error * Disallow requesting stake unlock if already unlocked client side * Add double spend checks for key image unlocks * Rename get_staking_requirement_lock_blocks To staking_initial_num_lock_blocks * Begin modifying output selection to not use locked outputs * Modify output selection to avoid locked/blacklisted key images * Cleanup and undoing some protocol breakages * Simplify expiration of nodes * Request unlock schedules entire node for expiration * Fix off by one in expiring nodes * Undo expiring code for pre v10 nodes * Fix RPC returning register as unlock height and not checking 0 * Rename key image unlock height const * Undo testnet hardfork debug changes * Remove is_type for get_type, fix missing var rename * Move serialisable data into public namespace * Serialise tx types properly * Fix typo in no service node known msg * Code review * Fix == to >= on serialising tx type * Code review 2 * Fix tests and key image unlock * Add additional test, fix assert * Remove debug code in wallet * Fix merge dev problem
2019-01-25 04:15:52 +01:00
// ASSERT_TRUE(tcd.use_rct);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// tcd.dests
ASSERT_TRUE(tcd.dests.size() == 1);
auto& dest = tcd.dests[0];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(dest.amount == 10000000000);
ASSERT_TRUE(cryptonote::get_account_address_as_str(nettype, false, dest.addr) == "T6TQ8id855U7YZFtT2wvCkPqCGhFujMfaE5NE15UWNKjMrxCHGsFLQPYbcSLjoF9xwYGFzbC6LdDw8Fhr5DNsjJe2cDkK1fSM");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// transfers
ASSERT_TRUE(exported_txs.transfers.size() == 3);
auto& td0 = exported_txs.transfers[0];
auto& td1 = exported_txs.transfers[1];
auto& td2 = exported_txs.transfers[2];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(td0.m_block_height == 8478);
ASSERT_TRUE(td1.m_block_height == 11034);
ASSERT_TRUE(td2.m_block_height == 11651);
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(td0.m_txid) == "b77633fe663a07283b071d16c3b783fe838389273fde373a569ad08cb214ab1b");
ASSERT_TRUE(tools::type_to_hex(td1.m_txid) == "4baf027b724623c524c539c5aec441a41c5c76730e4074280189005797a7329d");
ASSERT_TRUE(tools::type_to_hex(td2.m_txid) == "d986bc2e49ed83a990424ac42b2db9be0264be54c7ce13f7a8dca5177aa4781c");
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(td0.m_internal_output_index == 1);
ASSERT_TRUE(td1.m_internal_output_index == 1);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(td2.m_internal_output_index == 1);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(td0.m_global_output_index == 17104);
ASSERT_TRUE(td1.m_global_output_index == 22324);
ASSERT_TRUE(td2.m_global_output_index == 23560);
ASSERT_TRUE(td0.m_spent);
ASSERT_TRUE(td1.m_spent);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_FALSE(td2.m_spent);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(td0.m_spent_height == 11034);
ASSERT_TRUE(td1.m_spent_height == 11651);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(td2.m_spent_height == 0);
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(td0.m_key_image) == "05e1050df8262068682951b459a722495bfd5d070300e96a8d52c6255e300f11");
ASSERT_TRUE(tools::type_to_hex(td1.m_key_image) == "21dfe89b3dbde221eccd9b71e7f6383c81f9ada224a670956c895b230749a8d8");
ASSERT_TRUE(tools::type_to_hex(td2.m_key_image) == "92194cadfbb4f1317d25d39d6216cbf1030a2170a3edb47b5f008345a879150d");
ASSERT_TRUE(tools::type_to_hex(td0.m_mask) == "e87548646fdca2caf508c7036e975593063beb38ce6345dcebf6a4f78ac6690a");
ASSERT_TRUE(tools::type_to_hex(td1.m_mask) == "270fbc097ac0ce6d46f7d731ef8f6c28e7d29091106d50d8db5a96c2b43b0009");
ASSERT_TRUE(tools::type_to_hex(td2.m_mask) == "fedf66717b339fdcdd70809a20af7b4314645c859f3c71738567c0c0372f3509");
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(td0.m_amount == 100000000000);
ASSERT_TRUE(td1.m_amount == 47531982120);
ASSERT_TRUE(td2.m_amount == 35464004140);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(td0.m_rct);
ASSERT_TRUE(td1.m_rct);
ASSERT_TRUE(td2.m_rct);
ASSERT_TRUE(td0.m_key_image_known);
ASSERT_TRUE(td1.m_key_image_known);
ASSERT_TRUE(td2.m_key_image_known);
ASSERT_TRUE(td0.m_pk_index == 0);
ASSERT_TRUE(td1.m_pk_index == 0);
ASSERT_TRUE(td2.m_pk_index == 0);
}
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
#define SIGNED_TX_PREFIX "Loki signed tx set\004"
Rename serialization test category for consistency The vast majority of the tests use lower_case_category.lower_case_test, but Serialization did it differently for no reason (and even then wasn't consistent with the test names). Fix that.
2020-11-24 18:00:44 +01:00
TEST(serialization, portability_signed_tx)
portable serializer: tests added
2017-01-02 02:43:20 +01:00
{
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
const bool restricted = false;
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
tools::wallet2 w(cryptonote::network_type::TESTNET, restricted);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
compiles
2021-01-04 03:09:59 +01:00
const fs::path filename = unit_test::data_dir / "signed_oxen_tx";
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
const fs::path wallet_file = unit_test::data_dir / "wallet_testnet";
Overhaul binary serialization code (part 1) The serialization interface here had a lot going wrong with it. This overhauls it drastically and makes it a lot nicer to deal with. (This commit is in two parts: this first one updating the base serialization code, the subsequent one updating various places using it. There's a third part, depending on how you want to count, converting boost::variant to std::variant which relies on the serialization changes made here). - everything is now in the `serialization` namespace instead of having some things there and other things in the root namespace. - serialization failures now throw exceptions with reasons for the failure rather than needing to snake a bool back through the call stack (without any message). - the `template <bool W, template <bool> class Archive>` monstrosity is gone. Instead an Archive class has a Archive::is_serializer or Archive::is_deserializer constexpr bool that can be checked (there was something sort of similar before, but required messing around with boost::mpl crap in both the generic and specific serialization code). - the serialization code is significantly more flexible: instead of having to slam everything into a class itself, you can also serialize using with a free function in the same namespace as the class. - serialization macros are still provided, but now considered deprecated, replaced with (ADL callable) function names that don't hide the action from the caller. So: FIELD(a); VARINT_FIELD(b); FIELD_N("c", some_c); VARINT_FIELD_N("d", some_d); FIELDS(x); // (this is like the above, but doesn't write a tag) ENUM_FIELD(e, e < myenum::_count); FIELD(f); if (!W && f != 42) return false; becomes (all of this is documented in serialization.h): field(ar, "a", a); field_varint(ar, "b", b); field(ar, "c", some_c); field_varint(ar, "d", some_d); value(x); // enums just get passed to field_varint. It takes an optional // lambda to verify on deserialization: field_varint(ar, "e", f, [&] { return e < myenum::_count; }); // But the verifier isn't limited to enums: field(ar, "f", f, [&] { return f == 42; }); and all of this works without needing to `serialization::` qualify the beginning. In the rare case where you have a conflicting function defined (e.g. a local `field()`) you can qualify to disambiguate. - The messy eof hacks where you call `serialize_noeof` is cleaned up. You now call `serialization::serialize(ar, val)` to deserialize an *entire* value (which requires that the whole strem is consumed), and `serialization::value(ar, val)` to append a serialization. - Container serialization is significantly simplified; the various serialization/vector.h (and similar) are now extremely thin wrappers around the generic main container serialization code. - INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly identical yet more flexible (for both the caller and the compiler) relatively simple templated functions. - Drastically simplified the ability to serialize to/from a string via new `serialization::binary_string_archiver` and `serialization::binary_string_unarchiver` serializers. The former takes no arguments; the latter takes a string_view. This means you can serialize to binary using: serialization::binary_string_archiver ar; serialization::serialize(ar, myvalue); std::string serialized = ar.str(); and can deserialize using: MyType myvalue; serialization::binary_string_unarchiver ar{serialized}; serialization::serialize(ar, myvalue); (though really this interface is for slightly more complicated cases than these; see the next point) - the existing dump_binary() and parse_binary() are tweaked a bit: both now throw, and dump_binary() returns the result string instead of taking it as an output parameter. (parse_binary() is now void, rather than returning a bool, because there are many places where in-place deserialization is desirable). - make one_shot_read_buffer internal to binary_string_unarchiver, and use it there. The interfaces here means we no longer need to rely on the seeking behaviour, so the serialization issues on mac shouldn't happen now. - begin_array()/begin_object() now use an RAII interface to make serializing arrays much easier. Where previously we had a lot of code that did something like this: ar.begin_array(); for (auto it = whatever.begin(); it != whatever.end(); it++) { FIELDS(val); if (*(it+1) != whatever.end()) { ar.delimit_array(); } } ar.end_array(); Now an array serialization looks like this: auto arr = ar.begin_array(); for (auto& val : whatever) value(arr.element(), val); - serialized non-varint integer values weren't endian safe, now they are. - variant serialization converted to use std::variant instead of boost::variant and significantly cleaned up using C++17 features. - varint (no "a", i.e. variable length integers) was not easy to follow and badly documented (the given examples in the description were flat out wrong). Rewrote it, with substantially better documentation and unit tests, and taking advantage of C++14 `0b` and `'` in integer literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
const std::string password = "test";
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
w.load(wallet_file.string(), password);
Remove cryptonote_config macros - Replace all cryptonote_config macros with constexpr variables. Some become integer types, some become chrono types. - generally this involved removing a "CRYPTONOTE_" prefix since the values are now in the `cryptonote` namespace - some constants are grouped into sub-namespaces (e.g. cryptonote::p2p) - deprecated constants (i.e. for old HFs) are in the `cryptonote::old` namespace. - all the magic hash key domain separating strings are now in cryptonote::hashkey::WHATEVER. - Move some economy-related constants to oxen_economy.h instead - Replaced the BLOCKS_EXPECTED_IN_DAYS constexpr functions with more straightforward `BLOCKS_PER_DAY` value (i.e. old `BLOCKS_EXPECTED_IN_DAYS(10)` is now `BLOCKS_PER_DAY * 10`. - Replaced `network_version` unscoped enum with a scoped enum `cryptonote::hf`, replacing all the raw uint8_t values where it was currently accepted with the new `hf` type. - Made `network_type` a scoped enum so that it now has to be qualified (network_type::TESTNET) and can't be arbitrarily/unintentionally converted to/from an int. - HARDFORK_WHATEVER macros have become cryptonote::feature::WHATEVER constexpr hf values. - Add `revision` to rpc hard_fork_info response - Don't build trezor code at all (previously we were pointlessly building an empty dummy lib).
2022-05-16 22:55:05 +02:00
const cryptonote::network_type nettype = cryptonote::network_type::TESTNET;
portable serializer: tests added
2017-01-02 02:43:20 +01:00
std::string s;
std::filesystem Converts all use of boost::filesystem to std::filesystem. For macos and potentially other exotic systems where std::filesystem isn't available, we use ghc::filesystem instead (which is a drop-in replacement for std::filesystem, unlike boost::filesystem). This also greatly changes how we handle filenames internally by holding them in filesystem::path objects as soon as possible (using fs::u8path()), rather than strings, which avoids a ton of issues around unicode filenames. As a result this lets us drop the boost::locale dependency on Windows along with a bunch of messy Windows ifdef code, and avoids the need for doing gross boost locale codecvt calls.
2020-10-22 19:55:33 +02:00
bool r = tools::slurp_file(filename.string(), s);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(r);
Fix serialization.portability_wallet tests
2018-04-28 08:19:21 +02:00
size_t const magiclen = strlen(SIGNED_TX_PREFIX);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_FALSE(strncmp(s.c_str(), SIGNED_TX_PREFIX, magiclen));
tools::wallet2::signed_tx_set exported_txs;
s = s.substr(magiclen);
r = false;
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
try
{
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
s = w.decrypt_with_view_secret_key(s);
try
{
std::istringstream iss(s);
boost::archive::portable_binary_iarchive ar(iss);
ar >> exported_txs;
r = true;
}
catch (...)
{
}
portable serializer: tests added
2017-01-02 02:43:20 +01:00
}
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
catch (const std::exception &e)
{
}
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(r);
/*
fields of tools::wallet2::signed_tx_set to be checked:
std::vector<pending_tx> ptx
std::vector<crypto::key_image> key_images
fields of tools::walllet2::pending_tx to be checked:
cryptonote::transaction tx // TODO
uint64_t dust
uint64_t fee
bool dust_added_to_fee
cryptonote::tx_destination_entry change_dts
std::list<size_t> selected_transfers
std::string key_images
crypto::secret_key tx_key
std::vector<cryptonote::tx_destination_entry> dests
tx_construction_data construction_data
*/
// ptx
ASSERT_TRUE(exported_txs.ptx.size() == 1);
auto& ptx = exported_txs.ptx[0];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.{dust, fee, dust_added_to_fee}
ASSERT_TRUE (ptx.dust == 0);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE (ptx.fee == 67975320);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_FALSE(ptx.dust_added_to_fee);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.change.{amount, addr}
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(ptx.change_dts.amount == 25396028820);
ASSERT_TRUE(cryptonote::get_account_address_as_str(nettype, false, ptx.change_dts.addr) == "T6UC2BbT5qMV4QgLDd92aoSGTN43yye3eh7JuikVhGWBHSBjsdauotocF5rZK8E8cG5bKU36Mv75r8BwXr8M26ri1bCtF165e");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.selected_transfers
ASSERT_TRUE(ptx.selected_transfers.size() == 1);
ASSERT_TRUE(ptx.selected_transfers.front() == 2);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.{key_images, tx_key}
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(ptx.key_images == "<92194cadfbb4f1317d25d39d6216cbf1030a2170a3edb47b5f008345a879150d> ");
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(ptx.tx_key) == "0100000000000000000000000000000000000000000000000000000000000000");
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.dests
ASSERT_TRUE(ptx.dests.size() == 1);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(ptx.dests[0].amount == 10000000000);
ASSERT_TRUE(cryptonote::get_account_address_as_str(nettype, false, ptx.dests[0].addr) == "T6TQ8id855U7YZFtT2wvCkPqCGhFujMfaE5NE15UWNKjMrxCHGsFLQPYbcSLjoF9xwYGFzbC6LdDw8Fhr5DNsjJe2cDkK1fSM");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.construction_data
auto& tcd = ptx.construction_data;
ASSERT_TRUE(tcd.sources.size() == 1);
auto& tse = tcd.sources[0];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.construction_data.sources[0].outputs
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(tse.outputs.size() == 10);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
auto& out0 = tse.outputs[0];
auto& out1 = tse.outputs[1];
auto& out2 = tse.outputs[2];
auto& out3 = tse.outputs[3];
auto& out4 = tse.outputs[4];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
auto& out5 = tse.outputs[5];
auto& out6 = tse.outputs[6];
auto& out7 = tse.outputs[7];
auto& out8 = tse.outputs[8];
auto& out9 = tse.outputs[9];
ASSERT_TRUE(out0.first == 4905);
ASSERT_TRUE(out1.first == 7366);
ASSERT_TRUE(out2.first == 14589);
ASSERT_TRUE(out3.first == 16559);
ASSERT_TRUE(out4.first == 19748);
ASSERT_TRUE(out5.first == 21209);
ASSERT_TRUE(out6.first == 21695);
ASSERT_TRUE(out7.first == 23200);
ASSERT_TRUE(out8.first == 23236);
ASSERT_TRUE(out9.first == 23560);
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(out0.second) == "b92dd8689d86f3ddf4a21998f9f206dfa478b2944a5a272be6ac0896a406b0186b94ba5f17c6aa64a4cc1332825e1176952417f671ea04342fe6f156c951417d");
ASSERT_TRUE(tools::type_to_hex(out1.second) == "50cb891b4cda04360647b4e0c764de48c7ec82807f844f2fe5446d2684b8df403cba128103c46f40f1564e78b640d8fa11bb2a9d40579e792de30f4febae5ac0");
ASSERT_TRUE(tools::type_to_hex(out2.second) == "fb046f73105eca35aac2ea9befb1be75649456e8e69b78a7104c925f0546e890ea03efb93dddd97a9527248f423d3c1a7afffdf0efff8c844feecf0fe72449fe");
ASSERT_TRUE(tools::type_to_hex(out3.second) == "49c0d412f994b69d09458a8af7602f2ed2fd41d8e430a694d6c0fa5e17bc507ab096d83facce8e3f6381244fe97070e344d71a9d7380f74b9bef209c20308549");
ASSERT_TRUE(tools::type_to_hex(out4.second) == "f4c96b02d97e480fd20d9ad063f10787114ddda3d7a200b532283c4f25707d7ae49bc555dbd83d17a089c1c3acde66dce6a163f75e19835d58f15c2d2cb90c42");
ASSERT_TRUE(tools::type_to_hex(out5.second) == "0e7ca50fec28b7a6f47ee293b7ef9c9522f85fd5bb60ac3105edb9ef8d1e3c079ded4d5d3a45cf6a67200b0e434e7b057230274fed40305e96cab710319bf5cc");
ASSERT_TRUE(tools::type_to_hex(out6.second) == "806d57c9a2ab3402c171332c3fad13838dd125df846f076ca4545e0304b121525525ce94d662ab1eff88cbce06d1bcec37bf1042c3d9b20d04f743bd7392c05e");
ASSERT_TRUE(tools::type_to_hex(out7.second) == "92306e8714fb9c958e3e1df44e798b8c64bb09264d2c97e994b18af4c2fc89e4eab67da527dd194e087a811ae419e5012e32eea80d0c54ef6e39e389bad14edb");
ASSERT_TRUE(tools::type_to_hex(out8.second) == "e29aeceb86042442cbaf15e3907e8bcd254a8740810a75b5583f853a9fdc2228bc74f6a7198c89f7cf770f6c76755f7285fdb13761abaa72d5c79be33d0bd199");
ASSERT_TRUE(tools::type_to_hex(out9.second) == "b143a6f53cf20f986cbfe87ace7d33143275457dfaa5ea6f14cb78861302dbffd013ecf4ffecb0bc694eb4e12cf36b55c80b150acae7a3da42a99b9932dcdd22");
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.construction_data.sources[0].{real_output, real_out_tx_key, real_output_in_tx_index, amount, rct, mask}
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(tse.real_output == 9);
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(tse.real_out_tx_key) == "1e2092d2e8a72eaec814d8e7ecab9276f2779a5c58324a0fb057e24ffa11a14e");
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(tse.real_output_in_tx_index == 1);
ASSERT_TRUE(tse.amount == 35464004140);
portable serializer: tests added
2017-01-02 02:43:20 +01:00
ASSERT_TRUE(tse.rct);
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(tse.mask) == "fedf66717b339fdcdd70809a20af7b4314645c859f3c71738567c0c0372f3509");
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.construction_data.change_dts
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(tcd.change_dts.amount == 25396028820);
ASSERT_TRUE(cryptonote::get_account_address_as_str(nettype, false, tcd.change_dts.addr) == "T6UC2BbT5qMV4QgLDd92aoSGTN43yye3eh7JuikVhGWBHSBjsdauotocF5rZK8E8cG5bKU36Mv75r8BwXr8M26ri1bCtF165e");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.construction_data.splitted_dsts
ASSERT_TRUE(tcd.splitted_dsts.size() == 2);
auto& splitted_dst0 = tcd.splitted_dsts[0];
auto& splitted_dst1 = tcd.splitted_dsts[1];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(splitted_dst0.amount == 10000000000);
ASSERT_TRUE(splitted_dst1.amount == 25396028820);
ASSERT_TRUE(cryptonote::get_account_address_as_str(nettype, false, splitted_dst0.addr) == "T6TQ8id855U7YZFtT2wvCkPqCGhFujMfaE5NE15UWNKjMrxCHGsFLQPYbcSLjoF9xwYGFzbC6LdDw8Fhr5DNsjJe2cDkK1fSM");
ASSERT_TRUE(cryptonote::get_account_address_as_str(nettype, false, splitted_dst1.addr) == "T6UC2BbT5qMV4QgLDd92aoSGTN43yye3eh7JuikVhGWBHSBjsdauotocF5rZK8E8cG5bKU36Mv75r8BwXr8M26ri1bCtF165e");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.construction_data.selected_transfers
ASSERT_TRUE(tcd.selected_transfers.size() == 1);
ASSERT_TRUE(tcd.selected_transfers.front() == 2);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.construction_data.extra
ASSERT_TRUE(tcd.extra.size() == 68);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.construction_data.{unlock_time, use_rct}
ASSERT_TRUE(tcd.unlock_time == 0);
Infinite Staking Part 1 (#387) * Remove dead branches in hot-path check_tx_inputs Also renames #define for mixins to better match naming convention * Shuffle around some more code into common branches * Fix min/max tx version rules, since there 1 tx v2 on v9 fork * First draft infinite staking implementation * Actually generate the right key image and expire appropriately * Add framework to lock key images after expiry * Return locked key images for nodes, add request unlock option * Introduce transaction types for key image unlock * Update validation steps to accept tx types, key_image_unlock * Add mapping for lockable key images to amounts * Change inconsistent naming scheme of contributors * Create key image unlock transaction type and process it * Update tx params to allow v4 types and as a result construct_tx* * Fix some serialisation issues not sending all the information * Fix dupe tx extra tag causing incorrect deserialisation * Add warning comments * Fix key image unlocks parsing error * Simplify key image proof checks * Fix rebase errors * Correctly calculate the key image unlock times * Blacklist key image on deregistration * Serialise key image blacklist * Rollback blacklisted key images * Fix expiry logic error * Disallow requesting stake unlock if already unlocked client side * Add double spend checks for key image unlocks * Rename get_staking_requirement_lock_blocks To staking_initial_num_lock_blocks * Begin modifying output selection to not use locked outputs * Modify output selection to avoid locked/blacklisted key images * Cleanup and undoing some protocol breakages * Simplify expiration of nodes * Request unlock schedules entire node for expiration * Fix off by one in expiring nodes * Undo expiring code for pre v10 nodes * Fix RPC returning register as unlock height and not checking 0 * Rename key image unlock height const * Undo testnet hardfork debug changes * Remove is_type for get_type, fix missing var rename * Move serialisable data into public namespace * Serialise tx types properly * Fix typo in no service node known msg * Code review * Fix == to >= on serialising tx type * Code review 2 * Fix tests and key image unlock * Add additional test, fix assert * Remove debug code in wallet * Fix merge dev problem
2019-01-25 04:15:52 +01:00
// ASSERT_TRUE(tcd.use_rct);
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// ptx.construction_data.dests
ASSERT_TRUE(tcd.dests.size() == 1);
auto& dest = tcd.dests[0];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
ASSERT_TRUE(dest.amount == 10000000000);
ASSERT_TRUE(cryptonote::get_account_address_as_str(nettype, false, dest.addr) == "T6TQ8id855U7YZFtT2wvCkPqCGhFujMfaE5NE15UWNKjMrxCHGsFLQPYbcSLjoF9xwYGFzbC6LdDw8Fhr5DNsjJe2cDkK1fSM");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
// key_images
ASSERT_TRUE(exported_txs.key_images.size() == 3);
auto& ki0 = exported_txs.key_images[0];
auto& ki1 = exported_txs.key_images[1];
auto& ki2 = exported_txs.key_images[2];
Fix serialization un/signed tx unit tests
2018-04-29 10:19:35 +02:00
Remove deprecated epee code
2020-10-23 22:32:28 +02:00
ASSERT_TRUE(tools::type_to_hex(ki0) == "05e1050df8262068682951b459a722495bfd5d070300e96a8d52c6255e300f11");
ASSERT_TRUE(tools::type_to_hex(ki1) == "21dfe89b3dbde221eccd9b71e7f6383c81f9ada224a670956c895b230749a8d8");
ASSERT_TRUE(tools::type_to_hex(ki2) == "92194cadfbb4f1317d25d39d6216cbf1030a2170a3edb47b5f008345a879150d");
portable serializer: tests added
2017-01-02 02:43:20 +01:00
}
Don't build i686 targets, disable portability wallet unit test This has broken multiple times, should be fixed when we can since the testnet genesis block has been updated.
2019-02-25 08:09:17 +01:00
#endif
Reference in a new issue Copy permalink