Overhaul binary serialization code (part 1)
The serialization interface here had a lot going wrong with it. This
overhauls it drastically and makes it a lot nicer to deal with.
(This commit is in two parts: this first one updating the base
serialization code, the subsequent one updating various places using
it. There's a third part, depending on how you want to count,
converting boost::variant to std::variant which relies on the
serialization changes made here).
- everything is now in the `serialization` namespace instead of having
some things there and other things in the root namespace.
- serialization failures now throw exceptions with reasons for the
failure rather than needing to snake a bool back through the call stack
(without any message).
- the `template <bool W, template <bool> class Archive>` monstrosity is
gone. Instead an Archive class has a Archive::is_serializer or
Archive::is_deserializer constexpr bool that can be checked (there was
something sort of similar before, but required messing around with
boost::mpl crap in both the generic and specific serialization code).
- the serialization code is significantly more flexible: instead of
having to slam everything into a class itself, you can also serialize
using with a free function in the same namespace as the class.
- serialization macros are still provided, but now considered
deprecated, replaced with (ADL callable) function names that don't hide
the action from the caller. So:
FIELD(a);
VARINT_FIELD(b);
FIELD_N("c", some_c);
VARINT_FIELD_N("d", some_d);
FIELDS(x); // (this is like the above, but doesn't write a tag)
ENUM_FIELD(e, e < myenum::_count);
FIELD(f);
if (!W && f != 42) return false;
becomes (all of this is documented in serialization.h):
field(ar, "a", a);
field_varint(ar, "b", b);
field(ar, "c", some_c);
field_varint(ar, "d", some_d);
value(x);
// enums just get passed to field_varint. It takes an optional
// lambda to verify on deserialization:
field_varint(ar, "e", f, [&] { return e < myenum::_count; });
// But the verifier isn't limited to enums:
field(ar, "f", f, [&] { return f == 42; });
and all of this works without needing to `serialization::` qualify the
beginning. In the rare case where you have a conflicting function
defined (e.g. a local `field()`) you can qualify to disambiguate.
- The messy eof hacks where you call `serialize_noeof` is cleaned up.
You now call `serialization::serialize(ar, val)` to deserialize an
*entire* value (which requires that the whole strem is consumed), and
`serialization::value(ar, val)` to append a serialization.
- Container serialization is significantly simplified; the various
serialization/vector.h (and similar) are now extremely thin wrappers
around the generic main container serialization code.
- INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and
OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly
identical yet more flexible (for both the caller and the compiler)
relatively simple templated functions.
- Drastically simplified the ability to serialize to/from a string via
new `serialization::binary_string_archiver` and
`serialization::binary_string_unarchiver` serializers. The former takes
no arguments; the latter takes a string_view. This means you can
serialize to binary using:
serialization::binary_string_archiver ar;
serialization::serialize(ar, myvalue);
std::string serialized = ar.str();
and can deserialize using:
MyType myvalue;
serialization::binary_string_unarchiver ar{serialized};
serialization::serialize(ar, myvalue);
(though really this interface is for slightly more complicated cases
than these; see the next point)
- the existing dump_binary() and parse_binary() are tweaked a bit: both
now throw, and dump_binary() returns the result string instead of taking
it as an output parameter. (parse_binary() is now void, rather than
returning a bool, because there are many places where in-place
deserialization is desirable).
- make one_shot_read_buffer internal to binary_string_unarchiver, and
use it there. The interfaces here means we no longer need to rely on
the seeking behaviour, so the serialization issues on mac shouldn't
happen now.
- begin_array()/begin_object() now use an RAII interface to make
serializing arrays much easier. Where previously we had a lot of code
that did something like this:
ar.begin_array();
for (auto it = whatever.begin(); it != whatever.end(); it++)
{
FIELDS(val);
if (*(it+1) != whatever.end())
{
ar.delimit_array();
}
}
ar.end_array();
Now an array serialization looks like this:
auto arr = ar.begin_array();
for (auto& val : whatever)
value(arr.element(), val);
- serialized non-varint integer values weren't endian safe, now they
are.
- variant serialization converted to use std::variant instead of
boost::variant and significantly cleaned up using C++17 features.
- varint (no "a", i.e. variable length integers) was not easy to follow
and badly documented (the given examples in the description were flat
out wrong). Rewrote it, with substantially better documentation and
unit tests, and taking advantage of C++14 `0b` and `'` in integer
literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
|
|
|
// Copyright (c) 2020, The Loki Project
|
2023-04-13 15:50:13 +02:00
|
|
|
//
|
2017-12-22 20:47:12 +01:00
|
|
|
// All rights reserved.
|
2023-04-13 15:50:13 +02:00
|
|
|
//
|
2017-12-22 20:47:12 +01:00
|
|
|
// Redistribution and use in source and binary forms, with or without modification, are
|
|
|
|
|
// permitted provided that the following conditions are met:
|
2023-04-13 15:50:13 +02:00
|
|
|
//
|
2017-12-22 20:47:12 +01:00
|
|
|
// 1. Redistributions of source code must retain the above copyright notice, this list of
|
|
|
|
|
// conditions and the following disclaimer.
|
2023-04-13 15:50:13 +02:00
|
|
|
//
|
2017-12-22 20:47:12 +01:00
|
|
|
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
|
|
|
|
|
// of conditions and the following disclaimer in the documentation and/or other
|
|
|
|
|
// materials provided with the distribution.
|
2023-04-13 15:50:13 +02:00
|
|
|
//
|
2017-12-22 20:47:12 +01:00
|
|
|
// 3. Neither the name of the copyright holder nor the names of its contributors may be
|
|
|
|
|
// used to endorse or promote products derived from this software without specific
|
|
|
|
|
// prior written permission.
|
2023-04-13 15:50:13 +02:00
|
|
|
//
|
2017-12-22 20:47:12 +01:00
|
|
|
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
|
|
|
|
|
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
|
|
|
|
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
|
|
|
|
|
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
|
|
|
|
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
|
|
|
|
|
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
|
|
|
|
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
|
|
|
|
|
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
|
|
|
|
|
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
|
|
|
|
|
|
#pragma once
|
|
|
|
|
|
|
|
|
|
#include <deque>
|
2023-04-13 15:50:13 +02:00
|
|
|
|
Overhaul binary serialization code (part 1)
The serialization interface here had a lot going wrong with it. This
overhauls it drastically and makes it a lot nicer to deal with.
(This commit is in two parts: this first one updating the base
serialization code, the subsequent one updating various places using
it. There's a third part, depending on how you want to count,
converting boost::variant to std::variant which relies on the
serialization changes made here).
- everything is now in the `serialization` namespace instead of having
some things there and other things in the root namespace.
- serialization failures now throw exceptions with reasons for the
failure rather than needing to snake a bool back through the call stack
(without any message).
- the `template <bool W, template <bool> class Archive>` monstrosity is
gone. Instead an Archive class has a Archive::is_serializer or
Archive::is_deserializer constexpr bool that can be checked (there was
something sort of similar before, but required messing around with
boost::mpl crap in both the generic and specific serialization code).
- the serialization code is significantly more flexible: instead of
having to slam everything into a class itself, you can also serialize
using with a free function in the same namespace as the class.
- serialization macros are still provided, but now considered
deprecated, replaced with (ADL callable) function names that don't hide
the action from the caller. So:
FIELD(a);
VARINT_FIELD(b);
FIELD_N("c", some_c);
VARINT_FIELD_N("d", some_d);
FIELDS(x); // (this is like the above, but doesn't write a tag)
ENUM_FIELD(e, e < myenum::_count);
FIELD(f);
if (!W && f != 42) return false;
becomes (all of this is documented in serialization.h):
field(ar, "a", a);
field_varint(ar, "b", b);
field(ar, "c", some_c);
field_varint(ar, "d", some_d);
value(x);
// enums just get passed to field_varint. It takes an optional
// lambda to verify on deserialization:
field_varint(ar, "e", f, [&] { return e < myenum::_count; });
// But the verifier isn't limited to enums:
field(ar, "f", f, [&] { return f == 42; });
and all of this works without needing to `serialization::` qualify the
beginning. In the rare case where you have a conflicting function
defined (e.g. a local `field()`) you can qualify to disambiguate.
- The messy eof hacks where you call `serialize_noeof` is cleaned up.
You now call `serialization::serialize(ar, val)` to deserialize an
*entire* value (which requires that the whole strem is consumed), and
`serialization::value(ar, val)` to append a serialization.
- Container serialization is significantly simplified; the various
serialization/vector.h (and similar) are now extremely thin wrappers
around the generic main container serialization code.
- INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and
OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly
identical yet more flexible (for both the caller and the compiler)
relatively simple templated functions.
- Drastically simplified the ability to serialize to/from a string via
new `serialization::binary_string_archiver` and
`serialization::binary_string_unarchiver` serializers. The former takes
no arguments; the latter takes a string_view. This means you can
serialize to binary using:
serialization::binary_string_archiver ar;
serialization::serialize(ar, myvalue);
std::string serialized = ar.str();
and can deserialize using:
MyType myvalue;
serialization::binary_string_unarchiver ar{serialized};
serialization::serialize(ar, myvalue);
(though really this interface is for slightly more complicated cases
than these; see the next point)
- the existing dump_binary() and parse_binary() are tweaked a bit: both
now throw, and dump_binary() returns the result string instead of taking
it as an output parameter. (parse_binary() is now void, rather than
returning a bool, because there are many places where in-place
deserialization is desirable).
- make one_shot_read_buffer internal to binary_string_unarchiver, and
use it there. The interfaces here means we no longer need to rely on
the seeking behaviour, so the serialization issues on mac shouldn't
happen now.
- begin_array()/begin_object() now use an RAII interface to make
serializing arrays much easier. Where previously we had a lot of code
that did something like this:
ar.begin_array();
for (auto it = whatever.begin(); it != whatever.end(); it++)
{
FIELDS(val);
if (*(it+1) != whatever.end())
{
ar.delimit_array();
}
}
ar.end_array();
Now an array serialization looks like this:
auto arr = ar.begin_array();
for (auto& val : whatever)
value(arr.element(), val);
- serialized non-varint integer values weren't endian safe, now they
are.
- variant serialization converted to use std::variant instead of
boost::variant and significantly cleaned up using C++17 features.
- varint (no "a", i.e. variable length integers) was not easy to follow
and badly documented (the given examples in the description were flat
out wrong). Rewrote it, with substantially better documentation and
unit tests, and taking advantage of C++14 `0b` and `'` in integer
literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
|
|
|
#include "container.h"
|
2017-12-22 20:47:12 +01:00
|
|
|
|
2023-04-13 15:50:13 +02:00
|
|
|
namespace serialization {
|
2017-12-22 20:47:12 +01:00
|
|
|
|
Overhaul binary serialization code (part 1)
The serialization interface here had a lot going wrong with it. This
overhauls it drastically and makes it a lot nicer to deal with.
(This commit is in two parts: this first one updating the base
serialization code, the subsequent one updating various places using
it. There's a third part, depending on how you want to count,
converting boost::variant to std::variant which relies on the
serialization changes made here).
- everything is now in the `serialization` namespace instead of having
some things there and other things in the root namespace.
- serialization failures now throw exceptions with reasons for the
failure rather than needing to snake a bool back through the call stack
(without any message).
- the `template <bool W, template <bool> class Archive>` monstrosity is
gone. Instead an Archive class has a Archive::is_serializer or
Archive::is_deserializer constexpr bool that can be checked (there was
something sort of similar before, but required messing around with
boost::mpl crap in both the generic and specific serialization code).
- the serialization code is significantly more flexible: instead of
having to slam everything into a class itself, you can also serialize
using with a free function in the same namespace as the class.
- serialization macros are still provided, but now considered
deprecated, replaced with (ADL callable) function names that don't hide
the action from the caller. So:
FIELD(a);
VARINT_FIELD(b);
FIELD_N("c", some_c);
VARINT_FIELD_N("d", some_d);
FIELDS(x); // (this is like the above, but doesn't write a tag)
ENUM_FIELD(e, e < myenum::_count);
FIELD(f);
if (!W && f != 42) return false;
becomes (all of this is documented in serialization.h):
field(ar, "a", a);
field_varint(ar, "b", b);
field(ar, "c", some_c);
field_varint(ar, "d", some_d);
value(x);
// enums just get passed to field_varint. It takes an optional
// lambda to verify on deserialization:
field_varint(ar, "e", f, [&] { return e < myenum::_count; });
// But the verifier isn't limited to enums:
field(ar, "f", f, [&] { return f == 42; });
and all of this works without needing to `serialization::` qualify the
beginning. In the rare case where you have a conflicting function
defined (e.g. a local `field()`) you can qualify to disambiguate.
- The messy eof hacks where you call `serialize_noeof` is cleaned up.
You now call `serialization::serialize(ar, val)` to deserialize an
*entire* value (which requires that the whole strem is consumed), and
`serialization::value(ar, val)` to append a serialization.
- Container serialization is significantly simplified; the various
serialization/vector.h (and similar) are now extremely thin wrappers
around the generic main container serialization code.
- INSERT_INTO_JSON_OBJECT, GET_FROM_JSON_OBJECT, and
OBJECT_HAS_MEMBER_OR_THROW macros are gone, replaced with nearly
identical yet more flexible (for both the caller and the compiler)
relatively simple templated functions.
- Drastically simplified the ability to serialize to/from a string via
new `serialization::binary_string_archiver` and
`serialization::binary_string_unarchiver` serializers. The former takes
no arguments; the latter takes a string_view. This means you can
serialize to binary using:
serialization::binary_string_archiver ar;
serialization::serialize(ar, myvalue);
std::string serialized = ar.str();
and can deserialize using:
MyType myvalue;
serialization::binary_string_unarchiver ar{serialized};
serialization::serialize(ar, myvalue);
(though really this interface is for slightly more complicated cases
than these; see the next point)
- the existing dump_binary() and parse_binary() are tweaked a bit: both
now throw, and dump_binary() returns the result string instead of taking
it as an output parameter. (parse_binary() is now void, rather than
returning a bool, because there are many places where in-place
deserialization is desirable).
- make one_shot_read_buffer internal to binary_string_unarchiver, and
use it there. The interfaces here means we no longer need to rely on
the seeking behaviour, so the serialization issues on mac shouldn't
happen now.
- begin_array()/begin_object() now use an RAII interface to make
serializing arrays much easier. Where previously we had a lot of code
that did something like this:
ar.begin_array();
for (auto it = whatever.begin(); it != whatever.end(); it++)
{
FIELDS(val);
if (*(it+1) != whatever.end())
{
ar.delimit_array();
}
}
ar.end_array();
Now an array serialization looks like this:
auto arr = ar.begin_array();
for (auto& val : whatever)
value(arr.element(), val);
- serialized non-varint integer values weren't endian safe, now they
are.
- variant serialization converted to use std::variant instead of
boost::variant and significantly cleaned up using C++17 features.
- varint (no "a", i.e. variable length integers) was not easy to follow
and badly documented (the given examples in the description were flat
out wrong). Rewrote it, with substantially better documentation and
unit tests, and taking advantage of C++14 `0b` and `'` in integer
literals to make it far easier to follow.
2020-06-02 02:46:09 +02:00
|
|
|
template <class Archive, class T>
|
2023-04-13 15:50:13 +02:00
|
|
|
void serialize_value(Archive& ar, std::deque<T>& v) {
|
|
|
|
|
detail::serialize_container(ar, v);
|
2017-12-22 20:47:12 +01:00
|
|
|
}
|
|
|
|
|
|
2023-04-13 15:50:13 +02:00
|
|
|
} // namespace serialization
|
