From 0ab050647f21058aab13cf92cbac624cccb9b191 Mon Sep 17 00:00:00 2001 From: Jeff Becker Date: Thu, 6 Oct 2022 16:23:34 -0400 Subject: [PATCH 1/4] overview of refactor plan and new component structure. --- docs/refactor_notes.md | 97 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 docs/refactor_notes.md diff --git a/docs/refactor_notes.md b/docs/refactor_notes.md new file mode 100644 index 000000000..efc186ff4 --- /dev/null +++ b/docs/refactor_notes.md @@ -0,0 +1,97 @@ +# High Level Iterative Approach + +the desired outcome of this refactor will be splitting the existing code up into a stack of new components. +a layer hides all functionality of the layer below it to reduce the complexity like the OSI stack intends to. +the refactor starts at the top layer, wiring up the old implementation piecewise to the top layer. +once the top layer is wired up to the old implementation we will move down to the next layer. +this will repeat until we reach the bottom layer. +once the old implementation is wired up into these new clearly defined layers, we can fixup or replace different parts of each layer one at a time as needed. + +working down from each layer will let us pick apart the old implementation (if needed) that we would wire up to the new base classes for that layer we are defining now without worrying about what is below it (yet). + +this refactor is very able to be split up into small work units that (ideally) do not confict with each other. + + +PDU: https://en.wikipedia.org/wiki/Protocol_data_unit + +# The New Layers + +from top to bottom the new layers are: + +* Platform Layer +* Flow Layer +* Routing Layer +* Onion Layer +* Link Layer +* Wire Layer + + +## Platform Layer + +this is the top layer, it is responsibile ONLY to act as a handler of reading data from the "user" (via tun interface or whatever) to forward to the flow layer as desired, and to take data from the flow layer and send it to the "user". +any kind of IP/dns mapping or traffic isolation details are done here. embedded lokinet would be implemented in this layer as well, as it is without a full tun interface. + +Platform layer PDU are what the OS gives us and we internally convert them into flow layer PDU and hand them off to the flow layer. + + +## Flow Layer + +this layer is tl;dr mean to multiplex data from the platform layer across the routing layer and propagating PDU from the routing to the platform layer if needed. + +the flow layer is responsible for sending platform layer PDU across path we have already established. +this layer is informed by the routing layer below it of state changes in what paths are available for use. +the flow layer requests from the layer below to make new paths if it wishes to get new ones on demand. +this layer will recieve routing layer PDU from the routing layer and apply any congestion control needed to buffer things to the os if it is needed at all. + +flow layer PDU are (data, ethertype, src-pubkey, dst-pubkey, isolation-metric) tuples. +data is the datum we are tunneling over lokinet. ethertype tells us what kind of datum this is, e.g. plainquic/ipv4/ipv6/auth/etc. +src-pubkey and dst-pubkey are public the ed25519 public keys of each end of the flow in use. +the isolation metric is a piece of metadata we use to distinguish unique flows (convotag). in this new seperation convotags explicitly do not hand over across paths. + + +## Routing Layer + +this layer is tl;dr meant for path management but not path building. + +the routing layer is responsible for sending/recieving flow layer PDU, DHT requests/responses, latency testing PDU and any other kind of PDU we send/recieve over the onion layer. +this layer will be responsible for managing paths we have already built across lokinet. +the routing layer will periodically measure path status/latency, and do any other kinds of perioidic path related tasks post build. +this layer when asked for a new path from the flow layer will use one that has been prebuilt already and if the number of prebuilt paths is below a threshold we will tell the onion layer to build more paths. +the routing layer will recieve path build results be their success/fail/timeout from the onion layer that were requested and apply any congestion control needed at the pivot router. + +routing layer PDU are (data, src-path, dst-path) tuples. +data is the datum we are transferring between paths. +src-path and dst-path are (pathid, router id) tuples, the source being which path this routing layer PDU originated from, destination being which path it is going to. +in the old model, router id is always the router that recieves it as the pivot router, this remains the same unless we explicitly provide router-id. +this lets us propagate hints to DHT related PDU held inside the datum. + + +## Onion Layer + +the onion layer is repsonsible for path builds, path selection logic and low level details of encrypted/decrypting PDU that are onion routed over paths. +this layer is requested by the routing layer to build a path to a pivot router with an optional additional constraints (e.g. unique cidr/operator/geoip/etc, latency constaints, hop length, path lifetime). +the onion layer will encrypt PDU and send them to link layer as (frame/edge router id) tuples, and recieve link layer frames from edge routers, decrypt them and propagate them as needed to the routing layer. +this layer also handles transit onion traffic and transit path build responsibilities as a snode and apply congestion control as needed per transit path. + +the onion layer PDU are (data, src-path, dst-path) tuples. +src-path and dst-path are (router-id, path-id) tuples which contain the ed25519 pubkey of the node and the 128 bit path-id it was associated with. +data is some datum we are onion routing that we would apply symettric encryption as needed before propagating to upper or lower layers. + + +## Link Layer + +the link layer is responsbile for transmission of frames between nodes. +this layer will handle queuing and congestion control between wire proto sessions between nodes. +the link layer is will initate and recieve wire session to/from remote nodes. + +the link layer PDU is (data, src-router-id, dst-router-id) tuples. +data is a datum of a link layer frame. +src-router-id and dst-router-id are (ed25519-pubkey, net-addr, wire-proto-info) tuples. +the ed25519 pubkey is a .snode address, (clients have these too but they are ephemeral). +net-addr is an (ip, port) tuple the node is reachable via the wire protocol. +wire-proto-info is dialect specific wire protocol specific info. + +## Wire Layer + +the wire layer is responsible for transmitting link layer frames between nodes. +all details here are specific to each wire proto dialect. From 4508c59cd30d02862e59e47139c241c270830809 Mon Sep 17 00:00:00 2001 From: Jeff Becker Date: Mon, 9 Jan 2023 12:47:41 -0500 Subject: [PATCH 2/4] redo includes to be consistent --- llarp/config/config.cpp | 6 ++-- llarp/config/config.hpp | 6 ++-- llarp/constants/version.cpp.in | 4 +-- llarp/crypto/constants.hpp | 2 +- llarp/dns/rr.cpp | 2 +- llarp/exit/session.hpp | 2 +- llarp/handlers/exit.cpp | 2 +- llarp/layers/endpoint/kitchen_sink.cpp | 0 llarp/layers/flow/kitchen_sink.cpp | 0 llarp/messages/relay_commit.hpp | 2 +- llarp/path/pathbuilder.cpp | 4 +-- llarp/path/pathset.hpp | 2 +- llarp/path/transit_hop.cpp | 4 ++- llarp/path/transit_hop.hpp | 4 +-- llarp/quic/tunnel.cpp | 6 ++-- llarp/router/route_poker.cpp | 4 +-- llarp/service/convotag.cpp | 2 +- llarp/service/endpoint.cpp | 21 +++++++------- llarp/service/endpoint.hpp | 29 ++++++++++--------- llarp/service/intro.cpp | 2 +- llarp/service/outbound_context.cpp | 9 +++--- llarp/tooling/router_hive.hpp | 4 +-- llarp/vpn/win32.cpp | 2 +- pybind/llarp/context.cpp | 3 +- test/CMakeLists.txt | 4 --- test/check_main.cpp | 4 +-- test/config/test_llarp_config_definition.cpp | 2 +- test/config/test_llarp_config_ini.cpp | 2 +- test/config/test_llarp_config_output.cpp | 2 +- test/crypto/test_llarp_crypto.cpp | 2 +- test/crypto/test_llarp_crypto_types.cpp | 4 +-- test/crypto/test_llarp_key_manager.cpp | 11 +++---- test/dns/test_llarp_dns_dns.cpp | 14 ++++----- test/llarp_test.hpp | 2 +- test/net/test_ip_address.cpp | 2 +- test/net/test_llarp_net.cpp | 8 ++--- test/net/test_sock_addr.cpp | 6 ++-- test/nodedb/test_nodedb.cpp | 6 ++-- test/path/test_path.cpp | 2 +- test/peerstats/test_peer_db.cpp | 10 +++---- test/peerstats/test_peer_types.cpp | 2 +- test/router/test_llarp_router_version.cpp | 4 +-- .../test_llarp_routing_obtainexitmessage.cpp | 9 +++--- .../test_llarp_routing_transfer_traffic.cpp | 2 +- test/service/test_llarp_service_address.cpp | 2 +- test/service/test_llarp_service_identity.cpp | 16 +++++----- test/service/test_llarp_service_name.cpp | 4 +-- test/test_llarp_encrypted_frame.cpp | 15 ++++------ test/test_llarp_router_contact.cpp | 8 ++--- test/test_util.cpp | 2 +- test/test_util.hpp | 5 ++-- test/util/meta/test_llarp_util_memfn.cpp | 2 +- test/util/test_llarp_util_aligned.cpp | 2 +- test/util/test_llarp_util_bencode.cpp | 4 +-- test/util/test_llarp_util_bits.cpp | 2 +- .../util/test_llarp_util_decaying_hashset.cpp | 4 +-- test/util/test_llarp_util_log_level.cpp | 4 +-- test/util/test_llarp_util_str.cpp | 2 +- test/util/thread/test_llarp_util_queue.cpp | 6 ++-- .../thread/test_llarp_util_queue_manager.cpp | 2 +- 60 files changed, 148 insertions(+), 152 deletions(-) create mode 100644 llarp/layers/endpoint/kitchen_sink.cpp create mode 100644 llarp/layers/flow/kitchen_sink.cpp diff --git a/llarp/config/config.cpp b/llarp/config/config.cpp index 98a5f1571..eda628dda 100644 --- a/llarp/config/config.cpp +++ b/llarp/config/config.cpp @@ -1,8 +1,7 @@ -#include #include "config.hpp" - -#include "config/definition.hpp" +#include "definition.hpp" #include "ini.hpp" + #include #include #include @@ -18,6 +17,7 @@ #include +#include #include #include #include diff --git a/llarp/config/config.hpp b/llarp/config/config.hpp index 76540f19f..3165f0354 100644 --- a/llarp/config/config.hpp +++ b/llarp/config/config.hpp @@ -1,14 +1,15 @@ #pragma once +#include "ini.hpp" +#include "definition.hpp" #include + #include #include #include #include #include #include -#include "ini.hpp" -#include "definition.hpp" #include #include #include @@ -16,7 +17,6 @@ #include #include #include - #include #include diff --git a/llarp/constants/version.cpp.in b/llarp/constants/version.cpp.in index bca06675c..e07223705 100644 --- a/llarp/constants/version.cpp.in +++ b/llarp/constants/version.cpp.in @@ -1,5 +1,5 @@ -#include -#include +#include +#include namespace llarp { diff --git a/llarp/crypto/constants.hpp b/llarp/crypto/constants.hpp index 7f8eef90f..1d039bbc2 100644 --- a/llarp/crypto/constants.hpp +++ b/llarp/crypto/constants.hpp @@ -2,7 +2,7 @@ #include -#include +#include static constexpr uint32_t PUBKEYSIZE = 32; static constexpr uint32_t SECKEYSIZE = 64; diff --git a/llarp/dns/rr.cpp b/llarp/dns/rr.cpp index 495656cb2..8851ace30 100644 --- a/llarp/dns/rr.cpp +++ b/llarp/dns/rr.cpp @@ -1,6 +1,6 @@ #include "rr.hpp" #include "dns.hpp" -#include "util/formattable.hpp" +#include #include #include diff --git a/llarp/exit/session.hpp b/llarp/exit/session.hpp index def9115ac..7191cc104 100644 --- a/llarp/exit/session.hpp +++ b/llarp/exit/session.hpp @@ -1,7 +1,7 @@ #pragma once #include "exit_messages.hpp" -#include "service/protocol_type.hpp" +#include #include #include #include diff --git a/llarp/handlers/exit.cpp b/llarp/handlers/exit.cpp index 64e002e3a..a7840c090 100644 --- a/llarp/handlers/exit.cpp +++ b/llarp/handlers/exit.cpp @@ -11,7 +11,7 @@ #include #include -#include "service/protocol_type.hpp" +#include namespace llarp { diff --git a/llarp/layers/endpoint/kitchen_sink.cpp b/llarp/layers/endpoint/kitchen_sink.cpp new file mode 100644 index 000000000..e69de29bb diff --git a/llarp/layers/flow/kitchen_sink.cpp b/llarp/layers/flow/kitchen_sink.cpp new file mode 100644 index 000000000..e69de29bb diff --git a/llarp/messages/relay_commit.hpp b/llarp/messages/relay_commit.hpp index b019c1e53..164933917 100644 --- a/llarp/messages/relay_commit.hpp +++ b/llarp/messages/relay_commit.hpp @@ -2,7 +2,7 @@ #include #include -#include "link_message.hpp" +#include #include #include diff --git a/llarp/path/pathbuilder.cpp b/llarp/path/pathbuilder.cpp index abf149024..1710b9d33 100644 --- a/llarp/path/pathbuilder.cpp +++ b/llarp/path/pathbuilder.cpp @@ -1,10 +1,10 @@ #include "pathbuilder.hpp" +#include "path_context.hpp" #include #include #include -#include "path_context.hpp" -#include "util/logging.hpp" +#include #include #include #include diff --git a/llarp/path/pathset.hpp b/llarp/path/pathset.hpp index 3e431bf41..f7fe5690e 100644 --- a/llarp/path/pathset.hpp +++ b/llarp/path/pathset.hpp @@ -1,7 +1,7 @@ #pragma once #include "path_types.hpp" -#include "service/protocol_type.hpp" +#include #include #include #include diff --git a/llarp/path/transit_hop.cpp b/llarp/path/transit_hop.cpp index 4efe3540f..ff912aae0 100644 --- a/llarp/path/transit_hop.cpp +++ b/llarp/path/transit_hop.cpp @@ -33,7 +33,9 @@ namespace llarp } TransitHop::TransitHop() - : m_UpstreamGather(transit_hop_queue_size), m_DownstreamGather(transit_hop_queue_size) + : IHopHandler{} + , m_UpstreamGather{transit_hop_queue_size} + , m_DownstreamGather{transit_hop_queue_size} { m_UpstreamGather.enable(); m_DownstreamGather.enable(); diff --git a/llarp/path/transit_hop.hpp b/llarp/path/transit_hop.hpp index eb828af9b..fb628929f 100644 --- a/llarp/path/transit_hop.hpp +++ b/llarp/path/transit_hop.hpp @@ -1,8 +1,8 @@ #pragma once #include -#include "ihophandler.hpp" -#include "path_types.hpp" +#include +#include #include #include #include diff --git a/llarp/quic/tunnel.cpp b/llarp/quic/tunnel.cpp index 9b09e5e2f..914b963ea 100644 --- a/llarp/quic/tunnel.cpp +++ b/llarp/quic/tunnel.cpp @@ -1,7 +1,7 @@ #include "tunnel.hpp" -#include "service/convotag.hpp" -#include "service/endpoint.hpp" -#include "service/name.hpp" +#include +#include +#include #include "stream.hpp" #include #include diff --git a/llarp/router/route_poker.cpp b/llarp/router/route_poker.cpp index 3fdc0e4bb..1471ac911 100644 --- a/llarp/router/route_poker.cpp +++ b/llarp/router/route_poker.cpp @@ -1,6 +1,6 @@ #include "route_poker.hpp" -#include "abstractrouter.hpp" -#include "net/sock_addr.hpp" +#include +#include #include #include #include diff --git a/llarp/service/convotag.cpp b/llarp/service/convotag.cpp index 727751f00..b4b4eec61 100644 --- a/llarp/service/convotag.cpp +++ b/llarp/service/convotag.cpp @@ -1,5 +1,5 @@ #include "convotag.hpp" -#include "net/ip.hpp" +#include namespace llarp::service { diff --git a/llarp/service/endpoint.cpp b/llarp/service/endpoint.cpp index 24839ff4d..7768c8685 100644 --- a/llarp/service/endpoint.cpp +++ b/llarp/service/endpoint.cpp @@ -1,7 +1,15 @@ #include #include #include "endpoint.hpp" +#include "endpoint_state.hpp" +#include "endpoint_util.hpp" +#include "hidden_service_address_lookup.hpp" +#include "outbound_context.hpp" +#include "protocol.hpp" +#include "info.hpp" +#include "protocol_type.hpp" +#include #include #include #include @@ -16,27 +24,18 @@ #include #include #include -#include "endpoint_state.hpp" -#include "endpoint_util.hpp" -#include "hidden_service_address_lookup.hpp" -#include "net/ip.hpp" -#include "outbound_context.hpp" -#include "protocol.hpp" -#include "service/info.hpp" -#include "service/protocol_type.hpp" + #include #include #include #include #include +#include #include #include #include #include - -#include -#include #include #include diff --git a/llarp/service/endpoint.hpp b/llarp/service/endpoint.hpp index dd61c74b9..619f58fdd 100644 --- a/llarp/service/endpoint.hpp +++ b/llarp/service/endpoint.hpp @@ -6,24 +6,27 @@ #include #include #include -#include "address.hpp" -#include "handler.hpp" -#include "identity.hpp" -#include "pendingbuffer.hpp" -#include "protocol.hpp" -#include "sendcontext.hpp" -#include "service/protocol_type.hpp" -#include "session.hpp" -#include "lookup.hpp" #include + +// --- begin kitchen sink headers ---- +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +// ----- end kitchen sink headers ----- + #include #include #include #include -#include "endpoint_types.hpp" -#include "llarp/endpoint_base.hpp" - -#include "auth.hpp" #include #include diff --git a/llarp/service/intro.cpp b/llarp/service/intro.cpp index 54de28805..026d49dee 100644 --- a/llarp/service/intro.cpp +++ b/llarp/service/intro.cpp @@ -1,5 +1,5 @@ #include "intro.hpp" -#include "util/time.hpp" +#include namespace llarp { diff --git a/llarp/service/outbound_context.cpp b/llarp/service/outbound_context.cpp index 3494bceee..28b4a37af 100644 --- a/llarp/service/outbound_context.cpp +++ b/llarp/service/outbound_context.cpp @@ -1,16 +1,15 @@ #include "outbound_context.hpp" - -#include #include "async_key_exchange.hpp" #include "hidden_service_address_lookup.hpp" #include "endpoint.hpp" +#include "endpoint_util.hpp" +#include "protocol_type.hpp" + +#include #include #include #include -#include "endpoint_util.hpp" -#include "service/protocol_type.hpp" - #include #include diff --git a/llarp/tooling/router_hive.hpp b/llarp/tooling/router_hive.hpp index 31bfdffcf..760205ea6 100644 --- a/llarp/tooling/router_hive.hpp +++ b/llarp/tooling/router_hive.hpp @@ -3,8 +3,8 @@ #include "router_event.hpp" #include -#include -#include +#include +#include #include #include diff --git a/llarp/vpn/win32.cpp b/llarp/vpn/win32.cpp index 1562cef1c..f6f4a452e 100644 --- a/llarp/vpn/win32.cpp +++ b/llarp/vpn/win32.cpp @@ -1,4 +1,4 @@ -#include "vpn/win32.hpp" +#include "win32.hpp" #include #include #include diff --git a/pybind/llarp/context.cpp b/pybind/llarp/context.cpp index db9fad6f3..9a1cf194b 100644 --- a/pybind/llarp/context.cpp +++ b/pybind/llarp/context.cpp @@ -3,7 +3,8 @@ #include #include #include -#include "service/protocol_type.hpp" +#include + namespace llarp { void diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index c11104139..e6270599e 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -60,8 +60,4 @@ if(WIN32) target_link_libraries(testAll PUBLIC ws2_32 iphlpapi shlwapi) endif() -if(${CMAKE_SYSTEM_NAME} MATCHES "FreeBSD") - target_link_directories(testAll PRIVATE /usr/local/lib) -endif() - add_custom_target(check COMMAND testAll) diff --git a/test/check_main.cpp b/test/check_main.cpp index 87299c833..c98dadd9a 100644 --- a/test/check_main.cpp +++ b/test/check_main.cpp @@ -1,8 +1,8 @@ #define CATCH_CONFIG_RUNNER #include -#include -#include +#include +#include #ifdef _WIN32 #include diff --git a/test/config/test_llarp_config_definition.cpp b/test/config/test_llarp_config_definition.cpp index 80005d58c..29cd9b795 100644 --- a/test/config/test_llarp_config_definition.cpp +++ b/test/config/test_llarp_config_definition.cpp @@ -1,4 +1,4 @@ -#include +#include #include diff --git a/test/config/test_llarp_config_ini.cpp b/test/config/test_llarp_config_ini.cpp index 06c187e2c..b876e0d20 100644 --- a/test/config/test_llarp_config_ini.cpp +++ b/test/config/test_llarp_config_ini.cpp @@ -1,4 +1,4 @@ -#include +#include #include diff --git a/test/config/test_llarp_config_output.cpp b/test/config/test_llarp_config_output.cpp index 5ba47bf21..3324ef0f6 100644 --- a/test/config/test_llarp_config_output.cpp +++ b/test/config/test_llarp_config_output.cpp @@ -1,4 +1,4 @@ -#include +#include #include diff --git a/test/crypto/test_llarp_crypto.cpp b/test/crypto/test_llarp_crypto.cpp index 13105a87e..b78752375 100644 --- a/test/crypto/test_llarp_crypto.cpp +++ b/test/crypto/test_llarp_crypto.cpp @@ -1,4 +1,4 @@ -#include +#include #include diff --git a/test/crypto/test_llarp_crypto_types.cpp b/test/crypto/test_llarp_crypto_types.cpp index 8477352f7..359f1911b 100644 --- a/test/crypto/test_llarp_crypto_types.cpp +++ b/test/crypto/test_llarp_crypto_types.cpp @@ -1,9 +1,9 @@ -#include +#include #include #include -#include +#include "test_util.hpp" #include extern "C" { diff --git a/test/crypto/test_llarp_key_manager.cpp b/test/crypto/test_llarp_key_manager.cpp index e5082bf58..459ed309b 100644 --- a/test/crypto/test_llarp_key_manager.cpp +++ b/test/crypto/test_llarp_key_manager.cpp @@ -1,14 +1,15 @@ -#include +#include "llarp_test.hpp" +#include "test_util.hpp" -#include -#include -#include +#include + +#include +#include #include #include #include -#include #include using namespace ::llarp; diff --git a/test/dns/test_llarp_dns_dns.cpp b/test/dns/test_llarp_dns_dns.cpp index ff800b4ec..e130dc634 100644 --- a/test/dns/test_llarp_dns_dns.cpp +++ b/test/dns/test_llarp_dns_dns.cpp @@ -1,11 +1,11 @@ #include -#include -#include -#include -#include -#include -#include -#include +#include +#include +#include +#include +#include +#include +#include #include diff --git a/test/llarp_test.hpp b/test/llarp_test.hpp index 8199ff569..914c4a224 100644 --- a/test/llarp_test.hpp +++ b/test/llarp_test.hpp @@ -1,6 +1,6 @@ #pragma once -#include +#include #include namespace llarp::test diff --git a/test/net/test_ip_address.cpp b/test/net/test_ip_address.cpp index 1366ebb76..8cbc26b1e 100644 --- a/test/net/test_ip_address.cpp +++ b/test/net/test_ip_address.cpp @@ -1,4 +1,4 @@ -#include +#include #include diff --git a/test/net/test_llarp_net.cpp b/test/net/test_llarp_net.cpp index 0c237a90b..90f2f901c 100644 --- a/test/net/test_llarp_net.cpp +++ b/test/net/test_llarp_net.cpp @@ -1,7 +1,7 @@ -#include -#include -#include -#include +#include +#include +#include +#include #include #include diff --git a/test/net/test_sock_addr.cpp b/test/net/test_sock_addr.cpp index 9d13b48a4..c1c175d7e 100644 --- a/test/net/test_sock_addr.cpp +++ b/test/net/test_sock_addr.cpp @@ -1,6 +1,6 @@ -#include -#include -#include +#include +#include +#include #include #include diff --git a/test/nodedb/test_nodedb.cpp b/test/nodedb/test_nodedb.cpp index 2d23bd27d..9286e345b 100644 --- a/test/nodedb/test_nodedb.cpp +++ b/test/nodedb/test_nodedb.cpp @@ -1,8 +1,8 @@ #include -#include "config/config.hpp" -#include -#include +#include +#include +#include using llarp_nodedb = llarp::NodeDB; diff --git a/test/path/test_path.cpp b/test/path/test_path.cpp index d5ad3d56c..d25a1c68e 100644 --- a/test/path/test_path.cpp +++ b/test/path/test_path.cpp @@ -1,4 +1,4 @@ -#include +#include #include using Path_t = llarp::path::Path; diff --git a/test/peerstats/test_peer_db.cpp b/test/peerstats/test_peer_db.cpp index 52889514c..c3ed14c79 100644 --- a/test/peerstats/test_peer_db.cpp +++ b/test/peerstats/test_peer_db.cpp @@ -1,12 +1,12 @@ -#include +#include #include #include #include -#include "peerstats/types.hpp" -#include "router_contact.hpp" -#include "util/logging.hpp" -#include "util/time.hpp" +#include +#include +#include +#include TEST_CASE("Test PeerDb PeerStats memory storage", "[PeerDb]") { diff --git a/test/peerstats/test_peer_types.cpp b/test/peerstats/test_peer_types.cpp index 46b8bc834..b2c398563 100644 --- a/test/peerstats/test_peer_types.cpp +++ b/test/peerstats/test_peer_types.cpp @@ -1,5 +1,5 @@ #include -#include +#include #include #include diff --git a/test/router/test_llarp_router_version.cpp b/test/router/test_llarp_router_version.cpp index 260a16198..8f00ef03c 100644 --- a/test/router/test_llarp_router_version.cpp +++ b/test/router/test_llarp_router_version.cpp @@ -1,5 +1,5 @@ -#include -#include "router/router.hpp" +#include +#include #include diff --git a/test/routing/test_llarp_routing_obtainexitmessage.cpp b/test/routing/test_llarp_routing_obtainexitmessage.cpp index fca9842e6..631652072 100644 --- a/test/routing/test_llarp_routing_obtainexitmessage.cpp +++ b/test/routing/test_llarp_routing_obtainexitmessage.cpp @@ -1,8 +1,7 @@ -#include - -#include -#include -#include +#include "llarp_test.hpp" +#include +#include +#include #include diff --git a/test/routing/test_llarp_routing_transfer_traffic.cpp b/test/routing/test_llarp_routing_transfer_traffic.cpp index 915aecef0..3f7b44dee 100644 --- a/test/routing/test_llarp_routing_transfer_traffic.cpp +++ b/test/routing/test_llarp_routing_transfer_traffic.cpp @@ -1,4 +1,4 @@ -#include +#include #include diff --git a/test/service/test_llarp_service_address.cpp b/test/service/test_llarp_service_address.cpp index f95e398c9..62c6cd0d7 100644 --- a/test/service/test_llarp_service_address.cpp +++ b/test/service/test_llarp_service_address.cpp @@ -1,4 +1,4 @@ -#include +#include #include diff --git a/test/service/test_llarp_service_identity.cpp b/test/service/test_llarp_service_identity.cpp index 2fe8db9d7..560754280 100644 --- a/test/service/test_llarp_service_identity.cpp +++ b/test/service/test_llarp_service_identity.cpp @@ -1,13 +1,13 @@ -#include -#include +#include +#include #include -#include -#include -#include -#include -#include +#include +#include +#include +#include +#include -#include +#include "test_util.hpp" #include using namespace llarp; diff --git a/test/service/test_llarp_service_name.cpp b/test/service/test_llarp_service_name.cpp index 3f9daef61..86f0324f3 100644 --- a/test/service/test_llarp_service_name.cpp +++ b/test/service/test_llarp_service_name.cpp @@ -1,6 +1,6 @@ #include "catch2/catch.hpp" -#include -#include +#include +#include #include using namespace std::literals; diff --git a/test/test_llarp_encrypted_frame.cpp b/test/test_llarp_encrypted_frame.cpp index a1e850644..6648ba9b1 100644 --- a/test/test_llarp_encrypted_frame.cpp +++ b/test/test_llarp_encrypted_frame.cpp @@ -1,12 +1,9 @@ -#include - -#include -#include -#include -#include - -#include - +#include "llarp_test.hpp" +#include "test_util.hpp" +#include +#include +#include +#include #include using namespace ::llarp; diff --git a/test/test_llarp_router_contact.cpp b/test/test_llarp_router_contact.cpp index eac96f332..f94e7904b 100644 --- a/test/test_llarp_router_contact.cpp +++ b/test/test_llarp_router_contact.cpp @@ -1,9 +1,9 @@ #include -#include -#include -#include -#include +#include +#include +#include +#include #include namespace diff --git a/test/test_util.cpp b/test/test_util.cpp index 100af14a9..b73a0e445 100644 --- a/test/test_util.cpp +++ b/test/test_util.cpp @@ -1,4 +1,4 @@ -#include +#include "test_util.hpp" #include diff --git a/test/test_util.hpp b/test/test_util.hpp index 4433789ab..36f09a912 100644 --- a/test/test_util.hpp +++ b/test/test_util.hpp @@ -1,9 +1,8 @@ #ifndef TEST_UTIL_HPP #define TEST_UTIL_HPP -#include -#include - +#include +#include #include #include diff --git a/test/util/meta/test_llarp_util_memfn.cpp b/test/util/meta/test_llarp_util_memfn.cpp index 165115570..a75cde7c5 100644 --- a/test/util/meta/test_llarp_util_memfn.cpp +++ b/test/util/meta/test_llarp_util_memfn.cpp @@ -1,4 +1,4 @@ -#include +#include #include diff --git a/test/util/test_llarp_util_aligned.cpp b/test/util/test_llarp_util_aligned.cpp index 970a676d5..3e6cc0c5a 100644 --- a/test/util/test_llarp_util_aligned.cpp +++ b/test/util/test_llarp_util_aligned.cpp @@ -1,6 +1,6 @@ #include -#include +#include #include #include diff --git a/test/util/test_llarp_util_bencode.cpp b/test/util/test_llarp_util_bencode.cpp index 890ddce30..cff95e7d2 100644 --- a/test/util/test_llarp_util_bencode.cpp +++ b/test/util/test_llarp_util_bencode.cpp @@ -1,5 +1,5 @@ -#include -#include +#include +#include #include #include diff --git a/test/util/test_llarp_util_bits.cpp b/test/util/test_llarp_util_bits.cpp index befa3965f..e24eb33a4 100644 --- a/test/util/test_llarp_util_bits.cpp +++ b/test/util/test_llarp_util_bits.cpp @@ -1,5 +1,5 @@ #include -#include +#include using namespace llarp::bits; diff --git a/test/util/test_llarp_util_decaying_hashset.cpp b/test/util/test_llarp_util_decaying_hashset.cpp index 28cc9371c..5a9339bd8 100644 --- a/test/util/test_llarp_util_decaying_hashset.cpp +++ b/test/util/test_llarp_util_decaying_hashset.cpp @@ -1,5 +1,5 @@ -#include -#include +#include +#include #include TEST_CASE("DecayingHashSet test decay static time", "[decaying-hashset]") diff --git a/test/util/test_llarp_util_log_level.cpp b/test/util/test_llarp_util_log_level.cpp index 048c6324f..8335e986a 100644 --- a/test/util/test_llarp_util_log_level.cpp +++ b/test/util/test_llarp_util_log_level.cpp @@ -1,6 +1,6 @@ #include -#include -#include +#include +#include #include using TestString = std::string; diff --git a/test/util/test_llarp_util_str.cpp b/test/util/test_llarp_util_str.cpp index b53173660..caa324967 100644 --- a/test/util/test_llarp_util_str.cpp +++ b/test/util/test_llarp_util_str.cpp @@ -1,4 +1,4 @@ -#include +#include #include #include diff --git a/test/util/thread/test_llarp_util_queue.cpp b/test/util/thread/test_llarp_util_queue.cpp index 4410b5ab7..70f71a974 100644 --- a/test/util/thread/test_llarp_util_queue.cpp +++ b/test/util/thread/test_llarp_util_queue.cpp @@ -1,6 +1,6 @@ -#include -#include -#include +#include +#include +#include #include #include diff --git a/test/util/thread/test_llarp_util_queue_manager.cpp b/test/util/thread/test_llarp_util_queue_manager.cpp index 526c6b289..8085a3240 100644 --- a/test/util/thread/test_llarp_util_queue_manager.cpp +++ b/test/util/thread/test_llarp_util_queue_manager.cpp @@ -1,4 +1,4 @@ -#include +#include #include #include From 2498a085dbf319222c41e2d928f1ab320be7075a Mon Sep 17 00:00:00 2001 From: Jeff Becker Date: Mon, 9 Jan 2023 12:48:13 -0500 Subject: [PATCH 3/4] rearrange cmake libraries define relations and document them. make responsibilies clear and consice. --- CMakeLists.txt | 3 +- contrib/android-configure.sh | 1 - contrib/mac-configure.sh | 1 - contrib/windows-configure.sh | 1 - crypto/CMakeLists.txt | 2 +- external/CMakeLists.txt | 25 +- llarp/CMakeLists.txt | 479 +++++++++++++----- llarp/crypto/constants.hpp | 2 +- .../kitchen_sink.cpp => flow/stub.cpp} | 0 .../kitchen_sink.cpp => platform/stub.cpp} | 0 10 files changed, 371 insertions(+), 143 deletions(-) rename llarp/layers/{endpoint/kitchen_sink.cpp => flow/stub.cpp} (100%) rename llarp/layers/{flow/kitchen_sink.cpp => platform/stub.cpp} (100%) diff --git a/CMakeLists.txt b/CMakeLists.txt index 77d85b073..dac2c3d36 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -49,8 +49,7 @@ endif() option(USE_AVX2 "enable avx2 code" OFF) option(USE_NETNS "enable networking namespace support. Linux only" OFF) option(NATIVE_BUILD "optimise for host system and FPU" ON) -option(EMBEDDED_CFG "optimise for older hardware or embedded systems" OFF) -option(BUILD_LIBLOKINET "build liblokinet.so" ON) +option(WITH_EMBEDDED_LOKINET "build liblokinet.so for embedded lokinet" OFF) option(XSAN "use sanitiser, if your system has it (requires -DCMAKE_BUILD_TYPE=Debug)" OFF) option(USE_JEMALLOC "Link to jemalloc for memory allocations, if found" ON) option(TESTNET "testnet build" OFF) diff --git a/contrib/android-configure.sh b/contrib/android-configure.sh index 065e83e18..5e559b5a7 100755 --- a/contrib/android-configure.sh +++ b/contrib/android-configure.sh @@ -33,7 +33,6 @@ for abi in $build_abis; do -DBUILD_PACKAGE=ON \ -DBUILD_SHARED_LIBS=OFF \ -DBUILD_TESTING=OFF \ - -DBUILD_LIBLOKINET=OFF \ -DWITH_TESTS=OFF \ -DWITH_BOOTSTRAP=OFF \ -DNATIVE_BUILD=OFF \ diff --git a/contrib/mac-configure.sh b/contrib/mac-configure.sh index fa45a960b..50c49099a 100755 --- a/contrib/mac-configure.sh +++ b/contrib/mac-configure.sh @@ -13,7 +13,6 @@ cd build-mac cmake \ -G Ninja \ -DBUILD_STATIC_DEPS=ON \ - -DBUILD_LIBLOKINET=OFF \ -DWITH_TESTS=OFF \ -DWITH_BOOTSTRAP=OFF \ -DNATIVE_BUILD=OFF \ diff --git a/contrib/windows-configure.sh b/contrib/windows-configure.sh index f41ef3af8..fa57a1aa4 100755 --- a/contrib/windows-configure.sh +++ b/contrib/windows-configure.sh @@ -32,7 +32,6 @@ cmake \ -DBUILD_PACKAGE=ON \ -DBUILD_SHARED_LIBS=OFF \ -DBUILD_TESTING=OFF \ - -DBUILD_LIBLOKINET=OFF \ -DWITH_TESTS=OFF \ -DWITH_BOOTSTRAP=OFF \ -DNATIVE_BUILD=OFF \ diff --git a/crypto/CMakeLists.txt b/crypto/CMakeLists.txt index e4cedd9b6..c73a6ed14 100644 --- a/crypto/CMakeLists.txt +++ b/crypto/CMakeLists.txt @@ -20,7 +20,7 @@ add_library(lokinet-cryptography libntrup/src/ref/rq.c ) -target_include_directories(lokinet-cryptography PUBLIC libntrup/include) +target_include_directories(lokinet-cryptography PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/libntrup/include) # The avx implementation uses runtime CPU feature detection to enable itself, so we *always* want to # compile it with avx2/fma support when supported by the compiler even if we aren't compiling with diff --git a/external/CMakeLists.txt b/external/CMakeLists.txt index 9b33431f9..c2226bd77 100644 --- a/external/CMakeLists.txt +++ b/external/CMakeLists.txt @@ -1,4 +1,3 @@ - option(SUBMODULE_CHECK "Enables checking that vendored library submodules are up to date" ON) if(SUBMODULE_CHECK) find_package(Git) @@ -140,3 +139,27 @@ if(WITH_BOOTSTRAP) endif() endif() + + +# libcrypt defaults, only on with macos and non static linux +set(default_libcrypt OFF) + +if(LINUX AND NOT STATIC_LINK) + set(default_libcrypt ON) +endif() +if(MACOS) + set(default_libcrypt ON) +endif() + +option(WITH_LIBCRYPT "enable fast password hash with libcrypt" ${default_libcrypt}) + +add_library(lokinet-libcrypt INTERFACE) +if(WITH_LIBCRYPT) + pkg_check_modules(LIBCRYPT libcrypt IMPORTED_TARGET REQUIRED) + add_definitions(-DHAVE_CRYPT) + target_link_libraries(lokinet-libcrypt INTERFACE PkgConfig::LIBCRYPT) + message(STATUS "using libcrypt ${LIBCRYPT_VERSION}") +else() + # TODO static build lib crypt? + message(STATUS "not building with libcrypt") +endif() diff --git a/llarp/CMakeLists.txt b/llarp/CMakeLists.txt index bb4b3bef2..643c485ec 100644 --- a/llarp/CMakeLists.txt +++ b/llarp/CMakeLists.txt @@ -1,5 +1,12 @@ include(Version) +target_sources(lokinet-cryptography PRIVATE + crypto/crypto_libsodium.cpp + crypto/crypto.cpp + crypto/encrypted_frame.cpp + crypto/types.cpp +) + add_library(lokinet-util STATIC ${CMAKE_CURRENT_BINARY_DIR}/constants/version.cpp @@ -15,19 +22,9 @@ add_library(lokinet-util util/thread/threading.cpp util/time.cpp) - add_dependencies(lokinet-util genversion) -target_include_directories(lokinet-util PUBLIC ${CMAKE_CURRENT_SOURCE_DIR} ${PROJECT_SOURCE_DIR}/include ${PROJECT_SOURCE_DIR}) - -target_link_libraries(lokinet-util PUBLIC - lokinet-cryptography - nlohmann_json::nlohmann_json - filesystem - oxenc::oxenc - oxen::logging -) - +# lokinet-platform holds all platform specific code add_library(lokinet-platform STATIC # for networking @@ -45,9 +42,6 @@ add_library(lokinet-platform vpn/platform.cpp ) -target_link_libraries(lokinet-platform PUBLIC lokinet-cryptography lokinet-util Threads::Threads base_libs uvw) -target_link_libraries(lokinet-platform PRIVATE oxenmq::oxenmq) - if (ANDROID) target_sources(lokinet-platform PRIVATE android/ifaddrs.c util/nop_service_manager.cpp) endif() @@ -66,36 +60,26 @@ if (WIN32) net/win32.cpp vpn/win32.cpp win32/service_manager.cpp - win32/exec.cpp) - add_library(lokinet-win32 STATIC + win32/exec.cpp win32/dll.cpp - win32/exception.cpp) - add_library(lokinet-wintun STATIC - win32/wintun.cpp) - add_library(lokinet-windivert STATIC + win32/exception.cpp + win32/wintun.cpp win32/windivert.cpp) - - # wintun and windivert are privated linked by lokinet-platform - # this is so their details do not leak out to deps of lokinet-platform - # wintun and windivert still need things from lokinet-platform - target_compile_options(lokinet-wintun PUBLIC -I${CMAKE_BINARY_DIR}/wintun/include/) - target_compile_options(lokinet-windivert PUBLIC -I${CMAKE_BINARY_DIR}/WinDivert-${WINDIVERT_VERSION}/include/) - target_include_directories(lokinet-windivert PUBLIC ${PROJECT_SOURCE_DIR}) - target_link_libraries(lokinet-wintun PUBLIC lokinet-platform lokinet-util lokinet-config) - target_link_libraries(lokinet-win32 PUBLIC lokinet-util) - target_link_libraries(lokinet-windivert PUBLIC oxen-logging) - target_link_libraries(lokinet-windivert PRIVATE lokinet-win32) - target_link_libraries(lokinet-platform PRIVATE lokinet-win32 lokinet-wintun lokinet-windivert) + target_include_directories(lokinet-platform PRIVATE ${CMAKE_BINARY_DIR}/wintun/include/ ${CMAKE_BINARY_DIR}/WinDivert-${WINDIVERT_VERSION}/include/) else() target_sources(lokinet-platform PRIVATE net/posix.cpp) endif() - -if(CMAKE_SYSTEM_NAME MATCHES "FreeBSD") - target_include_directories(lokinet-platform SYSTEM PUBLIC /usr/local/include) +if(APPLE) + add_subdirectory(apple) + target_sources(lokinet-platform PRIVATE util/nop_service_manager.cpp) endif() +# lokinet-dns is the dns parsing and hooking library that we use to +# parse modify and reconstitute dns wire proto, dns queries and RR +# should have no concept of dns caching, this is left as an implementation +# detail of dns resolvers (LATER: make separate lib for dns resolvers) add_library(lokinet-dns STATIC dns/message.cpp @@ -107,13 +91,50 @@ add_library(lokinet-dns dns/server.cpp dns/srv_data.cpp) +# platform specific bits and bobs for setting dns +add_library(lokinet-dns-platform INTERFACE) if(WITH_SYSTEMD) - target_sources(lokinet-dns PRIVATE dns/nm_platform.cpp dns/sd_platform.cpp) + add_library(lokinet-dns-systemd STATIC dns/nm_platform.cpp dns/sd_platform.cpp) + target_link_libraries(lokinet-dns-platform INTERFACE lokinet-dns-systemd) endif() -target_link_libraries(lokinet-dns PUBLIC lokinet-platform uvw) -target_link_libraries(lokinet-dns PRIVATE libunbound lokinet-config) +# lokinet-nodedb holds all types and logic for storing parsing and constructing +# nodedb data published to the network and versions of it stored locally +add_library(lokinet-nodedb + STATIC + bootstrap.cpp + net/address_info.cpp + net/exit_info.cpp + net/traffic_policy.cpp + nodedb.cpp + pow.cpp + profiling.cpp + router_contact.cpp + router_id.cpp + router_version.cpp +) +set(BOOTSTRAP_FALLBACKS) +foreach(bs IN ITEMS MAINNET TESTNET) + if(BOOTSTRAP_FALLBACK_${bs}) + message(STATUS "Building with ${bs} fallback boostrap path \"${BOOTSTRAP_FALLBACK_${bs}}\"") + file(READ "${BOOTSTRAP_FALLBACK_${bs}}" bs_data HEX) + if(bs STREQUAL TESTNET) + set(network "gamma") + elseif(bs STREQUAL MAINNET) + set(network "lokinet") + else() + string(TOLOWER "${bs}" network) + endif() + string(REGEX REPLACE "([0-9a-f][0-9a-f])" "\\\\x\\1" bs_data "${bs_data}") + set(BOOTSTRAP_FALLBACKS "${BOOTSTRAP_FALLBACKS}{\"${network}\"s, \"${bs_data}\"sv},\n") + endif() +endforeach() +configure_file("bootstrap-fallbacks.cpp.in" "${CMAKE_CURRENT_BINARY_DIR}/bootstrap-fallbacks.cpp" @ONLY) +target_sources(lokinet-nodedb PRIVATE "${CMAKE_CURRENT_BINARY_DIR}/bootstrap-fallbacks.cpp") + + +# lokinet-config is for all configuration types and parsers add_library(lokinet-config STATIC config/config.cpp @@ -121,18 +142,15 @@ add_library(lokinet-config config/ini.cpp config/key_manager.cpp) -target_link_libraries(lokinet-config PUBLIC lokinet-dns lokinet-platform oxenmq::oxenmq) - -add_library(lokinet-amalgum +# lokinet-consensus is for deriving and tracking network consensus state for both service nodes and clients +add_library(lokinet-consensus STATIC consensus/reachability_testing.cpp +) - bootstrap.cpp - context.cpp - crypto/crypto_libsodium.cpp - crypto/crypto.cpp - crypto/encrypted_frame.cpp - crypto/types.cpp +# lokinet-dht holds all logic related to interacting with and participating in the DHT hashring +add_library(lokinet-dht + STATIC dht/context.cpp dht/dht.cpp dht/explorenetworkjob.cpp @@ -151,44 +169,56 @@ add_library(lokinet-amalgum dht/recursiverouterlookup.cpp dht/serviceaddresslookup.cpp dht/taglookup.cpp +) - endpoint_base.cpp +# lokinet-layer-flow is the flow layer which sits atop the routing layer which manages +# flows between lokinet snapp endpoints be they .loki or .snode +add_library(lokinet-layer-flow + STATIC + layers/flow/stub.cpp # todo: remove me +) - exit/context.cpp - exit/endpoint.cpp - exit/exit_messages.cpp - exit/policy.cpp - exit/session.cpp - handlers/exit.cpp - handlers/tun.cpp - iwp/iwp.cpp - iwp/linklayer.cpp - iwp/message_buffer.cpp - iwp/session.cpp - link/link_manager.cpp - link/session.cpp - link/server.cpp - messages/dht_immediate.cpp - messages/link_intro.cpp - messages/link_message_parser.cpp - messages/relay.cpp - messages/relay_commit.cpp - messages/relay_status.cpp - net/address_info.cpp - net/exit_info.cpp - net/traffic_policy.cpp - nodedb.cpp + +# lokinet-layer-onion is the "dumb" onion routing layer with builds manages and does i/o +# with onion paths. onion paths anonymize routing layer pdu. +add_library(lokinet-layer-onion + STATIC path/ihophandler.cpp path/path_context.cpp path/path.cpp path/pathbuilder.cpp path/pathset.cpp path/transit_hop.cpp - peerstats/peer_db.cpp - peerstats/types.cpp - pow.cpp - profiling.cpp - + messages/relay.cpp + messages/relay_commit.cpp + messages/relay_status.cpp +) + +# lokinet-layer-wire is a layer 1 analog which splits up +# layer 2 frames into layer 1 symbols which in the case of iwp are encrypted udp/ip packets +add_library(lokinet-layer-wire + STATIC + iwp/iwp.cpp + iwp/linklayer.cpp + iwp/message_buffer.cpp + iwp/session.cpp +) + +# lokinet-layer-link is for our layer 2 analog which splits up layer 2 frames into +# a series of layer 1 symbols which are then transmitted between lokinet instances +add_library(lokinet-layer-link + STATIC + link/link_manager.cpp + link/session.cpp + link/server.cpp + messages/dht_immediate.cpp + messages/link_intro.cpp + messages/link_message_parser.cpp +) + +# lokinet-plainquic is for holding the tunneled plainquic code, not quic wire protocol code +add_library(lokinet-plainquic + STATIC quic/address.cpp quic/client.cpp quic/connection.cpp @@ -197,27 +227,61 @@ add_library(lokinet-amalgum quic/server.cpp quic/stream.cpp quic/tunnel.cpp +) - router_contact.cpp - router_id.cpp - router_version.cpp - service/name.cpp +# lokinet-context holds the contextualized god objects for a lokinet instance +# it is what any main function would link to in practice but it is hidden behind an interface library (lokinet-amalgum) +add_library(lokinet-context + STATIC + context.cpp + link/link_manager.cpp router/outbound_message_handler.cpp router/outbound_session_maker.cpp router/rc_lookup_handler.cpp router/rc_gossiper.cpp router/router.cpp router/route_poker.cpp +) +# lokinet-rpc holds all rpc related compilation units +add_library(lokinet-rpc + STATIC + rpc/lokid_rpc_client.cpp + rpc/rpc_server.cpp + rpc/endpoint_rpc.cpp +) + +# optional peer stats library +add_library(lokinet-peerstats + STATIC + peerstats/peer_db.cpp + peerstats/types.cpp +) + +# lokinet-layer-routing holds logic related to the routing layer +# routing layer is anonymized over the onion layer +add_library(lokinet-layer-routing + STATIC routing/dht_message.cpp routing/message_parser.cpp routing/path_confirm_message.cpp routing/path_latency_message.cpp routing/path_transfer_message.cpp routing/transfer_traffic_message.cpp - rpc/lokid_rpc_client.cpp - rpc/rpc_server.cpp - rpc/endpoint_rpc.cpp +) + +# kitchen sink to be removed after refactor +add_library(lokinet-service-deprecated-kitchensink + STATIC + endpoint_base.cpp + exit/context.cpp + exit/endpoint.cpp + exit/exit_messages.cpp + exit/policy.cpp + exit/session.cpp + handlers/exit.cpp + handlers/tun.cpp + service/name.cpp service/address.cpp service/async_key_exchange.cpp service/auth.cpp @@ -242,65 +306,216 @@ add_library(lokinet-amalgum service/tag.cpp ) -set(BOOTSTRAP_FALLBACKS) -foreach(bs IN ITEMS MAINNET TESTNET) - if(BOOTSTRAP_FALLBACK_${bs}) - message(STATUS "Building with ${bs} fallback boostrap path \"${BOOTSTRAP_FALLBACK_${bs}}\"") - file(READ "${BOOTSTRAP_FALLBACK_${bs}}" bs_data HEX) - if(bs STREQUAL TESTNET) - set(network "gamma") - elseif(bs STREQUAL MAINNET) - set(network "lokinet") - else() - string(TOLOWER "${bs}" network) - endif() - string(REGEX REPLACE "([0-9a-f][0-9a-f])" "\\\\x\\1" bs_data "${bs_data}") - set(BOOTSTRAP_FALLBACKS "${BOOTSTRAP_FALLBACKS}{\"${network}\"s, \"${bs_data}\"sv},\n") - endif() -endforeach() -configure_file("bootstrap-fallbacks.cpp.in" "${CMAKE_CURRENT_BINARY_DIR}/bootstrap-fallbacks.cpp" @ONLY) -target_sources(lokinet-amalgum PRIVATE "${CMAKE_CURRENT_BINARY_DIR}/bootstrap-fallbacks.cpp") +add_library(lokinet-layer-platform + STATIC + layers/platform/stub.cpp # todo: remove me +) -if(WITH_PEERSTATS_BACKEND) - target_compile_definitions(lokinet-amalgum PRIVATE -DLOKINET_PEERSTATS_BACKEND) - target_link_libraries(lokinet-amalgum PUBLIC sqlite_orm) -endif() +# interal tooling for pybind +add_library(lokinet-tooling INTERFACE) if(WITH_HIVE) - target_sources(lokinet-amalgum PRIVATE + add_library(lokinet-hive-tooling + STATIC tooling/router_hive.cpp tooling/hive_router.cpp tooling/hive_context.cpp ) + target_link_libraries(lokinet-tooling INTERFACE lokinet-hive-tooling) endif() -# TODO: make libunbound hidden behind a feature flag like sqlite for embedded lokinet -target_link_libraries(lokinet-amalgum PRIVATE libunbound) -target_link_libraries(lokinet-amalgum PUBLIC - CLI11 - oxenc::oxenc - lokinet-platform - lokinet-config +# interface library for setting commone includes, linkage and flags. +add_library(lokinet-base INTERFACE) +target_include_directories(lokinet-base + INTERFACE ${PROJECT_SOURCE_DIR} ${PROJECT_SOURCE_DIR}/include +) +target_link_libraries(lokinet-base INTERFACE oxen::logging lokinet-cryptography) + +if(WITH_PEERSTATS) + target_compile_definitions(lokinet-base INTERFACE -DLOKINET_PEERSTATS_BACKEND) + target_link_libraries(lokinet-base INTERFACE sqlite_orm) +endif() + +# interface libraries for internal linkage +add_library(lokinet-layers INTERFACE) +add_library(lokinet-amalgum INTERFACE) + + +# helper function to link a library to lokinet-base, enable lto, add to lokinet-amalgum and then link to other libs +function(lokinet_link_lib libname) + message(DEBUG "created target: ${libname}") + enable_lto(${libname}) + target_link_libraries(${libname} PUBLIC lokinet-base ${ARGN}) + target_link_libraries(lokinet-amalgum INTERFACE ${libname}) +endfunction() + +# internal public linkages of components +lokinet_link_lib(lokinet-util) +lokinet_link_lib(lokinet-cryptography lokinet-libcrypt lokinet-util) +lokinet_link_lib(lokinet-peerstats lokinet-context) +lokinet_link_lib(lokinet-consensus lokinet-context) +lokinet_link_lib(lokinet-layer-link lokinet-peerstats) + +if(TARGET lokinet-hive-tooling) + lokinet_link_lib(lokinet-hive-tooling lokinet-context) +endif() + +if(TARGET lokinet-dns-systemd) + lokinet_link_lib(lokinet-dns-systemd lokinet-dns - lokinet-util - lokinet-cryptography - ngtcp2_static - oxenmq::oxenmq) - -enable_lto(lokinet-util lokinet-platform lokinet-dns lokinet-config lokinet-amalgum) - -pkg_check_modules(CRYPT libcrypt IMPORTED_TARGET) -if(CRYPT_FOUND AND NOT CMAKE_CROSSCOMPILING) - add_definitions(-DHAVE_CRYPT) - add_library(libcrypt INTERFACE) - target_link_libraries(libcrypt INTERFACE PkgConfig::CRYPT) - target_link_libraries(lokinet-amalgum PRIVATE libcrypt) - message(STATUS "using libcrypt ${CRYPT_VERSION}") + lokinet-platform + ) endif() +lokinet_link_lib(lokinet-platform lokinet-util) -if(BUILD_LIBLOKINET) +lokinet_link_lib(lokinet-config + lokinet-util + lokinet-nodedb + lokinet-dns + lokinet-platform +) + +lokinet_link_lib(lokinet-context + lokinet-config + lokinet-platform + lokinet-peerstats + lokinet-layers + lokinet-consensus + lokinet-rpc +) + +lokinet_link_lib(lokinet-dht + lokinet-util + lokinet-nodedb +) + +lokinet_link_lib(lokinet-plainquic + lokinet-platform + lokinet-config +) + +lokinet_link_lib(lokinet-dns + lokinet-platform + lokinet-dns-platform + lokinet-config +) + +lokinet_link_lib(lokinet-nodedb + lokinet-util + lokinet-platform +) + +lokinet_link_lib(lokinet-util + lokinet-nodedb + lokinet-platform +) + +lokinet_link_lib(lokinet-rpc + lokinet-context + lokinet-peerstats + lokinet-util +) + +# inter lokinet-layer public/private linkage. +# when linking each layer, we consider the layer directly below private linkage and the layer above public linkage. +# this lets us hide functionality of layers below us when depended on by another component. +# +# from highest to lowest layer, the above layers are stacked as follows: +# +# platform (what lokinet snapps interact with, be it l3 os interaction or embedded lokinet) +# flow (how we want to route and stripe over our onion routing) +# routing (what we are onion routing) +# onion (how the onion routing happens) +# link (what we want to send over the wire and to where) +# wire (what is actually sent over the wire) +# +function(link_lokinet_layers) + set(lib ${ARGV0}) + if(${ARGC} GREATER 1) + lokinet_link_lib(${ARGV1} ${lib}) + list(REMOVE_AT ARGV 1) + target_link_libraries(${lib} PRIVATE ${ARGV1}) + # recursion :D + link_lokinet_layers(${ARGV}) + else() + lokinet_link_lib(${lib}) + endif() +endfunction() + +link_lokinet_layers( + lokinet-layer-platform + lokinet-layer-flow + lokinet-layer-routing + lokinet-layer-onion + lokinet-layer-link + lokinet-layer-wire +) + +# set me to OFF to disable old codepath +set(use_old_impl ON) +if(use_old_impl) + # flow layer deprecated-kitchensink (remove me after refactor) + lokinet_link_lib(lokinet-service-deprecated-kitchensink + lokinet-dns + lokinet-nodedb + lokinet-context + lokinet-plainquic + lokinet-layer-routing + lokinet-layer-onion + lokinet-dht + lokinet-platform + lokinet-rpc + ) + target_link_libraries(lokinet-layers INTERFACE lokinet-service-deprecated-kitchensink) +endif() + +target_link_libraries(lokinet-layers INTERFACE + lokinet-layer-platform + lokinet-layer-flow + lokinet-layer-routing + lokinet-layer-onion + lokinet-layer-link + lokinet-layer-wire +) + + +# per component external deps + +target_link_libraries(lokinet-config PUBLIC oxenmq::oxenmq) +target_link_libraries(lokinet-platform PUBLIC oxenmq::oxenmq) +target_link_libraries(lokinet-dns PUBLIC libunbound) + +target_link_libraries(lokinet-cryptography PUBLIC + oxenc::oxenc + sodium +) + +target_link_libraries(lokinet-context PUBLIC + CLI11 + oxenmq::oxenmq + uvw +) + +target_link_libraries(lokinet-platform PUBLIC + Threads::Threads + base_libs + uvw +) + +target_link_libraries(lokinet-util PUBLIC + nlohmann_json::nlohmann_json + filesystem + oxenc::oxenc +) + +target_link_libraries(lokinet-plainquic PUBLIC + ngtcp2_static + uvw +) + +if(WITH_EMBEDDED_LOKINET) include(GNUInstallDirs) add_library(lokinet-shared SHARED lokinet_shared.cpp) target_link_libraries(lokinet-shared PUBLIC lokinet-amalgum) @@ -316,11 +531,5 @@ if(BUILD_LIBLOKINET) endif() endif() -if(APPLE) - add_subdirectory(apple) - target_sources(lokinet-platform PRIVATE util/nop_service_manager.cpp) -endif() - file(GLOB_RECURSE docs_SRC */*.hpp *.hpp) - set(DOCS_SRC ${docs_SRC} PARENT_SCOPE) diff --git a/llarp/crypto/constants.hpp b/llarp/crypto/constants.hpp index 1d039bbc2..7f8eef90f 100644 --- a/llarp/crypto/constants.hpp +++ b/llarp/crypto/constants.hpp @@ -2,7 +2,7 @@ #include -#include +#include static constexpr uint32_t PUBKEYSIZE = 32; static constexpr uint32_t SECKEYSIZE = 64; diff --git a/llarp/layers/endpoint/kitchen_sink.cpp b/llarp/layers/flow/stub.cpp similarity index 100% rename from llarp/layers/endpoint/kitchen_sink.cpp rename to llarp/layers/flow/stub.cpp diff --git a/llarp/layers/flow/kitchen_sink.cpp b/llarp/layers/platform/stub.cpp similarity index 100% rename from llarp/layers/flow/kitchen_sink.cpp rename to llarp/layers/platform/stub.cpp From cd8316e28773cf5ee2f98a29246432eafb079d8c Mon Sep 17 00:00:00 2001 From: Jeff Becker Date: Mon, 9 Jan 2023 12:49:03 -0500 Subject: [PATCH 4/4] add commented out line in clang-form for header reorg later --- .clang-format | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.clang-format b/.clang-format index 7109a2362..3f18c4219 100644 --- a/.clang-format +++ b/.clang-format @@ -54,3 +54,8 @@ PointerAlignment: Left # when wrapping function calls/declarations, force each parameter to have its own line BinPackParameters: 'false' BinPackArguments: 'false' + +# TODO: uncomment me when we are reading to rearrange the header includes +# IncludeBlocks: Regroup +# IncludeCategories: 'llarp/' +