vulkan-playground

xrmitter server grows here

Components

• server: Running directly on target device with OpenXR runtime. Receives layers and sound from clients and renders it. Contains all GUI settings
• streamer client: Driver or library, implementing openxr/openvr, receives tracking and streams video. There might be multiple client implementations (monado driver, monado in-process compositor, steamvr driver, OpenComposite driver)
• audio client: connects alsa/jack/pulseaudio to server with udp streams
• reconstructor/decomposer: stream encoding/decoding using Haar wavelet transform and video codec

Status

Server:

• TCP listener
• Receiving projection stream
• Connection/tracking with single client
• Very dumb audio streaming (does not use protocol)
• GUI prototype rendering (basic monitoring only, configuration does nothing)
• Android (Pico 4) and monado linux support (build scripts are not published yet)
• GUI server emulator (works without openxr runtime)

Clients:

• Audio with fifo pipes (tested with jack-stdio, should work with pulseaudio pipe sink/sources too)
• Monado driver with comp_target (monado interaction not published yet, WIP)
• Custom monado pacer synchronized with app for smoother framerate
• Monado builder with dynamic controller type switch (register all controller types)

Decomposer/Reconstructor:

• H264/HEVC VAAPI encoder (no ffmpeg used), linux only
• Haar decomposer compute pipeline
• BC1 reconstruction texture compression
• FXAA filter
• Android decoder
• ffmpeg decoder (software)
• Compute reconstruction pipeline (slow on Adreno)
• Graphics pipeline
• Foveated rendering (hard-coded scale)

Roadmap

Common:

• Choose better naming/branding
• PyroWave support
• Make reconstruction optional (allow using video only)
• Center position offset
• Refactoring, better project structure (split utilities)
• Choose build system
• Code style (it's AWFUL now!!!)
• Lost frame notification
• Imporve logging (use fmt_util if possible, remove extra \n)
• Shader combos to workaround SPIRV limitations
• Template combos (reduce unneeded branching)
• Improve vulkan utility
• New thread utility
• Optimize data layout to prevent false sharing

Server:

• Rec/Dec runtime configuration
• Configuration storage format
• GUI moving/toggle
• Improve gui monitoring (implot maybe)
• Interaction profile switches
• Emulate controllers in emulator GUI
• Timing information
• TCP streaming
• TCP keep-alive and reconnection
• UDP/mDNS broadcast
• SGSR/SGSR2 upscaling for reconstruction texture
• Foveated view configuration
• ffmpeg hwdec
• Multiple layers support (quad/cylinder layers)
• Depth/ASW in reconstruction data (16 bit or rgba block format)
• Disabling reconstruction (video rendering only)

Clients:

• Full jack/alsa/pulse audio clients
• Do something with shitty mediacodec on Android 14 (custom native window???)
• Make builder optional (restore static controller configuration)
• HMDInfo shared cache, so runtime can start without server connection
• Try send something from server to make ipv6 address availiable (udp broadcast???)
• Allow dynamic fov/resolution change (hack/bypass compositor?)
• UDP listener for autoconnect
• Monado compositor bypass (allow sending layers as is)
• Monado in-process compositor (do we need IPC server when we already have server on HMD?)
• Improve decomposer postprocessing
• Passing depth to server (Seems to be unsupported with OpenComposite/monado now)
• SteamVR driver
• Vulkan Video encoder
• External encoders (nvenc/amf/ffmpeg/x265) in separate process (TODO: can we use GPL software with IPC such way???)
• OpenComposite driver (at least, we can send textures without extra swapchain copy and update overlays without xrEndFrame call, so this should be useful)
• Windows support (at least in vulkan video/pyrowave mode)