Resource files are no longer packaged and preloaded all in advance.
Instead, they are downloaded while the game is running.
The implementation is less than ideal, but it works. Resources are
requested on demand and cached into IDBFS. Unfortunately, since the
resource system was not designed with web-style asynchrony in mind and
we aren't ready for threads either, we have no way to do any useful work
while a required file is downloading. To somewhat offset that, we also
pre-fetch files that were not requested yet. The pre-fetching is limited
to 4 files at a time, which seems to be a reasonable compromise between
throughput and cold-cache load latency on slow connections. Also
unfortunately, it's dumb as rocks: we have no way of knowing which files
we will actually be needing soon, so the pre-fetching is done in
whatever order the files were indexed.
Possibly the easiest way to improve this system would be to bundle (and
compress) all of the tiny text files together to alleviate the overhead
of hundreds of HTTP requests.
This is currently unused. It's going to be the backbone for a new
on-the-fly resource fetching system for the Emscripten port.
In this mode, the virtual resource directory structure is embedded into
the executable. Files are referenced by their "content IDs", which are
currently their sha256 hashes. The "resindex" VFS backend allows one to
define a custom file open function, which connects the content IDs to
actual data. A example implementation is provided in resindex_layered,
which simply opens the content ID as a file under a specified VFS path.
VFSNode no longer has the generic void* data1 and data2 fields.
Introduced a macro system that allows backends to easily "subclass"
VFSNode to add extra fields.
Introduces wrappers around memory allocation functions in `memory.h`
that should be used instead of the standard C ones.
These never return NULL and, with the exception of `mem_realloc()`,
zero-initialize the allocated memory like `calloc()` does.
All allocations made with the memory.h API must be deallocated with
`mem_free()`. Although standard `free()` will work on some platforms,
it's not portable (currently it won't work on Windows). Likewise,
`mem_free()` must not be used to free foreign allocations.
The standard C allocation functions are now diagnosed as deprecated.
They are, however, available with the `libc_` prefix in case interfacing
with foreign APIs is required. So far they are only used to implement
`memory.h`.
Perhaps the most important change is the introduction of the `ALLOC()`,
`ALLOC_ARRAY()`, and `ALLOC_FLEX()` macros. They take a type as a
parameter, and allocate enough memory with the correct alignment for
that type. That includes overaligned types as well. In most
circumstances you should prefer to use these macros. See the `memory.h`
header for some usage examples.
Squashed commit of the following:
commit 9ab0197ee1
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 28 23:05:58 2022 +0100
build: add dummy descriptions for deprecated options
commit b0c41d4719
Author: Alice D <alice@starwitch.productions>
Date: Mon Dec 26 15:18:09 2022 -0500
update docker containers with ones containing meson 0.63.3
commit a2dd4173fc
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Sun Dec 25 19:20:44 2022 +0100
build: featurize audio options
commit 7c59e2a569
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Sun Dec 25 19:20:35 2022 +0100
build: remove dead code
commit cf500af137
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Sun Dec 25 19:07:37 2022 +0100
doc: update BUILD.rst
commit 9764ac706f
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Sun Dec 25 17:12:37 2022 +0100
ci: disable WGI joystick backend for SDL
Fails to build with latest llvm-mingw
commit 2d4f310b90
Author: p-sam <p-sam@d3vs.net>
Date: Sun Dec 25 17:03:47 2022 +0100
switch: fix missing include
commit 9264f1649d
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Sat Dec 24 14:36:41 2022 +0100
ci: update meson to 0.63.3 due to a bug
commit 3e37be3d8f
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Sat Dec 24 13:42:34 2022 +0100
ci: update build options
commit 281a39a4f5
Author: Alice D <alice@starwitch.productions>
Date: Sun Dec 18 00:17:07 2022 -0500
ci: change when apt is cleaned up for switch image
commit 2f2100a84c
Author: Alice D <alice@starwitch.productions>
Date: Sun Dec 18 00:11:08 2022 -0500
ci: bump switch homebrew container version
commit ebaff59260
Author: Alice D <alice@starwitch.productions>
Date: Sat Dec 17 22:55:39 2022 -0500
ci: bump windows container version
commit 7ce6f23419
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Sat Dec 17 07:36:28 2022 +0100
build: featurize renderer options
commit a345f21b35
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Sat Dec 17 07:33:30 2022 +0100
build: revert to explicit fallback for libzstd
Implicit fallback causes conflicts in the basis_universal subproject
https://github.com/mesonbuild/meson/issues/11189
commit b2de61f5c5
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Sat Dec 17 05:29:29 2022 +0100
build: remove debug_opengl
This option was always misdocumented, as it only controlled whether GL
debugging is on by default. You can still control it with the
TAISEI_GL_DEBUG environment variable, which is set to 1 in meson devenv.
commit 4c659ba195
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Fri Dec 16 08:03:37 2022 +0100
build: set up useful env vars for meson devenv
Makes Taisei load resources from the source tree, making live reload
work; enables OpenGL debugging by default; works around ASan stack use
after return false positives.
commit 1ddc58b84a
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 08:06:32 2022 +0100
build: featurize docs; always install licenses
commit c3a94d7f5b
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 07:46:28 2022 +0100
build: featurize shader_transpiler
commit 619d82ae36
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 07:38:30 2022 +0100
build: allow libzip fallback by default
commit 79ab72292a
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 07:35:51 2022 +0100
build: featurize use_libcrypto
commit b393077f71
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 05:03:20 2022 +0100
build: featurize validate_glsl
commit 7d76484079
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 04:34:31 2022 +0100
build: featurize some more installation options
commit 17c0bda205
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 03:24:46 2022 +0100
build: deprecate enable_zip for vfs_zip; convert package_data into feature
commit 9a4ef8c091
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 02:37:55 2022 +0100
build: deprecate developer=auto; default to false
commit 807240e508
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 02:24:04 2022 +0100
build: move subproject fallback info into wraps
commit 08c953f487
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 01:57:33 2022 +0100
build: deprecate 'static' option
Meson has a built-in `prefer_static` now
commit c575601537
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 01:49:31 2022 +0100
build: remove complicated shaderc detection logic
Meson does all of this internally since 0.51.0
commit cd2d460d4b
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Wed Dec 14 01:45:17 2022 +0100
build: Require meson >=0.63.0
commit 10abd790f7
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Mon Dec 12 05:16:47 2022 +0100
ci: remove meson dist version workaround from release workflow
Meson should know the correct version now
commit 31a07d8c3c
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Mon Dec 12 03:46:28 2022 +0100
build: use built-in summary() function
commit 0c47f6d14b
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Mon Dec 12 04:33:01 2022 +0100
build: remove -Dversion_fallback and introduce .VERSION file
The version script will now first try to read the version from a
.VERSION file in the source root. If that file does not exist, it will
query git. The .VERSION file is not included in the git repository, it
is generated for inclusion in source tarballs.
Package maintainers who had any use for -Dversion_fallback may want to
create the .VERSION file themselves.
Additionally, some git-specific files have been removed from source
tarballs.
commit d32332a5e3
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Mon Dec 12 03:00:07 2022 +0100
build: set the "real" version in project()
commit f9a3fbc729
Author: Andrei Alexeyev <akari@taisei-project.org>
Date: Mon Dec 12 01:50:08 2022 +0100
build: raise meson_version and remove some legacy workarounds
Effectively 0.56.0 was already required since we started using some of
its features at some point.
This also fixes all current meson warnings.
The output of __FILE__ previously depended on where the build directory
is and was needlessly verbose. With this change, all filenames are
relative to the src/ directory.
Logging of source file names is now also enabled for release builds
(although this is not used yet).
Removed the use_gnu_ext option as well as fallback paths for compilers
that don't support GNU extensions. To my knowledge, none of those
compilers support C11 to a sufficient extent to compile Taisei anyway,
and those fallbacks are very poorly tested.
Pedantic warnings are now disabled, and extensions that are common to
reasonably recent versions of GCC and clang are permitted to be relied
on (list of allowed extensions TBA).
A zstd-compressed file "foobar.foo.zst" will appear in the filesystem as
a normal uncompressed file "foobar.foo". The raw compressed data is
still available under the original .zst filename.
If both "foobar.foo" and "foobar.foo.zst" exist, the former is
preferred and won't be shadowed.
This is implemented as a wrapper layer, much like readonly_wrapper.
It is currently applied on top of plain directories that contribute to
/res, but not to zipped packages. Zip files support zstd as a
compression method, so we'll probably rely on that instead.
Windows: %LOCALAPPDATA%\taisei\cache
Linux and *BSD: $XDG_CACHE_HOME/taisei (xdg basedir spec); usually
~/.cache/taisei
macOS: NSCachesDirectory; probably ~/Library/Caches/taisei
Fallback: $storage/cache (old behavior)
This commit also introduces a more modular and buildsystem-driven
approach to specifying those paths, and makes vfs initialization more
resilient. The game will try to create missing directories recursively,
and will not crash if storage or cache can not be mounted.
Co-authored-by: Alice D <alice@starwitch.productions>
* WIP compressed textures, swizzles, sRGB sampling, ...
* refactor texture type info & fix random bugs
* fix preprocessing of sRGB textures
* handle y-flipped basis textures
* glcommon: better WebGL compat for compressed format detection
* missed WEBGL_compressed_texture_pvrtc
* implement compressed texture xcoding and uploading
* Add basis_universal submodule
* Reorganize texture loader code
Clean up some code
Isolate Basis Universal loader into a separate module
* Add wrapper script for encoding .basis textures
* basisu: honor custom metadata written by the mkbasis.py script
* mkbasis.py: add --incredibly-slow and --dry-run
* Move pixmap code from util/ to pixmap/
* Add an on-disk transcode cache for basis textures to speed up loads
* Compress texture cache with zlib
* Use readable format names for basisu cache filenames
* basisu: mip bias test code
* basisu: small caching cleanup
* add TAISEI_BASISU_MIP_BIAS env variable
* Improve OpenGL format matching heuristics
* Document considerations for compressed format priority
* Remove dead code
* Enable two forgotten formats, BC3_RGBA and ATC_RGBA
Also prefer BC7 over BC1/BC3
* Recognize GL_ANGLE_compressed_texture_etc for ETC2 textures
* Default depth buffers to 24-bit; remove ANGLE hack
* Fix glcommon_check_extension for GLES2/legacy gl
* Add renderer feature bit for texture swizzle masks
* glcommon: Fixup internal formats for GLES2
Sized internal formats are not allowed in GLES2
* Fix emscripten compile errors
* Update basis_universal
* remove more dead code
* revert irrelevant stage4 change
* shut up UBSan
* basisu: shut up some debug spam
* Add normalmap sampling helper to util.glslh
* basisu: add a gray-alpha mode
* mkbasis.py: Abort if image dimansions aren't multiples of 4
* Add basic Basis Universal encoding documentation (WIP)
* doc/basisu: Add paragraph about modes; minor tweaks
* basisu: workarounds for GL texture size requirements
* gles20: fix uncompressed sRGB formats
* Partial workaround for missing swizzles in gles2 and webgl
* remove invalid assertion
* New renderer API to expose glDrawBuffers-like functionality
* stagedraw: disable all color outputs for copy_depth pass
required for WebGL compatibility
* support GL_ANGLE_request_extension
* emscripten: include *.basis in gfx package
Also fix a potential problem when more than one .pkgdir is used to
construct emscripten packages
* Don't rely on emscripten runtime to enable webgl extensions
This also introduces `float32`, `float64`, and `real` typedefs to be
used in place of `float` and `double` later. `real` is for game code and
other places where we don't particularly care about the precision and
format of the underlying type, and is currently defined to `double`.
`float32` and `float64` should replace `float` and `double` respectively
* Initial port
* Switch specific video modes
* Handle clean exit
* Hide option to disable gamepads, force it enabled
* Match buttons layout with Switch controllers
* Hide player name setting (to avoid getting stuck)
* Switch specific VFS setup instead of env bootstrap
* Clean up, get rid of warnings and a few ifdefs
* Add Switch specific build script and assets
* Make vfs setup mount packages
* Re-enable packaging on Switch
* Transpile shaders to es and install them
* Add applet warning and shader deps to README
* Remove script; instead using meson build scripts
* Strict prototypes
* Build script compat with project minimum meson ver
Refactor of pack.py exclude option
* Uniformise header inclusion on arch_switch.c
* Allow input for any dev; hide the option on Switch
* Silence unsused function warnings
* Major refactoring of the main loop(s) and control flow (WIP)
run_at_fps() is gone 🦀
Instead of nested blocking event loops, there is now an eventloop API
that manages an explicit stack of scenes. This makes Taisei a lot more
portable to async environments where spinning a loop forever without
yielding control simply is not an option, and that is the entire point
of this change.
A prime example of such an environment is the Web (via emscripten).
Taisei was able to run there through a terrible hack: inserting
emscripten_sleep calls into the loop, which would yield to the browser.
This has several major drawbacks: first of all, every function that
could possibly call emscripten_sleep must be compiled into a special
kind of bytecode, which then has to be interpreted at runtime, *much*
slower than JITed WebAssembly. And that includes *everything* down the
call stack, too! For more information, see
https://emscripten.org/docs/porting/emterpreter.html
Even though that method worked well enough for experimenting, despite
suboptimal performance, there is another obvious drawback:
emscripten_sleep is implemented via setTimeout(), which can be very
imprecise and is generally not reliable for fluid animation. Browsers
actually have an API specifically for that use case:
window.requestAnimationFrame(), but Taisei's original blocking control
flow style is simply not compatible with it. Emscripten exposes this API
with its emscripten_set_main_loop(), which the eventloop backend now
uses on that platform.
Unfortunately, C is still C, with no fancy closures or coroutines.
With blocking calls into menu/scene loops gone, the control flow is
reimplemented via so-called (pun intended) "call chains". That is
basically an euphemism for callback hell. With manual memory management
and zero type-safety. Not that the menu system wasn't shitty enough
already. I'll just keep telling myself that this is all temporary and
will be replaced with scripts in v1.4.
* improve build system for emscripten + various fixes
* squish menu bugs
* improve emscripten event loop; disable EMULATE_FUNCTION_POINTER_CASTS
Note that stock freetype does not work without
EMULATE_FUNCTION_POINTER_CASTS; use a patched version from the
"emscripten" branch here:
https://github.com/taisei-project/freetype2/tree/emscripten
* Enable -Wcast-function-type
Calling functions through incompatible pointers is nasal demons and
doesn't work in WASM.
* webgl: workaround a crash on some browsers
* emscripten improvements:
* Persist state (config, progress, replays, ...) in local IndexDB
* Simpler HTML shell (temporary)
* Enable more optimizations
* fix build if validate_glsl=false
* emscripten: improve asset packaging, with local cache
Note that even though there are rules to build audio bundles, audio
does *not* work yet. It looks like SDL2_mixer can not work without
threads, which is a problem. Yet another reason to write an OpenAL
backend - emscripten supports that natively.
* emscripten: customize the html shell
* emscripten: force "show log" checkbox unchecked initially
* emscripten: remove quit shortcut from main menu (since there's no quit)
* emscripten: log area fixes
* emscripten/webgl: workaround for fullscreen viewport issue
* emscripten: implement frameskip
* emscripter: improve framerate limiter
* align List to at least 8 bytes (shut up warnings)
* fix non-emscripten builds
* improve fullscreen handling, mainly for emscripten
* Workaround to make audio work in chromium
emscripten-core/emscripten#6511
* emscripten: better vsync handling; enable vsync & disable fxaa by default
