mistletoe is a Markdown parser in pure Python, designed to be fast,
spec-compliant and fully customizable.
Apart from being the fastest CommonMark-compliant Markdown parser implementation
in pure Python, mistletoe also supports easy definitions of custom tokens.
Parsing Markdown into an abstract syntax tree also allows us to swap out
renderers for different output formats, without touching any of the core
components.
Remember to spell mistletoe in lowercase!
detect-secrets is an aptly named module for (surprise, surprise) detecting
secrets within a code base.
However, unlike other similar packages that solely focus on finding secrets,
this package is designed with the enterprise client in mind: providing a
backwards compatible, systematic means of:
1. Preventing new secrets from entering the code base,
2. Detecting if such preventions are explicitly bypassed, and
3. Providing a checklist of secrets to roll, and migrate off to a more secure
storage.
This way, you create a separation of concern: accepting that there may currently
be secrets hiding in your large repository (this is what we refer to as a
baseline), but preventing this issue from getting any larger, without dealing
with the potentially gargantuan effort of moving existing secrets away.
It does this by running periodic diff outputs against heuristically crafted
regex statements, to identify whether any new secret has been committed. This
way, it avoids the overhead of digging through all git history, as well as the
need to scan the entire repository every time.
VisiData provides a terminal interface for exploring and arranging tabular data.
It supports tsv, csv, sqlite, json, xlsx (Excel), hdf5, and many other formats.
Requirements Detector is a simple Python tool which attempts to find and list
the requirements of a Python project.
When run from the root of a Python project, it will try to ascertain which
libraries and the versions of those libraries that the project depends on.
It uses the following methods in order, in the root of the project:
1. Parse setup.py (if this is successful, the remaining steps are skipped)
2. Parse pyproject.yoml (if a tool.poetry.dependencies section is found, the
remaining steps are skipped)
3. Parse requirements.txt or requirements.pip
4. Parse all *.txt and *.pip files inside a folder called requirements
5. Parse all files in the root folder matching *requirements*.txt or reqs.txt
(so for example, pip_requirements.txt would match, as would
requirements_common.txt)
This is the backend for PDM projects that is fully-compatible with PEP 517 spec,
but you can also use it alone. It reads the metadata of PEP 621 format and
coverts it to Core metadata.
Frictionless CKAN Mapper is a library for mapping CKAN metadata from/to
Frictionless metadata.
The library has zero dependencies (not even on Data Package libs). You can use
it directly or use it for inspiration. Detailed outline of the algorithm is in
the docs or you can read the code.
Dodgy is a very basic tool to run against your codebase to search for "dodgy"
looking values. It is a series of simple regular expressions designed to detect
things such as accidental SCM diff checkins, or passwords or secret keys hard
coded into files.
While this is primarily aimed at open source projects (for whom a publicly
available secret key is pretty dangerous), it can also be used in private
projects, with the caveat that it will point out things which are not a problem
for private projects and is not configurable enough currently to change that.
Another note - this tool is probably best run pre-commit, since it will
hopefully prevent dodgy things being checked in. To automatically execute before
a commit use a git pre-commit hook.
fastparquet is a python implementation of the parquet format, aiming integrate
into python-based big data work-flows. It is used implicitly by the projects
Dask, Pandas and intake-parquet.
We offer a high degree of support for the features of the parquet format, and
very competitive performance, in a small install size and codebase.
sozipfile is a fork of Python zipfile module, from its implementation in CPython
3.11, which implements the SOZip optimization, when writing deflate compressed
files whose size exceeds the chunk size (defaults to 32768 bytes).
Extremely thin Python bindings to de/compression algorithms in Rust. Allows for
using algorithms such as Snappy, without any system dependencies.
This is handy when being used in environments like AWS Lambda, where installing
packages like python-snappy becomes difficult because of system level
dependencies.
CGIF is a fast and lightweight GIF encoder that can create GIF animations and
images. Summary of the main features:
- user-defined global or local color-palette with up to 256 colors (limit of the
GIF format)
- size-optimizations for GIF animations:
- option to set a pixel to transparent if it has identical color in the
previous frame (transparency optimization)
- do encoding just for the rectangular area that differs from the previous
frame (width/height optimization)
- fast: a GIF with 256 colors and 1024x1024 pixels can be created in below 50 ms
even on a minimalistic system
- MIT license (permissive)
- different options for GIF animations: static image, N repetitions, infinite
repetitions
- additional source-code for verifying the encoder after making changes
- user-defined delay time from one frame to the next (can be set independently
for each frame)
- source-code conforms to the C99 standard
libddwaf is Datadog's implementation of a Web Application Firewall (WAF) engine,
with a goal of low performance and memory overhead, and embeddability in a wide
variety of language runtimes through a C API.
- Make GetTempDir() function Unix-aware, where TMP environment
variable is preferred over Windows' TEMP, and fallback to /tmp
when it is unset
- Change GetAppPath() function to uphold that we do not install
the main program in another subdirectory (bin) since the whole
project already lives under $DATADIR (Mono/.NET programs are
not always hier(7)-conforming)
- Fix provided *.desktop file (respect PREFIX and replace badly
named startup wrapper script with ours)
Remove build workaround for 12.3 re-added in last commit by mistake that
was removed in 5693229 commit.
Cleanup port and delete scripts dir used by workaround.
PR: 270625
MFH: 2023Q2 (buildfixes)
