librewolf
/
bunkerized-nginx
mirror of https://github.com/bunkerity/bunkerized-nginx
4
0
Fork 1
Code Issues Projects Releases Wiki Activity

Merge pull request #441 from bunkerity/dev 

Merge branch "dev" into branch "ui"
This commit is contained in:
Théophile Diot 2023-04-24 16:59:34 +02:00 committed by GitHub
parent dffc770a99 2ea726c222
commit 52806afe73
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
85 changed files with 2463 additions and 1422 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
TODO
View File

@ -1,6 +1,3 @@
- utils refactoring
- load inline values for white/black/grey list core
- check if correct setting is set to yes in new() before loading stuff in self
- store object in ngx.ctx
- bwcli with redis
- move bans to cachestore
- stream refactoring
- stream examples

58
docs/backup-new-integrations.md
View File

@ -96,7 +96,7 @@ vagrant ssh
python3 -m http.server -b 127.0.0.1
```
Configuration of BunkerWeb is done by editing the `/opt/bunkerweb/variables.env` file.
Configuration of BunkerWeb is done by editing the `/etc/bunkerweb/variables.env` file.
Connect to your vagrant machine :
```shell
@ -159,7 +159,7 @@ vagrant ssh
vagrant ssh
```
Configuration of BunkerWeb is done by editing the /opt/bunkerweb/variables.env file :
Configuration of BunkerWeb is done by editing the /etc/bunkerweb/variables.env file :
```conf
SERVER_NAME=app1.example.com app2.example.com app3.example.com
HTTP_PORT=80
@ -190,7 +190,7 @@ vagrant ssh
=== "Vagrant"
You will need to add the settings to the `/opt/bunkerweb/variables.env` file :
You will need to add the settings to the `/etc/bunkerweb/variables.env` file :
```conf
...
@ -204,7 +204,7 @@ vagrant ssh
=== "Vagrant"
You will need to add the settings to the `/opt/bunkerweb/variables.env` file :
You will need to add the settings to the `/etc/bunkerweb/variables.env` file :
```conf
...
@ -219,7 +219,7 @@ vagrant ssh
=== "Vagrant"
When using the [Vagrant integration](/1.4/integrations/#vagrant), custom configurations must be written to the `/opt/bunkerweb/configs` folder.
When using the [Vagrant integration](/1.4/integrations/#vagrant), custom configurations must be written to the `/etc/bunkerweb/configs` folder.
Here is an example for server-http/hello-world.conf :
```conf
@ -233,8 +233,8 @@ vagrant ssh
Because BunkerWeb runs as an unprivileged user (nginx:nginx), you will need to edit the permissions :
```shell
chown -R root:nginx /opt/bunkerweb/configs && \
chmod -R 770 /opt/bunkerweb/configs
chown -R root:nginx /etc/bunkerweb/configs && \
chmod -R 770 /etc/bunkerweb/configs
```
Don't forget to restart the BunkerWeb service once it's done.
@ -243,9 +243,9 @@ vagrant ssh
We will assume that you already have the [Vagrant integration](/1.4/integrations/#vagrant) stack running on your machine.
By default, BunkerWeb will search for web files inside the `/opt/bunkerweb/www` folder. You can use it to store your PHP application. Please note that you will need to configure your PHP-FPM service to get or set the user/group of the running processes and the UNIX socket file used to communicate with BunkerWeb.
By default, BunkerWeb will search for web files inside the `/var/www/html` folder. You can use it to store your PHP application. Please note that you will need to configure your PHP-FPM service to get or set the user/group of the running processes and the UNIX socket file used to communicate with BunkerWeb.
First of all, you will need to make sure that your PHP-FPM instance can access the files inside the `/opt/bunkerweb/www` folder and also that BunkerWeb can access the UNIX socket file in order to communicate with PHP-FPM. We recommend to set a different user like `www-data` for the PHP-FPM service and to give the nginx group access to the UNIX socket file. Here is corresponding PHP-FPM configuration :
First of all, you will need to make sure that your PHP-FPM instance can access the files inside the `/var/www/html` folder and also that BunkerWeb can access the UNIX socket file in order to communicate with PHP-FPM. We recommend to set a different user like `www-data` for the PHP-FPM service and to give the nginx group access to the UNIX socket file. Here is corresponding PHP-FPM configuration :
```ini
...
[www]
@ -263,14 +263,14 @@ vagrant ssh
systemctl restart php8.1-fpm
```
Once your application is copied to the `/opt/bunkerweb/www` folder, you will need to fix the permissions so BunkerWeb (user/group nginx) can at least read files and list folders and PHP-FPM (user/group www-data) is the owner of the files and folders :
Once your application is copied to the `/var/www/html` folder, you will need to fix the permissions so BunkerWeb (user/group nginx) can at least read files and list folders and PHP-FPM (user/group www-data) is the owner of the files and folders :
```shell
chown -R www-data:nginx /opt/bunkerweb/www && \
find /opt/bunkerweb/www -type f -exec chmod 0640 {} \; && \
find /opt/bunkerweb/www -type d -exec chmod 0750 {} \;
chown -R www-data:nginx /var/www/html && \
find /var/www/html -type f -exec chmod 0640 {} \; && \
find /var/www/html -type d -exec chmod 0750 {} \;
```
You can now edit the `/opt/bunkerweb/variable.env` file :
You can now edit the `/etc/bunkerweb/variable.env` file :
```env
HTTP_PORT=80
HTTPS_PORT=443
@ -278,7 +278,7 @@ vagrant ssh
SERVER_NAME=www.example.com
AUTO_LETS_ENCRYPT=yes
LOCAL_PHP=/run/php/php-fpm.sock
LOCAL_PHP_PATH=/opt/bunkerweb/www/
LOCAL_PHP_PATH=/var/www/html/
```
Let's check the status of BunkerWeb :
@ -299,9 +299,9 @@ vagrant ssh
We will assume that you already have the [Vagrant integration](/1.4/integrations/#vagrant) stack running on your machine.
By default, BunkerWeb will search for web files inside the `/opt/bunkerweb/www` folder. You can use it to store your PHP applications : each application will be in its own subfolder named the same as the primary server name. Please note that you will need to configure your PHP-FPM service to get or set the user/group of the running processes and the UNIX socket file used to communicate with BunkerWeb.
By default, BunkerWeb will search for web files inside the `/var/www/html` folder. You can use it to store your PHP applications : each application will be in its own subfolder named the same as the primary server name. Please note that you will need to configure your PHP-FPM service to get or set the user/group of the running processes and the UNIX socket file used to communicate with BunkerWeb.
First of all, you will need to make sure that your PHP-FPM instance can access the files inside the `/opt/bunkerweb/www` folder and also that BunkerWeb can access the UNIX socket file in order to communicate with PHP-FPM. We recommend to set a different user like `www-data` for the PHP-FPM service and to give the nginx group access to the UNIX socket file. Here is corresponding PHP-FPM configuration :
First of all, you will need to make sure that your PHP-FPM instance can access the files inside the `/var/www/html` folder and also that BunkerWeb can access the UNIX socket file in order to communicate with PHP-FPM. We recommend to set a different user like `www-data` for the PHP-FPM service and to give the nginx group access to the UNIX socket file. Here is corresponding PHP-FPM configuration :
```ini
...
[www]
@ -319,14 +319,14 @@ vagrant ssh
systemctl restart php8.1-fpm
```
Once your application is copied to the `/opt/bunkerweb/www` folder, you will need to fix the permissions so BunkerWeb (user/group nginx) can at least read files and list folders and PHP-FPM (user/group www-data) is the owner of the files and folders :
Once your application is copied to the `/var/www/html` folder, you will need to fix the permissions so BunkerWeb (user/group nginx) can at least read files and list folders and PHP-FPM (user/group www-data) is the owner of the files and folders :
```shell
chown -R www-data:nginx /opt/bunkerweb/www && \
find /opt/bunkerweb/www -type f -exec chmod 0640 {} \; && \
find /opt/bunkerweb/www -type d -exec chmod 0750 {} \;
chown -R www-data:nginx /var/www/html && \
find /var/www/html -type f -exec chmod 0640 {} \; && \
find /var/www/html -type d -exec chmod 0750 {} \;
```
You can now edit the `/opt/bunkerweb/variable.env` file :
You can now edit the `/etc/bunkerweb/variable.env` file :
```env
HTTP_PORT=80
HTTPS_PORT=443
@ -335,11 +335,11 @@ vagrant ssh
MULTISITE=yes
AUTO_LETS_ENCRYPT=yes
app1.example.com_LOCAL_PHP=/run/php/php-fpm.sock
app1.example.com_LOCAL_PHP_PATH=/opt/bunkerweb/www/app1.example.com
app1.example.com_LOCAL_PHP_PATH=/var/www/html/app1.example.com
app2.example.com_LOCAL_PHP=/run/php/php-fpm.sock
app2.example.com_LOCAL_PHP_PATH=/opt/bunkerweb/www/app2.example.com
app2.example.com_LOCAL_PHP_PATH=/var/www/html/app2.example.com
app3.example.com_LOCAL_PHP=/run/php/php-fpm.sock
app3.example.com_LOCAL_PHP_PATH=/opt/bunkerweb/www/app3.example.com
app3.example.com_LOCAL_PHP_PATH=/var/www/html/app3.example.com
```
Let's check the status of BunkerWeb :
@ -360,7 +360,7 @@ vagrant ssh
=== "Vagrant"
When using the [Linux integration](/1.4/integrations/#linux), plugins must be written to the `/opt/bunkerweb/plugins` folder :
When using the [Linux integration](/1.4/integrations/#linux), plugins must be written to the `/etc/bunkerweb/plugins` folder :
```shell
git clone https://github.com/bunkerity/bunkerweb-plugins && \
cp -rp ./bunkerweb-plugins/* /data/plugins
@ -372,7 +372,7 @@ vagrant ssh
The installation of the web UI using the [Vagrant integration](/1.4/integrations/#vagrant) is pretty straightforward because it is installed with BunkerWeb.
The first thing to do is to edit the BunkerWeb configuration located at **/opt/bunkerweb/variables.env** to add settings related to the web UI :
The first thing to do is to edit the BunkerWeb configuration located at **/etc/bunkerweb/variables.env** to add settings related to the web UI :
```conf
HTTP_PORT=80
HTTPS_PORT=443
@ -401,7 +401,7 @@ vagrant ssh
systemctl restart bunkerweb
```
You can edit the **/opt/bunkerweb/ui.env** file containing the settings of the web UI :
You can edit the **/etc/bunkerweb/ui.env** file containing the settings of the web UI :
```conf
ADMIN_USERNAME=admin
ADMIN_PASSWORD=changeme
@ -410,7 +410,7 @@ vagrant ssh
Important things to note :
* `http(s)://bwadmin.example.com/changeme/` is the full base URL of the web UI (must match the sub(domain) and /changeme URL used in **/opt/bunkerweb/variables.env**)
* `http(s)://bwadmin.example.com/changeme/` is the full base URL of the web UI (must match the sub(domain) and /changeme URL used in **/etc/bunkerweb/variables.env**)
* replace the username `admin` and password `changeme` with strong ones
Restart the BunkerWeb UI service and you are now ready to access it :

142
docs/integrations.md
View File

@ -3,7 +3,7 @@
## Docker
<figure markdown>
![Overwiew](assets/img/integration-docker.svg){ align=center }
![Overview](assets/img/integration-docker.svg){ align=center }
<figcaption>Docker integration</figcaption>
</figure>
@ -174,7 +174,7 @@ networks:
## Docker autoconf
<figure markdown>
![Overwiew](assets/img/integration-autoconf.svg){ align=center }
![Overview](assets/img/integration-autoconf.svg){ align=center }
<figcaption>Docker autoconf integration</figcaption>
</figure>
@ -325,7 +325,7 @@ networks:
## Swarm
<figure markdown>
![Overwiew](assets/img/integration-swarm.svg){ align=center }
![Overview](assets/img/integration-swarm.svg){ align=center }
<figcaption>Docker Swarm integration</figcaption>
</figure>
@ -486,7 +486,7 @@ networks:
## Kubernetes
<figure markdown>
![Overwiew](assets/img/integration-kubernetes.svg){ align=center }
![Overview](assets/img/integration-kubernetes.svg){ align=center }
<figcaption>Kubernetes integration</figcaption>
</figure>
@ -580,7 +580,7 @@ spec:
livenessProbe:
exec:
command:
- /opt/bunkerweb/helpers/healthcheck.sh
- /usr/share/bunkerweb/helpers/healthcheck.sh
initialDelaySeconds: 30
periodSeconds: 5
timeoutSeconds: 1
@ -588,7 +588,7 @@ spec:
readinessProbe:
exec:
command:
- /opt/bunkerweb/helpers/healthcheck.sh
- /usr/share/bunkerweb/helpers/healthcheck.sh
initialDelaySeconds: 30
periodSeconds: 1
timeoutSeconds: 1
@ -673,7 +673,7 @@ spec:
## Linux
<figure markdown>
![Overwiew](assets/img/integration-linux.svg){ align=center }
![Overview](assets/img/integration-linux.svg){ align=center }
<figcaption>Linux integration</figcaption>
</figure>
@ -806,9 +806,9 @@ Repositories of Linux packages for BunkerWeb are available on [PackageCloud](htt
The first step is to install NGINX 1.20.2 using the repository of your choice or by [compiling it from source](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/#compiling-and-installing-from-source).
The target installation folder of BunkerWeb is located at `/opt/bunkerweb`, let's create it :
The target installation folder of BunkerWeb is located at `/usr/share/bunkerweb`, let's create it :
```shell
mkdir /opt/bunkerweb
mkdir /usr/share/bunkerweb
```
You can now clone the BunkerWeb project to the `/tmp` folder :
@ -816,40 +816,42 @@ Repositories of Linux packages for BunkerWeb are available on [PackageCloud](htt
https://github.com/bunkerity/bunkerweb.git /tmp/bunkerweb
```
BunkerWeb needs some dependencies to be compiled and installed to `/opt/bunkerweb/deps`, the easiest way to do it is by executing the [install.sh helper script](https://github.com/bunkerity/bunkerweb/blob/master/deps/install.sh) (please note that you will need to install additional packages which is not covered in this procedure and depends on your own system) :
BunkerWeb needs some dependencies to be compiled and installed to `/usr/share/bunkerweb/deps`, the easiest way to do it is by executing the [install.sh helper script](https://github.com/bunkerity/bunkerweb/blob/master/deps/install.sh) (please note that you will need to install additional packages which is not covered in this procedure and depends on your own system) :
```
mkdir /opt/bunkerweb/deps && \
mkdir /usr/share/bunkerweb/deps && \
/tmp/bunkerweb/deps/install.sh
```
Additional Python dependencies needs to be installed into the `/opt/bunkerweb/deps/python` folder :
Additional Python dependencies needs to be installed into the `/usr/share/bunkerweb/deps/python` folder :
```shell
mkdir /opt/bunkerweb/deps/python && \
pip install --no-cache-dir --require-hashes --target /opt/bunkerweb/deps/python -r /tmp/bunkerweb/deps/requirements.txt && \
pip install --no-cache-dir --target /opt/bunkerweb/deps/python -r /tmp/bunkerweb/ui/requirements.txt
mkdir /usr/share/bunkerweb/deps/python && \
pip install --no-cache-dir --require-hashes --target /usr/share/bunkerweb/deps/python -r /tmp/bunkerweb/deps/requirements.txt && \
pip install --no-cache-dir --target /usr/share/bunkerweb/deps/python -r /tmp/bunkerweb/ui/requirements.txt
```
Once dependencies are installed, you will be able to copy the BunkerWeb sources to the target `/opt/bunkerweb` folder :
Once dependencies are installed, you will be able to copy the BunkerWeb sources to the target `/usr/share/bunkerweb` folder :
```shell
for src in api cli confs core gen helpers job lua misc utils ui settings.json VERSION linux/variables.env linux/ui.env linux/scripts ; do
cp -r /tmp/bunkerweb/${src} /opt/bunkerweb
cp -r /tmp/bunkerweb/${src} /usr/share/bunkerweb
done
cp /opt/bunkerweb/helpers/bwcli /usr/local/bin
cp /usr/share/bunkerweb/helpers/bwcli /usr/local/bin
```
Additional folders also need to be created :
```shell
mkdir /opt/bunkerweb/{configs,cache,plugins,tmp}
mkdir -p /etc/bunkerweb/{configs,plugins} && \
mkdir -p /var/cache/bunkerweb && \
mkdir -p /var/tmp/bunkerweb
```
Permissions needs to be fixed :
```shell
find /opt/bunkerweb -path /opt/bunkerweb/deps -prune -o -type f -exec chmod 0740 {} \; && \
find /opt/bunkerweb -path /opt/bunkerweb/deps -prune -o -type d -exec chmod 0750 {} \; && \
find /opt/bunkerweb/core/*/jobs/* -type f -exec chmod 750 {} \; && \
chmod 770 /opt/bunkerweb/cache /opt/bunkerweb/tmp && \
chmod 750 /opt/bunkerweb/gen/main.py /opt/bunkerweb/job/main.py /opt/bunkerweb/cli/main.py /opt/bunkerweb/helpers/*.sh /opt/bunkerweb/scripts/*.sh /usr/local/bin/bwcli /opt/bunkerweb/ui/main.py && \
chown -R root:nginx /opt/bunkerweb
find /usr/share/bunkerweb -path /usr/share/bunkerweb/deps -prune -o -type f -exec chmod 0740 {} \; && \
find /usr/share/bunkerweb -path /usr/share/bunkerweb/deps -prune -o -type d -exec chmod 0750 {} \; && \
find /usr/share/bunkerweb/core/*/jobs/* -type f -exec chmod 750 {} \; && \
chmod 770 /var/cache/bunkerweb /var/tmp/bunkerweb && \
chmod 750 /usr/share/bunkerweb/gen/main.py /usr/share/bunkerweb/scheduler/main.py /usr/share/bunkerweb/cli/main.py /usr/share/bunkerweb/helpers/*.sh /usr/share/bunkerweb/scripts/*.sh /usr/bin/bwcli /usr/share/bunkerweb/ui/main.py && \
chown -R root:nginx /usr/share/bunkerweb
```
Last but not least, you will need to set up systemd unit files :
@ -862,7 +864,7 @@ Repositories of Linux packages for BunkerWeb are available on [PackageCloud](htt
systemctl enable bunkerweb-ui
```
The configuration of BunkerWeb is done by editing the `/opt/bunkerweb/variables.env` file :
The configuration of BunkerWeb is done by editing the `/etc/bunkerweb/variables.env` file :
```conf
MY_SETTING_1=value1
@ -880,7 +882,7 @@ BunkerWeb is managed using systemctl :
## Ansible
<figure markdown>
![Overwiew](assets/img/integration-ansible.svg){ align=center }
![Overview](assets/img/integration-ansible.svg){ align=center }
<figcaption>Ansible integration</figcaption>
</figure>
@ -938,4 +940,88 @@ Configuration of BunkerWeb is done by using specific role variables :
| `custom_www` | string | Path of the www directory to upload. | empty value |
| `custom_plugins` | string | Path of the plugins directory to upload. | empty value |
| `custom_www_owner` | string | Default owner for www files and folders. | `nginx` |
| `custom_www_group` | string | Default group for www files and folders. | `nginx` |
| `custom_www_group` | string | Default group for www files and folders. | `nginx` |
## Vagrant
<figure markdown>
![Overview](assets/img/integration-vagrant.svg){ align=center }
<figcaption>BunkerWeb integration with Vagrant</figcaption>
</figure>
List of supported providers :
- vmware_desktop
- virtualbox
- libvirt
**_Note on Supported Base Images_**
Please be aware that the provided Vagrant boxes are based **exclusively on Ubuntu 22.04 "Jammy"**. While BunkerWeb supports other Linux distributions, the Vagrant setup currently only supports Ubuntu 22.04 as the base operating system. This ensures a consistent and reliable environment for users who want to deploy BunkerWeb using Vagrant.
Similar to other BunkerWeb integrations, the Vagrant setup uses **NGINX version 1.20.2**. This specific version is required to ensure compatibility and smooth functioning with BunkerWeb. Additionally, the Vagrant box includes **PHP** pre-installed, providing a ready-to-use environment for hosting PHP-based applications alongside BunkerWeb.
By using the provided Vagrant box based on Ubuntu 22.04 "Jammy", you benefit from a well-configured and integrated setup, allowing you to focus on developing and securing your applications with BunkerWeb without worrying about the underlying infrastructure.
Here are the steps to install BunkerWeb using Vagrant on Ubuntu with the supported virtualization providers (VirtualBox, VMware, and libvirt):
1. Make sure you have Vagrant and one of the supported virtualization providers (VirtualBox, VMware, or libvirt) installed on your system.
2. There are two ways to install the Vagrant box with BunkerWeb: either by using a provided Vagrantfile to configure your virtual machine or by creating a new box based on the existing BunkerWeb Vagrant box, offering you flexibility in how you set up your development environment.
=== "Vagrantfile"
```shell
Vagrant.configure("2") do |config|
config.vm.box = "bunkerity/bunkerity"
end
```
Depending on the virtualization provider you choose, you may need to install additional plugins:
* For **VMware**, install the `vagrant-vmware-desktop` plugin. For more information, see the [Vagrant documentation](https://www.vagrantup.com/docs/providers).
* For **libvirt**, install the `vagrant-libvirt plugin`. For more information, see the [Vagrant documentation](https://www.vagrantup.com/docs/providers).
* For **VirtualBox**, install the `vagrant-vbguest` plugin. For more information, see the [Vagrant documentation](https://www.vagrantup.com/docs/providers).
=== "New Vagrant Box"
```shell
vagrant init bunkerity/bunkerity
```
Depending on the virtualization provider you choose, you may need to install additional plugins:
* For **VMware**, install the `vagrant-vmware-desktop` plugin. For more information, see the [Vagrant documentation](https://www.vagrantup.com/docs/providers).
* For **libvirt**, install the `vagrant-libvirt plugin`. For more information, see the [Vagrant documentation](https://www.vagrantup.com/docs/providers).
* For **VirtualBox**, install the `vagrant-vbguest` plugin. For more information, see the [Vagrant documentation](https://www.vagrantup.com/docs/providers).
After installing the necessary plugins for your chosen virtualization provider, run the following command to start the virtual machine and install BunkerWeb:
```shell
vagrant up --provider=virtualbox # or --provider=vmware_desktop or --provider=libvirt
```
Finally, to access the virtual machine using SSH, execute the following command:
```shell
vagrant ssh
```
**Example Vagrantfile**
Here is an example `Vagrantfile` for installing BunkerWeb on Ubuntu 22.04 "Jammy" using the different supported virtualization providers:
```shell
Vagrant.configure("2") do |config|
# Ubuntu 22.04 "Jammy"
config.vm.box = "bunkerity/bunkerity"
# Uncomment the desired virtualization provider
# For VirtualBox (default)
config.vm.provider "virtualbox"
# For VMware
# config.vm.provider "vmware_desktop" # Windows
# config.vm.provider "vmware_workstation" # Linux
# For libvirt
# config.vm.provider "libvirt"
end
```

42
docs/json2md.py
View File

@ -1,11 +1,12 @@
#!/usr/bin/python3
from io import StringIO
from json import loads
from glob import glob
from pytablewriter import MarkdownTableWriter
def print_md_table(settings):
def print_md_table(settings) -> MarkdownTableWriter:
writer = MarkdownTableWriter(
headers=["Setting", "Default", "Context", "Multiple", "Description"],
value_matrix=[
@ -19,37 +20,52 @@ def print_md_table(settings):
for setting, data in settings.items()
],
)
writer.write_table()
print()
return writer
print("# Settings\n")
doc = StringIO()
print("# Settings\n", file=doc)
print(
'!!! info "Settings generator tool"\n\n To help you tune BunkerWeb, we have made an easy-to-use settings generator tool available at [config.bunkerweb.io](https://config.bunkerweb.io).\n'
'!!! info "Settings generator tool"\n\n To help you tune BunkerWeb, we have made an easy-to-use settings generator tool available at [config.bunkerweb.io](https://config.bunkerweb.io).\n',
file=doc,
)
print(
"This section contains the full list of settings supported by BunkerWeb. If you are not yet familiar with BunkerWeb, you should first read the [concepts](/1.4/concepts) section of the documentation. Please follow the instructions for your own [integration](/1.4/integrations) on how to apply the settings.\n"
"This section contains the full list of settings supported by BunkerWeb. If you are not yet familiar with BunkerWeb, you should first read the [concepts](/1.4/concepts) section of the documentation. Please follow the instructions for your own [integration](/1.4/integrations) on how to apply the settings.\n",
file=doc,
)
print(
"As a general rule when multisite mode is enabled, if you want to apply settings with multisite context to a specific server, you will need to add the primary (first) server name as a prefix like `www.example.com_USE_ANTIBOT=captcha` or `myapp.example.com_USE_GZIP=yes` for example.\n"
"As a general rule when multisite mode is enabled, if you want to apply settings with multisite context to a specific server, you will need to add the primary (first) server name as a prefix like `www.example.com_USE_ANTIBOT=captcha` or `myapp.example.com_USE_GZIP=yes` for example.\n",
file=doc,
)
print(
'When settings are considered as "multiple", it means that you can have multiple groups of settings for the same feature by adding numbers as suffix like `REVERSE_PROXY_URL_1=/subdir`, `REVERSE_PROXY_HOST_1=http://myhost1`, `REVERSE_PROXY_URL_2=/anotherdir`, `REVERSE_PROXY_HOST_2=http://myhost2`, ... for example.\n'
'When settings are considered as "multiple", it means that you can have multiple groups of settings for the same feature by adding numbers as suffix like `REVERSE_PROXY_URL_1=/subdir`, `REVERSE_PROXY_HOST_1=http://myhost1`, `REVERSE_PROXY_URL_2=/anotherdir`, `REVERSE_PROXY_HOST_2=http://myhost2`, ... for example.\n',
file=doc,
)
# Print global settings
print("## Global settings\n")
print("## Global settings\n", file=doc)
with open("src/common/settings.json", "r") as f:
print_md_table(loads(f.read()))
print(print_md_table(loads(f.read())), file=doc)
print(file=doc)
# Print core settings
print("## Core settings\n")
print("## Core settings\n", file=doc)
core_settings = {}
for core in glob("src/common/core/*/plugin.json"):
with open(core, "r") as f:
core_plugin = loads(f.read())
if len(core_plugin["settings"]) > 0:
core_settings[core_plugin["name"]] = core_plugin["settings"]
for name, settings in dict(sorted(core_settings.items())).items():
print(f"### {name}\n")
print_md_table(settings)
print(f"### {name}\n", file=doc)
print(print_md_table(settings), file=doc)
doc.seek(0)
content = doc.read()
doc = StringIO(content.replace("\\|", "|"))
doc.seek(0)
with open("docs/settings.md", "w") as f:
f.write(doc.read())

2
docs/requirements.txt
View File

@ -1,5 +1,5 @@
mkdocs==1.4.2
mkdocs-material==9.1.6
mkdocs-material==9.1.7
pytablewriter==0.64.2
mike==1.1.2
jinja2<3.1.0

39
docs/security-tuning.md
View File

@ -118,9 +118,9 @@ If you want to use your own certificates, here is the list of related settings :
| Setting | Default | Description |
| :-----------------: | :-----: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `USE_CUSTOM_HTTPS` | `no` | When set to `yes`, HTTPS will be enabled with custom certificates. |
| `CUSTOM_HTTPS_CERT` | | Full path to the certificate. If you have one or more intermediate certificate(s) in your chain of trust, you will need to provide the bundle (more info [here](https://nginx.org/en/docs/http/configuring_https_servers.html#chains)). |
| `CUSTOM_HTTPS_KEY` | | Full path to the private key. |
| `USE_CUSTOM_SSL` | `no` | When set to `yes`, HTTPS will be enabled with custom certificates. |
| `CUSTOM_SSL_CERT` | | Full path to the certificate. If you have one or more intermediate certificate(s) in your chain of trust, you will need to provide the bundle (more info [here](https://nginx.org/en/docs/http/configuring_https_servers.html#chains)). |
| `CUSTOM_SSL_KEY` | | Full path to the private key. |
When `USE_CUSTOM_HTTPS` is set to `yes`, BunkerWeb will check every day if the custom certificate specified in `CUSTOM_HTTPS_CERT` is modified and will reload NGINX if that's the case.
@ -221,17 +221,28 @@ You can use the following settings to set up blacklisting :
| Setting | Default | Description |
| :-------------------------: | :----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------- |
| `USE_BLACKLIST` | `yes` | When set to `yes`, will enable blacklisting based on various criteria. |
| `BLACKLIST_IP` | | List of IPs and networks to blacklist. |
| `BLACKLIST_IP_URLS` | `https://www.dan.me.uk/torlist/?exit` | List of URL containing IP and network to blacklist. The default list contains TOR exit nodes. |
| `BLACKLIST_RDNS` | `.shodan.io .censys.io` | List of reverse DNS to blacklist. |
| `BLACKLIST_RDNS_URLS` | | List of URLs containing reverse DNS to blacklist. |
| `BLACKLIST_ASN` | | List of ASN to blacklist. |
| `BLACKLIST_ASN_URLS` | | List of URLs containing ASN to blacklist. |
| `BLACKLIST_USER_AGENT` | | List of User-Agents to blacklist. |
| `BLACKLIST_USER_AGENT_URLS` | `https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list` | List of URLs containing User-Agent(s) to blacklist. |
| `BLACKLIST_URI` | | List of requests URI to blacklist. |
| `BLACKLIST_URI_URLS` | | List of URLs containing request URI to blacklist. |
|`USE_BLACKLIST` |`yes` |Activate blacklist feature. |
|`BLACKLIST_IP` | |List of IP/network, separated with spaces, to block. |
|`BLACKLIST_IP_URLS` |`https://www.dan.me.uk/torlist/?exit` |List of URLs, separated with spaces, containing bad IP/network to block. |
|`BLACKLIST_RDNS_GLOBAL` |`yes` |Only perform RDNS blacklist checks on global IP addresses. |
|`BLACKLIST_RDNS` |`.shodan.io .censys.io` |List of reverse DNS suffixes, separated with spaces, to block. |
|`BLACKLIST_RDNS_URLS` | |List of URLs, separated with spaces, containing reverse DNS suffixes to block. |
|`BLACKLIST_ASN` | |List of ASN numbers, separated with spaces, to block. |
|`BLACKLIST_ASN_URLS` | |List of URLs, separated with spaces, containing ASN to block. |
|`BLACKLIST_USER_AGENT` | |List of User-Agent, separated with spaces, to block. |
|`BLACKLIST_USER_AGENT_URLS` |`https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list`|List of URLs, separated with spaces, containing bad User-Agent to block. |
|`BLACKLIST_URI` | |List of URI, separated with spaces, to block. |
|`BLACKLIST_URI_URLS` | |List of URLs, separated with spaces, containing bad URI to block. |
|`BLACKLIST_IGNORE_IP` | |List of IP/network, separated with spaces, to ignore in the blacklist. |
|`BLACKLIST_IGNORE_IP_URLS` | |List of URLs, separated with spaces, containing IP/network to ignore in the blacklist. |
|`BLACKLIST_IGNORE_RDNS` | |List of reverse DNS suffixes, separated with spaces, to ignore in the blacklist. |
|`BLACKLIST_IGNORE_RDNS_URLS` | |List of URLs, separated with spaces, containing reverse DNS suffixes to ignore in the blacklist.|
|`BLACKLIST_IGNORE_ASN` | |List of ASN numbers, separated with spaces, to ignore in the blacklist. |
|`BLACKLIST_IGNORE_ASN_URLS` | |List of URLs, separated with spaces, containing ASN to ignore in the blacklist. |
|`BLACKLIST_IGNORE_USER_AGENT` | |List of User-Agent, separated with spaces, to ignore in the blacklist. |
|`BLACKLIST_IGNORE_USER_AGENT_URLS`| |List of URLs, separated with spaces, containing User-Agent to ignore in the blacklist. |
|`BLACKLIST_IGNORE_URI` | |List of URI, separated with spaces, to ignore in the blacklist. |
|`BLACKLIST_IGNORE_URI_URLS` | |List of URLs, separated with spaces, containing URI to ignore in the blacklist. |
### Greylisting

65
docs/settings.md
View File

@ -12,34 +12,39 @@ When settings are considered as "multiple", it means that you can have multiple
## Global settings
| Setting | Default | Context |Multiple| Description |
|------------------------|------------------------------------------------------------------------------------------------------------------------|---------|--------|--------------------------------------------------|
|`IS_LOADING` |`no` |global |no |Internal use : set to yes when BW is loading. |
|`NGINX_PREFIX` |`/etc/nginx/` |global |no |Where nginx will search for configurations. |
|`HTTP_PORT` |`8080` |global |no |HTTP port number which bunkerweb binds to. |
|`HTTPS_PORT` |`8443` |global |no |HTTPS port number which bunkerweb binds to. |
|`MULTISITE` |`no` |global |no |Multi site activation. |
|`SERVER_NAME` |`www.example.com` |multisite|no |List of the virtual hosts served by bunkerweb. |
|`WORKER_PROCESSES` |`auto` |global |no |Number of worker processes. |
|`WORKER_RLIMIT_NOFILE` |`2048` |global |no |Maximum number of open files for worker processes.|
|`WORKER_CONNECTIONS` |`1024` |global |no |Maximum number of connections per worker. |
|`LOG_FORMAT` |`$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"`|global |no |The format to use for access logs. |
|`LOG_LEVEL` |`notice` |global |no |The level to use for error logs. |
|`DNS_RESOLVERS` |`127.0.0.11` |global |no |DNS addresses of resolvers to use. |
|`DATASTORE_MEMORY_SIZE` |`256m` |global |no |Size of the internal datastore. |
|`USE_API` |`yes` |global |no |Activate the API to control BunkerWeb. |
|`API_HTTP_PORT` |`5000` |global |no |Listen port number for the API. |
|`API_LISTEN_IP` |`0.0.0.0` |global |no |Listen IP address for the API. |
|`API_SERVER_NAME` |`bwapi` |global |no |Server name (virtual host) for the API. |
|`API_WHITELIST_IP` |`127.0.0.0/8` |global |no |List of IP/network allowed to contact the API. |
|`AUTOCONF_MODE` |`no` |global |no |Enable Autoconf Docker integration. |
|`SWARM_MODE` |`no` |global |no |Enable Docker Swarm integration. |
|`KUBERNETES_MODE` |`no` |global |no |Enable Kubernetes integration. |
|`SERVER_TYPE` |`http` |multisite|no |Server type : http or stream. |
|`LISTEN_STREAM` |`yes` |multisite|no |Enable listening for non-ssl (passthrough). |
|`LISTEN_STREAM_PORT` |`1337` |multisite|no |Listening port for non-ssl (passthrough). |
|`LISTEN_STREAM_PORT_SSL`|`4242` |multisite|no |Listening port for ssl (passthrough). |
|`USE_UDP` |`no` |multisite|no |UDP listen instead of TCP (stream). |
| Setting | Default | Context |Multiple| Description |
|------------------------------|------------------------------------------------------------------------------------------------------------------------|---------|--------|--------------------------------------------------|
|`IS_LOADING` |`no` |global |no |Internal use : set to yes when BW is loading. |
|`NGINX_PREFIX` |`/etc/nginx/` |global |no |Where nginx will search for configurations. |
|`HTTP_PORT` |`8080` |global |no |HTTP port number which bunkerweb binds to. |
|`HTTPS_PORT` |`8443` |global |no |HTTPS port number which bunkerweb binds to. |
|`MULTISITE` |`no` |global |no |Multi site activation. |
|`SERVER_NAME` |`www.example.com` |multisite|no |List of the virtual hosts served by bunkerweb. |
|`WORKER_PROCESSES` |`auto` |global |no |Number of worker processes. |
|`WORKER_RLIMIT_NOFILE` |`2048` |global |no |Maximum number of open files for worker processes.|
|`WORKER_CONNECTIONS` |`1024` |global |no |Maximum number of connections per worker. |
|`LOG_FORMAT` |`$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"`|global |no |The format to use for access logs. |
|`LOG_LEVEL` |`notice` |global |no |The level to use for error logs. |
|`DNS_RESOLVERS` |`127.0.0.11` |global |no |DNS addresses of resolvers to use. |
|`DATASTORE_MEMORY_SIZE` |`64m` |global |no |Size of the internal datastore. |
|`CACHESTORE_MEMORY_SIZE` |`64m` |global |no |Size of the internal cachestore. |
|`CACHESTORE_IPC_MEMORY_SIZE` |`16m` |global |no |Size of the internal cachestore (ipc). |
|`CACHESTORE_MISS_MEMORY_SIZE` |`16m` |global |no |Size of the internal cachestore (miss). |
|`CACHESTORE_LOCKS_MEMORY_SIZE`|`16m` |global |no |Size of the internal cachestore (locks). |
|`USE_API` |`yes` |global |no |Activate the API to control BunkerWeb. |
|`API_HTTP_PORT` |`5000` |global |no |Listen port number for the API. |
|`API_LISTEN_IP` |`0.0.0.0` |global |no |Listen IP address for the API. |
|`API_SERVER_NAME` |`bwapi` |global |no |Server name (virtual host) for the API. |
|`API_WHITELIST_IP` |`127.0.0.0/8` |global |no |List of IP/network allowed to contact the API. |
|`AUTOCONF_MODE` |`no` |global |no |Enable Autoconf Docker integration. |
|`SWARM_MODE` |`no` |global |no |Enable Docker Swarm integration. |
|`KUBERNETES_MODE` |`no` |global |no |Enable Kubernetes integration. |
|`SERVER_TYPE` |`http` |multisite|no |Server type : http or stream. |
|`LISTEN_STREAM` |`yes` |multisite|no |Enable listening for non-ssl (passthrough). |
|`LISTEN_STREAM_PORT` |`1337` |multisite|no |Listening port for non-ssl (passthrough). |
|`LISTEN_STREAM_PORT_SSL` |`4242` |multisite|no |Listening port for ssl (passthrough). |
|`USE_UDP` |`no` |multisite|no |UDP listen instead of TCP (stream). |
## Core settings
@ -135,7 +140,7 @@ When settings are considered as "multiple", it means that you can have multiple
| Setting | Default | Context |Multiple| Description |
|-------------------------|------------------------------------------------------------|---------|--------|--------------------------------------------------------------------|
|`USE_CLIENT_CACHE` |`no` |multisite|no |Tell client to store locally static files. |
|`CLIENT_CACHE_EXTENSIONS`|`jpg\|jpeg\|png\|bmp\|ico\|svg\|tif\|css\|js\|otf\|ttf\|eot\|woff\|woff2`|global |no |List of file extensions, separated with pipes that should be cached.|
|`CLIENT_CACHE_EXTENSIONS`|`jpg|jpeg|png|bmp|ico|svg|tif|css|js|otf|ttf|eot|woff|woff2`|global |no |List of file extensions, separated with pipes that should be cached.|
|`CLIENT_CACHE_ETAG` |`yes` |multisite|no |Send the HTTP ETag header for static resources. |
|`CLIENT_CACHE_CONTROL` |`public, max-age=15552000` |multisite|no |Value of the Cache-Control HTTP header. |
@ -249,7 +254,7 @@ When settings are considered as "multiple", it means that you can have multiple
|`DISABLE_DEFAULT_SERVER` |`no` |global |no |Close connection if the request vhost is unknown. |
|`REDIRECT_HTTP_TO_HTTPS` |`no` |multisite|no |Redirect all HTTP request to HTTPS. |
|`AUTO_REDIRECT_HTTP_TO_HTTPS`|`yes` |multisite|no |Try to detect if HTTPS is used and activate HTTP to HTTPS redirection if that's the case. |
|`ALLOWED_METHODS` |`GET\|POST\|HEAD` |multisite|no |Allowed HTTP and WebDAV methods, separated with pipes to be sent by clients. |
|`ALLOWED_METHODS` |`GET|POST|HEAD` |multisite|no |Allowed HTTP and WebDAV methods, separated with pipes to be sent by clients. |
|`MAX_CLIENT_SIZE` |`10m` |multisite|no |Maximum body size (0 for infinite). |
|`SERVE_FILES` |`yes` |multisite|no |Serve files from the local folder. |
|`ROOT_FOLDER` | |multisite|no |Root folder containing files to serve (/var/www/html/{server_name} if unset). |

8
docs/web-ui.md
View File

@ -69,7 +69,7 @@ Because the web UI is a web application, the recommended installation procedure
-e bwadm.example.com_REVERSE_PROXY_URL=/changeme/ \
-e bwadm.example.com_REVERSE_PROXY_HOST=http://bw-ui:7000 \
-e "bwadm.example.com_REVERSE_PROXY_HEADERS=X-Script-Name /changeme" \
-e bwadm.example.com_INTERCEPTED_ERROR_CODES="400 401.5.0-beta 413 429 500 501 502 503 504" \
-e bwadm.example.com_INTERCEPTED_ERROR_CODES="400 401 405 413 429 500 501 502 503 504" \
-l bunkerweb.INSTANCE \
bunkerity/bunkerweb:1.5.0-beta && \
docker network connect bw-universe bunkerweb
@ -294,7 +294,7 @@ Because the web UI is a web application, the recommended installation procedure
-l "bunkerweb.REVERSE_PROXY_URL=/changeme" \
-l "bunkerweb.REVERSE_PROXY_HOST=http://bw-ui:7000" \
-l "bunkerweb.REVERSE_PROXY_HEADERS=X-Script-Name /changeme" \
-l "bunkerweb.INTERCEPTED_ERROR_CODES=400 401.5.0-beta 405 413 429 500 501 502 503 504" \
-l "bunkerweb.INTERCEPTED_ERROR_CODES=400 401 405 405 413 429 500 501 502 503 504" \
bunkerity/bunkerweb-ui:1.5.0-beta && \
docker network connect bw-docker bw-ui
```
@ -379,7 +379,7 @@ Because the web UI is a web application, the recommended installation procedure
- "bunkerweb.REVERSE_PROXY_URL=/changeme"
- "bunkerweb.REVERSE_PROXY_HOST=http://bw-ui:7000"
- "bunkerweb.REVERSE_PROXY_HEADERS=X-Script-Name /changeme"
- "bunkerweb.INTERCEPTED_ERROR_CODES=400 401.5.0-beta 405 413 429 500 501 502 503 504"
- "bunkerweb.INTERCEPTED_ERROR_CODES=400 401 405 405 413 429 500 501 502 503 504"
volumes:
bw-data:
@ -526,7 +526,7 @@ Because the web UI is a web application, the recommended installation procedure
-l "bunkerweb.REVERSE_PROXY_URL=/changeme" \
-l "bunkerweb.REVERSE_PROXY_HOST=http://bw-ui:7000" \
-l "bunkerweb.REVERSE_PROXY_HEADERS=X-Script-Name /changeme" \
-l "bunkerweb.INTERCEPTED_ERROR_CODES=400 401.5.0-beta 405 413 429 500 501 502 503 504" \
-l "bunkerweb.INTERCEPTED_ERROR_CODES=400 401 405 405 413 429 500 501 502 503 504" \
bunkerity/bunkerweb-ui:1.5.0-beta
```

6
examples/php-cookie-flags/docker-compose.yml
View File

@ -13,7 +13,7 @@ services:
# another example for existing folder : chown -R root:101 folder && chmod -R 770 folder
# more info at https://docs.bunkerweb.io
volumes:
- ./www/var/www/html # contains web files (PHP, assets, ...)
- ./www:/var/www/html # contains web files (PHP, assets, ...)
environment:
- SERVER_NAME=www.example.com # replace with your domain
- API_WHITELIST_IP=127.0.0.0/8 10.20.30.0/24
@ -62,6 +62,10 @@ services:
networks:
- bw-services
volumes:
bw-data:
networks:
bw-universe:
ipam:

5
misc/integrations/k8s.mariadb.ui.yml
View File

@ -141,9 +141,10 @@ spec:
labels:
app: bunkerweb-scheduler
spec:
serviceAccountName: sa-bunkerweb
containers:
- name: bunkerweb-controller
image: bunkerity/bunkerweb-autoconf:1.4.6
- name: bunkerweb-scheduler
image: bunkerity/bunkerweb-scheduler:1.4.6
imagePullPolicy: Always
env:
- name: KUBERNETES_MODE

5
misc/integrations/k8s.mysql.ui.yml
View File

@ -141,9 +141,10 @@ spec:
labels:
app: bunkerweb-scheduler
spec:
serviceAccountName: sa-bunkerweb
containers:
- name: bunkerweb-controller
image: bunkerity/bunkerweb-autoconf:1.4.6
- name: bunkerweb-scheduler
image: bunkerity/bunkerweb-scheduler:1.4.6
imagePullPolicy: Always
env:
- name: KUBERNETES_MODE

5
misc/integrations/k8s.postgres.ui.yml
View File

@ -141,9 +141,10 @@ spec:
labels:
app: bunkerweb-scheduler
spec:
serviceAccountName: sa-bunkerweb
containers:
- name: bunkerweb-controller
image: bunkerity/bunkerweb-autoconf:1.4.6
- name: bunkerweb-scheduler
image: bunkerity/bunkerweb-scheduler:1.4.6
imagePullPolicy: Always
env:
- name: KUBERNETES_MODE

3
src/autoconf/Dockerfile
View File

@ -53,6 +53,9 @@ RUN apk add --no-cache bash && \
chown root:nginx /var/log/letsencrypt /var/lib/letsencrypt && \
chmod 770 /var/log/letsencrypt /var/lib/letsencrypt
# Fix CVEs
RUN apk add "libcrypto3>=3.0.8-r4" "libssl3>=3.0.8-r4"
VOLUME /data /etc/nginx
WORKDIR /usr/share/bunkerweb/autoconf

3
src/bw/Dockerfile
View File

@ -72,6 +72,9 @@ RUN apk add --no-cache pcre bash python3 && \
ln -s /proc/1/fd/1 /var/log/nginx/access.log && \
ln -s /proc/1/fd/1 /var/log/nginx/jobs.log
# Fix CVEs
RUN apk add "libcrypto3>=3.0.8-r4" "libssl3>=3.0.8-r4"
VOLUME /data /etc/nginx
EXPOSE 8080/tcp 8443/tcp

1
src/bw/lua/bunkerweb/api.lua
View File

@ -188,6 +188,7 @@ function api:do_api_call()
local status, resp = self:response(ngx.HTTP_INTERNAL_SERVER_ERROR, "error", "can't list loaded plugins : " .. err)
return false, resp["msg"], ngx.HTTP_INTERNAL_SERVER_ERROR, resp
end
list = cjson.decode(list)
for i, plugin in ipairs(list) do
if pcall(require, plugin.id .. "/" .. plugin.id) then
local plugin_lua = require(plugin.id .. "/" .. plugin.id)

87
src/bw/lua/bunkerweb/cachestore.lua
View File

@ -47,48 +47,41 @@ function cachestore:initialize(use_redis)
end
function cachestore:get(key)
local function callback(key)
local callback = function(key)
-- Connect to redis
local clusterstore = require "bunkerweb.clusterstore"
local ok, err = clusterstore:new()
if not ok then
return nil, "clusterstore:new() failed : " .. err, nil
end
local clusterstore = require "bunkerweb.clusterstore":new()
local ok, err = clusterstore:connect()
if not ok then
return nil, "can't connect to redis : " .. err, nil
end
-- Exec transaction
local calls = {
{"get", {key}},
{"ttl", {key}}
}
-- Exec transaction
local exec, err = clusterstore:multi(calls)
if err then
-- Redis script to get value + ttl
local redis_script = [[
local ret_get = redis.pcall("GET", KEYS[1])
if type(ret_get) == "table" and ret_get["err"] ~= nil then
redis.log(redis.LOG_WARNING, "BUNKERWEB CACHESTORE GET error : " .. ret_get["err"])
return ret_get
end
local ret_ttl = redis.pcall("TTL", KEYS[1])
if type(ret_ttl) == "table" and ret_ttl["err"] ~= nil then
redis.log(redis.LOG_WARNING, "BUNKERWEB CACHESTORE DEL error : " .. ret_ttl["err"])
return ret_ttl
end
return {ret_get, ret_ttl}
]]
local ret, err = clusterstore:call("eval", redis_script, 1, key)
if not ret then
clusterstore:close()
return nil, "exec() failed : " .. err, nil
return nil, err, nil
end
-- Get results
local value = exec[1]
if type(value) == "table" then
clusterstore:close(redis)
return nil, "GET error : " .. value[2], nil
-- Extract values
clusterstore:close()
if ret[1] == ngx.null then
ret[1] = nil
end
local ttl = exec[2]
if type(ttl) == "table" then
clusterstore:close(redis)
return nil, "TTL error : " .. ttl[2], nil
if ret[2] < 0 then
ret[2] = ret[2] + 1
end
-- Return value
clusterstore:close(redis)
if value == ngx.null then
value = nil
end
if ttl < 0 then
ttl = ttl + 1
end
return value, nil, ttl
return ret[1], nil, ret[2]
end
local value, err, hit_level
if self.use_redis then
@ -96,7 +89,7 @@ function cachestore:get(key)
else
value, err, hit_level = self.cache:get(key)
end
if value == nil and hit_level == nil then
if value == nil and err ~= nil then
return false, err
end
self.logger:log(ngx.INFO, "hit level for " .. key .. " = " .. tostring(hit_level))
@ -124,18 +117,19 @@ end
function cachestore:set_redis(key, value, ex)
-- Connect to redis
local redis, err = clusterstore:connect()
if not redis then
local clusterstore = require "bunkerweb.clusterstore":new()
local ok, err = clusterstore:connect()
if not ok then
return false, "can't connect to redis : " .. err
end
-- Set value with ttl
local default_ex = ttl or 30
local ok, err = redis:set(key, value, "EX", ex)
local default_ex = ex or 30
local ok, err = clusterstore:call("set", key, value, "EX", default_ex)
if err then
clusterstore:close(redis)
return false, "GET failed : " .. err
clusterstore:close()
return false, "SET failed : " .. err
end
clusterstore:close(redis)
clusterstore:close()
return true
end
@ -155,17 +149,18 @@ end
function cachestore:del_redis(key)
-- Connect to redis
local redis, err = clusterstore:connect()
if not redis then
local clusterstore = require "bunkerweb.clusterstore":new()
local ok, err = clusterstore:connect()
if not ok then
return false, "can't connect to redis : " .. err
end
-- Set value with ttl
local ok, err = redis:del(key)
local ok, err = clusterstore:del(key)
if err then
clusterstore:close(redis)
clusterstore:close()
return false, "DEL failed : " .. err
end
clusterstore:close(redis)
clusterstore:close()
return true
end

7
src/bw/lua/bunkerweb/clusterstore.lua
View File

@ -62,7 +62,7 @@ function clusterstore:connect()
return false, err
end
if times == 0 then
local select, err = redis_client:select(tonumber(variables["REDIS_DATABASE"]))
local select, err = redis_client:select(tonumber(self.variables["REDIS_DATABASE"]))
if err then
self:close()
return false, err
@ -74,8 +74,9 @@ end
function clusterstore:close()
if self.redis_client then
-- Equivalent to close but keep a pool of connections
local ok, err = self.redis_client:set_keepalive(tonumber(self.variables["REDIS_KEEPALIVE_IDLE"]), tonumber(self.variables["REDIS_KEEPALIVE_POOL"]))
self.redis_client = nil
return self.redis_client:set_keepalive(tonumber(self.variables["REDIS_KEEPALIVE_IDLE"]), tonumber(self.variables["REDIS_KEEPALIVE_POOL"]))
return ok, err
end
return false, "not connected"
end
@ -102,7 +103,7 @@ function clusterstore:multi(calls)
-- Loop on calls
for i, call in ipairs(calls) do
local method = call[1]
local args = table.unpack(call[2])
local args = unpack(call[2])
local ok, err = self.redis_client[method](self.redis_client, args)
if not ok then
return false, method + "() failed : " .. err

2
src/bw/lua/bunkerweb/datastore.lua
View File

@ -30,7 +30,7 @@ function datastore:keys()
return self.dict:get_keys(0)
end
function datastore:exp(key)
function datastore:ttl(key)
local ttl, err = self.dict:ttl(key)
if not ttl then
return false, err

11
src/bw/lua/bunkerweb/helpers.lua
View File

@ -99,12 +99,15 @@ helpers.fill_ctx = function()
if not ngx.shared.cachestore then
data.kind = "stream"
end
data.ip = ngx.var.remote_addr
data.remote_addr = ngx.var.remote_addr
data.uri = ngx.var.uri
data.original_uri = ngx.var.original_uri
data.user_agent = ngx.var.http_user_agent
data.request_uri = ngx.var.request_uri
data.request_method = ngx.var.request_method
data.http_user_agent = ngx.var.http_user_agent
data.http_host = ngx.var.http_host
data.server_name = ngx.var.server_name
-- IP data : global
local ip_is_global, err = utils.ip_is_global(data.ip)
local ip_is_global, err = utils.ip_is_global(data.remote_addr)
if ip_is_global == nil then
table.insert(errors, "can't check if IP is global : " .. err)
else

6
src/bw/lua/bunkerweb/plugin.lua
View File

@ -27,6 +27,12 @@ function plugin:initialize(id)
end
self.variables[k] = value
end
-- Is loading
local is_loading, err = utils.get_variable("IS_LOADING", false)
if is_loading == nil then
self.logger:log(ngx.ERR, "can't get IS_LOADING variable : " .. err)
end
self.is_loading = is_loading == "yes"
end
function plugin:get_id()

158
src/bw/lua/bunkerweb/utils.lua
View File

@ -338,7 +338,7 @@ utils.get_rdns = function(ip)
return false, nil
end
utils.get_ips = function(fqdn, resolvers)
utils.get_ips = function(fqdn)
-- Get resolvers
local resolvers, err = utils.get_resolvers()
if not resolvers then
@ -433,57 +433,153 @@ end
utils.get_session = function()
-- Session already in context
if ngx.ctx.bw.session then
return ngx.ctx.bw.session, ngx.ctx.bw.session_err, ngx.ctx.bw.session_exists
return ngx.ctx.bw.session, ngx.ctx.bw.session_err, ngx.ctx.bw.session_exists, ngx.ctx.bw.session_refreshed
end
-- Open session
local _session, err, exists = session.start()
if err then
logger:log(ngx.ERR, "can't start session : " .. err)
-- Open session and fill ctx
local _session, err, exists, refreshed = session.start()
ngx.ctx.bw.session_err = nil
if err and err ~= "missing session cookie" and err ~= "no session" then
logger:log(ngx.WARN, "can't start session : " .. err)
ngx.ctx.bw.session_err = err
end
-- Fill ctx
ngx.ctx.session = _session
ngx.ctx.session_err = err
ngx.ctx.session_exists = exists
ngx.ctx.session_saved = false
ngx.ctx.session_data = _session.get_data()
if not ngx.ctx.session_data then
ngx.ctx.session_data = {}
ngx.ctx.bw.session = _session
ngx.ctx.bw.session_exists = exists
ngx.ctx.bw.session_refreshed = refreshed
ngx.ctx.bw.session_saved = false
ngx.ctx.bw.session_data = _session:get_data()
if not ngx.ctx.bw.session_data then
ngx.ctx.bw.session_data = {}
end
return _session, err, exists
return _session, ngx.ctx.bw.session_err, exists, refreshed
end
utils.save_session = function()
-- Check if save is needed
if ngx.ctx.session and not ngx.ctx.session_err and not ngx.ctx.session_saved then
ngx.ctx.session:set_data(ngx.ctx.session_data)
local ok, err = ngx.ctx.session:save()
if ngx.ctx.bw.session and not ngx.ctx.bw.session_saved then
ngx.ctx.bw.session:set_data(ngx.ctx.bw.session_data)
local ok, err = ngx.ctx.bw.session:save()
if err then
logger:log(ngx.ERR, "can't save session : " .. err)
return false, "can't save session : " .. err
end
ngx.ctx.session_saved = true
ngx.ctx.bw.session_saved = true
return true, "session saved"
elseif ngx.ctx.session_saved then
elseif ngx.ctx.bw.session_saved then
return true, "session already saved"
end
return true, "no session"
end
utils.set_session = function(key, value)
utils.set_session_var = function(key, value)
-- Set new data
if ngx.ctx.session and not ngx.ctx.session_err then
ngx.ctx.session_data[key] = value
if ngx.ctx.bw.session then
ngx.ctx.bw.session_data[key] = value
return true, "value set"
end
return true, "no session"
end
utils.get_session = function(key)
-- Get data
if ngx.ctx.session and not ngx.ctx.session_err then
return true, "value get", ngx.ctx.session_data[key]
end
return false, "no session"
end
utils.get_session_var = function(key)
-- Get data
if ngx.ctx.bw.session then
if ngx.ctx.bw.session_data[key] then
return true, "data present", ngx.ctx.bw.session_data[key]
end
return true, "no data"
end
return false, "no session"
end
utils.is_banned = function(ip)
-- Check on local datastore
local reason, err = datastore:get("bans_ip_" .. ip)
if not reason and err ~= "not found" then
return nil, "datastore:get() error : " .. reason
elseif reason and err ~= "not found" then
local ok, ttl = datastore:ttl("bans_ip_" .. ip)
if not ok then
return true, reason, -1
end
return true, reason, ttl
end
-- Redis case
local use_redis, err = utils.get_variable("USE_REDIS", false)
if not use_redis then
return nil, "can't get USE_REDIS variable : " .. err
elseif use_redis ~= "yes" then
return false, "not banned"
end
-- Connect
local clusterstore = require "bunkerweb.clusterstore":new()
local ok, err = clusterstore:connect()
if not ok then
return nil, "can't connect to redis server : " .. err
end
-- Redis atomic script : GET+TTL
local redis_script = [[
local ret_get = redis.pcall("GET", KEYS[1])
if type(ret_get) == "table" and ret_get["err"] ~= nil then
redis.log(redis.LOG_WARNING, "access GET error : " .. ret_get["err"])
return ret_get
end
local ret_ttl = nil
if ret_get ~= nil then
ret_ttl = redis.pcall("TTL", KEYS[1])
if type(ret_ttl) == "table" and ret_ttl["err"] ~= nil then
redis.log(redis.LOG_WARNING, "access TTL error : " .. ret_ttl["err"])
return ret_ttl
end
end
return {ret_get, ret_ttl}
]]
-- Execute redis script
local data, err = clusterstore:call("eval", redis_script, 1, "bans_ip_" .. ip)
if not data then
clusterstore:close()
return nil, "redis call error : " .. err
elseif data.err then
clusterstore:close()
return nil, "redis script error : " .. data.err
elseif data[1] ~= ngx.null then
clusterstore:close()
-- Update local cache
local ok, err = datastore:set("bans_ip_" .. ip, data[1], data[2])
if not ok then
return nil, "datastore:set() error : " .. err
end
return true, data[1], data[2]
end
clusterstore:close()
return false, "not banned"
end
utils.add_ban = function(ip, reason, ttl)
-- Set on local datastore
local ok, err = datastore:set("bans_ip_" .. ip, reason, ttl)
if not ok then
return false, "datastore:set() error : " .. err
end
-- Set on redis
local use_redis, err = utils.get_variable("USE_REDIS", false)
if not use_redis then
return nil, "can't get USE_REDIS variable : " .. err
elseif use_redis ~= "yes" then
return true, "success"
end
-- Connect
local clusterstore = require "bunkerweb.clusterstore":new()
local ok, err = clusterstore:connect()
if not ok then
return false, "can't connect to redis server : " .. err
end
-- SET call
local ok, err = clusterstore:call("set", "bans_ip_" .. ip, reason, "EX", ttl)
if not ok then
clusterstore:close()
return false, "redis SET failed : " .. err
end
clusterstore:close()
return true, "success"
end
return utils

68
src/bw/lua/plugins.lua
View File

@ -1,68 +0,0 @@
local datastore = require "datastore"
local cjson = require "cjson"
local plugins = {}
plugins.load = function(self, path)
-- Read plugin.json file
local file = io.open(path .. "/plugin.json")
if not file then
return false, "can't read plugin.json file"
end
-- Decode plugin.json
-- TODO : check return value of file:read and cjson.encode
local data = cjson.decode(file:read("*a"))
file:close()
-- Check required fields
local required_fields = {"id", "order", "name", "description", "version", "settings"}
for i, field in ipairs(required_fields) do
if data[field] == nil then
return false, "missing field " .. field .. " in plugin.json"
end
-- TODO : check values and types with regex
end
-- Get existing plugins
local list, err = plugins:list()
if not list then
return false, err
end
-- Add our plugin to existing list and sort it
table.insert(list, data)
table.sort(list, function (a, b)
return a.order < b.order
end)
-- Save new plugin list in datastore
local ok, err = datastore:set("plugins", cjson.encode(list))
if not ok then
return false, "can't save new plugin list"
end
-- Save default settings value
for variable, value in pairs(data.settings) do
ok, err = datastore:set("plugin_" .. data.id .. "_" .. variable, value["default"])
if not ok then
return false, "can't save default variable value of " .. variable .. " into datastore"
end
end
-- Return the plugin
return data, "success"
end
plugins.list = function(self)
-- Get encoded plugins from datastore
local encoded_plugins, err = datastore:get("plugins")
if not encoded_plugins then
return false, "can't get encoded plugins from datastore"
end
-- Decode and return the list
return cjson.decode(encoded_plugins), "success"
end
return plugins

73
src/bw/lua/redisutils.lua
View File

@ -1,73 +0,0 @@
local clusterstore = require "clusterstore"
local datastore = require "datastore"
local utils = require "utils"
local redisutils = {}
redisutils.ban = function(ip)
-- Connect
local redis_client, err = clusterstore:connect()
if not redis_client then
return nil, "can't connect to redis server : " .. err
end
-- Start transaction
local ok, err = redis_client:multi()
if not ok then
clusterstore:close(redis_client)
return nil, "MULTI failed : " .. err
end
-- Get ban
ok, err = redis_client:get("ban_" .. ip)
if not ok then
clusterstore:close(redis_client)
return nil, "GET failed : " .. err
end
-- Get ttl
ok, err = redis_client:ttl("ban_" .. ip)
if not ok then
clusterstore:close(redis_client)
return nil, "TTL failed : " .. err
end
-- Exec transaction
local exec, err = redis_client:exec()
if err then
clusterstore:close(redis_client)
return nil, "EXEC failed : " .. err
end
if type(exec) ~= "table" then
clusterstore:close(redis_client)
return nil, "EXEC result is not a table"
end
-- Extract ban reason
local reason = exec[1]
if type(reason) == "table" then
clusterstore:close(redis_client)
return nil, "GET failed : " .. reason[2]
end
if reason == ngx.null then
clusterstore:close(redis_client)
datastore:delete("bans_ip_" .. ip)
return false
end
-- Extract ttl
local ttl = exec[2]
if type(ttl) == "table" then
clusterstore:close(redis_client)
return nil, "TTL failed : " .. ttl[2]
end
if ttl <= 0 then
clusterstore:close(redis_client)
return nil, "TTL returned invalid value : " .. tostring(ttl)
end
ok, err = datastore:set("bans_ip_" .. ip, reason, ttl)
if not ok then
clusterstore:close(redis_client)
datastore:delete("bans_ip_" .. ip)
return nil, "can't save ban to local datastore : " .. err
end
-- Return reason
clusterstore:close(redis_client)
return true, reason
end
return redisutils

46
src/common/confs/api.conf
View File

@ -15,26 +15,52 @@ server {
# check IP and do the API call
access_by_lua_block {
local capi = require "bunkerweb.api"
local clogger = require "bunkerweb.logger"
local logger = clogger:new("API")
local api = capi:new()
if not ngx.var.http_host or ngx.var.http_host ~= "{{ API_SERVER_NAME }}" then
logger:log(ngx.WARN, "wrong Host header from IP " .. ngx.var.remote_addr)
-- Instantiate objects and import required modules
local logger = require "bunkerweb.logger":new("API")
local api = require "bunkerweb.api":new()
local helpers = require "bunkerweb.helpers"
-- Start API handler
logger:log(ngx.INFO, "API handler started")
-- Fill ctx
logger:log(ngx.INFO, "filling ngx.ctx ...")
local ok, ret, errors = helpers.fill_ctx()
if not ok then
logger:log(ngx.ERR, "fill_ctx() failed : " .. ret)
elseif errors then
for i, error in ipairs(errors) do
logger:log(ngx.ERR, "fill_ctx() error " .. tostring(i) .. " : " .. error)
end
end
logger:log(ngx.INFO, "ngx.ctx filled (ret = " .. ret .. ")")
-- Check host header
if not ngx.ctx.bw.http_host or ngx.ctx.bw.http_host ~= "{{ API_SERVER_NAME }}" then
logger:log(ngx.WARN, "wrong Host header from IP " .. ngx.ctx.bw.remote_addr)
return ngx.exit(ngx.HTTP_CLOSE)
end
-- Check IP
local ok, err = api:is_allowed_ip()
if not ok then
logger:log(ngx.WARN, "can't validate access from IP " .. ngx.var.remote_addr .. " : " .. err)
logger:log(ngx.WARN, "can't validate access from IP " .. ngx.ctx.bw.remote_addr .. " : " .. err)
return ngx.exit(ngx.HTTP_CLOSE)
end
logger:log(ngx.NOTICE, "validated access from IP " .. ngx.var.remote_addr)
logger:log(ngx.NOTICE, "validated access from IP " .. ngx.ctx.bw.remote_addr)
-- Do API call
local ok, err, status, resp = api:do_api_call()
if not ok then
logger:log(ngx.WARN, "call from " .. ngx.var.remote_addr .. " on " .. ngx.var.uri .. " failed : " .. err)
logger:log(ngx.WARN, "call from " .. ngx.ctx.bw.remote_addr .. " on " .. ngx.ctx.bw.uri .. " failed : " .. err)
else
logger:log(ngx.NOTICE, "successful call from " .. ngx.var.remote_addr .. " on " .. ngx.var.uri .. " : " .. err)
logger:log(ngx.NOTICE, "successful call from " .. ngx.ctx.bw.remote_addr .. " on " .. ngx.ctx.bw.uri .. " : " .. err)
end
-- Start API handler
logger:log(ngx.INFO, "API handler ended")
-- Send response
ngx.status = status
ngx.say(resp)
return ngx.exit(status)

12
src/common/confs/default-server-http.conf
View File

@ -44,6 +44,18 @@ server {
local datastore = cdatastore:new()
logger:log(ngx.INFO, "log_default phase started")
-- Fill ctx
logger:log(ngx.INFO, "filling ngx.ctx ...")
local ok, ret, errors = helpers.fill_ctx()
if not ok then
logger:log(ngx.ERR, "fill_ctx() failed : " .. ret)
elseif errors then
for i, error in ipairs(errors) do
logger:log(ngx.ERR, "fill_ctx() error " .. tostring(i) .. " : " .. error)
end
end
logger:log(ngx.INFO, "ngx.ctx filled (ret = " .. ret .. ")")
-- Get plugins
local plugins, err = datastore:get("plugins")
if not plugins then

3
src/common/confs/http.conf
View File

@ -53,6 +53,9 @@ lua_shared_dict cachestore_locks {{ CACHESTORE_LOCKS_MEMORY_SIZE }};
# LUA init block
include /etc/nginx/init-lua.conf;
# LUA init worker block
include /etc/nginx/init-worker-lua.conf;
# API server
{% if USE_API == "yes" %}include /etc/nginx/api.conf;{% endif +%}

94
src/common/confs/init-lua.conf
View File

@ -1,76 +1,76 @@
init_by_lua_block {
local class = require "middleclass"
local logger = require "bunkerweb.logger"
local helpers = require "bunkerweb.helpers"
local datastore = require "bunkerweb.datastore"
local cjson = require "cjson"
local class = require "middleclass"
local clogger = require "bunkerweb.logger"
local helpers = require "bunkerweb.helpers"
local cdatastore = require "bunkerweb.datastore"
local cjson = require "cjson"
-- Start init phase
local init_logger = logger:new("INIT")
local ds = datastore:new()
init_logger:log(ngx.NOTICE, "init phase started")
local logger = clogger:new("INIT")
local datastore = cdatastore:new()
logger:log(ngx.NOTICE, "init phase started")
-- Remove previous data from the datastore
init_logger:log(ngx.NOTICE, "deleting old keys from datastore ...")
logger:log(ngx.NOTICE, "deleting old keys from datastore ...")
local data_keys = {"^plugin_", "^variable_", "^plugins$", "^api_", "^misc_"}
for i, key in pairs(data_keys) do
local ok, err = ds:delete_all(key)
local ok, err = datastore:delete_all(key)
if not ok then
init_logger:log(ngx.ERR, "can't delete " .. key .. " from datastore : " .. err)
logger:log(ngx.ERR, "can't delete " .. key .. " from datastore : " .. err)
return false
end
init_logger:log(ngx.INFO, "deleted " .. key .. " from datastore")
logger:log(ngx.INFO, "deleted " .. key .. " from datastore")
end
init_logger:log(ngx.NOTICE, "deleted old keys from datastore")
logger:log(ngx.NOTICE, "deleted old keys from datastore")
-- Load variables into the datastore
init_logger:log(ngx.NOTICE, "saving variables into datastore ...")
logger:log(ngx.NOTICE, "saving variables into datastore ...")
local file = io.open("/etc/nginx/variables.env")
if not file then
init_logger:log(ngx.ERR, "can't open /etc/nginx/variables.env file")
logger:log(ngx.ERR, "can't open /etc/nginx/variables.env file")
return false
end
file:close()
for line in io.lines("/etc/nginx/variables.env") do
local variable, value = line:match("(.+)=(.*)")
local ok, err = ds:set("variable_" .. variable, value)
local ok, err = datastore:set("variable_" .. variable, value)
if not ok then
init_logger:log(ngx.ERR, "can't save variable " .. variable .. " into datastore : " .. err)
logger:log(ngx.ERR, "can't save variable " .. variable .. " into datastore : " .. err)
return false
end
init_logger:log(ngx.INFO, "saved variable " .. variable .. "=" .. value .. " into datastore")
logger:log(ngx.INFO, "saved variable " .. variable .. "=" .. value .. " into datastore")
end
init_logger:log(ngx.NOTICE, "saved variables into datastore")
logger:log(ngx.NOTICE, "saved variables into datastore")
-- Set API values into the datastore
init_logger:log(ngx.NOTICE, "saving API values into datastore ...")
local value, err = ds:get("variable_USE_API")
logger:log(ngx.NOTICE, "saving API values into datastore ...")
local value, err = datastore:get("variable_USE_API")
if not value then
init_logger:log(ngx.ERR, "can't get variable USE_API from the datastore : " .. err)
logger:log(ngx.ERR, "can't get variable USE_API from the datastore : " .. err)
return false
end
if value == "yes" then
local value, err = ds:get("variable_API_WHITELIST_IP")
local value, err = datastore:get("variable_API_WHITELIST_IP")
if not value then
init_logger:log(ngx.ERR, "can't get variable API_WHITELIST_IP from the datastore : " .. err)
logger:log(ngx.ERR, "can't get variable API_WHITELIST_IP from the datastore : " .. err)
return false
end
local whitelists = {}
for whitelist in value:gmatch("%S+") do
table.insert(whitelists, whitelist)
end
local ok, err = ds:set("api_whitelist_ip", cjson.encode(whitelists))
local ok, err = datastore:set("api_whitelist_ip", cjson.encode(whitelists))
if not ok then
init_logger:log(ngx.ERR, "can't save API whitelist_ip to datastore : " .. err)
logger:log(ngx.ERR, "can't save API whitelist_ip to datastore : " .. err)
return false
end
init_logger:log(ngx.INFO, "saved API whitelist_ip into datastore")
logger:log(ngx.INFO, "saved API whitelist_ip into datastore")
end
init_logger:log(ngx.NOTICE, "saved API values into datastore")
logger:log(ngx.NOTICE, "saved API values into datastore")
-- Load plugins into the datastore
init_logger:log(ngx.NOTICE, "saving plugins into datastore ...")
logger:log(ngx.NOTICE, "saving plugins into datastore ...")
local plugins = {}
local plugin_paths = {"/usr/share/bunkerweb/core", "/etc/bunkerweb/plugins"}
for i, plugin_path in ipairs(plugin_paths) do
@ -78,61 +78,61 @@ for i, plugin_path in ipairs(plugin_paths) do
for path in paths:lines() do
local ok, plugin = helpers.load_plugin(path .. "/plugin.json")
if not ok then
init_logger:log(ngx.ERR, plugin)
logger:log(ngx.ERR, plugin)
else
local ok, err = ds:set("plugin_" .. plugin.id, cjson.encode(plugin))
local ok, err = datastore:set("plugin_" .. plugin.id, cjson.encode(plugin))
if not ok then
init_logger:log(ngx.ERR, "can't save " .. plugin.id .. " into datastore : " .. err)
logger:log(ngx.ERR, "can't save " .. plugin.id .. " into datastore : " .. err)
else
table.insert(plugins, plugin)
table.sort(plugins, function (a, b)
return a.order < b.order
end)
init_logger:log(ngx.NOTICE, "loaded plugin " .. plugin.id .. " v" .. plugin.version)
logger:log(ngx.NOTICE, "loaded plugin " .. plugin.id .. " v" .. plugin.version)
end
end
end
end
local ok, err = ds:set("plugins", cjson.encode(plugins))
local ok, err = datastore:set("plugins", cjson.encode(plugins))
if not ok then
init_logger:log(ngx.ERR, "can't save plugins into datastore : " .. err)
logger:log(ngx.ERR, "can't save plugins into datastore : " .. err)
return false
end
init_logger:log(ngx.NOTICE, "saved plugins into datastore")
logger:log(ngx.NOTICE, "saved plugins into datastore")
-- Call init() methods
init_logger:log(ngx.NOTICE, "calling init() methods of plugins ...")
-- Call init() methodatastore
logger:log(ngx.NOTICE, "calling init() methods of plugins ...")
for i, plugin in ipairs(plugins) do
-- Require call
local plugin_lua, err = helpers.require_plugin(plugin.id)
if plugin_lua == false then
init_logger:log(ngx.ERR, err)
logger:log(ngx.ERR, err)
elseif plugin_lua == nil then
init_logger:log(ngx.NOTICE, err)
logger:log(ngx.NOTICE, err)
else
-- Check if plugin has init method
if plugin_lua.init ~= nil then
-- New call
local ok, plugin_obj = helpers.new_plugin(plugin_lua)
if not ok then
init_logger:log(ngx.ERR, plugin_obj)
logger:log(ngx.ERR, plugin_obj)
else
local ok, ret = helpers.call_plugin(plugin_obj, "init")
if not ok then
init_logger:log(ngx.ERR, ret)
logger:log(ngx.ERR, ret)
elseif not ret.ret then
init_logger:log(ngx.ERR, plugin.id .. ":init() call failed : " .. ret.msg)
logger:log(ngx.ERR, plugin.id .. ":init() call failed : " .. ret.msg)
else
init_logger:log(ngx.NOTICE, plugin.id .. ":init() call successful : " .. ret.msg)
logger:log(ngx.NOTICE, plugin.id .. ":init() call successful : " .. ret.msg)
end
end
else
init_logger:log(ngx.NOTICE, "skipped execution of " .. plugin.id .. " because method init() is not defined")
logger:log(ngx.NOTICE, "skipped execution of " .. plugin.id .. " because method init() is not defined")
end
end
end
init_logger:log(ngx.NOTICE, "called init() methods of plugins")
logger:log(ngx.NOTICE, "called init() methods of plugins")
init_logger:log(ngx.NOTICE, "init phase ended")
logger:log(ngx.NOTICE, "init phase ended")
}

39
src/common/confs/init-worker-lua.conf Normal file
View File

@ -0,0 +1,39 @@
lua_shared_dict ready_lock 16k;
init_worker_by_lua_block {
-- Our timer function
local ready_log = function(premature)
-- Instantiate objects
local logger = require "bunkerweb.logger":new("INIT")
local datastore = require "bunkerweb.datastore":new()
local lock = require "resty.lock":new("ready_lock")
if not lock then
logger:log(ngx.ERR, "lock:new() failed : " .. err)
return
end
-- Acquire lock
local elapsed, err = lock:lock("ready")
if elapsed == nil then
logger:log(ngx.ERR, "lock:lock() failed : " .. err)
else
-- Display ready log
local ok, err = datastore:get("misc_ready")
if not ok and err ~= "not found" then
logger:log(ngx.ERR, "datastore:get() failed : " .. err)
elseif not ok and err == "not found" then
logger:log(ngx.NOTICE, "BunkerWeb is ready to fool hackers ! 🚀")
local ok, err = datastore:set("misc_ready", "ok")
if not ok then
logger:log(ngx.ERR, "datastore:set() failed : " .. err)
end
end
end
-- Release lock
lock:unlock()
end
-- Start timer
ngx.timer.at(5, ready_log)
}

50
src/common/confs/server-http/access-lua.conf
View File

@ -5,7 +5,7 @@ local clogger = require "bunkerweb.logger"
local helpers = require "bunkerweb.helpers"
local utils = require "bunkerweb.utils"
local cdatastore = require "bunkerweb.datastore"
local ccachestore = require "bunkerweb.cachestore"
local cclusterstore = require "bunkerweb.clusterstore"
local cjson = require "cjson"
-- Don't process internal requests
@ -17,19 +17,8 @@ end
-- Start access phase
local datastore = cdatastore:new()
local use_redis, err = utils.get_variable("USE_REDIS", false)
if not use_redis then
logger:log(ngx.ERR, err)
end
local cachestore = ccachestore:new(use_redis == "yes")
logger:log(ngx.INFO, "access phase started")
-- Update cachestore only once and before any other code
local ok, err = cachestore.cache:update()
if not ok then
logger:log(ngx.ERR, "can't update cachestore : " .. err)
end
-- Fill ctx
logger:log(ngx.INFO, "filling ngx.ctx ...")
local ok, ret, errors = helpers.fill_ctx()
@ -43,13 +32,14 @@ end
logger:log(ngx.INFO, "ngx.ctx filled (ret = " .. ret .. ")")
-- Process bans as soon as possible
local ok, reason = datastore:get("bans_ip_" .. ngx.var.remote_addr)
if not ok and reason ~= "not found" then
logger:log(ngx.INFO, "error while checking if client is banned : " .. reason)
return false
elseif ok and reason ~= "not found" then
logger:log(ngx.WARN, "IP " .. ngx.var.remote_addr .. " is banned with reason : " .. reason)
local banned, reason, ttl = utils.is_banned(ngx.ctx.bw.remote_addr)
if banned == nil then
logger:log(ngx.ERR, "can't check if IP " .. ngx.ctx.bw.remote_addr .. " is banned : " .. reason)
elseif banned then
logger:log(ngx.WARN, "IP " .. ngx.ctx.bw.remote_addr .. " is banned with reason " .. reason .. " (" .. tostring(ttl) .. "s remaining)")
return ngx.exit(utils.get_deny_status())
else
logger:log(ngx.INFO, "IP " .. ngx.ctx.bw.remote_addr .. " is not banned")
end
-- Get plugins
@ -62,6 +52,8 @@ plugins = cjson.decode(plugins)
-- Call access() methods
logger:log(ngx.INFO, "calling access() methods of plugins ...")
local status = nil
local redirect = nil
for i, plugin in ipairs(plugins) do
-- Require call
local plugin_lua, err = helpers.require_plugin(plugin.id)
@ -83,20 +75,20 @@ for i, plugin in ipairs(plugins) do
elseif not ret.ret then
logger:log(ngx.ERR, plugin.id .. ":access() call failed : " .. ret.msg)
else
logger:log(ngx.NOTICE, plugin.id .. ":access() call successful : " .. ret.msg)
logger:log(ngx.INFO, plugin.id .. ":access() call successful : " .. ret.msg)
end
if ret.status then
if ret.status == utils.get_deny_status() then
ngx.ctx.reason = plugin.id
logger:log(ngx.WARN, "denied access from " .. plugin.id .. " : " .. err)
logger:log(ngx.WARN, "denied access from " .. plugin.id .. " : " .. ret.msg)
else
logger:log(ngx.NOTICE, plugin.id .. " returned status " .. tostring(ret.status))
logger:log(ngx.NOTICE, plugin.id .. " returned status " .. tostring(ret.status) .. " : " .. ret.msg)
end
ngx.ctx.status = ret.status
status = ret.status
break
elseif ret.redirect then
logger:log(ngx.NOTICE, plugin.id .. " redirect to " .. ret.redirect .. " : " .. err)
ngx.ctx.redirect = ret.redirect
logger:log(ngx.NOTICE, plugin.id .. " redirect to " .. ret.redirect .. " : " .. ret.msg)
redirect = ret.redirect
break
end
end
@ -111,18 +103,20 @@ logger:log(ngx.INFO, "called access() methods of plugins")
local ok, err = utils.save_session()
if not ok then
logger:log(ngx.ERR, "can't save session : " .. err)
else
logger:log(ngx.INFO, "session save return : " .. err)
end
logger:log(ngx.INFO, "access phase ended")
-- Return status if needed
if ngx.ctx.status then
return ngx.exit(ngx.ctx.status)
if status then
return ngx.exit(status)
end
-- Redirect if needed
if ngx.ctx.redirect then
return ngx.redirect(ngx.ctx.redirect)
if redirect then
return ngx.redirect(redirect)
end
return true

14
src/common/confs/server-http/header-lua.conf
View File

@ -17,6 +17,18 @@ end
local datastore = cdatastore:new()
logger:log(ngx.INFO, "header phase started")
-- Fill ctx
logger:log(ngx.INFO, "filling ngx.ctx ...")
local ok, ret, errors = helpers.fill_ctx()
if not ok then
logger:log(ngx.ERR, "fill_ctx() failed : " .. ret)
elseif errors then
for i, error in ipairs(errors) do
logger:log(ngx.ERR, "fill_ctx() error " .. tostring(i) .. " : " .. error)
end
end
logger:log(ngx.INFO, "ngx.ctx filled (ret = " .. ret .. ")")
-- Get plugins
local plugins, err = datastore:get("plugins")
if not plugins then
@ -48,7 +60,7 @@ for i, plugin in ipairs(plugins) do
elseif not ret.ret then
logger:log(ngx.ERR, plugin.id .. ":header() call failed : " .. ret.msg)
else
logger:log(ngx.NOTICE, plugin.id .. ":header() call successful : " .. ret.msg)
logger:log(ngx.INFO, plugin.id .. ":header() call successful : " .. ret.msg)
end
end
else

14
src/common/confs/server-http/log-lua.conf
View File

@ -11,6 +11,18 @@ local logger = clogger:new("LOG")
local datastore = cdatastore:new()
logger:log(ngx.INFO, "log phase started")
-- Fill ctx
logger:log(ngx.INFO, "filling ngx.ctx ...")
local ok, ret, errors = helpers.fill_ctx()
if not ok then
logger:log(ngx.ERR, "fill_ctx() failed : " .. ret)
elseif errors then
for i, error in ipairs(errors) do
logger:log(ngx.ERR, "fill_ctx() error " .. tostring(i) .. " : " .. error)
end
end
logger:log(ngx.INFO, "ngx.ctx filled (ret = " .. ret .. ")")
-- Get plugins
local plugins, err = datastore:get("plugins")
if not plugins then
@ -42,7 +54,7 @@ for i, plugin in ipairs(plugins) do
elseif not ret.ret then
logger:log(ngx.ERR, plugin.id .. ":log() call failed : " .. ret.msg)
else
logger:log(ngx.NOTICE, plugin.id .. ":log() call successful : " .. ret.msg)
logger:log(ngx.INFO, plugin.id .. ":log() call successful : " .. ret.msg)
end
end
else

22
src/common/confs/server-http/set-lua.conf
View File

@ -5,6 +5,7 @@ local class = require "middleclass"
local clogger = require "bunkerweb.logger"
local helpers = require "bunkerweb.helpers"
local cdatastore = require "bunkerweb.datastore"
local ccachestore = require "bunkerweb.cachestore"
local cjson = require "cjson"
-- Don't process internal requests
@ -18,6 +19,25 @@ end
local datastore = cdatastore:new()
logger:log(ngx.INFO, "set phase started")
-- Update cachestore only once and before any other code
local cachestore = ccachestore:new()
local ok, err = cachestore.cache:update()
if not ok then
logger:log(ngx.ERR, "can't update cachestore : " .. err)
end
-- Fill ctx
logger:log(ngx.INFO, "filling ngx.ctx ...")
local ok, ret, errors = helpers.fill_ctx()
if not ok then
logger:log(ngx.ERR, "fill_ctx() failed : " .. ret)
elseif errors then
for i, error in ipairs(errors) do
logger:log(ngx.ERR, "fill_ctx() error " .. tostring(i) .. " : " .. error)
end
end
logger:log(ngx.INFO, "ngx.ctx filled (ret = " .. ret .. ")")
-- Get plugins
local plugins, err = datastore:get("plugins")
if not plugins then
@ -49,7 +69,7 @@ for i, plugin in ipairs(plugins) do
elseif not ret.ret then
logger:log(ngx.ERR, plugin.id .. ":set() call failed : " .. ret.msg)
else
logger:log(ngx.NOTICE, plugin.id .. ":set() call successful : " .. ret.msg)
logger:log(ngx.INFO, plugin.id .. ":set() call successful : " .. ret.msg)
end
end
else

167
src/common/core/antibot/antibot.lua
View File

@ -23,46 +23,59 @@ function antibot:access()
return self:ret(true, "antibot not activated")
end
-- Prepare challenge
local ok, err = self:prepare_challenge(antibot, challenge_uri)
if not ok then
return self:ret(false, "can't prepare challenge : " .. err, ngx.HTTP_INTERNAL_SERVER_ERROR)
end
-- Don't go further if client resolved the challenge
local resolved, err, original_uri = self:challenge_resolved(antibot)
local resolved, err, original_uri = self:challenge_resolved()
if resolved == nil then
return self:ret(false, "can't check if challenge is resolved : " .. err)
end
if resolved then
if ngx.var.uri == challenge_uri then
if ngx.ctx.bw.uri == self.variables["ANTIBOT_URI"] then
return self:ret(true, "client already resolved the challenge", nil, original_uri)
end
return self:ret(true, "client already resolved the challenge")
end
-- Redirect to challenge page
if ngx.var.uri ~= challenge_uri then
return self:ret(true, "redirecting client to the challenge uri", nil, challenge_uri)
if ngx.ctx.bw.uri ~= self.variables["ANTIBOT_URI"] then
-- Prepare challenge
local ok, err = self:prepare_challenge()
if not ok then
return self:ret(false, "can't prepare challenge : " .. err, ngx.HTTP_INTERNAL_SERVER_ERROR)
end
return self:ret(true, "redirecting client to the challenge uri", nil, self.variables["ANTIBOT_URI"])
end
-- Direct access without session => prepare challenge
if not self:challenge_prepared() then
local ok, err = self:prepare_challenge()
if not ok then
return self:ret(false, "can't prepare challenge : " .. err, ngx.HTTP_INTERNAL_SERVER_ERROR)
end
end
-- Display challenge needed
if ngx.var.request_method == "GET" then
ngx.ctx.antibot_display_content = true
return self:ret(true, "displaying challenge to client", ngx.HTTP_OK)
if ngx.ctx.bw.request_method == "GET" then
ngx.ctx.bw.antibot_display_content = true
return self:ret(true, "displaying challenge to client", ngx.OK)
end
-- Check challenge
if ngx.var.request_method == "POST" then
local ok, err, redirect = self:check_challenge(antibot)
if ngx.ctx.bw.request_method == "POST" then
local ok, err, redirect = self:check_challenge()
if ok == nil then
return self:ret(false, "check challenge error : " .. err, ngx.HTTP_INTERNAL_SERVER_ERROR)
elseif not ok then
self.logger:log(ngx.WARN, "client failed challenge : " .. err)
local ok, err = self:prepare_challenge()
if not ok then
return self:ret(false, "can't prepare challenge : " .. err, ngx.HTTP_INTERNAL_SERVER_ERROR)
end
end
if redirect then
return self:ret(true, "check challenge redirect : " .. redirect, nil, redirect)
end
ngx.ctx.antibot_display_content = true
return self:ret(true, "displaying challenge to client", ngx.HTTP_OK)
ngx.ctx.bw.antibot_display_content = true
return self:ret(true, "displaying challenge to client", ngx.OK)
end
-- Method is suspicious, let's deny the request
@ -70,20 +83,16 @@ function antibot:access()
end
function antibot:content()
-- Check if access is needed
local antibot, err = utils.get_variable("USE_ANTIBOT")
if antibot == nil then
return self:ret(false, err)
end
if antibot == "no" then
-- Check if content is needed
if not self.variables["USE_ANTIBOT"] or self.variables["USE_ANTIBOT"] == "no" then
return self:ret(true, "antibot not activated")
end
-- Check if display content is needed
if not ngx.ctx.antibot_display_content then
return self:ret(true, "display content not needed")
if not ngx.ctx.bw.antibot_display_content then
return self:ret(true, "display content not needed", nil, "/")
end
-- Display content
local ok, err = self:display_challenge(antibot)
local ok, err = self:display_challenge()
if not ok then
return self:ret(false, "display challenge error : " .. err)
end
@ -91,41 +100,50 @@ function antibot:content()
end
function antibot:challenge_resolved()
local session, err, exists = utils.get_session()
if err then
return false, "session error : " .. err
local session, err, exists, refreshed = utils.get_session()
if not exists then
return false, "no session set"
end
local raw_data = get_session("antibot")
local ok, err, raw_data = utils.get_session_var("antibot")
if not raw_data then
return false, "session is set but no antibot data", nil
return false, "session is set but no antibot data"
end
local data = cjson.decode(raw_data)
if data.resolved and self.variables["USE_ANTIBOT"] == data.antibot then
local data = raw_data
if data.resolved and self.variables["USE_ANTIBOT"] == data.type then
return true, "challenge resolved", data.original_uri
end
return false, "challenge not resolved", data.original_uri
end
function antibot:prepare_challenge()
local session, err, exists = utils.get_session()
if err then
return false, "session error : " .. err
function antibot:challenge_prepared()
local session, err, exists, refreshed = utils.get_session()
if not exists then
return false
end
local ok, err, raw_data = utils.get_session_var("antibot")
if not raw_data then
return false
end
return self.variables["USE_ANTIBOT"] == raw_data.type
end
function antibot:prepare_challenge()
local session, err, exists, refreshed = utils.get_session()
local set_needed = false
local data = nil
if exists then
local raw_data = get_session("antibot")
local ok, err, raw_data = utils.get_session_var("antibot")
if raw_data then
data = cjson.decode(raw_data)
data = raw_data
end
end
if not data or current_data.antibot ~= self.variables["USE_ANTIBOT"] then
if not data or data.type ~= self.variables["USE_ANTIBOT"] then
data = {
type = self.variables["USE_ANTIBOT"],
resolved = self.variables["USE_ANTIBOT"] == "cookie",
original_uri = ngx.var.request_uri
original_uri = ngx.ctx.bw.request_uri
}
if ngx.var.original_uri == challenge_uri then
if ngx.ctx.bw.uri == self.variables["ANTIBOT_URI"] then
data.original_uri = "/"
end
set_needed = true
@ -144,24 +162,27 @@ function antibot:prepare_challenge()
end
end
if set_needed then
utils.set_session("antibot", cjson.encode(data))
local ok, err = utils.set_session_var("antibot", data)
if not ok then
return false, "error while setting session antibot : " .. err
end
end
return true, "prepared"
end
function antibot:display_challenge(challenge_uri)
function antibot:display_challenge()
-- Open session
local session, err, exists = utils.get_session()
if err then
return false, "can't open session : " .. err
local session, err, exists, refreshed = utils.get_session()
if not exists then
return false, "no session set"
end
-- Get data
local raw_data = get_session("antibot")
local ok, err, raw_data = utils.get_session_var("antibot")
if not raw_data then
return false, "session is set but no data"
end
local data = cjson.decode(raw_data)
local data = raw_data
-- Check if session type is equal to antibot type
if self.variables["USE_ANTIBOT"] ~= data.type then
@ -201,20 +222,20 @@ end
function antibot:check_challenge()
-- Open session
local session, err, exists = utils.get_session()
if err then
return nil, "can't open session : " .. err, nil
local session, err, exists, refreshed = utils.get_session()
if not exists then
return false, "no session set"
end
-- Get data
local raw_data = get_session("antibot")
local ok, err, raw_data = utils.get_session_var("antibot")
if not raw_data then
return false, "session is set but no data", nil
end
local data = cjson.decode(raw_data)
local data = raw_data
-- Check if session type is equal to antibot type
if elf.variables["USE_ANTIBOT"] ~= data.type then
if self.variables["USE_ANTIBOT"] ~= data.type then
return nil, "session type is different from antibot type", nil
end
@ -227,7 +248,7 @@ function antibot:check_challenge()
ngx.req.read_body()
local args, err = ngx.req.get_post_args(1)
if err == "truncated" or not args or not args["challenge"] then
return false, "missing challenge arg", nil
return nil, "missing challenge arg", nil
end
local hash = sha256:new()
hash:update(data.random .. args["challenge"])
@ -237,7 +258,10 @@ function antibot:check_challenge()
return false, "wrong value", nil
end
data.resolved = true
utils.set_session("antibot", cjson.encode(data))
local ok, err = utils.set_session_var("antibot", data)
if not ok then
return nil, "error while setting session antibot : " .. err
end
return true, "resolved", data.original_uri
end
@ -246,13 +270,16 @@ function antibot:check_challenge()
ngx.req.read_body()
local args, err = ngx.req.get_post_args(1)
if err == "truncated" or not args or not args["captcha"] then
return false, "missing challenge arg", nil
return nil, "missing challenge arg", nil
end
if data.text ~= args["captcha"] then
return false, "wrong value", nil
end
data.resolved = true
utils.set_session("antibot", cjson.encode(data))
local ok, err = utils.set_session_var("antibot", data)
if not ok then
return nil, "error while setting session antibot : " .. err
end
return true, "resolved", data.original_uri
end
@ -261,15 +288,15 @@ function antibot:check_challenge()
ngx.req.read_body()
local args, err = ngx.req.get_post_args(1)
if err == "truncated" or not args or not args["token"] then
return false, "missing challenge arg", nil
return nil, "missing challenge arg", nil
end
local httpc, err = http.new()
if not httpc then
return false, "can't instantiate http object : " .. err, nil, nil
return nil, "can't instantiate http object : " .. err, nil, nil
end
local res, err = httpc:request_uri("https://www.google.com/recaptcha/api/siteverify", {
method = "POST",
body = "secret=" .. self.variables["ANTIBOT_RECAPTCHA_SECRET"] .. "&response=" .. args["token"] .. "&remoteip=" .. ngx.var.remote_addr,
body = "secret=" .. self.variables["ANTIBOT_RECAPTCHA_SECRET"] .. "&response=" .. args["token"] .. "&remoteip=" .. ngx.ctx.bw.remote_addr,
headers = {
["Content-Type"] = "application/x-www-form-urlencoded"
}
@ -286,7 +313,10 @@ function antibot:check_challenge()
return false, "client failed challenge with score " .. tostring(rdata.score), nil
end
data.resolved = true
utils.set_session("antibot", cjson.encode(data))
local ok, err = utils.set_session_var("antibot", data)
if not ok then
return nil, "error while setting session antibot : " .. err
end
return true, "resolved", data.original_uri
end
@ -295,15 +325,15 @@ function antibot:check_challenge()
ngx.req.read_body()
local args, err = ngx.req.get_post_args(1)
if err == "truncated" or not args or not args["token"] then
return false, "missing challenge arg", nil
return nil, "missing challenge arg", nil
end
local httpc, err = http.new()
if not httpc then
return false, "can't instantiate http object : " .. err, nil, nil
return nil, "can't instantiate http object : " .. err, nil, nil
end
local res, err = httpc:request_uri("https://hcaptcha.com/siteverify", {
method = "POST",
body = "secret=" .. self.variables["ANTIBOT_HCAPTCHA_SECRET"] .. "&response=" .. args["token"] .. "&remoteip=" .. ngx.var.remote_addr,
body = "secret=" .. self.variables["ANTIBOT_HCAPTCHA_SECRET"] .. "&response=" .. args["token"] .. "&remoteip=" .. ngx.ctx.bw.remote_addr,
headers = {
["Content-Type"] = "application/x-www-form-urlencoded"
}
@ -320,7 +350,10 @@ function antibot:check_challenge()
return false, "client failed challenge", nil
end
data.resolved = true
utils.set_session("antibot", cjson.encode(data))
local ok, err = utils.set_session_var("antibot", data)
if not ok then
return nil, "error while setting session antibot : " .. err
end
return true, "resolved", data.original_uri
end

14
src/common/core/antibot/confs/server-http/antibot.conf
View File

@ -1,16 +1,18 @@
{% if USE_ANTIBOT == "yes" +%}
location /{{ ANTIBOT_URI }} {
{% if USE_ANTIBOT != "no" +%}
location {{ ANTIBOT_URI }} {
default_type 'text/html';
root /usr/share/bunkerweb/core/antibot/files;
content_by_lua_block {
local cantibot = require "antibot.antibot"
local clogger = require "bunkerweb.logger"
local antibot = cantibot:new()
local logger = clogger:new("ANTIBOT")
local ok, err = antibot:content()
if not ok then
logger:log(ngx.ERR, "antibot:content() failed : " .. err)
local ret = antibot:content()
if not ret.ret then
logger:log(ngx.ERR, "antibot:content() failed : " .. ret.msg)
else
logger:log(ngx.INFO, "antibot:content() success : " .. err)
logger:log(ngx.INFO, "antibot:content() success : " .. ret.msg)
end
}
}

482
src/common/core/antibot/files/captcha.html
View File

File diff suppressed because one or more lines are too long

2
src/common/core/antibot/plugin.json
View File

@ -1,6 +1,6 @@
{
"id": "antibot",
"order": 8,
"order": 9,
"name": "Antibot",
"description": "Bot detection by using a challenge.",
"version": "0.1",

177
src/common/core/badbehavior/badbehavior.lua
View File

@ -1,8 +1,6 @@
local class = require "middleclass"
local plugin = require "bunkerweb.plugin"
local utils = require "bunkerweb.utils"
local datastore = require "bunkerweb.datastore"
local clusterstore = require "bunkerweb.clusterstore"
local badbehavior = class("badbehavior", plugin)
@ -19,7 +17,7 @@ end
function badbehavior:log()
-- Check if we are whitelisted
if ngx.var.is_whitelisted == "yes" then
if ngx.ctx.bw.is_whitelisted == "yes" then
return self:ret(true, "client is whitelisted")
end
-- Check if bad behavior is activated
@ -31,12 +29,12 @@ function badbehavior:log()
return self:ret(true, "not increasing counter")
end
-- Check if we are already banned
local banned, err = self.datastore:get("bans_ip_" .. ngx.var.remote_addr)
local banned, err = self.datastore:get("bans_ip_" .. ngx.ctx.bw.remote_addr)
if banned then
return self:ret(true, "already banned")
end
-- Call increase function later and with cosocket enabled
local ok, err = ngx.timer.at(0, badbehavior.increase, self, ngx.var.remote_addr)
local ok, err = ngx.timer.at(0, badbehavior.increase, ngx.ctx.bw.remote_addr, tonumber(self.variables["BAD_BEHAVIOR_COUNT_TIME"]), tonumber(self.variables["BAD_BEHAVIOR_BAN_TIME"]), tonumber(self.variables["BAD_BEHAVIOR_THRESHOLD"]), self.use_redis)
if not ok then
return self:ret(false, "can't create increase timer : " .. err)
end
@ -47,27 +45,26 @@ function badbehavior:log_default()
return self:log()
end
function badbehavior.increase(premature, obj, ip)
-- Our vars
local count_time = tonumber(obj.variables["BAD_BEHAVIOR_COUNT_TIME"])
local ban_time = tonumber(obj.variables["BAD_BEHAVIOR_BAN_TIME"])
local threshold = tonumber(obj.variables["BAD_BEHAVIOR_THRESHOLD"])
function badbehavior.increase(premature, ip, count_time, ban_time, threshold, use_redis)
-- Instantiate objects
local logger = require "bunkerweb.logger":new("badbehavior")
local datastore = require "bunkerweb.datastore":new()
-- Declare counter
local counter = false
-- Redis case
if obj.use_redis then
local redis_counter, err = obj:redis_increase(ip)
if use_redis then
local redis_counter, err = badbehavior.redis_increase(ip, count_time, ban_time)
if not redis_counter then
obj.logger:log(ngx.ERR, "(increase) redis_increase failed, falling back to local : " .. err)
logger:log(ngx.ERR, "(increase) redis_increase failed, falling back to local : " .. err)
else
counter = redis_counter
end
end
-- Local case
if not counter then
local local_counter, err = obj.datastore:get("plugin_badbehavior_count_" .. ip)
local local_counter, err = datastore:get("plugin_badbehavior_count_" .. ip)
if not local_counter and err ~= "not found" then
obj.logger:log(ngx.ERR, "(increase) can't get counts from the datastore : " .. err)
logger:log(ngx.ERR, "(increase) can't get counts from the datastore : " .. err)
end
if local_counter == nil then
local_counter = 0
@ -75,48 +72,48 @@ function badbehavior.increase(premature, obj, ip)
counter = local_counter + 1
end
-- Call decrease later
local ok, err = ngx.timer.at(count_time, badbehavior.decrease, obj, ip)
local ok, err = ngx.timer.at(count_time, badbehavior.decrease, ip, count_time, threshold, use_redis)
if not ok then
obj.logger:log(ngx.ERR, "(increase) can't create decrease timer : " .. err)
logger:log(ngx.ERR, "(increase) can't create decrease timer : " .. err)
end
-- Store local counter
local ok, err = obj.datastore:set("plugin_badbehavior_count_" .. ip, counter)
local ok, err = datastore:set("plugin_badbehavior_count_" .. ip, counter, count_time)
if not ok then
obj.logger:log(ngx.ERR, "(increase) can't save counts to the datastore : " .. err)
logger:log(ngx.ERR, "(increase) can't save counts to the datastore : " .. err)
return
end
-- Store local ban
if counter > threshold then
local ok, err = obj.datastore:set("bans_ip_" .. ip, "bad behavior", ban_time)
local ok, err = utils.add_ban(ip, "bad behavior", ban_time)
if not ok then
obj.logger:log(ngx.ERR, "(increase) can't save ban to the datastore : " .. err)
logger:log(ngx.ERR, "(increase) can't save ban : " .. err)
return
end
obj.logger:log(ngx.WARN, "IP " .. ip .. " is banned for " .. ban_time .. "s (" .. tostring(counter) .. "/" .. tostring(threshold) .. ")")
logger:log(ngx.WARN, "IP " .. ip .. " is banned for " .. ban_time .. "s (" .. tostring(counter) .. "/" .. tostring(threshold) .. ")")
end
logger:log(ngx.NOTICE, "increased counter for IP " .. ip .. " (" .. tostring(counter) .. "/" .. tostring(threshold) .. ")")
end
function badbehavior.decrease(premature, obj, ip)
-- Our vars
local count_time = tonumber(obj.variables["BAD_BEHAVIOR_COUNT_TIME"])
local ban_time = tonumber(obj.variables["BAD_BEHAVIOR_BAN_TIME"])
local threshold = tonumber(obj.variables["BAD_BEHAVIOR_THRESHOLD"])
function badbehavior.decrease(premature, ip, count_time, threshold, use_redis)
-- Instantiate objects
local logger = require "bunkerweb.logger":new("badbehavior")
local datastore = require "bunkerweb.datastore":new()
-- Declare counter
local counter = false
-- Redis case
if obj.use_redis then
local redis_counter, err = obj:redis_decrease(ip)
if use_redis then
local redis_counter, err = badbehavior.redis_decrease(ip, count_time)
if not redis_counter then
obj.logger:log(ngx.ERR, "(increase) redis_increase failed, falling back to local : " .. err)
logger:log(ngx.ERR, "(decrease) redis_decrease failed, falling back to local : " .. err)
else
counter = redis_counter
end
end
-- Local case
if not counter then
local local_counter, err = obj.datastore:get("plugin_badbehavior_count_" .. ip)
local local_counter, err = datastore:get("plugin_badbehavior_count_" .. ip)
if not local_counter and err ~= "not found" then
obj.logger:log(ngx.ERR, "(increase) can't get counts from the datastore : " .. err)
logger:log(ngx.ERR, "(decrease) can't get counts from the datastore : " .. err)
end
if local_counter == nil or local_counter <= 1 then
counter = 0
@ -126,92 +123,92 @@ function badbehavior.decrease(premature, obj, ip)
end
-- Store local counter
if counter <= 0 then
local ok, err = obj.datastore:delete("plugin_badbehavior_count_" .. ip)
counter = 0
local ok, err = datastore:delete("plugin_badbehavior_count_" .. ip)
else
local ok, err = obj.datastore:delete("plugin_badbehavior_count_" .. ip, counter)
local ok, err = datastore:set("plugin_badbehavior_count_" .. ip, counter, count_time)
if not ok then
obj.logger:log(ngx.ERR, "(increase) can't save counts to the datastore : " .. err)
logger:log(ngx.ERR, "(decrease) can't save counts to the datastore : " .. err)
return
end
end
logger:log(ngx.NOTICE, "decreased counter for IP " .. ip .. " (" .. tostring(counter) .. "/" .. tostring(threshold) .. ")")
end
function badbehavior:redis_increase(ip)
-- Our vars
local count_time = tonumber(self.variables["BAD_BEHAVIOR_COUNT_TIME"])
local ban_time = tonumber(self.variables["BAD_BEHAVIOR_BAN_TIME"])
function badbehavior.redis_increase(ip, count_time, ban_time)
-- Instantiate objects
local clusterstore = require "bunkerweb.clusterstore":new()
-- Our LUA script to execute on redis
local redis_script = [[
local ret_incr = redis.pcall("INCR", KEYS[1])
if type(ret_incr) == "table" and ret_incr["err"] ~= nil then
redis.log(redis.LOG_WARNING, "Bad behavior increase INCR error : " .. ret_incr["err"])
return ret_incr
end
local ret_expire = redis.pcall("EXPIRE", KEYS[1], ARGV[1])
if type(ret_expire) == "table" and ret_expire["err"] ~= nil then
redis.log(redis.LOG_WARNING, "Bad behavior increase EXPIRE error : " .. ret_expire["err"])
return ret_expire
end
if ret_incr > tonumber(ARGV[2]) then
local ret_set = redis.pcall("SET", KEYS[2], "bad behavior", "EX", ARGV[2])
if type(ret_set) == "table" and ret_set["err"] ~= nil then
redis.log(redis.LOG_WARNING, "Bad behavior increase SET error : " .. ret_set["err"])
return ret_set
end
end
return ret_incr
]]
-- Connect to server
local cstore, err = clusterstore:new()
if not cstore then
return false, err
end
local ok, err = clusterstore:connect()
if not ok then
return false, err
end
-- Exec transaction
local calls = {
{"incr", {"bad_behavior_" .. ip}},
{"expire", {"bad_behavior_" .. ip, count_time}}
}
local ok, err, exec = clusterstore:multi(calls)
if not ok then
-- Execute LUA script
local counter, err = clusterstore:call("eval", redis_script, 2, "bad_behavior_" .. ip, "bans_ip" .. ip, count_time, ban_time)
if not counter then
clusterstore:close()
return false, err
end
-- Extract counter
local counter = exec[1]
if type(counter) == "table" then
clusterstore:close()
return false, counter[2]
end
-- Check expire result
local expire = exec[2]
if type(expire) == "table" then
clusterstore:close()
return false, expire[2]
end
-- Add IP to redis bans if needed
if counter > threshold then
local ok, err = clusterstore:call("set", "ban_" .. ip, "bad behavior", "EX", ban_time)
if err then
clusterstore:close()
return false, err
end
end
-- End connection
clusterstore:close()
return counter
end
function badbehavior:redis_decrease(ip)
function badbehavior.redis_decrease(ip, count_time)
-- Instantiate objects
local clusterstore = require "bunkerweb.clusterstore":new()
-- Our LUA script to execute on redis
local redis_script = [[
local ret_decr = redis.pcall("DECR", KEYS[1])
if type(ret_decr) == "table" and ret_decr["err"] ~= nil then
redis.log(redis.LOG_WARNING, "Bad behavior decrease DECR error : " .. ret_decr["err"])
return ret_decr
end
local ret_expire = redis.pcall("EXPIRE", KEYS[1], ARGV[1])
if type(ret_expire) == "table" and ret_expire["err"] ~= nil then
redis.log(redis.LOG_WARNING, "Bad behavior decrease EXPIRE error : " .. ret_expire["err"])
return ret_expire
end
if ret_decr <= 0 then
local ret_del = redis.pcall("DEL", KEYS[1])
if type(ret_del) == "table" and ret_del["err"] ~= nil then
redis.log(redis.LOG_WARNING, "Bad behavior decrease DEL error : " .. ret_del["err"])
return ret_del
end
end
return ret_decr
]]
-- Connect to server
local cstore, err = clusterstore:new()
if not cstore then
return false, err
end
local ok, err = clusterstore:connect()
if not ok then
return false, err
end
-- Decrement counter
local counter, err = clusterstore:call("decr", "bad_behavior_" .. ip)
if err then
local counter, err = clusterstore:call("eval", redis_script, 1, "bad_behavior_" .. ip, count_time)
if not counter then
clusterstore:close()
return false, err
end
-- Delete counter
if counter < 0 then
counter = 0
end
if counter == 0 then
local ok, err = clusterstore:call("del", "bad_behavior_" .. ip)
if err then
clusterstore:close()
return false, err
end
end
-- End connection
clusterstore:close()
return counter
end

81
src/common/core/blacklist/blacklist.lua
View File

@ -17,21 +17,31 @@ function blacklist:initialize()
self.logger:log(ngx.ERR, err)
end
self.use_redis = use_redis == "yes"
-- Check if init is needed
if ngx.get_phase() == "init" then
local init_needed, err = utils.has_variable("USE_BLACKLIST", "yes")
if init_needed == nil then
self.logger:log(ngx.ERR, err)
end
self.init_needed = init_needed
-- Decode lists
else
if ngx.get_phase() ~= "init" and self.variables["USE_BLACKLIST"] == "yes" then
local lists, err = self.datastore:get("plugin_blacklist_lists")
if not lists then
self.logger:log(ngx.ERR, err)
else
self.lists = cjson.decode(lists)
end
local kinds = {
["IP"] = {},
["RDNS"] = {},
["ASN"] = {},
["USER_AGENT"] = {},
["URI"] = {},
["IGNORE_IP"] = {},
["IGNORE_RDNS"] = {},
["IGNORE_ASN"] = {},
["IGNORE_USER_AGENT"] = {},
["IGNORE_URI"] = {},
}
for kind, _ in pairs(kinds) do
for data in self.variables["BLACKLIST_" .. kind]:gmatch("%S+") do
table.insert(self.lists[kind], data)
end
end
end
-- Instantiate cachestore
self.cachestore = cachestore:new(self.use_redis)
@ -39,9 +49,14 @@ end
function blacklist:init()
-- Check if init is needed
if not self.init_needed then
local init_needed, err = utils.has_variable("USE_BLACKLIST", "yes")
if init_needed == nil then
return self:ret(false, "can't check USE_BLACKLIST variable : " .. err)
end
if not init_needed or self.is_loading then
return self:ret(true, "init not needed")
end
-- Read blacklists
local blacklists = {
["IP"] = {},
@ -81,13 +96,13 @@ function blacklist:access()
end
-- Check the caches
local checks = {
["IP"] = "ip" .. ngx.var.remote_addr
["IP"] = "ip" .. ngx.ctx.bw.remote_addr
}
if ngx.var.http_user_agent then
checks["UA"] = "ua" .. ngx.var.http_user_agent
if ngx.ctx.bw.http_user_agent then
checks["UA"] = "ua" .. ngx.ctx.bw.http_user_agent
end
if ngx.var.uri then
checks["URI"] = "uri" .. ngx.var.uri
if ngx.ctx.bw.uri then
checks["URI"] = "uri" .. ngx.ctx.bw.uri
end
local already_cached = {
["IP"] = false,
@ -99,7 +114,7 @@ function blacklist:access()
if not ok then
self.logger:log(ngx.ERR, "error while checking cache : " .. cached)
elseif cached and cached ~= "ok" then
return self:ret(true, k + " is in cached blacklist (info : " .. cached .. ")", utils.get_deny_status())
return self:ret(true, k .. " is in cached blacklist (info : " .. cached .. ")", utils.get_deny_status())
end
if cached then
already_cached[k] = true
@ -121,7 +136,7 @@ function blacklist:access()
self.logger:log(ngx.ERR, "error while adding element to cache : " .. err)
end
if blacklisted ~= "ok" then
return self:ret(true, k + " is blacklisted (info : " .. blacklisted .. ")", utils.get_deny_status())
return self:ret(true, k .. " is blacklisted (info : " .. blacklisted .. ")", utils.get_deny_status())
end
end
end
@ -138,11 +153,11 @@ end
function blacklist:kind_to_ele(kind)
if kind == "IP" then
return "ip" .. ngx.var.remote_addr
return "ip" .. ngx.ctx.bw.remote_addr
elseif kind == "UA" then
return "ua" .. ngx.var.http_user_agent
return "ua" .. ngx.ctx.bw.http_user_agent
elseif kind == "URI" then
return "uri" .. ngx.var.uri
return "uri" .. ngx.ctx.bw.uri
end
end
@ -179,7 +194,7 @@ function blacklist:is_blacklisted_ip()
if not ipm then
return nil, err
end
local match, err = ipm:match(ngx.var.remote_addr)
local match, err = ipm:match(ngx.ctx.bw.remote_addr)
if err then
return nil, err
end
@ -189,7 +204,7 @@ function blacklist:is_blacklisted_ip()
if not ipm then
return nil, err
end
local match, err = ipm:match(ngx.var.remote_addr)
local match, err = ipm:match(ngx.ctx.bw.remote_addr)
if err then
return nil, err
end
@ -200,18 +215,12 @@ function blacklist:is_blacklisted_ip()
-- Check if rDNS is needed
local check_rdns = true
local is_global, err = utils.ip_is_global(ngx.var.remote_addr)
if self.variables["BLACKLIST_RDNS_GLOBAL"] == "yes" then
if is_global == nil then
return nil, err
end
if not is_global then
check_rdns = false
end
if self.variables["BLACKLIST_RDNS_GLOBAL"] == "yes" and not ngx.ctx.bw.ip_is_global then
check_rdns = false
end
if check_rdns then
-- Get rDNS
local rdns_list, err = utils.get_rdns(ngx.var.remote_addr)
local rdns_list, err = utils.get_rdns(ngx.ctx.bw.remote_addr)
if not rdns_list then
return false, err
end
@ -241,8 +250,8 @@ function blacklist:is_blacklisted_ip()
end
-- Check if ASN is in ignore list
if is_global then
local asn, err = utils.get_asn(ngx.var.remote_addr)
if ngx.ctx.bw.ip_is_global then
local asn, err = utils.get_asn(ngx.ctx.bw.remote_addr)
if not asn then
self.logger:log(ngx.ERR, "7")
return nil, err
@ -272,7 +281,7 @@ function blacklist:is_blacklisted_uri()
-- Check if URI is in ignore list
local ignore = false
for i, ignore_uri in ipairs(self.lists["IGNORE_URI"]) do
if ngx.var.uri:match(ignore_uri) then
if ngx.ctx.bw.uri:match(ignore_uri) then
ignore = true
break
end
@ -280,7 +289,7 @@ function blacklist:is_blacklisted_uri()
-- Check if URI is in blacklist
if not ignore then
for i, uri in ipairs(self.lists["URI"]) do
if ngx.var.uri:match(uri) then
if ngx.ctx.bw.uri:match(uri) then
return true, "URI " .. uri
end
end
@ -293,7 +302,7 @@ function blacklist:is_blacklisted_ua()
-- Check if UA is in ignore list
local ignore = false
for i, ignore_ua in ipairs(self.lists["IGNORE_USER_AGENT"]) do
if ngx.var.http_user_agent:match(ignore_ua) then
if ngx.ctx.bw.http_user_agent:match(ignore_ua) then
ignore = true
break
end
@ -301,7 +310,7 @@ function blacklist:is_blacklisted_ua()
-- Check if UA is in blacklist
if not ignore then
for i, ua in ipairs(self.lists["USER_AGENT"]) do
if ngx.var.http_user_agent:match(ua) then
if ngx.ctx.bw.http_user_agent:match(ua) then
return true, "UA " .. ua
end
end

23
src/common/core/blacklist/jobs/blacklist-download.py
View File

@ -6,7 +6,6 @@ from os import _exit, getenv
from pathlib import Path
from re import IGNORECASE, compile as re_compile
from sys import exit as sys_exit, path as sys_path
from threading import Lock
from traceback import format_exc
from typing import Tuple
@ -84,7 +83,6 @@ try:
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
lock = Lock()
# Create directories if they don't exist
Path("/var/cache/bunkerweb/blacklist").mkdir(parents=True, exist_ok=True)
@ -108,7 +106,9 @@ try:
}
all_fresh = True
for kind in kinds_fresh:
if not is_cached_file(f"/var/cache/bunkerweb/blacklist/{kind}.list", "hour"):
if not is_cached_file(
f"/var/cache/bunkerweb/blacklist/{kind}.list", "hour", db
):
kinds_fresh[kind] = False
all_fresh = False
logger.info(
@ -172,7 +172,7 @@ try:
logger.info(f"Downloaded {i} bad {kind}")
# Check if file has changed
new_hash = file_hash(f"/var/tmp/bunkerweb/blacklist/{kind}.list")
old_hash = cache_hash(f"/var/cache/bunkerweb/blacklist/{kind}.list")
old_hash = cache_hash(f"/var/cache/bunkerweb/blacklist/{kind}.list", db)
if new_hash == old_hash:
logger.info(
f"New file {kind}.list is identical to cache file, reload is not needed",
@ -186,25 +186,12 @@ try:
f"/var/tmp/bunkerweb/blacklist/{kind}.list",
f"/var/cache/bunkerweb/blacklist/{kind}.list",
new_hash,
db,
)
if not cached:
logger.error(f"Error while caching blacklist : {err}")
status = 2
else:
# Update db
with lock:
err = db.update_job_cache(
"blacklist-download",
None,
f"{kind}.list",
content,
checksum=new_hash,
)
if err:
logger.warning(f"Couldn't update db cache: {err}")
status = 1
except:
status = 2
logger.error(

44
src/common/core/bunkernet/bunkernet.lua
View File

@ -10,29 +10,27 @@ local bunkernet = class("bunkernet", plugin)
function bunkernet:initialize()
-- Call parent initialize
plugin.initialize(self, "bunkernet")
-- Check if init is needed
if ngx.get_phase() == "init" then
local init_needed, err = utils.has_variable("USE_BUNKERNET", "yes")
if init_needed == nil then
self.logger:log(ngx.ERR, err)
end
self.init_needed = init_needed
-- Get BunkerNet ID
else
if ngx.get_phase() ~= "init" and self.variables["USE_BUNKERNET"] == "yes" then
local id, err = self.datastore:get("plugin_bunkernet_id")
if not id then
self.bunkernet_id = nil
else
if id then
self.bunkernet_id = id
else
self.logger:log(ngx.ERR, "can't get BunkerNet ID from datastore : " .. err)
end
end
end
function bunkernet:init()
-- Check if init is needed
if not self.init_needed then
local init_needed, err = utils.has_variable("USE_BUNKERNET", "yes")
if init_needed == nil then
return self:ret(false, "can't check USE_BUNKERNET variable : " .. err)
end
if not init_needed or self.is_loading then
return self:ret(true, "no service uses bunkernet, skipping init")
end
-- Check if instance ID is present
local f, err = io.open("/var/cache/bunkerweb/bunkernet/instance.id", "r")
if not f then
@ -83,7 +81,7 @@ function bunkernet:log(bypass_use_bunkernet)
end
-- Check if BunkerNet ID is generated
if not self.bunkernet_id then
return self:ret(true, "bunkernet ID is not generated")
return self:ret(false, "bunkernet ID is not generated")
end
-- Check if IP has been blocked
local reason = utils.get_reason()
@ -94,16 +92,14 @@ function bunkernet:log(bypass_use_bunkernet)
return self:ret(true, "skipping report because the reason is bunkernet")
end
-- Check if IP is global
local is_global, err = utils.ip_is_global(ngx.var.remote_addr)
if is_global == nil then
return self:ret(false, "error while checking if IP is global " .. err)
end
if not is_global then
if not ngx.ctx.bw.ip_is_global then
return self:ret(true, "IP is not global")
end
-- TODO : check if IP has been reported recently
self.integration = ngx.ctx.bw.integration
self.version = ngx.ctx.bw.version
local function report_callback(premature, obj, ip, reason, method, url, headers)
local ok, err, status, data = obj:report(ip, reason, method, url, headers)
local ok, err, status, data = obj:report(ip, reason, method, url, headers, obj.ctx.integration, obj.ctx.version)
if status == 429 then
obj.logger:log(ngx.WARN, "bunkernet API is rate limiting us")
elseif not ok then
@ -113,8 +109,8 @@ function bunkernet:log(bypass_use_bunkernet)
end
end
local hdr, err = ngx.timer.at(0, report_callback, self, ngx.var.remote_addr, reason, ngx.var.request_method,
ngx.var.request_uri, ngx.req.get_headers())
local hdr, err = ngx.timer.at(0, report_callback, self, ngx.ctx.bw.remote_addr, reason, ngx.ctx.bw.request_method,
ngx.ctx.bw.request_uri, ngx.req.get_headers())
if not hdr then
return self:ret(false, "can't create report timer : " .. err)
end
@ -149,8 +145,8 @@ function bunkernet:request(method, url, data)
end
local all_data = {
id = self.id,
integration = utils.get_integration(),
version = utils.get_version()
integration = self.integration,
version = self.version
}
for k, v in pairs(data) do
all_data[k] = v
@ -160,7 +156,7 @@ function bunkernet:request(method, url, data)
body = cjson.encode(all_data),
headers = {
["Content-Type"] = "application/json",
["User-Agent"] = "BunkerWeb/" .. utils.get_version()
["User-Agent"] = "BunkerWeb/" .. self.version
}
})
httpc:close()

19
src/common/core/bunkernet/jobs/bunkernet-data.py
View File

@ -47,7 +47,6 @@ try:
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
lock = Lock()
# Create directory if it doesn't exist
Path("/var/cache/bunkerweb/bunkernet").mkdir(parents=True, exist_ok=True)
@ -64,7 +63,7 @@ try:
_exit(2)
# Don't go further if the cache is fresh
if is_cached_file("/var/cache/bunkerweb/bunkernet/ip.list", "day"):
if is_cached_file("/var/cache/bunkerweb/bunkernet/ip.list", "day", db):
logger.info(
"BunkerNet list is already in cache, skipping download...",
)
@ -111,7 +110,7 @@ try:
# Check if file has changed
new_hash = file_hash("/var/tmp/bunkerweb/bunkernet-ip.list")
old_hash = cache_hash("/var/cache/bunkerweb/bunkernet/ip.list")
old_hash = cache_hash("/var/cache/bunkerweb/bunkernet/ip.list", db)
if new_hash == old_hash:
logger.info(
"New file is identical to cache file, reload is not needed",
@ -123,24 +122,12 @@ try:
"/var/tmp/bunkerweb/bunkernet-ip.list",
"/var/cache/bunkerweb/bunkernet/ip.list",
new_hash,
db,
)
if not cached:
logger.error(f"Error while caching BunkerNet data : {err}")
_exit(2)
# Update db
with lock:
err = db.update_job_cache(
"bunkernet-data",
None,
"ip.list",
content,
checksum=new_hash,
)
if err:
logger.warning(f"Couldn't update db ip.list cache: {err}")
logger.info("Successfully saved BunkerNet data")
status = 1

2
src/common/core/bunkernet/plugin.json
View File

@ -1,6 +1,6 @@
{
"id": "bunkernet",
"order": 6,
"order": 7,
"name": "BunkerNet",
"description": "Share threat data with other BunkerWeb instances via BunkerNet.",
"version": "0.1",

2
src/common/core/cors/cors.lua
View File

@ -14,7 +14,7 @@ function cors:header()
if self.variables["USE_CORS"] ~= "yes" then
return self:ret(true, "service doesn't use CORS")
end
if ngx.var.request_method ~= "OPTIONS" then
if ngx.ctx.bw.request_method ~= "OPTIONS" then
return self:ret(true, "method is not OPTIONS")
end
-- Add headers

38
src/common/core/country/country.lua
View File

@ -2,6 +2,7 @@ local class = require "middleclass"
local plugin = require "bunkerweb.plugin"
local utils = require "bunkerweb.utils"
local cachestore = require "bunkerweb.cachestore"
local cjson = require "cjson"
local country = class("country", plugin)
@ -23,69 +24,66 @@ function country:access()
return self:ret(true, "country not activated")
end
-- Check if IP is in cache
local data, err = self:is_in_cache(ngx.var.remote_addr)
local data, err = self:is_in_cache(ngx.ctx.bw.remote_addr)
if data then
if data.result == "ok" then
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is in country cache (not blacklisted, country = " .. data.country .. ")")
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is in country cache (not blacklisted, country = " .. data.country .. ")")
end
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is in country cache (blacklisted, country = " .. data.country .. ")", utils.get_deny_status())
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is in country cache (blacklisted, country = " .. data.country .. ")", utils.get_deny_status())
end
-- Don't go further if IP is not global
local is_global, err = utils.ip_is_global(ngx.var.remote_addr)
if is_global == nil then
return self:ret(false, "error while checking if ip is global : " .. err)
elseif not is_global then
local ok, err = self:add_to_cache(ngx.var.remote_addr, "unknown", "ok")
if not ngx.ctx.bw.ip_is_global then
local ok, err = self:add_to_cache(ngx.ctx.bw.remote_addr, "unknown", "ok")
if not ok then
return self:ret(false, "error while adding ip to cache : " .. err)
end
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is not global, skipping check")
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is not global, skipping check")
end
-- Get the country of client
local country, err = utils.get_country(ngx.var.remote_addr)
local country, err = utils.get_country(ngx.ctx.bw.remote_addr)
if not country then
return self:ret(false, "can't get country of client IP " .. ngx.var.remote_addr .. " : " .. err)
return self:ret(false, "can't get country of client IP " .. ngx.ctx.bw.remote_addr .. " : " .. err)
end
-- Process whitelist first
if self.variables["WHITELIST_COUNTRY"] ~= "" then
for wh_country in self.variables["WHITELIST_COUNTRY"]:gmatch("%S+") do
if wh_country == country then
local ok, err = self:add_to_cache(ngx.var.remote_addr, country, "ok")
local ok, err = self:add_to_cache(ngx.ctx.bw.remote_addr, country, "ok")
if not ok then
return self:ret(false, "error while adding item to cache : " .. err)
end
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is whitelisted (country = " .. country .. ")")
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is whitelisted (country = " .. country .. ")")
end
end
local ok, err = self:add_to_cache(ngx.var.remote_addr, country, "ko")
local ok, err = self:add_to_cache(ngx.ctx.bw.remote_addr, country, "ko")
if not ok then
return self:ret(false, "error while adding item to cache : " .. err)
end
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is not whitelisted (country = " .. country .. ")", utils.get_deny_status())
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is not whitelisted (country = " .. country .. ")", utils.get_deny_status())
end
-- And then blacklist
if self.variables["BLACKLIST_COUNTRY"] ~= "" then
for bl_country in self.variables["BLACKLIST_COUNTRY"]:gmatch("%S+") do
if bl_country == country then
local ok, err = self:add_to_cache(ngx.var.remote_addr, country, "ko")
local ok, err = self:add_to_cache(ngx.ctx.bw.remote_addr, country, "ko")
if not ok then
return self:ret(false, "error while adding item to cache : " .. err)
end
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is blacklisted (country = " .. country .. ")", true, utils.get_deny_status())
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is blacklisted (country = " .. country .. ")", true, utils.get_deny_status())
end
end
end
-- Country IP is not in blacklist
local ok, err = self:add_to_cache(ngx.var.remote_addr, country, "ok")
local ok, err = self:add_to_cache(ngx.ctx.bw.remote_addr, country, "ok")
if not ok then
return self:ret(false, "error while caching IP " .. ngx.var.remote_addr .. " : " .. err)
return self:ret(false, "error while caching IP " .. ngx.ctx.bw.remote_addr .. " : " .. err)
end
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is not blacklisted (country = " .. country .. ")")
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is not blacklisted (country = " .. country .. ")")
end
function country:preread()

34
src/common/core/customcert/jobs/custom-cert.py
View File

@ -81,30 +81,28 @@ def check_cert(cert_path, key_path, first_server: Optional[str] = None) -> bool:
if old_hash != key_hash:
copy(key_path, key_cache_path.replace(".hash", ""))
with open(key_path, "r") as f:
with lock:
err = db.update_job_cache(
"custom-cert",
first_server,
key_cache_path.replace(".hash", "").split("/")[-1],
f.read().encode("utf-8"),
checksum=key_hash,
)
with lock:
err = db.update_job_cache(
"custom-cert",
first_server,
key_cache_path.replace(".hash", "").split("/")[-1],
Path(key_path).read_bytes(),
checksum=key_hash,
)
if err:
logger.warning(
f"Couldn't update db cache for {key_path.replace('/', '_')}.hash: {err}"
)
with open(cert_path, "r") as f:
with lock:
err = db.update_job_cache(
"custom-cert",
first_server,
cert_cache_path.replace(".hash", "").split("/")[-1],
f.read().encode("utf-8"),
checksum=cert_hash,
)
with lock:
err = db.update_job_cache(
"custom-cert",
first_server,
cert_cache_path.replace(".hash", "").split("/")[-1],
Path(cert_path).read_bytes(),
checksum=cert_hash,
)
if err:
logger.warning(

24
src/common/core/dnsbl/dnsbl.lua
View File

@ -28,22 +28,18 @@ function dnsbl:access()
return self:ret(true, "dnsbl list is empty")
end
-- Check if IP is in cache
local ok, cached = self:is_in_cache(ngx.var.remote_addr)
local ok, cached = self:is_in_cache(ngx.ctx.bw.remote_addr)
if not ok then
return self:ret(false, "error while checking cache : " .. err)
return self:ret(false, "error while checking cache : " .. cached)
elseif cached then
if cached == "ok" then
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is in DNSBL cache (not blacklisted)")
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is in DNSBL cache (not blacklisted)")
end
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is in DNSBL cache (server = " .. cached .. ")", utils.get_deny_status())
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is in DNSBL cache (server = " .. cached .. ")", utils.get_deny_status())
end
-- Don't go further if IP is not global
local is_global, err = utils.ip_is_global(ngx.var.remote_addr)
if is_global == nil then
return self:ret(false, "can't check if client IP is global : " .. err)
end
if not is_global then
local ok, err = self:add_to_cache(ngx.var.remote_addr, "ok")
if not ngx.ctx.bw.ip_is_global then
local ok, err = self:add_to_cache(ngx.ctx.bw.remote_addr, "ok")
if not ok then
return self:ret(false, "error while adding element to cache : " .. err)
end
@ -51,12 +47,12 @@ function dnsbl:access()
end
-- Loop on DNSBL list
for server in self.variables["DNSBL_LIST"]:gmatch("%S+") do
local result, err = self:is_in_dnsbl(server)
local result, err = self:is_in_dnsbl(ngx.ctx.bw.remote_addr, server)
if result == nil then
self.logger:log(ngx.ERR, "error while sending DNS request to " .. server .. " : " .. err)
end
if result then
local ok, err self:add_to_cache(ngx.var.remote_addr, server)
local ok, err self:add_to_cache(ngx.ctx.bw.remote_addr, server)
if not ok then
return self:ret(false, "error while adding element to cache : " .. err)
end
@ -64,7 +60,7 @@ function dnsbl:access()
end
end
-- IP is not in DNSBL
local ok, err = self:add_to_cache(ngx.var.remote_addr, "ok")
local ok, err = self:add_to_cache(ngx.ctx.bw.remote_addr, "ok")
if not ok then
return self:ret(false, "IP is not in DNSBL (error = " .. err .. ")")
end
@ -91,7 +87,7 @@ function dnsbl:add_to_cache(ip, value)
return true
end
function dnsbl:is_in_dnsbl(server)
function dnsbl:is_in_dnsbl(ip, server)
local request = resolver.arpa_str(ip) .. "." .. server
local ips, err = utils.get_ips(request)
if not ips then

174
src/common/core/errors/files/error.html
View File

File diff suppressed because one or more lines are too long

75
src/common/core/greylist/greylist.lua
View File

@ -16,21 +16,26 @@ function greylist:initialize()
self.logger:log(ngx.ERR, err)
end
self.use_redis = use_redis == "yes"
-- Check if init is needed
if ngx.get_phase() == "init" then
local init_needed, err = utils.has_variable("USE_GREYLIST", "yes")
if init_needed == nil then
self.logger:log(ngx.ERR, err)
end
self.init_needed = init_needed
-- Decode lists
elseif self.variables["USE_GREYLIST"] == "yes" then
if ngx.get_phase() ~= "init" and self.variables["USE_GREYLIST"] == "yes" then
local lists, err = self.datastore:get("plugin_greylist_lists")
if not lists then
self.logger:log(ngx.ERR, err)
else
self.lists = cjson.decode(lists)
end
local kinds = {
["IP"] = {},
["RDNS"] = {},
["ASN"] = {},
["USER_AGENT"] = {},
["URI"] = {}
}
for kind, _ in pairs(kinds) do
for data in self.variables["GREYLIST_" .. kind]:gmatch("%S+") do
table.insert(self.lists[kind], data)
end
end
end
-- Instantiate cachestore
self.cachestore = cachestore:new(self.use_redis)
@ -38,10 +43,14 @@ end
function greylist:init()
-- Check if init is needed
if not self.init_needed then
local init_needed, err = utils.has_variable("USE_GREYLIST", "yes")
if init_needed == nil then
return self:ret(false, "can't check USE_GREYLIST variable : " .. err)
end
if not init_needed or self.is_loading then
return self:ret(true, "init not needed")
end
-- Read blacklists
-- Read greylists
local greylists = {
["IP"] = {},
["RDNS"] = {},
@ -75,13 +84,13 @@ function greylist:access()
end
-- Check the caches
local checks = {
["IP"] = "ip" .. ngx.var.remote_addr
["IP"] = "ip" .. ngx.ctx.bw.remote_addr
}
if ngx.var.http_user_agent then
checks["UA"] = "ua" .. ngx.var.http_user_agent
if ngx.ctx.bw.http_user_agent then
checks["UA"] = "ua" .. ngx.ctx.bw.http_user_agent
end
if ngx.var.uri then
checks["URI"] = "uri" .. ngx.var.uri
if ngx.ctx.bw.uri then
checks["URI"] = "uri" .. ngx.ctx.bw.uri
end
local already_cached = {
["IP"] = false,
@ -93,7 +102,7 @@ function greylist:access()
if not cached and err ~= "success" then
self.logger:log(ngx.ERR, "error while checking cache : " .. err)
elseif cached and cached ~= "ok" then
return self:ret(true, k + " is in cached greylist", utils.get_deny_status())
return self:ret(true, k .. " is in cached greylist", utils.get_deny_status())
end
if cached then
already_cached[k] = true
@ -115,7 +124,7 @@ function greylist:access()
self.logger:log(ngx.ERR, "error while adding element to cache : " .. err)
end
if greylisted == "ko" then
return self:ret(true, k + " is not in greylist", utils.get_deny_status())
return self:ret(true, k .. " is not in greylist", utils.get_deny_status())
end
end
end
@ -131,11 +140,11 @@ end
function greylist:kind_to_ele(kind)
if kind == "IP" then
return "ip" .. ngx.var.remote_addr
return "ip" .. ngx.ctx.bw.remote_addr
elseif kind == "UA" then
return "ua" .. ngx.var.http_user_agent
return "ua" .. ngx.ctx.bw.http_user_agent
elseif kind == "URI" then
return "uri" .. ngx.var.uri
return "uri" .. ngx.ctx.bw.uri
end
end
@ -151,12 +160,12 @@ function greylist:is_greylisted(kind)
end
function greylist:is_greylisted_ip()
-- Check if IP is in blacklist
-- Check if IP is in greylist
local ipm, err = ipmatcher.new(self.lists["IP"])
if not ipm then
return nil, err
end
local match, err = ipm:match(ngx.var.remote_addr)
local match, err = ipm:match(ngx.ctx.bw.remote_addr)
if err then
return nil, err
end
@ -166,18 +175,12 @@ function greylist:is_greylisted_ip()
-- Check if rDNS is needed
local check_rdns = true
local is_global, err = utils.ip_is_global(ngx.var.remote_addr)
if self.variables["BLACKLIST_RDNS_GLOBAL"] == "yes" then
if is_global == nil then
return nil, err
end
if not is_global then
check_rdns = false
end
if self.variables["GREYLIST_RDNS_GLOBAL"] == "yes" and not ngx.ctx.bw.ip_is_global then
check_rdns = false
end
if check_rdns then
-- Get rDNS
local rdns_list, err = utils.get_rdns(ngx.var.remote_addr)
local rdns_list, err = utils.get_rdns(ngx.ctx.bw.remote_addr)
if not rdns_list then
return nil, err
end
@ -192,8 +195,8 @@ function greylist:is_greylisted_ip()
end
-- Check if ASN is in greylist
if is_global then
local asn, err = utils.get_asn(ngx.var.remote_addr)
if ngx.ctx.bw.ip_is_global then
local asn, err = utils.get_asn(ngx.ctx.bw.remote_addr)
if not asn then
return nil, err
end
@ -209,9 +212,9 @@ function greylist:is_greylisted_ip()
end
function greylist:is_greylisted_uri()
-- Check if URI is in blacklist
-- Check if URI is in greylist
for i, uri in ipairs(self.lists["URI"]) do
if ngx.var.uri:match(uri) then
if ngx.ctx.bw.uri:match(uri) then
return true, "URI " .. uri
end
end
@ -222,7 +225,7 @@ end
function greylist:is_greylisted_ua()
-- Check if UA is in greylist
for i, ua in ipairs(self.lists["USER_AGENT"]) do
if ngx.var.http_user_agent:match(ua) then
if ngx.ctx.bw.http_user_agent:match(ua) then
return true, "UA " .. ua
end
end

21
src/common/core/greylist/jobs/greylist-download.py
View File

@ -6,7 +6,6 @@ from os import _exit, getenv
from pathlib import Path
from re import IGNORECASE, compile as re_compile
from sys import exit as sys_exit, path as sys_path
from threading import Lock
from traceback import format_exc
from typing import Tuple
@ -84,7 +83,6 @@ try:
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
lock = Lock()
# Create directories if they don't exist
Path("/var/cache/bunkerweb/greylist").mkdir(parents=True, exist_ok=True)
@ -103,7 +101,7 @@ try:
}
all_fresh = True
for kind in kinds_fresh:
if not is_cached_file(f"/var/cache/bunkerweb/greylist/{kind}.list", "hour"):
if not is_cached_file(f"/var/cache/bunkerweb/greylist/{kind}.list", "hour", db):
kinds_fresh[kind] = False
all_fresh = False
logger.info(
@ -156,7 +154,7 @@ try:
logger.info(f"Downloaded {i} grey {kind}")
# Check if file has changed
new_hash = file_hash(f"/var/tmp/bunkerweb/greylist/{kind}.list")
old_hash = cache_hash(f"/var/cache/bunkerweb/greylist/{kind}.list")
old_hash = cache_hash(f"/var/cache/bunkerweb/greylist/{kind}.list", db)
if new_hash == old_hash:
logger.info(
f"New file {kind}.list is identical to cache file, reload is not needed",
@ -170,25 +168,12 @@ try:
f"/var/tmp/bunkerweb/greylist/{kind}.list",
f"/var/cache/bunkerweb/greylist/{kind}.list",
new_hash,
db,
)
if not cached:
logger.error(f"Error while caching greylist : {err}")
status = 2
else:
# Update db
with lock:
err = db.update_job_cache(
"greylist-download",
None,
f"{kind}.list",
content,
checksum=new_hash,
)
if err:
logger.warning(f"Couldn't update db cache: {err}")
status = 1
except:
status = 2
logger.error(

29
src/common/core/jobs/jobs/mmdb-asn.py
View File

@ -5,7 +5,6 @@ from gzip import decompress
from os import _exit, getenv
from pathlib import Path
from sys import exit as sys_exit, path as sys_path
from threading import Lock
from traceback import format_exc
sys_path.extend(
@ -25,10 +24,14 @@ from jobs import cache_file, cache_hash, file_hash, is_cached_file
logger = setup_logger("JOBS.mmdb-asn", getenv("LOG_LEVEL", "INFO"))
status = 0
db = Database(
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
try:
# Don't go further if the cache is fresh
if is_cached_file("/var/cache/bunkerweb/asn.mmdb", "month"):
if is_cached_file("/var/cache/bunkerweb/asn.mmdb", "month", db):
logger.info("asn.mmdb is already in cache, skipping download...")
_exit(0)
@ -52,8 +55,7 @@ try:
# Decompress it
logger.info("Decompressing mmdb file ...")
file_content = decompress(file_content)
Path(f"/var/tmp/bunkerweb/asn.mmdb").write_bytes(file_content)
Path(f"/var/tmp/bunkerweb/asn.mmdb").write_bytes(decompress(file_content))
# Try to load it
logger.info("Checking if mmdb file is valid ...")
@ -62,7 +64,7 @@ try:
# Check if file has changed
new_hash = file_hash("/var/tmp/bunkerweb/asn.mmdb")
old_hash = cache_hash("/var/cache/bunkerweb/asn.mmdb")
old_hash = cache_hash("/var/cache/bunkerweb/asn.mmdb", db)
if new_hash == old_hash:
logger.info("New file is identical to cache file, reload is not needed")
_exit(0)
@ -70,27 +72,12 @@ try:
# Move it to cache folder
logger.info("Moving mmdb file to cache ...")
cached, err = cache_file(
"/var/tmp/bunkerweb/asn.mmdb", "/var/cache/bunkerweb/asn.mmdb", new_hash
"/var/tmp/bunkerweb/asn.mmdb", "/var/cache/bunkerweb/asn.mmdb", new_hash, db
)
if not cached:
logger.error(f"Error while caching mmdb file : {err}")
_exit(2)
db = Database(
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
lock = Lock()
# Update db
with lock:
err = db.update_job_cache(
"mmdb-asn", None, "asn.mmdb", file_content, checksum=new_hash
)
if err:
logger.warning(f"Couldn't update db cache: {err}")
# Success
logger.info(f"Downloaded new mmdb from {mmdb_url}")

55
src/common/core/jobs/jobs/mmdb-country.py
View File

@ -5,7 +5,6 @@ from gzip import decompress
from os import _exit, getenv
from pathlib import Path
from sys import exit as sys_exit, path as sys_path
from threading import Lock
from traceback import format_exc
sys_path.extend(
@ -27,8 +26,35 @@ logger = setup_logger("JOBS.mmdb-country", getenv("LOG_LEVEL", "INFO"))
status = 0
try:
# Only download mmdb if the country blacklist or whitelist is enabled
dl_mmdb = False
# Multisite case
if getenv("MULTISITE", "no") == "yes":
for first_server in getenv("SERVER_NAME", "").split(" "):
if getenv(
f"{first_server}_BLACKLIST_COUNTRY", getenv("BLACKLIST_COUNTRY")
) or getenv(
f"{first_server}_WHITELIST_COUNTRY", getenv("WHITELIST_COUNTRY")
):
dl_mmdb = True
break
# Singlesite case
elif getenv("BLACKLIST_COUNTRY") or getenv("WHITELIST_COUNTRY"):
dl_mmdb = True
if not dl_mmdb:
logger.info(
"Country blacklist or whitelist is not enabled, skipping download..."
)
_exit(0)
db = Database(
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
# Don't go further if the cache is fresh
if is_cached_file("/var/cache/bunkerweb/country.mmdb", "month"):
if is_cached_file("/var/cache/bunkerweb/country.mmdb", "month", db):
logger.info("country.mmdb is already in cache, skipping download...")
_exit(0)
@ -52,8 +78,7 @@ try:
# Decompress it
logger.info("Decompressing mmdb file ...")
file_content = decompress(file_content)
Path(f"/var/tmp/bunkerweb/country.mmdb").write_bytes(file_content)
Path(f"/var/tmp/bunkerweb/country.mmdb").write_bytes(decompress(file_content))
# Try to load it
logger.info("Checking if mmdb file is valid ...")
@ -62,7 +87,7 @@ try:
# Check if file has changed
new_hash = file_hash("/var/tmp/bunkerweb/country.mmdb")
old_hash = cache_hash("/var/cache/bunkerweb/country.mmdb")
old_hash = cache_hash("/var/cache/bunkerweb/country.mmdb", db)
if new_hash == old_hash:
logger.info("New file is identical to cache file, reload is not needed")
_exit(0)
@ -70,27 +95,15 @@ try:
# Move it to cache folder
logger.info("Moving mmdb file to cache ...")
cached, err = cache_file(
"/var/tmp/bunkerweb/country.mmdb", "/var/cache/bunkerweb/country.mmdb", new_hash
"/var/tmp/bunkerweb/country.mmdb",
"/var/cache/bunkerweb/country.mmdb",
new_hash,
db,
)
if not cached:
logger.error(f"Error while caching mmdb file : {err}")
_exit(2)
db = Database(
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
lock = Lock()
# Update db
with lock:
err = db.update_job_cache(
"mmdb-country", None, "country.mmdb", file_content, checksum=new_hash
)
if err:
logger.warning(f"Couldn't update db cache: {err}")
# Success
logger.info(f"Downloaded new mmdb from {mmdb_url}")

16
src/common/core/letsencrypt/jobs/certbot-new.py
View File

@ -99,17 +99,15 @@ try:
)
if Path(f"/etc/letsencrypt/live/{first_server}/cert.pem").exists():
cert = Path(
f"/etc/letsencrypt/live/{first_server}/cert.pem"
).read_bytes()
# Update db
with lock:
err = db.update_job_cache(
"certbot-new",
first_server,
"cert.pem",
cert,
Path(
f"/etc/letsencrypt/live/{first_server}/cert.pem"
).read_bytes(),
)
if err:
@ -139,17 +137,15 @@ try:
)
if Path(f"/etc/letsencrypt/live/{first_server}/cert.pem").exists():
cert = Path(
f"/etc/letsencrypt/live/{first_server}/cert.pem"
).read_bytes()
# Update db
with lock:
err = db.update_job_cache(
"certbot-new",
first_server,
"cert.pem",
cert,
Path(
f"/etc/letsencrypt/live/{first_server}/cert.pem"
).read_bytes(),
)
if err:

10
src/common/core/letsencrypt/letsencrypt.lua
View File

@ -11,7 +11,7 @@ function letsencrypt:initialize()
end
function letsencrypt:access()
if string.sub(ngx.var.uri, 1, string.len("/.well-known/acme-challenge/")) == "/.well-known/acme-challenge/" then
if string.sub(ngx.ctx.bw.uri, 1, string.len("/.well-known/acme-challenge/")) == "/.well-known/acme-challenge/" then
self.logger:log(ngx.NOTICE, "got a visit from Let's Encrypt, let's whitelist it")
return self:ret(true, "visit from LE", ngx.OK)
end
@ -19,8 +19,8 @@ function letsencrypt:access()
end
function letsencrypt:api()
if not string.match(ngx.var.uri, "^/lets%-encrypt/challenge$") or
(ngx.var.request_method ~= "POST" and ngx.var.request_method ~= "DELETE") then
if not string.match(ngx.ctx.bw.uri, "^/lets%-encrypt/challenge$") or
(ngx.ctx.bw.request_method ~= "POST" and ngx.ctx.bw.request_method ~= "DELETE") then
return false, nil, nil
end
local acme_folder = "/var/tmp/bunkerweb/lets-encrypt/.well-known/acme-challenge/"
@ -30,7 +30,7 @@ function letsencrypt:api()
return true, ngx.HTTP_BAD_REQUEST, { status = "error", msg = "json body decoding failed" }
end
os.execute("mkdir -p " .. acme_folder)
if ngx.var.request_method == "POST" then
if ngx.ctx.bw.request_method == "POST" then
local file, err = io.open(acme_folder .. data.token, "w+")
if not file then
return true, ngx.HTTP_INTERNAL_SERVER_ERROR, { status = "error", msg = "can't write validation token : " .. err }
@ -38,7 +38,7 @@ function letsencrypt:api()
file:write(data.validation)
file:close()
return true, ngx.HTTP_OK, { status = "success", msg = "validation token written" }
elseif ngx.var.request_method == "DELETE" then
elseif ngx.ctx.bw.request_method == "DELETE" then
local ok, err = os.remove(acme_folder .. data.token)
if not ok then
return true, ngx.HTTP_INTERNAL_SERVER_ERROR, { status = "error", msg = "can't remove validation token : " .. err }

144
src/common/core/limit/limit.lua
View File

@ -16,29 +16,28 @@ function limit:initialize()
self.logger:log(ngx.ERR, err)
end
self.use_redis = use_redis == "yes"
self.clusterstore = clusterstore:new()
-- Load rules if needed
if ngx.get_phase() == "access" then
if self.variables["USE_LIMIT_REQ"] == "yes" then
-- Get all rules from datastore
local limited = false
local all_rules, err = self.datastore:get("plugin_limit_rules")
if not all_rules then
self.logger:log(ngx.ERR, err)
return
if ngx.get_phase() ~= "init" and self.variables["USE_LIMIT_REQ"] == "yes" then
-- Get all rules from datastore
local limited = false
local all_rules, err = self.datastore:get("plugin_limit_rules")
if not all_rules then
self.logger:log(ngx.ERR, err)
return
end
all_rules = cjson.decode(all_rules)
self.rules = {}
-- Extract global rules
if all_rules.global then
for k, v in pairs(all_rules.global) do
self.rules[k] = v
end
all_rules = cjson.decode(all_rules)
self.rules = {}
-- Extract global rules
if all_rules.global then
for k, v in pairs(all_rules.global) do
self.rules[k] = v
end
end
-- Extract and overwrite if needed server rules
if all_rules[ngx.var.server_name] then
for k, v in pairs(all_rules[ngx.var.server_name]) do
self.rules[k] = v
end
end
-- Extract and overwrite if needed server rules
if all_rules[ngx.ctx.bw.server_name] then
for k, v in pairs(all_rules[ngx.ctx.bw.server_name]) do
self.rules[k] = v
end
end
end
@ -50,7 +49,7 @@ function limit:init()
if init_needed == nil then
return self:ret(false, err)
end
if not init_needed then
if not init_needed or self.is_loading then
return self:ret(true, "no service uses Limit for requests, skipping init")
end
-- Get variables
@ -83,7 +82,7 @@ end
function limit:access()
-- Check if we are whitelisted
if ngx.var.is_whitelisted == "yes" then
if ngx.ctx.bw.is_whitelisted == "yes" then
return self:ret(true, "client is whitelisted")
end
-- Check if access is needed
@ -94,7 +93,7 @@ function limit:access()
local rate = nil
local uri = nil
for k, v in pairs(self.rules) do
if k ~= "/" and ngx.var.uri:match(k) then
if k ~= "/" and ngx.ctx.bw.uri:match(k) then
rate = v
uri = k
break
@ -105,7 +104,7 @@ function limit:access()
rate = self.rules["/"]
uri = "/"
else
return self:ret(true, "no rule for " .. ngx.var.uri)
return self:ret(true, "no rule for " .. ngx.ctx.bw.uri)
end
end
-- Check if limit is reached
@ -116,10 +115,10 @@ function limit:access()
end
-- Limit reached
if limited then
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is limited for URL " .. ngx.var.uri .. " (current rate = " .. current_rate .. "r/" .. rate_time .. " and max rate = " .. rate .. ")", ngx.HTTP_TOO_MANY_REQUESTS)
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is limited for URL " .. ngx.ctx.bw.uri .. " (current rate = " .. current_rate .. "r/" .. rate_time .. " and max rate = " .. rate .. ")", ngx.HTTP_TOO_MANY_REQUESTS)
end
-- Limit not reached
return self:ret(true, "client IP " .. ngx.var.remote_addr .. " is not limited for URL " .. ngx.var.uri .. " (current rate = " .. current_rate .. "r/" .. rate_time .. " and max rate = " .. rate .. ")")
return self:ret(true, "client IP " .. ngx.ctx.bw.remote_addr .. " is not limited for URL " .. ngx.ctx.bw.uri .. " (current rate = " .. current_rate .. "r/" .. rate_time .. " and max rate = " .. rate .. ")")
end
function limit:limit_req(rate_max, rate_time)
@ -132,7 +131,7 @@ function limit:limit_req(rate_max, rate_time)
else
timestamps = redis_timestamps
-- Save the new timestamps
local ok, err = self.datastore:set("plugin_limit_cache_" .. ngx.var.server_name .. ngx.var.remote_addr .. ngx.var.uri, cjson.encode(timestamps), delay)
local ok, err = self.datastore:set("plugin_limit_cache_" .. ngx.ctx.bw.server_name .. ngx.ctx.bw.remote_addr .. ngx.ctx.bw.uri, cjson.encode(timestamps), delay)
if not ok then
return nil, "can't update timestamps : " .. err
end
@ -154,7 +153,7 @@ end
function limit:limit_req_local(rate_max, rate_time)
-- Get timestamps
local timestamps, err = self.datastore:get("plugin_limit_cache_" .. ngx.var.server_name .. ngx.var.remote_addr .. ngx.var.uri)
local timestamps, err = self.datastore:get("plugin_limit_cache_" .. ngx.ctx.bw.server_name .. ngx.ctx.bw.remote_addr .. ngx.ctx.bw.uri)
if not timestamps and err ~= "not found" then
return nil, err
elseif err == "not found" then
@ -165,7 +164,7 @@ function limit:limit_req_local(rate_max, rate_time)
local updated, new_timestamps, delay = self:limit_req_timestamps(rate_max, rate_time, timestamps)
-- Save new timestamps if needed
if updated then
local ok, err = self.datastore:set("plugin_limit_cache_" .. ngx.var.server_name .. ngx.var.remote_addr .. ngx.var.uri, cjson.encode(timestamps), delay)
local ok, err = self.datastore:set("plugin_limit_cache_" .. ngx.ctx.bw.server_name .. ngx.ctx.bw.remote_addr .. ngx.ctx.bw.uri, cjson.encode(new_timestamps), delay)
if not ok then
return nil, err
end
@ -174,38 +173,69 @@ function limit:limit_req_local(rate_max, rate_time)
end
function limit:limit_req_redis(rate_max, rate_time)
-- Connect to server
local cstore, err = clusterstore:new()
if not cstore then
return nil, err
end
local ok, err = clusterstore:connect()
-- Redis atomic script
local redis_script = [[
local ret_get = redis.pcall("GET", KEYS[1])
if type(ret_get) == "table" and ret_get["err"] ~= nil then
redis.log(redis.LOG_WARNING, "limit GET error : " .. ret_get["err"])
return ret_get
end
local timestamps = {}
if ret_get then
timestamps = cjson.decode(ret_get)
end
-- Keep only timestamps within the delay
local updated = false
local new_timestamps = {}
local rate_max = tonumber(ARGV[1])
local rate_time = ARGV[2]
local current_timestamp = tonumber(ARGV[3])
local delay = 0
if rate_time == "s" then
delay = 1
elseif rate_time == "m" then
delay = 60
elseif rate_time == "h" then
delay = 3600
elseif rate_time == "d" then
delay = 86400
end
for i, timestamp in ipairs(timestamps) do
if current_timestamp - timestamp <= delay then
table.insert(new_timestamps, timestamp)
else
updated = true
end
end
-- Only insert the new timestamp if client is not limited already to avoid infinite insert
if #new_timestamps <= rate_max then
table.insert(new_timestamps, current_timestamp)
updated = true
end
-- Save new timestamps if needed
if updated then
local ret_set = redis.pcall("SET", KEYS[1], cjson.encode(new_timestamps), "EX", delay)
if type(ret_set) == "table" and ret_set["err"] ~= nil then
redis.log(redis.LOG_WARNING, "limit SET error : " .. ret_set["err"])
return ret_set
end
end
return new_timestamps
]]
-- Connect
local ok, err = self.clusterstore:connect()
if not ok then
return nil, err
end
-- Get timestamps
local timestamps, err = clusterstore:call("get", "limit_" .. ngx.var.server_name .. ngx.var.remote_addr .. ngx.var.uri)
if err then
clusterstore:close()
-- Execute script
local timestamps, err = self.clusterstore:call("eval", redis_script, 1, "limit_" .. ngx.ctx.bw.server_name .. ngx.ctx.bw.remote_addr .. ngx.ctx.bw.uri, rate_max, rate_time, os.time(os.date("!*t")))
if not timestamps then
self.clusterstore:close()
return nil, err
end
if timestamps then
timestamps = cjson.decode(timestamps)
else
timestamps = {}
end
-- Compute new timestamps
local updated, new_timestamps, delay = self:limit_req_timestamps(rate_max, rate_time, timestamps)
-- Save new timestamps if needed
if updated then
local ok, err = clusterstore:call("set", "limit_" .. ngx.var.server_name .. ngx.var.remote_addr .. ngx.var.uri, cjson.encode(new_timestamps), "EX", delay)
if not ok then
clusterstore:close()
return nil, err
end
end
lusterstore:close()
return new_timestamps, "success"
-- Return timestamps
self.clusterstore:close()
return timestamps, "success"
end
function limit:limit_req_timestamps(rate_max, rate_time, timestamps)

2
src/common/core/limit/plugin.json
View File

@ -1,6 +1,6 @@
{
"id": "limit",
"order": 7,
"order": 8,
"name": "Limit",
"description": "Limit maximum number of requests and connections.",
"version": "0.1",

29
src/common/core/realip/jobs/realip-download.py
View File

@ -67,8 +67,13 @@ try:
# Create directory if it doesn't exist
Path("/var/cache/bunkerweb/realip").mkdir(parents=True, exist_ok=True)
db = Database(
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
# Don't go further if the cache is fresh
if is_cached_file("/var/cache/bunkerweb/realip/combined.list", "hour"):
if is_cached_file("/var/cache/bunkerweb/realip/combined.list", "hour", db):
logger.info("RealIP list is already in cache, skipping download...")
_exit(0)
@ -106,7 +111,7 @@ try:
# Check if file has changed
new_hash = file_hash("/var/tmp/bunkerweb/realip-combined.list")
old_hash = cache_hash("/var/cache/bunkerweb/realip/combined.list")
old_hash = cache_hash("/var/cache/bunkerweb/realip/combined.list", db)
if new_hash == old_hash:
logger.info("New file is identical to cache file, reload is not needed")
_exit(0)
@ -116,30 +121,12 @@ try:
"/var/tmp/bunkerweb/realip-combined.list",
"/var/cache/bunkerweb/realip/combined.list",
new_hash,
db,
)
if not cached:
logger.error(f"Error while caching list : {err}")
_exit(2)
db = Database(
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
lock = Lock()
# Update db
with lock:
err = db.update_job_cache(
"realip-download",
None,
"combined.list",
content,
checksum=new_hash,
)
if err:
logger.warning(f"Couldn't update db cache: {err}")
logger.info(f"Downloaded {i} trusted IP/net")
status = 1

6
src/common/core/redis/redis.lua → src/common/core/redis/redis.lua.backup
View File

@ -13,10 +13,10 @@ end
function redis:init()
-- Check if init is needed
if self.variables["USE_REDIS"] then
return self:ret(true, "redis not used")
if self.variables["USE_REDIS"] ~= "yes" or self.is_loading then
return self:ret(true, "init not needed")
end
-- Check redis connection
-- Check redis connection ()
local ok, err = clusterstore:connect()
if not ok then
return self:ret(false, "redis connect error : " .. err)

2
src/common/core/reversescan/plugin.json
View File

@ -1,6 +1,6 @@
{
"id": "reversescan",
"order": 5,
"order": 6,
"name": "Reverse scan",
"description": "Scan clients ports to detect proxies or servers.",
"version": "0.1",

12
src/common/core/reversescan/reversescan.lua
View File

@ -25,28 +25,28 @@ function reversescan:access()
-- Loop on ports
for port in self.variables["REVERSE_SCAN_PORTS"]:gmatch("%S+") do
-- Check if the scan is already cached
local cached, err = self:is_in_cache(ngx.var.remote_addr .. ":" .. port)
local cached, err = self:is_in_cache(ngx.ctx.bw.remote_addr .. ":" .. port)
if cached == nil then
return self:ret(false, "error getting cache from datastore : " .. err)
end
if cached == "open" then
return self:ret(true, "port " .. port .. " is opened for IP " .. ngx.var.remote_addr, utils.get_deny_status())
return self:ret(true, "port " .. port .. " is opened for IP " .. ngx.ctx.bw.remote_addr, utils.get_deny_status())
elseif not cached then
-- Do the scan
local res, err = self:scan(ngx.var.remote_addr, tonumber(port), tonumber(self.variables["REVERSE_SCAN_TIMEOUT"]))
local res, err = self:scan(ngx.ctx.bw.remote_addr, tonumber(port), tonumber(self.variables["REVERSE_SCAN_TIMEOUT"]))
-- Cache the result
local ok, err = self:add_to_cache(ngx.var.remote_addr .. ":" .. port, res)
local ok, err = self:add_to_cache(ngx.ctx.bw.remote_addr .. ":" .. port, res)
if not ok then
return self:ret(false, "error updating cache from datastore : " .. err)
end
-- Deny request if port is open
if res == "open" then
return self:ret(true, "port " .. port .. " is opened for IP " .. ngx.var.remote_addr, utils.get_deny_status())
return self:ret(true, "port " .. port .. " is opened for IP " .. ngx.ctx.bw.remote_addr, utils.get_deny_status())
end
end
end
-- No port opened
return self:ret(true, "no port open for IP " .. ngx.var.remote_addr)
return self:ret(true, "no port open for IP " .. ngx.ctx.bw.remote_addr)
end
function reversescan:scan(ip, port, timeout)

14
src/common/core/selfsigned/jobs/self-signed.py
View File

@ -42,21 +42,23 @@ def generate_cert(first_server, days, subj):
return False, 2
# Update db
key_data = Path(f"/var/cache/bunkerweb/selfsigned/{first_server}.key").read_bytes()
with lock:
err = db.update_job_cache(
"self-signed", first_server, f"{first_server}.key", key_data
"self-signed",
first_server,
f"{first_server}.key",
Path(f"/var/cache/bunkerweb/selfsigned/{first_server}.key").read_bytes(),
)
if err:
logger.warning(f"Couldn't update db cache for {first_server}.key file: {err}")
pem_data = Path(f"/var/cache/bunkerweb/selfsigned/{first_server}.pem").read_bytes()
with lock:
err = db.update_job_cache(
"self-signed", first_server, f"{first_server}.pem", pem_data
"self-signed",
first_server,
f"{first_server}.pem",
Path(f"/var/cache/bunkerweb/selfsigned/{first_server}.pem").read_bytes(),
)
if err:

11
src/common/core/sessions/sessions.lua
View File

@ -11,21 +11,26 @@ function sessions:initialize()
end
function sessions:init()
if self.is_loading then
return self:ret(true, "init not needed")
end
-- Get redis vars
local redis_vars = {
["USE_REDIS"] = "",
["REDIS_HOST"] = "",
["REDIS_PORT"] = "",
["REDIS_DATABASE"] = "",
["REDIS_SSL"] = "",
["REDIS_TIMEOUT"] = "",
["REDIS_KEEPALIVE_IDLE"] = "",
["REDIS_KEEPALIVE_POOL"] = ""
}
for k, v in pairs(redis_vars) do
local var, err = utils.get_variable(k, false)
if var == nil then
local value, err = utils.get_variable(k, false)
if value == nil then
return self:ret(false, "can't get " .. k .. " variable : " .. err)
end
redis_vars[k] = value
end
-- Init configuration
local config = {
@ -55,7 +60,7 @@ function sessions:init()
pool_size = tonumber(redis_vars["REDIS_KEEPALIVE_POOL"]),
ssl = redis_vars["REDIS_SSL"] == "yes",
host = redis_vars["REDIS_HOST"],
port = tonumber(redis_vars["REDIS_HOST"]),
port = tonumber(redis_vars["REDIS_PORT"]),
database = tonumber(redis_vars["REDIS_DATABASE"])
}
end

32
src/common/core/whitelist/jobs/whitelist-download.py
View File

@ -6,7 +6,6 @@ from os import _exit, getenv
from pathlib import Path
from re import IGNORECASE, compile as re_compile
from sys import exit as sys_exit, path as sys_path
from threading import Lock
from traceback import format_exc
from typing import Tuple
@ -80,6 +79,11 @@ try:
logger.info("Whitelist is not activated, skipping downloads...")
_exit(0)
db = Database(
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
# Create directories if they don't exist
Path("/var/cache/bunkerweb/whitelist").mkdir(parents=True, exist_ok=True)
Path("/var/tmp/bunkerweb/whitelist").mkdir(parents=True, exist_ok=True)
@ -97,7 +101,9 @@ try:
}
all_fresh = True
for kind in kinds_fresh:
if not is_cached_file(f"/var/cache/bunkerweb/whitelist/{kind}.list", "hour"):
if not is_cached_file(
f"/var/cache/bunkerweb/whitelist/{kind}.list", "hour", db
):
kinds_fresh[kind] = False
all_fresh = False
logger.info(
@ -150,7 +156,7 @@ try:
logger.info(f"Downloaded {i} good {kind}")
# Check if file has changed
new_hash = file_hash(f"/var/tmp/bunkerweb/whitelist/{kind}.list")
old_hash = cache_hash(f"/var/cache/bunkerweb/whitelist/{kind}.list")
old_hash = cache_hash(f"/var/cache/bunkerweb/whitelist/{kind}.list", db)
if new_hash == old_hash:
logger.info(
f"New file {kind}.list is identical to cache file, reload is not needed",
@ -164,30 +170,12 @@ try:
f"/var/tmp/bunkerweb/whitelist/{kind}.list",
f"/var/cache/bunkerweb/whitelist/{kind}.list",
new_hash,
db,
)
if not cached:
logger.error(f"Error while caching whitelist : {err}")
status = 2
else:
db = Database(
logger,
sqlalchemy_string=getenv("DATABASE_URI", None),
)
lock = Lock()
# Update db
with lock:
err = db.update_job_cache(
"whitelist-download",
None,
f"{kind}.list",
content,
checksum=new_hash,
)
if err:
logger.warning(f"Couldn't update db cache: {err}")
status = 1
except:
status = 2
logger.error(

75
src/common/core/whitelist/whitelist.lua
View File

@ -18,29 +18,38 @@ function whitelist:initialize()
self.logger:log(ngx.ERR, err)
end
self.use_redis = use_redis == "yes"
-- Check if init is needed
if ngx.get_phase() == "init" then
local init_needed, err = utils.has_variable("USE_WHITELIST", "yes")
if init_needed == nil then
self.logger:log(ngx.ERR, err)
end
self.init_needed = init_needed
-- Decode lists
else
if ngx.get_phase() ~= "init" and self.variables["USE_WHITELIST"] == "yes" then
local lists, err = self.datastore:get("plugin_whitelist_lists")
if not lists then
self.logger:log(ngx.ERR, err)
else
self.lists = cjson.decode(lists)
end
local kinds = {
["IP"] = {},
["RDNS"] = {},
["ASN"] = {},
["USER_AGENT"] = {},
["URI"] = {}
}
for kind, _ in pairs(kinds) do
for data in self.variables["WHITELIST_" .. kind]:gmatch("%S+") do
table.insert(self.lists[kind], data)
end
end
end
-- Instantiate cachestore
self.cachestore = cachestore:new(self.use_redis)
self.cachestore = cachestore:new(self.use_redis and ngx.get_phase() == "access")
end
function whitelist:init()
-- Check if init is needed
if not self.init_needed then
local init_needed, err = utils.has_variable("USE_WHITELIST", "yes")
if init_needed == nil then
return self:ret(false, "can't check USE_WHITELIST variable : " .. err)
end
if not init_needed or self.is_loading then
return self:ret(true, "init not needed")
end
-- Read whitelists
@ -73,6 +82,7 @@ end
function whitelist:set()
-- Set default value
ngx.var.is_whitelisted = "no"
ngx.ctx.bw.is_whitelisted = "no"
env.set("is_whitelisted", "no")
-- Check if set is needed
if self.variables["USE_WHITELIST"] ~= "yes" then
@ -84,6 +94,7 @@ function whitelist:set()
return self:ret(false, err)
elseif whitelisted then
ngx.var.is_whitelisted = "yes"
ngx.ctx.bw.is_whitelisted = "yes"
env.set("is_whitelisted", "yes")
return self:ret(true, err)
end
@ -101,6 +112,7 @@ function whitelist:access()
return self:ret(false, err)
elseif whitelisted then
ngx.var.is_whitelisted = "yes"
ngx.ctx.bw.is_whitelisted = "yes"
env.set("is_whitelisted", "yes")
return self:ret(true, err, ngx.OK)
end
@ -117,8 +129,9 @@ function whitelist:access()
end
if whitelisted ~= "ok" then
ngx.var.is_whitelisted = "yes"
ngx.ctx.bw.is_whitelisted = "yes"
env.set("is_whitelisted", "yes")
return self:ret(true, k + " is whitelisted (info : " .. whitelisted .. ")", ngx.OK)
return self:ret(true, k .. " is whitelisted (info : " .. whitelisted .. ")", ngx.OK)
end
end
end
@ -133,24 +146,24 @@ end
function whitelist:kind_to_ele(kind)
if kind == "IP" then
return "ip" .. ngx.var.remote_addr
return "ip" .. ngx.ctx.bw.remote_addr
elseif kind == "UA" then
return "ua" .. ngx.var.http_user_agent
return "ua" .. ngx.ctx.bw.http_user_agent
elseif kind == "URI" then
return "uri" .. ngx.var.uri
return "uri" .. ngx.ctx.bw.uri
end
end
function whitelist:check_cache()
-- Check the caches
local checks = {
["IP"] = "ip" .. ngx.var.remote_addr
["IP"] = "ip" .. ngx.ctx.bw.remote_addr
}
if ngx.var.http_user_agent then
checks["UA"] = "ua" .. ngx.var.http_user_agent
if ngx.ctx.bw.http_user_agent then
checks["UA"] = "ua" .. ngx.ctx.bw.http_user_agent
end
if ngx.var.uri then
checks["URI"] = "uri" .. ngx.var.uri
if ngx.ctx.bw.uri then
checks["URI"] = "uri" .. ngx.ctx.bw.uri
end
local already_cached = {
["IP"] = false,
@ -162,7 +175,7 @@ function whitelist:check_cache()
if not ok then
self.logger:log(ngx.ERR, "error while checking cache : " .. cached)
elseif cached and cached ~= "ok" then
return true, k + " is in cached whitelist (info : " .. cached .. ")"
return true, k .. " is in cached whitelist (info : " .. cached .. ")"
end
if cached then
already_cached[k] = true
@ -209,7 +222,7 @@ function whitelist:is_whitelisted_ip()
if not ipm then
return nil, err
end
local match, err = ipm:match(ngx.var.remote_addr)
local match, err = ipm:match(ngx.ctx.bw.remote_addr)
if err then
return nil, err
end
@ -219,18 +232,12 @@ function whitelist:is_whitelisted_ip()
-- Check if rDNS is needed
local check_rdns = true
local is_global, err = utils.ip_is_global(ngx.var.remote_addr)
if self.variables["WHITELIST_RDNS_GLOBAL"] == "yes" then
if is_global == nil then
return nil, err
end
if not is_global then
check_rdns = false
end
if self.variables["WHITELIST_RDNS_GLOBAL"] == "yes" and not ngx.ctx.bw.ip_is_global then
check_rdns = false
end
if check_rdns then
-- Get rDNS
local rdns_list, err = utils.get_rdns(ngx.var.remote_addr)
local rdns_list, err = utils.get_rdns(ngx.ctx.bw.remote_addr)
if not rdns_list then
return nil, err
end
@ -245,8 +252,8 @@ function whitelist:is_whitelisted_ip()
end
-- Check if ASN is in whitelist
if is_global then
local asn, err = utils.get_asn(ngx.var.remote_addr)
if ngx.ctx.bw.ip_is_global then
local asn, err = utils.get_asn(ngx.ctx.bw.remote_addr)
if not asn then
return nil, err
end
@ -264,7 +271,7 @@ end
function whitelist:is_whitelisted_uri()
-- Check if URI is in whitelist
for i, uri in ipairs(self.lists["URI"]) do
if ngx.var.uri:match(uri) then
if ngx.ctx.bw.uri:match(uri) then
return true, "URI " .. uri
end
end
@ -275,7 +282,7 @@ end
function whitelist:is_whitelisted_ua()
-- Check if UA is in whitelist
for i, ua in ipairs(self.lists["USER_AGENT"]) do
if ngx.var.http_user_agent:match(ua) then
if ngx.ctx.bw.http_user_agent:match(ua) then
return true, "UA " .. ua
end
end

17
src/common/db/Database.py
View File

@ -1501,12 +1501,25 @@ class Database:
)
}
def get_job_cache_file(self, job_name: str, file_name: str) -> Optional[Any]:
def get_job_cache_file(
self,
job_name: str,
file_name: str,
*,
with_info: bool = False,
with_data: bool = True,
) -> Optional[Any]:
"""Get job cache file."""
entities = []
if with_info:
entities.extend([Jobs_cache.last_update, Jobs_cache.checksum])
if with_data:
entities.append(Jobs_cache.data)
with self.__db_session() as session:
return (
session.query(Jobs_cache)
.with_entities(Jobs_cache.data)
.with_entities(*entities)
.filter_by(job_name=job_name, file_name=file_name)
.first()
)

2
src/common/db/requirements.in
View File

@ -1,4 +1,4 @@
sqlalchemy==2.0.9
sqlalchemy==2.0.10
psycopg2-binary==2.9.6
PyMySQL==1.0.3
cryptography==40.0.2

84
src/common/db/requirements.txt
View File

@ -225,48 +225,48 @@ pymysql==1.0.3 \
--hash=sha256:3dda943ef3694068a75d69d071755dbecacee1adf9a1fc5b206830d2b67d25e8 \
--hash=sha256:89fc6ae41c0aeb6e1f7710cdd623702ea2c54d040565767a78b00a5ebb12f4e5
# via -r requirements.in
sqlalchemy==2.0.9 \
--hash=sha256:07950fc82f844a2de67ddb4e535f29b65652b4d95e8b847823ce66a6d540a41d \
--hash=sha256:0a865b5ec4ba24f57c33b633b728e43fde77b968911a6046443f581b25d29dd9 \
--hash=sha256:0b49f1f71d7a44329a43d3edd38cc5ee4c058dfef4487498393d16172007954b \
--hash=sha256:13f984a190d249769a050634b248aef8991acc035e849d02b634ea006c028fa8 \
--hash=sha256:1b69666e25cc03c602d9d3d460e1281810109e6546739187044fc256c67941ef \
--hash=sha256:1d06e119cf79a3d80ab069f064a07152eb9ba541d084bdaee728d8a6f03fd03d \
--hash=sha256:246712af9fc761d6c13f4f065470982e175d902e77aa4218c9cb9fc9ff565a0c \
--hash=sha256:34eb96c1de91d8f31e988302243357bef3f7785e1b728c7d4b98bd0c117dafeb \
--hash=sha256:4c3020afb144572c7bfcba9d7cce57ad42bff6e6115dffcfe2d4ae6d444a214f \
--hash=sha256:4f759eccb66e6d495fb622eb7f4ac146ae674d829942ec18b7f5a35ddf029597 \
--hash=sha256:68ed381bc340b4a3d373dbfec1a8b971f6350139590c4ca3cb722fdb50035777 \
--hash=sha256:6b72dccc5864ea95c93e0a9c4e397708917fb450f96737b4a8395d009f90b868 \
--hash=sha256:6e84ab63d25d8564d7a8c05dc080659931a459ee27f6ed1cf4c91f292d184038 \
--hash=sha256:734805708632e3965c2c40081f9a59263c29ffa27cba9b02d4d92dfd57ba869f \
--hash=sha256:78612edf4ba50d407d0eb3a64e9ec76e6efc2b5d9a5c63415d53e540266a230a \
--hash=sha256:7e472e9627882f2d75b87ff91c5a2bc45b31a226efc7cc0a054a94fffef85862 \
--hash=sha256:865392a50a721445156809c1a6d6ab6437be70c1c2599f591a8849ed95d3c693 \
--hash=sha256:8d118e233f416d713aac715e2c1101e17f91e696ff315fc9efbc75b70d11e740 \
--hash=sha256:8d3ece5960b3e821e43a4927cc851b6e84a431976d3ffe02aadb96519044807e \
--hash=sha256:93c78d42c14aa9a9e0866eacd5b48df40a50d0e2790ee377af7910d224afddcf \
--hash=sha256:95719215e3ec7337b9f57c3c2eda0e6a7619be194a5166c07c1e599f6afc20fa \
--hash=sha256:9838bd247ee42eb74193d865e48dd62eb50e45e3fdceb0fdef3351133ee53dcf \
--hash=sha256:aa5c270ece17c0c0e0a38f2530c16b20ea05d8b794e46c79171a86b93b758891 \
--hash=sha256:ac6a0311fb21a99855953f84c43fcff4bdca27a2ffcc4f4d806b26b54b5cddc9 \
--hash=sha256:ad5363a1c65fde7b7466769d4261126d07d872fc2e816487ae6cec93da604b6b \
--hash=sha256:b3e5864eba71a3718236a120547e52c8da2ccb57cc96cecd0480106a0c799c92 \
--hash=sha256:bbda1da8d541904ba262825a833c9f619e93cb3fd1156be0a5e43cd54d588dcd \
--hash=sha256:c6e27189ff9aebfb2c02fd252c629ea58657e7a5ff1a321b7fc9c2bf6dc0b5f3 \
--hash=sha256:c8239ce63a90007bce479adf5460d48c1adae4b933d8e39a4eafecfc084e503c \
--hash=sha256:d209594e68bec103ad5243ecac1b40bf5770c9ebf482df7abf175748a34f4853 \
--hash=sha256:d5327f54a9c39e7871fc532639616f3777304364a0bb9b89d6033ad34ef6c5f8 \
--hash=sha256:db4bd1c4792da753f914ff0b688086b9a8fd78bb9bc5ae8b6d2e65f176b81eb9 \
--hash=sha256:e4780be0f19e5894c17f75fc8de2fe1ae233ab37827125239ceb593c6f6bd1e2 \
--hash=sha256:e4a019f723b6c1e6b3781be00fb9e0844bc6156f9951c836ff60787cc3938d76 \
--hash=sha256:e62c4e762d6fd2901692a093f208a6a6575b930e9458ad58c2a7f080dd6132da \
--hash=sha256:e730603cae5747bc6d6dece98b45a57d647ed553c8d5ecef602697b1c1501cf2 \
--hash=sha256:ebc4eeb1737a5a9bdb0c24f4c982319fa6edd23cdee27180978c29cbb026f2bd \
--hash=sha256:ee2946042cc7851842d7a086a92b9b7b494cbe8c3e7e4627e27bc912d3a7655e \
--hash=sha256:f005245e1cb9b8ca53df73ee85e029ac43155e062405015e49ec6187a2e3fb44 \
--hash=sha256:f49c5d3c070a72ecb96df703966c9678dda0d4cb2e2736f88d15f5e1203b4159 \
--hash=sha256:f61ab84956dc628c8dfe9d105b6aec38afb96adae3e5e7da6085b583ff6ea789
sqlalchemy==2.0.10 \
--hash=sha256:04020aba2c0266ec521095ddd5cb760fc0067b0088828ccbf6b323c900a62e59 \
--hash=sha256:06401013dad015e6f6f72c946f66d750fe4c5ef852ed2f15537d572cb92d7a75 \
--hash=sha256:096d9f72882035b4c6906172bf5c5afe4caefbfe0e028ab0c83dfdaa670cc193 \
--hash=sha256:1f5638aac94c8f3fe04ca030e2b3e84d52d70f15d67f35f794fd2057284abced \
--hash=sha256:1fa90ed075ebc5fefc504c0e35b84fde1880d7c095473c5aa0c01f63eb37beae \
--hash=sha256:207c2cc9b946f832fd45fbdd6276c28e3e80b206909a028cd163e87f4080a333 \
--hash=sha256:23e3e1cc3634a70bba2ab10c144d4f11cf0ddeca239bbdaf646770873030c600 \
--hash=sha256:28c79289b4bf21cf09fb770b124cfae2432bbafb2ffd6758ac280bc1cacabfac \
--hash=sha256:2bd944dc701be15a91ec965c6634ab90998ca2d14e4f1f568545547a3a3adc16 \
--hash=sha256:2fdccadc9359784ae12ae9199849b724c7165220ae93c6066e841b66c6823742 \
--hash=sha256:300e8165bc78a0a917b39617730caf2c08c399302137c562e5ce7a37780ad10f \
--hash=sha256:39869cf2cfe73c8ad9a6f15712a2ed8c13c1f87646611882efb6a8ec80d180e8 \
--hash=sha256:3e77ed2e6d911aafc931c92033262d2979a44317294328b071a53aa10e2a9614 \
--hash=sha256:4a1ec8fcbe7e6a6ec28e161c6030d8cf5077e31efc3d08708d8de5aa8314b345 \
--hash=sha256:5892afc393ecd5f20910ff5a6b90d56620ec2ef3e36e3358eaedbae2aa36816d \
--hash=sha256:5e8abd2ce0745a2819f3e41a17570c9d74b634a5b5ab5a04de5919e55d5d8601 \
--hash=sha256:61ea1af2d01e709dcd4edc0d994db42bac6b2673c093cc35df3875e54cad9cef \
--hash=sha256:631ea4d1a8d78b43126773fa2de5472d97eb54dc4b9fbae4d8bd910f72f31f25 \
--hash=sha256:6b15cadba33d77e6fcee4f4f7706913d143d20e48ce26e9b6578b5cd07d4a353 \
--hash=sha256:70aed8f508f6c2f4da63ee6fa853534bb97d47bc82e28d56442f62a0b6ad2660 \
--hash=sha256:736e92fa4d6e020fc780b915bcdd69749ad32c79bc6b031e85dcd2b8069f8de1 \
--hash=sha256:7a8ca39fbc2dfe357f03e398bf5c1421b9b6614a8cf69ccada9ab3ef7e036073 \
--hash=sha256:7da5bf86746ddbf8d68f1a3f9d1efee1d95e07d5ad63f47b839f4db799e12566 \
--hash=sha256:88df3327c32468716a52c10e7991268afb552a0a7ef36130925864f28873d2e0 \
--hash=sha256:89e7a05639b3ae4fd17062a37b0ee336ea50ac9751e98e3330a6ed95daa4880c \
--hash=sha256:8a3e3f34468a512b3886ac5584384aed8bef388297c710509a842fb1468476f3 \
--hash=sha256:8c3366be42bca5c066703af54b856e00f23b8fbef9ab0346a58d34245af695a5 \
--hash=sha256:9a77e29a96779f373eb144040e5fae1e3944916c13360715e74f73b186f0d8d2 \
--hash=sha256:a4cdac392547dec07d69c5e8b05374b0357359ebc58ab2bbcb9fa0370ecb715f \
--hash=sha256:a9aa445201754a49b7ddb0b99fbe5ccf98f6900548fc60a0a07dde2253dd541e \
--hash=sha256:af525e9fbcf7da7404fc4b91ca4ce6172457d3f4390b93941fb97bfe29afb7dc \
--hash=sha256:b608ad640ac70e2901d111a69ad975e6b0ca39947e08cc28691b0de00831a787 \
--hash=sha256:d46edd508123413595a17bb64655db7c4bfefa83e721a3064f66e046e9a6a103 \
--hash=sha256:d975ac2bc513f530fa2574eb58e0ca731357d4686de2fb644af3036fca4f3fd6 \
--hash=sha256:dcd5793b98eb043703895443cc399fb8e2ce21c9b09757e954e425c8415c541b \
--hash=sha256:dd40fbf4f916a41b4afe50665e2d029a1c9f74967fd3b7422475529641d31ef5 \
--hash=sha256:dddbe2c012d712873fb9f203512db57d3cbdd20803f0792aa01bc513da8a2380 \
--hash=sha256:e9d7e65c2c4f313524399f6b8ec14bfa8f4e9fccd999ff585e10e073cfd21429 \
--hash=sha256:ec910449c70b0359dbe08a5e8c63678c7ef0113ab61cd0bb2e80ed09ea8ce6ab \
--hash=sha256:ed368ee7b1c119d5f6321cc9a3ea806adacf522bb4c2e9e398cbfc2e2cc68a2a \
--hash=sha256:faa6d2e6d6d46d2d58c5a4713148300b44fcfc911341ec82d8731488d0757f96
# via -r requirements.in
typing-extensions==4.5.0 \
--hash=sha256:5cb5f4a79139d699607b3ef622a1dedafa84e115ab0024e0d9c044a9479ca7cb \

23
src/common/gen/Configurator.py
View File

@ -146,14 +146,21 @@ class Configurator:
ret, err = self.__check_var(variable)
if ret:
config[variable] = value
elif not variable.startswith("PYTHON") and variable not in (
"GPG_KEY",
"LANG",
"PATH",
"NGINX_VERSION",
"NJS_VERSION",
"PKG_RELEASE",
"DOCKER_HOST",
elif (
not variable.startswith("PYTHON")
and not variable.startswith("KUBERNETES_SERVICE_")
and not variable.startswith("KUBERNETES_PORT_")
and not variable.startswith("SVC_")
and variable
not in (
"GPG_KEY",
"LANG",
"PATH",
"NGINX_VERSION",
"NJS_VERSION",
"PKG_RELEASE",
"DOCKER_HOST",
)
):
self.__logger.warning(f"Ignoring variable {variable} : {err}")
# Expand variables to each sites if MULTISITE=yes and if not present

18
src/common/gen/requirements.txt
View File

@ -171,15 +171,15 @@ packaging==23.1 \
--hash=sha256:994793af429502c4ea2ebf6bf664629d07c1a9fe974af92966e4b8d2df7edc61 \
--hash=sha256:a392980d2b6cffa644431898be54b0045151319d1e7ec34f0cfed48767dd334f
# via docker
pyasn1==0.4.8 \
--hash=sha256:39c7e2ec30515947ff4e87fb6f456dfc6e84857d34be479c9d4a4ba4bf46aa5d \
--hash=sha256:aef77c9fb94a3ac588e87841208bdec464471d9871bd5050a287cc9a475cd0ba
pyasn1==0.5.0 \
--hash=sha256:87a2121042a1ac9358cabcaf1d07680ff97ee6404333bacca15f76aa8ad01a57 \
--hash=sha256:97b7290ca68e62a832558ec3976f15cbf911bf5d7c7039d8b861c2a0ece69fde
# via
# pyasn1-modules
# rsa
pyasn1-modules==0.2.8 \
--hash=sha256:905f84c712230b2c592c19470d3ca8d552de726050d1d1716282a1f6146be65e \
--hash=sha256:a50b808ffeb97cb3601dd25981f6b016cbb3d31fbf57a8b8a87428e6158d0c74
pyasn1-modules==0.3.0 \
--hash=sha256:5bd01446b736eb9d31512a30d46c1ac3395d676c6f3cafa4c03eb54b9925631c \
--hash=sha256:d3ccd6ed470d9ffbc716be08bd90efbd44d0734bc9303818f7336070984a162d
# via google-auth
python-dateutil==2.8.2 \
--hash=sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86 \
@ -269,7 +269,7 @@ websocket-client==1.5.1 \
# kubernetes
# The following packages are considered to be unsafe in a requirements file:
setuptools==67.6.1 \
--hash=sha256:257de92a9d50a60b8e22abfcbb771571fde0dbf3ec234463212027a4eeecbe9a \
--hash=sha256:e728ca814a823bf7bf60162daf9db95b93d532948c4c0bea762ce62f60189078
setuptools==67.7.1 \
--hash=sha256:6f0839fbdb7e3cfef1fc38d7954f5c1c26bf4eebb155a55c9bf8faf997b9fb67 \
--hash=sha256:bb16732e8eb928922eabaa022f881ae2b7cdcfaf9993ef1f5e841a96d32b8e0c
# via kubernetes

88
src/common/utils/jobs.py
View File

@ -1,11 +1,16 @@
from contextlib import suppress
from datetime import datetime
from hashlib import sha512
from inspect import getsourcefile
from json import dumps, loads
from os.path import basename
from pathlib import Path
from shutil import copy
from sys import _getframe
from threading import Lock
from traceback import format_exc
from typing import Optional, Tuple
lock = Lock()
"""
{
@ -15,29 +20,46 @@ from traceback import format_exc
"""
def is_cached_file(file, expire):
def is_cached_file(file: str, expire: str, db=None) -> bool:
is_cached = False
cached_file = None
try:
if not Path(f"{file}.md").is_file():
return False
if not db:
return False
cached_file = db.get_job_cache_file(
basename(getsourcefile(_getframe(1))).replace(".py", ""),
basename(file),
with_info=True,
)
if not cached_file:
return False
cached_time = cached_file.last_update.timestamp()
else:
cached_time = loads(Path(f"{file}.md").read_text())["date"]
cached_time = loads(Path(f"{file}.md").read_text())["date"]
current_time = datetime.now().timestamp()
if current_time < cached_time:
return False
diff_time = current_time - cached_time
if expire == "hour":
is_cached = diff_time < 3600
elif expire == "day":
is_cached = diff_time < 86400
elif expire == "month":
is_cached = diff_time < 2592000
is_cached = False
else:
diff_time = current_time - cached_time
if expire == "hour":
is_cached = diff_time < 3600
elif expire == "day":
is_cached = diff_time < 86400
elif expire == "month":
is_cached = diff_time < 2592000
except:
is_cached = False
if is_cached and cached_file:
Path(file).write_bytes(cached_file.data)
return is_cached
def file_hash(file):
def file_hash(file: str) -> str:
_sha512 = sha512()
with open(file, "rb") as f:
while True:
@ -48,19 +70,47 @@ def file_hash(file):
return _sha512.hexdigest()
def cache_hash(cache):
def cache_hash(cache: str, db=None) -> Optional[str]:
with suppress(BaseException):
return loads(Path(f"{cache}.md").read_text())["checksum"]
return loads(Path(f"{cache}.md").read_text()).get("checksum", None)
if db:
cached_file = db.get_job_cache_file(
basename(getsourcefile(_getframe(1))).replace(".py", ""),
basename(cache),
with_info=True,
with_data=False,
)
if cached_file:
return cached_file.checksum
return None
def cache_file(file, cache, _hash):
def cache_file(
file: str, cache: str, _hash: str, db=None, *, service_id: Optional[str] = None
) -> Tuple[bool, str]:
ret, err = True, "success"
try:
copy(file, cache)
content = Path(file).read_bytes()
Path(cache).write_bytes(content)
Path(file).unlink()
md = {"date": datetime.timestamp(datetime.now()), "checksum": _hash}
Path(f"{cache}.md").write_text(dumps(md))
if db:
with lock:
err = db.update_job_cache(
basename(getsourcefile(_getframe(1))).replace(".py", ""),
service_id,
basename(cache),
content,
checksum=_hash,
)
if err:
ret = False
else:
Path(f"{cache}.md").write_text(
dumps(dict(date=datetime.now().timestamp(), checksum=_hash))
)
except:
return False, f"exception :\n{format_exc()}"
return ret, err

4
src/deps/clone.sh
View File

@ -276,6 +276,10 @@ if [ "$dopatch" == "yes" ] ; then
do_and_check_cmd rm -r deps/src/lua-resty-openssl/t
fi
# lua-ffi-zlib v0.5.0
echo "ℹ️ Downloading lua-ffi-zlib"
git_secure_clone "https://github.com/hamishforbes/lua-ffi-zlib.git" "1fb69ca505444097c82d2b72e87904f3ed923ae9"
# ModSecurity v3.0.9
echo "ℹ️ Downloading ModSecurity"
dopatch="no"

5
src/deps/install.sh
View File

@ -142,6 +142,11 @@ CHANGE_DIR="/tmp/bunkerweb/deps/src/lua-pack" do_and_check_cmd make INST_LIBDIR=
# Installing lua-resty-openssl
echo "ℹ️ Installing lua-resty-openssl"
CHANGE_DIR="/tmp/bunkerweb/deps/src/lua-resty-openssl" do_and_check_cmd make LUA_LIB_DIR=/usr/share/bunkerweb/deps/lib/lua install
do_and_check_cmd cp /tmp/bunkerweb/deps/src/lua-resty-openssl/lib/resty/openssl.lua /usr/share/bunkerweb/deps/lib/lua/resty
# Installing lua-ffi-zlib
echo "ℹ️ Installing lua-ffi-zlib"
do_and_check_cmd cp /tmp/bunkerweb/deps/src/lua-ffi-zlib/lib/ffi-zlib.lua /usr/share/bunkerweb/deps/lib/lua
# Compile dynamic modules
echo "ℹ️ Compiling and installing dynamic modules"

21
src/deps/src/lua-ffi-zlib/LICENSE.txt Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2016 Hamish Forbes
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

133
src/deps/src/lua-ffi-zlib/README.md Normal file
View File

@ -0,0 +1,133 @@
# lua-ffi-zlib
A [Lua](http://www.lua.org) module using LuaJIT's [FFI](http://luajit.org/ext_ffi.html) feature to access zlib.
Intended primarily for use within [OpenResty](http://openresty.org) to allow manipulation of gzip encoded HTTP responses.
# Methods
Basic methods allowing for simple compression or decompression of gzip data
## inflateGzip
`Syntax: ok, err = inflateGzip(input, output, chunk?, windowBits?)`
* `input` should be a function that accepts a chunksize as its only argument and return that many bytes of the gzip stream
* `output` will receive a string of decompressed data as its only argument, do with it as you will!
* `chunk` is the size of the input and output buffers, optional and defaults to 16KB
* `windowBits` is passed to `inflateInit2()`, should be left as default for most cases.
See [zlib manual](http://zlib.net/manual.html) for details
On error returns `false` and the error message, otherwise `true` and the last status message
## deflateGzip
`Syntax: ok, err = deflateGzip(input, output, chunk?, options?)`
* `input` should be a function that accepts a chunksize as its only argument and return that many bytes of uncompressed data.
* `output` will receive a string of compressed data as its only argument, do with it as you will!
* `chunk` is the size of the input and output buffers, optional and defaults to 16KB
* `options` is a table of options to pass to `deflateInit2()`
Valid options are level, memLevel, strategy and windowBits, see [zlib manual](http://zlib.net/manual.html) for details
On error returns `false` and the error message, otherwise `true` and the last status message
# Example
Reads a file and output the decompressed version.
Roughly equivalent to running `gzip -dc file.gz > out_file | tee`
```lua
local table_insert = table.insert
local table_concat = table.concat
local zlib = require('lib.ffi-zlib')
local f = io.open(arg[1], "rb")
local out_f = io.open(arg[2], "w")
local input = function(bufsize)
-- Read the next chunk
local d = f:read(bufsize)
if d == nil then
return nil
end
return d
end
local output_table = {}
local output = function(data)
table_insert(output_table, data)
local ok, err = out_f:write(data)
if not ok then
-- abort decompression when error occurs
return nil, err
end
end
-- Decompress the data
local ok, err = zlib.inflateGzip(input, output)
if not ok then
print(err)
return
end
local decompressed = table_concat(output_table,'')
print(decompressed)
```
# Advanced Usage
Several other methods are available for advanced usage.
Some of these map directly to functions in the zlib library itself, see the [manual](http://zlib.net/manual.html) for full details.
Others are lower level utility functions.
## createStream
`Synax: stream, inbuf, outbuf = createStream(bufsize)`
Returns a z_stream struct, input buffer and output buffer of length `bufsize`
## initInflate
`Syntax: ok = initInflate(stream, windowBits?)`
Calls zlib's inflateInit2 with given stream, defaults to automatic header detection.
## initDeflate
`Syntax: ok = initDeflate(stream, options?)`
Calls zlib's deflateInit2 with the given stream.
`options` is an optional table that can set level, memLevel, strategy and windowBits
## deflate
`Syntax: ok, err = deflate(input, output, bufsize, stream, inbuf, outbuf)`
* `input` is a function that takes a chunk size argument and returns at most that many input bytes
* `output` is a function that takes a string argument of output data
* `bufsize` is the length of the output buffer
* `inbuf` cdata input buffer
* `outpuf` ccdata output buffer
This function will loop until all input data is consumed (`input` returns nil) or an error occurs.
It will then clean up the stream and return an error code
## inflate
`Syntax: ok, err = inflate(input, output, bufsize, stream, inbuf, outbuf)`
* `input` is a function that takes a chunk size argument and returns at most that many input bytes
* `output` is a function that takes a string argument of output data
* `bufsize` is the length of the output buffer
* `inbuf` cdata input buffer
* `outpuf` ccdata output buffer
This function will loop until all input data is consumed (`input` returns nil) or an error occurs.
It will then clean up the stream and return an error code
## adler
`Syntax: chksum = adler(str, chksum?)`
Computes an adler32 checksum for a string, updates an existing checksum if provided
## crc
`Syntax: chksum = crc(str, chksum?)`
Computes an crc32 checksum for a string, updates an existing checksum if provided
## zlib_err
`Syntax: err = zlib_err(code)`
Returns the string representation of a zlib error code

9
src/deps/src/lua-ffi-zlib/dist.ini Normal file
View File

@ -0,0 +1,9 @@
name=lua-ffi-zlib
abstract=Luajit FFI binding for zlib
author=Hamish Forbes
is_original=yes
license=mit
lib_dir=lib
repo_link=https://github.com/hamishforbes/lua-ffi-zlib
main_module=lib/ffi-zlib.lua
requires = luajit

330
src/deps/src/lua-ffi-zlib/lib/ffi-zlib.lua Normal file
View File

@ -0,0 +1,330 @@
local ffi = require "ffi"
local ffi_new = ffi.new
local ffi_str = ffi.string
local ffi_sizeof = ffi.sizeof
local ffi_copy = ffi.copy
local tonumber = tonumber
local _M = {
_VERSION = '0.5.0',
}
local mt = { __index = _M }
ffi.cdef([[
enum {
Z_NO_FLUSH = 0,
Z_PARTIAL_FLUSH = 1,
Z_SYNC_FLUSH = 2,
Z_FULL_FLUSH = 3,
Z_FINISH = 4,
Z_BLOCK = 5,
Z_TREES = 6,
/* Allowed flush values; see deflate() and inflate() below for details */
Z_OK = 0,
Z_STREAM_END = 1,
Z_NEED_DICT = 2,
Z_ERRNO = -1,
Z_STREAM_ERROR = -2,
Z_DATA_ERROR = -3,
Z_MEM_ERROR = -4,
Z_BUF_ERROR = -5,
Z_VERSION_ERROR = -6,
/* Return codes for the compression/decompression functions. Negative values
* are errors, positive values are used for special but normal events.
*/
Z_NO_COMPRESSION = 0,
Z_BEST_SPEED = 1,
Z_BEST_COMPRESSION = 9,
Z_DEFAULT_COMPRESSION = -1,
/* compression levels */
Z_FILTERED = 1,
Z_HUFFMAN_ONLY = 2,
Z_RLE = 3,
Z_FIXED = 4,
Z_DEFAULT_STRATEGY = 0,
/* compression strategy; see deflateInit2() below for details */
Z_BINARY = 0,
Z_TEXT = 1,
Z_ASCII = Z_TEXT, /* for compatibility with 1.2.2 and earlier */
Z_UNKNOWN = 2,
/* Possible values of the data_type field (though see inflate()) */
Z_DEFLATED = 8,
/* The deflate compression method (the only one supported in this version) */
Z_NULL = 0, /* for initializing zalloc, zfree, opaque */
};
typedef void* (* z_alloc_func)( void* opaque, unsigned items, unsigned size );
typedef void (* z_free_func) ( void* opaque, void* address );
typedef struct z_stream_s {
char* next_in;
unsigned avail_in;
unsigned long total_in;
char* next_out;
unsigned avail_out;
unsigned long total_out;
char* msg;
void* state;
z_alloc_func zalloc;
z_free_func zfree;
void* opaque;
int data_type;
unsigned long adler;
unsigned long reserved;
} z_stream;
const char* zlibVersion();
const char* zError(int);
int inflate(z_stream*, int flush);
int inflateEnd(z_stream*);
int inflateInit2_(z_stream*, int windowBits, const char* version, int stream_size);
int deflate(z_stream*, int flush);
int deflateEnd(z_stream* );
int deflateInit2_(z_stream*, int level, int method, int windowBits, int memLevel,int strategy, const char *version, int stream_size);
unsigned long adler32(unsigned long adler, const char *buf, unsigned len);
unsigned long crc32(unsigned long crc, const char *buf, unsigned len);
unsigned long adler32_combine(unsigned long, unsigned long, long);
unsigned long crc32_combine(unsigned long, unsigned long, long);
]])
local zlib = ffi.load(ffi.os == "Windows" and "zlib1" or "z")
_M.zlib = zlib
-- Default to 16k output buffer
local DEFAULT_CHUNK = 16384
local Z_OK = zlib.Z_OK
local Z_NO_FLUSH = zlib.Z_NO_FLUSH
local Z_STREAM_END = zlib.Z_STREAM_END
local Z_FINISH = zlib.Z_FINISH
local Z_NEED_DICT = zlib.Z_NEED_DICT
local Z_BUF_ERROR = zlib.Z_BUF_ERROR
local Z_STREAM_ERROR = zlib.Z_STREAM_ERROR
local function zlib_err(err)
return ffi_str(zlib.zError(err))
end
_M.zlib_err = zlib_err
local function createStream(bufsize)
-- Setup Stream
local stream = ffi_new("z_stream")
-- Create input buffer var
local inbuf = ffi_new('char[?]', bufsize+1)
stream.next_in, stream.avail_in = inbuf, 0
-- create the output buffer
local outbuf = ffi_new('char[?]', bufsize)
stream.next_out, stream.avail_out = outbuf, 0
return stream, inbuf, outbuf
end
_M.createStream = createStream
local function initInflate(stream, windowBits)
-- Setup inflate process
local windowBits = windowBits or (15 + 32) -- +32 sets automatic header detection
local version = ffi_str(zlib.zlibVersion())
return zlib.inflateInit2_(stream, windowBits, version, ffi_sizeof(stream))
end
_M.initInflate = initInflate
local function initDeflate(stream, options)
-- Setup deflate process
local method = zlib.Z_DEFLATED
local level = options.level or zlib.Z_DEFAULT_COMPRESSION
local memLevel = options.memLevel or 8
local strategy = options.strategy or zlib.Z_DEFAULT_STRATEGY
local windowBits = options.windowBits or (15 + 16) -- +16 sets gzip wrapper not zlib
local version = ffi_str(zlib.zlibVersion())
return zlib.deflateInit2_(stream, level, method, windowBits, memLevel, strategy, version, ffi_sizeof(stream))
end
_M.initDeflate = initDeflate
local function flushOutput(stream, bufsize, output, outbuf)
-- Calculate available output bytes
local out_sz = bufsize - stream.avail_out
if out_sz == 0 then
return
end
-- Read bytes from output buffer and pass to output function
local ok, err = output(ffi_str(outbuf, out_sz))
if not ok then
return err
end
end
local function inflate(input, output, bufsize, stream, inbuf, outbuf)
local zlib_flate = zlib.inflate
local zlib_flateEnd = zlib.inflateEnd
-- Inflate a stream
local err = 0
repeat
-- Read some input
local data = input(bufsize)
if data ~= nil then
ffi_copy(inbuf, data)
stream.next_in, stream.avail_in = inbuf, #data
else
-- no more input data
stream.avail_in = 0
end
if stream.avail_in == 0 then
-- When decompressing we *must* have input bytes
zlib_flateEnd(stream)
return false, "INFLATE: Data error, no input bytes"
end
-- While the output buffer is being filled completely just keep going
repeat
stream.next_out = outbuf
stream.avail_out = bufsize
-- Process the stream, always Z_NO_FLUSH in inflate mode
err = zlib_flate(stream, Z_NO_FLUSH)
-- Buffer errors are OK here
if err == Z_BUF_ERROR then
err = Z_OK
end
if err < Z_OK or err == Z_NEED_DICT then
-- Error, clean up and return
zlib_flateEnd(stream)
return false, "INFLATE: "..zlib_err(err), stream
end
-- Write the data out
local err = flushOutput(stream, bufsize, output, outbuf)
if err then
zlib_flateEnd(stream)
return false, "INFLATE: "..err
end
until stream.avail_out ~= 0
until err == Z_STREAM_END
-- Stream finished, clean up and return
zlib_flateEnd(stream)
return true, zlib_err(err)
end
_M.inflate = inflate
local function deflate(input, output, bufsize, stream, inbuf, outbuf)
local zlib_flate = zlib.deflate
local zlib_flateEnd = zlib.deflateEnd
-- Deflate a stream
local err = 0
local mode = Z_NO_FLUSH
repeat
-- Read some input
local data = input(bufsize)
if data ~= nil then
ffi_copy(inbuf, data)
stream.next_in, stream.avail_in = inbuf, #data
else
-- EOF, try and finish up
mode = Z_FINISH
stream.avail_in = 0
end
-- While the output buffer is being filled completely just keep going
repeat
stream.next_out = outbuf
stream.avail_out = bufsize
-- Process the stream
err = zlib_flate(stream, mode)
-- Only possible *bad* return value here
if err == Z_STREAM_ERROR then
-- Error, clean up and return
zlib_flateEnd(stream)
return false, "DEFLATE: "..zlib_err(err), stream
end
-- Write the data out
local err = flushOutput(stream, bufsize, output, outbuf)
if err then
zlib_flateEnd(stream)
return false, "DEFLATE: "..err
end
until stream.avail_out ~= 0
-- In deflate mode all input must be used by this point
if stream.avail_in ~= 0 then
zlib_flateEnd(stream)
return false, "DEFLATE: Input not used"
end
until err == Z_STREAM_END
-- Stream finished, clean up and return
zlib_flateEnd(stream)
return true, zlib_err(err)
end
_M.deflate = deflate
local function adler(str, chksum)
local chksum = chksum or 0
local str = str or ""
return zlib.adler32(chksum, str, #str)
end
_M.adler = adler
local function crc(str, chksum)
local chksum = chksum or 0
local str = str or ""
return zlib.crc32(chksum, str, #str)
end
_M.crc = crc
function _M.inflateGzip(input, output, bufsize, windowBits)
local bufsize = bufsize or DEFAULT_CHUNK
-- Takes 2 functions that provide input data from a gzip stream and receives output data
-- Returns uncompressed string
local stream, inbuf, outbuf = createStream(bufsize)
local init = initInflate(stream, windowBits)
if init == Z_OK then
return inflate(input, output, bufsize, stream, inbuf, outbuf)
else
-- Init error
zlib.inflateEnd(stream)
return false, "INIT: "..zlib_err(init)
end
end
function _M.deflateGzip(input, output, bufsize, options)
local bufsize = bufsize or DEFAULT_CHUNK
options = options or {}
-- Takes 2 functions that provide plain input data and receives output data
-- Returns gzip compressed string
local stream, inbuf, outbuf = createStream(bufsize)
local init = initDeflate(stream, options)
if init == Z_OK then
return deflate(input, output, bufsize, stream, inbuf, outbuf)
else
-- Init error
zlib.deflateEnd(stream)
return false, "INIT: "..zlib_err(init)
end
end
function _M.version()
return ffi_str(zlib.zlibVersion())
end
return _M

20
src/deps/src/lua-ffi-zlib/lua-ffi-zlib-0.4-0.rockspec Normal file
View File

@ -0,0 +1,20 @@
package = "lua-ffi-zlib"
version = "0.4-0"
source = {
url = "git://github.com/hamishforbes/lua-ffi-zlib",
tag = "v0.4"
}
description = {
summary = "A Lua module using LuaJIT's FFI feature to access zlib.",
homepage = "https://github.com/hamishforbes/lua-ffi-zlib",
maintainer = "Hamish Forbes"
}
dependencies = {
"lua >= 5.1",
}
build = {
type = "builtin",
modules = {
["ffi-zlib"] = "lib/ffi-zlib.lua",
}
}

20
src/deps/src/lua-ffi-zlib/lua-ffi-zlib-0.5-0.rockspec Normal file
View File

@ -0,0 +1,20 @@
package = "lua-ffi-zlib"
version = "0.5-0"
source = {
url = "git://github.com/hamishforbes/lua-ffi-zlib",
tag = "v0.5"
}
description = {
summary = "A Lua module using LuaJIT's FFI feature to access zlib.",
homepage = "https://github.com/hamishforbes/lua-ffi-zlib",
maintainer = "Hamish Forbes"
}
dependencies = {
"lua >= 5.1",
}
build = {
type = "builtin",
modules = {
["ffi-zlib"] = "lib/ffi-zlib.lua",
}
}

145
src/deps/src/lua-ffi-zlib/test.lua Normal file
View File

@ -0,0 +1,145 @@
local table_insert = table.insert
local table_concat = table.concat
local zlib = require('lib.ffi-zlib')
local chunk = tonumber(arg[2]) or 16384
local uncompressed = ''
local input
local f
local passing = true
local in_adler
local out_adler
local in_crc
local out_crc
if arg[1] == nil then
print("No file provided")
return
else
f = io.open(arg[1], "rb")
input = function(bufsize)
local d = f:read(bufsize)
if d == nil then
return nil
end
in_crc = zlib.crc(d, in_crc)
in_adler = zlib.adler(d, in_adler)
uncompressed = uncompressed..d
return d
end
end
print('zlib version: '..zlib.version())
print()
local output_table = {}
local output = function(data)
out_crc = zlib.crc(data, out_crc)
out_adler = zlib.adler(data, out_adler)
table_insert(output_table, data)
end
-- Compress the data
print('Compressing')
local ok, err = zlib.deflateGzip(input, output, chunk)
if not ok then
-- Err message
print(err)
end
local compressed = table_concat(output_table,'')
local orig_in_crc = in_crc
local orig_in_adler = in_adler
print('Input crc32: ', in_crc)
print('Output crc32: ', out_crc)
print('Input adler32: ', in_adler)
print('Output adler32: ', out_adler)
-- Decompress it again
print()
print('Decompressing')
-- Reset vars
in_adler = nil
out_adler = nil
in_crc = nil
out_crc = nil
output_table = {}
local count = 0
local input = function(bufsize)
local start = count > 0 and bufsize*count or 1
local finish = (bufsize*(count+1)-1)
count = count + 1
if bufsize == 1 then
start = count
finish = count
end
local data = compressed:sub(start, finish)
in_crc = zlib.crc(data, in_crc)
in_adler = zlib.adler(data, in_adler)
return data
end
local ok, err = zlib.inflateGzip(input, output, chunk)
if not ok then
-- Err message
print(err)
end
local output_data = table_concat(output_table,'')
print('Input crc32: ', in_crc)
print('Output crc32: ', out_crc)
print('Input adler32: ', in_adler)
print('Output adler32: ', out_adler)
print()
if output_data ~= uncompressed then
passing = false
print("inflateGzip / deflateGzip failed")
end
if orig_in_adler ~= out_adler then
passing = false
print("Adler checksum failed")
end
if orig_in_crc ~= out_crc then
passing = false
print("CRC checksum failed")
end
local bad_output = function(data)
return nil, "bad output"
end
if not passing then
print(":(")
else
print(":)")
end
local dump_input = function(bufsize)
return compressed
end
local ok, err = zlib.deflateGzip(dump_input, bad_output, chunk)
if not ok then
if err ~= "DEFLATE: bad output" then
print(err)
else
print("abort deflation: ok")
end
end
local ok, err = zlib.inflateGzip(dump_input, bad_output, chunk)
if not ok then
if err ~= "INFLATE: bad output" then
print(err)
else
print("abort inflation: ok")
end
end

3
src/scheduler/Dockerfile
View File

@ -63,6 +63,9 @@ RUN apk add --no-cache bash libgcc libstdc++ openssl && \
ln -s /proc/1/fd/1 /var/log/letsencrypt/letsencrypt.log && \
chmod 660 /usr/share/bunkerweb/INTEGRATION
# Fix CVEs
RUN apk add "libcrypto3>=3.0.8-r4" "libssl3>=3.0.8-r4"
VOLUME /data /etc/nginx
WORKDIR /usr/share/bunkerweb/scheduler

11
src/scheduler/main.py
View File

@ -106,7 +106,7 @@ def generate_custom_configs(
Path(dirname(tmp_path)).mkdir(parents=True, exist_ok=True)
Path(tmp_path).write_bytes(custom_config["data"])
if integration not in ("Autoconf", "Swarm", "Kubernetes", "Docker"):
if integration in ("Autoconf", "Swarm", "Kubernetes", "Docker"):
logger.info("Sending custom configs to BunkerWeb")
ret = api_caller._send_files("/data/configs", "/custom_configs")
@ -137,7 +137,7 @@ def generate_external_plugins(
st = stat(job_file)
chmod(job_file, st.st_mode | S_IEXEC)
if integration not in ("Autoconf", "Swarm", "Kubernetes", "Docker"):
if integration in ("Autoconf", "Swarm", "Kubernetes", "Docker"):
logger.info("Sending plugins to BunkerWeb")
ret = api_caller._send_files("/data/plugins", "/plugins")
@ -461,7 +461,12 @@ if __name__ == "__main__":
# reload nginx
logger.info("Reloading nginx ...")
if integration not in ("Autoconf", "Swarm", "Kubernetes", "Docker"):
if integration not in (
"Autoconf",
"Swarm",
"Kubernetes",
"Docker",
):
# Reloading the nginx server.
proc = subprocess_run(
# Reload nginx

6
src/scheduler/requirements.txt
View File

@ -254,9 +254,9 @@ urllib3==1.26.15 \
# via requests
# The following packages are considered to be unsafe in a requirements file:
setuptools==67.6.1 \
--hash=sha256:257de92a9d50a60b8e22abfcbb771571fde0dbf3ec234463212027a4eeecbe9a \
--hash=sha256:e728ca814a823bf7bf60162daf9db95b93d532948c4c0bea762ce62f60189078
setuptools==67.7.1 \
--hash=sha256:6f0839fbdb7e3cfef1fc38d7954f5c1c26bf4eebb155a55c9bf8faf997b9fb67 \
--hash=sha256:bb16732e8eb928922eabaa022f881ae2b7cdcfaf9993ef1f5e841a96d32b8e0c
# via
# acme
# certbot

3
src/ui/Dockerfile
View File

@ -49,6 +49,9 @@ RUN apk add --no-cache bash && \
chmod 750 /usr/share/bunkerweb/gen/*.py /usr/share/bunkerweb/ui/*.py /usr/share/bunkerweb/ui/src/*.py /usr/share/bunkerweb/deps/python/bin/* && \
chmod 660 /usr/share/bunkerweb/INTEGRATION
# Fix CVEs
RUN apk add "libcrypto3>=3.0.8-r4" "libssl3>=3.0.8-r4"
VOLUME /data /etc/nginx
EXPOSE 7000

24
src/ui/requirements.txt
View File

@ -1,6 +1,6 @@
#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
# pip-compile --allow-unsafe --generate-hashes --resolver=backtracking
#
@ -170,6 +170,10 @@ gunicorn==20.1.0 \
--hash=sha256:9dcc4547dbb1cb284accfb15ab5667a0e5d1881cc443e0677b4882a4067a807e \
--hash=sha256:e0a968b5ba15f8a328fdfd7ab1fcb5af4470c28aaf7e55df02a99bc13138e6e8
# via -r requirements.in
importlib-metadata==6.6.0 \
--hash=sha256:43dd286a2cd8995d5eaef7fee2066340423b818ed3fd70adf0bad5f1fac53fed \
--hash=sha256:92501cdf9cc66ebd3e612f1b4f0c0765dfa42f0fa38ffb319b6bd84dd675d705
# via flask
itsdangerous==2.1.2 \
--hash=sha256:2c2349112351b88699d8d4b6b075022c0808887cb7ad10069318a8b0bc88db44 \
--hash=sha256:5dbbc68b317e5e42f327f9021763545dc3fc3bfe22e6deb96aaf1fc38874156a
@ -243,9 +247,9 @@ six==1.16.0 \
--hash=sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926 \
--hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254
# via python-dateutil
soupsieve==2.4 \
--hash=sha256:49e5368c2cda80ee7e84da9dbe3e110b70a4575f196efb74e51b94549d921955 \
--hash=sha256:e28dba9ca6c7c00173e34e4ba57448f0688bb681b7c5e8bf4971daafc093d69a
soupsieve==2.4.1 \
--hash=sha256:1c1bfee6819544a3447586c889157365a27e10d88cde3ad3da0cf0ddf646feb8 \
--hash=sha256:89d12b2d5dfcd2c9e8c22326da9d9aa9cb3dfab0a83a024f05704076ee8d35ea
# via beautifulsoup4
werkzeug==2.2.3 \
--hash=sha256:2e1ccc9417d4da358b9de6f174e3ac094391ea1d4fbef2d667865d819dfd0afe \
@ -257,6 +261,10 @@ wtforms==3.0.1 \
--hash=sha256:6b351bbb12dd58af57ffef05bc78425d08d1914e0fd68ee14143b7ade023c5bc \
--hash=sha256:837f2f0e0ca79481b92884962b914eba4e72b7a2daaf1f939c890ed0124b834b
# via flask-wtf
zipp==3.15.0 \
--hash=sha256:112929ad649da941c23de50f356a2b5570c954b65150642bccdd66bf194d224b \
--hash=sha256:48904fc76a60e542af151aded95726c1a5c34ed43ab4134b597665c86d7ad556
# via importlib-metadata
zope-event==4.6 \
--hash=sha256:73d9e3ef750cca14816a9c322c7250b0d7c9dbc337df5d1b807ff8d3d0b9e97c \
--hash=sha256:81d98813046fc86cc4136e3698fee628a3282f9c320db18658c21749235fce80
@ -295,9 +303,9 @@ zope-interface==6.0 \
# via gevent
# The following packages are considered to be unsafe in a requirements file:
setuptools==67.6.1 \
--hash=sha256:257de92a9d50a60b8e22abfcbb771571fde0dbf3ec234463212027a4eeecbe9a \
--hash=sha256:e728ca814a823bf7bf60162daf9db95b93d532948c4c0bea762ce62f60189078
setuptools==67.7.1 \
--hash=sha256:6f0839fbdb7e3cfef1fc38d7954f5c1c26bf4eebb155a55c9bf8faf997b9fb67 \
--hash=sha256:bb16732e8eb928922eabaa022f881ae2b7cdcfaf9993ef1f5e841a96d32b8e0c
# via
# gevent
# gunicorn

4
src/ui/src/Config.py
View File

@ -28,14 +28,14 @@ class Config:
self.__logger.warning(
"Database is not initialized, retrying in 5s ...",
)
sleep(3)
sleep(5)
env = self.__db.get_config()
while not self.__db.is_first_config_saved() or not env:
self.__logger.warning(
"Database doesn't have any config saved yet, retrying in 5s ...",
)
sleep(3)
sleep(5)
env = self.__db.get_config()
self.__logger.info("Database is ready")

2
tests/ui/requirements.txt
View File

@ -1,2 +1,2 @@
selenium==4.8.3
selenium==4.9.0
requests==2.28.2