LogDoctor/README.md

608 lines
21 KiB
Markdown

<h1 align="center">LogDoctor</h1>
<p align="center">Parse Apache2 / Nginx / IIS logs and view dynamically generated statistics</p>
<div align="center">
<img src="https://img.shields.io/badge/version-2.03-fff"/>
<img src="https://img.shields.io/badge/C%2B%2B-17-blue"/>
<img src="https://img.shields.io/badge/Qt-5.15-blue"/>
<br/>
<img src="https://img.shields.io/badge/Linux-supported-brightgreen"/>
<img src="https://img.shields.io/badge/BSD-supported-brightgreen"/>
<img src="https://img.shields.io/badge/Windows-supported-brightgreen"/>
<img src="https://img.shields.io/badge/Mac%20OS%20X-supported-brightgreen"/>
<br/>
<h3>🇬🇧 🇪🇸 🇫🇷 🇮🇹 🇯🇵 🇧🇷</h3>
</div>
<br/><br/>
## Table of contents
- [Overview](#overview)
- [Installation and usage](#installation-and-usage)
- [Requirements / dependencies](#requirements--dependencies)
- [Usage without installation](#usage-without-installation)
- [Usage with installation](#usage-with-installation)
- [How to compile](#how-to-compile)
- [Updates](#updates)
- [Version check](#version-check)
- [How to update](#how-to-update)
- [Before to start](#before-to-start)
- [Logs data](#logs-data)
- [Storage](#storage)
- [Examined fields](#examined-fields)
- [Logs options](#logs-options)
- [Usage control](#usage-control)
- [Logs path](#logs-path)
- [Logs format](#logs-format)
- [Apache2](#apache2)
- [Nginx](#nginx)
- [IIS](#iis)
- [Blacklist](#blacklist)
- [Warnlist](#warnlist)
- [Statistics](#statistics)
- [Warnings](#warnings)
- [Counts](#counts)
- [Speed](#speed)
- [Time of day](#time-of-day)
- [Relational](#relational)
- [Extra features](#extra-features)
- [Log files viewer](#log-files-viewer)
- [Block note](#block-note)
- [Games](#games)
- [Final considerations](#final-considerations)
- [Backups](#backups)
- [Estimated working speed](#estimated-working-speed)
- [Languages](#languages)
- [Contributions](#contributions)
- [Translations](#translations)
<br/><br/>
## Overview
LogDoctor is a web servers' access logs parser which allows to view dynamic satistics of the collected data.<br/>
Supported web servers are **Apache2**, **Nginx** and **IIS**.
<br/>
![screenshot](https://git.disroot.org/elB4RTO/screenshots/raw/branch/main/LogDoctor/log_files.png)
![screenshot](https://git.disroot.org/elB4RTO/screenshots/raw/branch/main/LogDoctor/make_stats.png)
<br/><br/>
LogDoctor is a hard fork of [Craplog](https://git.disroot.org/elB4RTO/CRAPLOG).
<br/>
## Installation and usage
### Requirements / Dependencies
- **From binary**:
- C++ 17
- Qt5 *(Framework 5.15+, Linguist, Widgets, Charts, Sql, Network)*<br/><br/>
- **From source**:
- *all the above*
- Cmake
- g++ / gcc / clang<br/><br/>
<br/>
### Usage without installation
- Download a pre-compiled [Release](https://git.disroot.org/elB4RTO/LogDoctor/releases)
<br/>*or*<br/>
Follow the step-by-step "[How to compile](#how-to-compile)" guide
- Run the executable
<br/>
### Usage with installation
#### From source
- Download and unzip this repo
<br/>*or*<br/>
`git clone https://git.disroot.org/elB4RTO/LogDoctor`<br/><br/>
- Step inside inside "*LogDoctor-main*"
<br/>*or*<br/>
`cd LogDoctor`<br/><br/>
- Run the installation script
- Linux/BSD:
- `chmod +x ./build_install.sh`
- `./build_install.sh`
- Windows:
- run `WIN_build_install_1.bat` as normal user
- right-click on `WIN_build_install_2.bat` and select **Run as Administrator**
- Mac OS:
- `chmod +x ./MAC_build_install.sh`
- `./MAC_build_install.sh`<br/><br/>
#### From package
**Arch-based distributions**
- Pre-made package:
- Step in the [Release](https://git.disroot.org/elB4RTO/LogDoctor/releases) page
- Download `logdoctor-<VERSION>-x86_64.pkg.tar.zst`
- Run `sudo pacman -U logdoctor-<VERSION>-x86_64.pkg.tar.zst`<br/><br/>
- From the AUR:
- Using **yay**:
- `yay -S logdoctor`<br/>
- Manually:
- `git clone https://aur.archlinux.org/logdoctor.git`
- `cd logdoctor`
- `makepkg -sci`<br/><br/>
**Debian-based distributions**
- Pre-made package:
- Step in the [Release](https://git.disroot.org/elB4RTO/LogDoctor/releases) page
- Download `logdoctor_<VERSION>_amd64.deb`
- Run `sudo apt install ./logdoctor_<VERSION>_amd64.deb`<br/><br/>
#### From binary
- Download a pre-compiled [Release](https://git.disroot.org/elB4RTO/LogDoctor/releases)
- Run the installation executable, or the installation script if you prefer it<br/><br/>
<br/>
### How to compile
- Install the *dependencies* you're missing (usually the list reduces to just *Qt*, and *cmake* at least).<br/><br/>
- Download and unzip this repo
<br/>*or*<br/>
`git clone https://git.disroot.org/elB4RTO/LogDoctor`<br/><br/>
- Open a terminal inside "*LogDoctor-main/*"
<br/>*or*<br/>
`cd LogDoctor/`<br/><br/>
- Prepare a build folder:
<br/>`mkdir build && cd build`<br/><br/>
- Prepare **Cmake**'s build files:
<br/>`cmake ../logdoctor -DCMAKE_BUILD_TYPE=MinSizeRel`<br/><br/>
- Use **Cmake** to compile the entire project:
<br/>`cmake --build ./ --target all`<br/><br/>
If compilation fails, use the following command before to rebuild:
<br/>`cmake --build ./ --target clean`
<br/><br/>
#### Additional steps
- **Linux**:
- No additional steps. Once compiling is done, you can move the executable file wherever you want and execute it from there.<br/><br/>
- **Windows**:
- Create a new folder and move the executable in it:
<br/>`mkdir LogDoctor && move LogDoctor.exe .\LogDoctor`<br/><br/>
- Add the needed libraries to the executable's folder:<br/>
- You need to know the path of your Qt installation, default is *C:\Qt*<br/>
- You need to know which compiler you used, usually *MinGW*
<br/>`cd C:\<path>\<to>\Qt\<version>\<compiler>\bin`<br/><br/>
- Deploy Qt's libraries using Qt's additional tool:
<br/>`windeployqt.exe C:\<path>\<to>\<LogDoctor>`<br/><br/>
- Deploy C++ libraries by copying them:
<br/>`copy "libstdc++-6.dll" C:\<path>\<to>\<LogDoctor>`
<br/>`copy "libwinpthread-1.dll" C:\<path>\<to>\<LogDoctor>`
<br/>`copy "libgcc_s_seh-1.dll" C:\<path>\<to>\<LogDoctor>`<br/><br/>
- You can now move the executable's folder wherever you want and execute LogDoctor from there.<br/><br/>
- **Mac OS**:
- No additional steps. Once compiling is done, you can use the app bundle to execute LogDoctor.
<br/><br/>
## Updates
### Version check
A version check utility is available while running LogDoctor to check the availability of a new version.<br/>
To check for updates, open the menu `Utilities`→`Version check`.
<br/>
### How to update
At the moment of writing, the only supported method is the manual update.
#### From source
- Download and unzip this repo
<br/>*or*<br/>
`git clone https://git.disroot.org/elB4RTO/LogDoctor`<br/><br/>
- Step inside inside "*LogDoctor-main*"
<br/>*or*<br/>
`cd LogDoctor`<br/><br/>
- Run the update script
- Linux/BSD:
- `chmod +x ./build_update.sh`
- `./build_update.sh`
- Windows:
- run `WIN_build_update_1.bat` as normal user
- right-click on `WIN_build_update_2.bat` and select **Run as Administrator**
- Mac OS:
- `chmod +x ./MAC_build_update.sh`
- `./MAC_build_update.sh`<br/><br/>
#### From package
Follow the same process as for installing
#### From binary
- Download a pre-compiled [Release](https://git.disroot.org/elB4RTO/LogDoctor/releases)<br/>
- Run the update script
<br/><br/><br/>
## Before to start
When you run LogDoctor for the first time, please take a minute to set-up the things it needs.<br/>
Head to the **configurations** section and give a look at least at the [logs format](#logs-format) settings. You have to tell the doctor what he'll be dealing with!
<br/><br/>
## Logs data
Archived (**gzipped**) log files can be used as well as normal files.
<br/>
### Storage
Parsed data will be stored in an [SQLite](https://www.sqlite.org/about.html) database, which makes it easy to transport/view/edit it as you please.<br/>
If LogDoctor's funcionalities aren't enough for your needs, you can always use a *DB manager* or the SQLite *API* to make your own queries and retrieve the data you need.
<br/>
### Examined fields
Not all the available log fields (expecially for *Apache2* and *Nginx*) are taken into consideration.<br/>
The considered fields are:
- **Date** and **Time**
- Request stuff: **Protocol**, **Method**, **URI** and **Query**
- Server stuff: **Bytes received**, **Bytes sent** and **Time taken**
- Client stuff: **User-agent**, **IP address**, **Cookie** and **Referrer site**
Further informations can be found in the [wiki](https://git.disroot.org/elB4RTO/LogDoctor/wiki/Examined-fields) or while running LogDoctor.
<br/>
### Logs options
Various options can be configured about log files.
<br/>
#### Usage control
When you parse a file, it will be hashed using the **SHA256** algorithm and the hash will be stored in another database, to keep track of which files you've already used and help you not parsing them twice.<br/>
##### Note
If you don't know, *SHA256* produces an irreversible hash, which means that no information about the file can be retrieved from the hash.<br/>
You have full control on the hashes database (same as for the logs-data database): you can move, delete, view or edit it the way you want (but you must keep the original file-name).<br/>
LogDoctor will **never** grab and/or use any information about you or the usage you make of it.
<br/>
#### Logs path
A different logs path can be used for any of the three supported *Web Servers*.<br/>
It can be the default system folder or any folder you decide to use, just set it in the options.
<br/>
#### Logs format
Before to start parsing logs, you must set-up the *log format* that LogDoctor will have to use.<br/>
Head to the **configurations** section, tap `Logs`, select the **Web Server** you want to configure and tap `Format`.<br/>
Once inside the **Format** section, you can insert the *log format string* you're using. Don't forget to use the `Generete preview` button to generate a *log line sample* and **check the correctness** of the format!<br/>
For reliability reasons, LogDoctor **does not** support the usage of the **Carriage Return** inside the log format string.
<br/>
##### Apache2
The log format string must be specified. Any format is supported, if valid.<br/>
To retrieve your format string:
- open the configuration file `/etc/apache2/apache2.conf`
- *usually*, the line you're looking for is the one starting with `LogFormat` and ending with `combined`. It should be somewhere near to the end of the file.
- you must not paste the whole line, just the part holding the *format string*.<br/>
Example:<br/>
- this is the whole line:<br/>
```
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
```
- this is the *format string*:<br/>
```
%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"
```
please notice that you have to remove the enclosing quotes/apostrophes as well<br/>
More informations can be found in the [wiki](https://git.disroot.org/elB4RTO/LogDoctor/wiki/Apache2) or while setting the format.
<br/>
##### Nginx
The log format string must be specified. Any format is supported, if valid.<br/>
To retrieve your format string:<br/>
- open the configuration file `/usr/local/etc/nginx/nginx.conf`
- *usually*, the line you're looking for is the one starting with `log_format main`. It should be somwehere in the middle of the file
- one **important** thing: don't paste the indentations and new lines! The default line is usualy declared in consecutive lines, and indented. You must reduce it to a one consecutive string (by also removing the *apostrophes* in the middle of it). The best way is to do this job inside the configuration file, then save and restart Nginx to see if any error is thrown.<br/>
Example:
- this is the whole line:<br/>
```
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" "$gzip_ratio"';
```
- this is the resulting *format string*:<br/>
```
$remote_addr - $remote_user [$time_local] "$request" $status $bytes_sent "$http_referer" "$http_user_agent" "$gzip_ratio"
```
please notice that you have to remove the enclosing apostrophes/quotes as well<br/>
More informations can be found in the [wiki](https://git.disroot.org/elB4RTO/LogDoctor/wiki/Nginx) or while setting the format.
<br/>
##### IIS
Supported log formats are: **W3C**, **NCSA** and **IIS**.<br/>
The *NCSA* and *IIS* modules doesn't allow any modification from the user, so nothing more have to be specified.
The *W3C* module instead allows the user to decide which fields to log, and thus you must declare the *log format string* you're using.
To retrieve your format string (for the *W3C* module only):
- open any of the log files which have been generated by this module
- the line you're looking for is the one starting with `#Fields:`, usually at the beginning of the file.<br/>
Example:<br/>
- this is the whole line:<br/>
```
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
```
- this is the *format string*:<br/>
```
date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
```
More informations can be found in the [wiki](https://git.disroot.org/elB4RTO/LogDoctor/wiki/IIS) or while setting the format.
<br/><br/>
#### Blacklist
You can add elements to the **blacklist** to avoid storing the lines containing those elements.
Each web server has its own list.
<br/>
#### Warnlist
As for the *blacklist*, you can add elements to the **warnlist**.<br/>
*Warnlists* will mark with a **warning** the lines triggering them. Warnings can be viewed and modified in the relative [statistics](#warnings) section.
Each web server has its own lists.
<br/>
## Statistics
Most of the *statistics sections* allows you to set filters to the log fields, to skim data by only including lines matching those parameters.<br/>
<br/>
### Warnings
In the *warning* section you can view the lines which triggered a warning, as well as remove any of the warnings and/or add your own.<br/>
![screenshot](https://git.disroot.org/elB4RTO/screenshots/raw/branch/main/LogDoctor/stats_warnings.png)
<br/>
### Speed
In the *speed* section you can view how fast has been your server at serving contents (if you logged the *time taken*, of course).<br/>
![screenshot](https://git.disroot.org/elB4RTO/screenshots/raw/branch/main/LogDoctor/stats_speed.png)
<br/>
### Counts
The *count* section is very simple. It just shows the recurrence of the elements for a specific field.<br/>
![screenshot](https://git.disroot.org/elB4RTO/screenshots/raw/branch/main/LogDoctor/stats_count.png)
<br/>
### Time of day
In the *time of day* section you can see the traffic, in terms of number of requests logged.<br/>
When viewing a period of time, the mean value (of all the logged days in that period) is shown.<br/>
![screenshot](https://git.disroot.org/elB4RTO/screenshots/raw/branch/main/LogDoctor/stats_daytime.png)
<br/>
### Relational
In the *relational* section you can view how many times a specific field brought to another.<br/>
This section is more suited for long periods of time.<br/>
![screenshot](https://git.disroot.org/elB4RTO/screenshots/raw/branch/main/LogDoctor/stats_relational.png)
<br/>
### Globals
In the *globals* section you can have an overview of your logs history.<br/>
![screenshot](https://git.disroot.org/elB4RTO/screenshots/raw/branch/main/LogDoctor/stats_globals.png)
<br/><br/>
## Extra features
### Log files viewer
Use the built-in logs viewer to inspect the content of your log files.<br/>
Color schemes will be applied using the currently set log format.
<br/>
### Block-note
A block-note utility is available at `Tools`→`BlockNote` which can be used to temporary write text, notes, etc.
<br/>
### Games
Simple mini-games to pass the time<br/><br/>
#### CrissCross
<img height="300px" src="https://git.disroot.org/elB4RTO/screenshots/raw/branch/main/LogDoctor/game_crisscross.png" /><br/><br/>
#### Snake
<img height="350px" src="https://git.disroot.org/elB4RTO/screenshots/raw/branch/main/LogDoctor/game_snake.png" /><br/><br/>
<br/><br/>
## Final considerations
### Backups
LogDoctor can automatically do a backup of your **logs database** file, so you can recover your data in case something goes wrong.<br/>
Move inside LogDoctor's folder (if you don't know/remember the path, open the `Utilities`→`Infos`>`Paths` menu to view it) and open the folder named "**backups**'.<br/>
Here you will find the backups with an increasing index, where '.1' represents the newest.
A new backup is made every time you quit LogDoctor after doing a job which affected the database in any way.
#### Note
Only the *logs-data database* will be backed-up, the *hashes database* **won't**.<br/>
This is because it is unlikely (supposedly impossible) that a hash equals another, therefore they're supposed to be useful for a short period of time (that is, since you or your web server delete the original log files).
<br/>
### Estimated working speed
1~15 MB/s
May be higher or lower depending on the complexity of the logs, the complexity of the blacklist/warnlists, your hardware and the workload of your system during the execution.
<br/><br/>
## Languages
LogDoctor is available in:
- 🇬🇧 **English** (100%)
- 🇮🇹 **Italian** (90%, *wanna [contribute](#translations)?*)
- 🇪🇸 **Spanish** (90%, *wanna [contribute](#translations)?*)
- 🇫🇷 **French** (90%, *wanna [contribute](#translations)?*)
- 🇧🇷 **Portuguese** [**Brazil**] (90%, *wanna [contribute](#translations)?*)
- 🇯🇵 **Japanese** (90%, *wanna [contribute](#translations)?*)
<br/><br/>
## Contributions
LogDoctor is under development.
If you have suggestions about how to improve it, please open an ![issue](https://git.disroot.org/elB4RTO/LogDoctor/issues).
If you want to contribute to the code, please read the [Contribution Guidelines](https://git.disroot.org/elB4RTO/LogDoctor/src/branch/main/CONTRIBUTING.md).
If you want to contribute to the translation, please read the [Translation Guidelines](#translation-guidelines).
<br/>
### Translations
Current translations under developement:
- `it_IT` : **90%**
- `es_ES` : **90%** *(auditor needed)*
- `fr_FR` : **90%** *(auditor needed)*
- `pt_BR` : **90%** *(auditor needed)*
- `ja_JP` : **90%** *(auditor needed)*
If you have a request for a missing language or you're willing to contribute, please refer to [this issue](https://git.disroot.org/elB4RTO/LogDoctor/issues/10).
<br/>
#### How to contribute to translations
Since the whole application is build upon Qt, translations are made throught `.ts` [translation files](https://git.disroot.org/elB4RTO/LogDoctor/src/branch/main/logdoctor/translations).
<br/>
The easiest way to go is to use **Qt Linguist**:
- Download or clone this repo<br/>
- Open the `.ts` translation file of your language using QtLinguist<br/>
- Translate (*don't know how? Follow [this video](https://www.youtube.com/watch?v=GNyfkuDchNQ)*)<br/>
- **Push only the `.ts` files, you don't have to release them**: pull request containing `.qm` binaries won't be accepted.<br/>
<br/><br/>
If you don't want to install QtLinguist, you can do it the hard way, by opening the files with a text editor and do it manually:
- Here is a sample of some text waiting for translation<br/>
You can see the original text enclosed in the `<source>` tags. Don't edit it.<br/>
```
<message>
<location filename="..." line="n"/>
<source>This is the original text</source>
</message>
```
- Everything you need to do is to add the missing line containing the translated text, without editing the other lines.<br/>
Here is a sample after having translated it<br/>
```
<message>
<location filename="..." line="n"/>
<source>Original text</source>
<translation type="unfinished">Translated text goes here</translation>
</message>
```
<br/>
If you don't feel comfortable with any of the above solutions, please open an ![issue](https://git.disroot.org/elB4RTO/LogDoctor/issues) and write your translations/corrections there, in a clear way.
<br/>
#### Translation guidelines
*[hints from Qt](https://doc.qt.io/qt-6/linguist-overview.html)*
Just follow some simple guidelines to ensure a correct and clear contribution:
- Respect the structure of the phrase: if *(for example)* it starts or ends with a whitespace or something, please do the same.
- Please leave your translations marked as *unfinished*, they will be checked and un-marked after having been verified.
- If you see a translation marked as *finished*, please edit it only if you're sure that your definition suits better.
<br/>