librewolf
/
search-privacy
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

add searxng fork

This commit is contained in:
Secven 2021-11-06 20:50:34 +00:00
parent 8aaa56a832
commit 92f65b909e
1153 changed files with 222539 additions and 197458 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
README.md
View File

@ -2,6 +2,12 @@
![logo](./logo.png)
<h1 align="center">Welcome to search-privacy 🥳🥳🥳</h1>
<p>
<img alt="Version" src="https://img.shields.io/badge/version-1.0-blue.svg?cacheSeconds=2592000" />
<a href="#" target="_blank">
<img alt="License: GPLv3" src="https://img.shields.io/badge/License-GPLv3-yellow.svg" />
</a>
</p>
### Помочь Донатом на сервак

2
docker-compose.cloudflare.yml
View File

@ -49,7 +49,7 @@ services:
- ${WHOOGLE_DOMAIN}_REVERSE_PROXY_HOST=http://${SERVER_IP}:5050
search:
image: searx/searx
image: secven/searxng
restart: always
ports:
- 6060:8080

2
docker-compose.prod.yml
View File

@ -46,7 +46,7 @@ services:
- ${WHOOGLE_DOMAIN}_REVERSE_PROXY_HOST=http://${SERVER_IP}:5050
search:
image: searx/searx
image: secven/searxng
restart: always
ports:
- 6060:8080

31
src/searx/.codecov.yml
View File

@ -1,31 +0,0 @@
comment: false
coverage:
status:
project:
default:
# basic
target: auto
threshold: null
base: auto
# advanced
branches: null
if_no_uploads: error
if_not_found: success
if_ci_failed: error
only_pulls: false
flags: null
paths: null
patch:
default:
# basic
target: auto
threshold: null
base: auto
# advanced
branches: null
if_no_uploads: error
if_not_found: success
if_ci_failed: error
only_pulls: false
flags: null
paths: null

59
src/searx/.config.sh
View File

@ -1,59 +0,0 @@
# -*- coding: utf-8; mode: sh -*-
# SPDX-License-Identifier: AGPL-3.0-or-later
# shellcheck shell=bash disable=SC2034
#
# This environment is used by ./utils scripts like filtron.sh or searx.sh. The
# default values are *most flexible* and *best maintained*, you normally not
# need to change the defaults (except PUBLIC_URL).
#
# Before you change any value here you have to uninstall any previous
# installation. Further is it recommended to backup your changes simply by
# adding them to you local brand (git branch)::
#
# git add .config
# The public URL of the searx instance: PUBLIC_URL="https://mydomain.xy/searx"
# The default is taken from ./utils/brand.env.
PUBLIC_URL="${SEARX_URL}"
if [[ ${PUBLIC_URL} == "https://searx.me" ]]; then
# hint: Linux containers do not have DNS entries, lets use IPs
PUBLIC_URL="http://$(primary_ip)/searx"
fi
# searx.sh
# ---------
# SEARX_INTERNAL_URL="127.0.0.1:8888"
# SEARX_SETTINGS_TEMPLATE="${REPO_ROOT}/utils/templates/etc/searx/use_default_settings.yml"
# Only change, if you maintain a searx brand in your searx fork (GIT_URL) which
# is not hold by branch 'master'. The branch has to be a local branch, in the
# repository from which you install (which is most often the case). If you want
# to install branch 'foo', don't forget to run 'git branch foo origin/foo' once.
# GIT_BRANCH="${GIT_BRANCH:-master}"
# filtron.sh
# ----------
# FILTRON_API="127.0.0.1:4005"
# FILTRON_LISTEN="127.0.0.1:4004"
# FILTRON_TARGET="127.0.0.1:8888"
# morty.sh
# --------
# morty listen address
# MORTY_LISTEN="127.0.0.1:3000"
# PUBLIC_URL_PATH_MORTY="/morty/"
# system services
# ---------------
# Common $HOME folder of the service accounts
# SERVICE_HOME_BASE="/usr/local"
# **experimental**: Set SERVICE_USER to run all services by one account, but be
# aware that removing discrete components might conflict!
# SERVICE_USER=searx

133
src/searx/.dir-locals.el
View File

@ -1,133 +0,0 @@
;;; .dir-locals.el
;;
;; If you get ``*** EPC Error ***`` (even after a jedi:install-server) in your
;; emacs session, mostly you have jedi-mode enabled but the python enviroment is
;; missed. The python environment has to be next to the
;; ``<repo>/.dir-locals.el`` in::
;;
;; ./local/py3
;;
;; In Emacs, some buffer locals are referencing the project environment:
;;
;; - prj-root --> <repo>/
;; - python-environment-directory --> <repo>/local
;; - python-environment-default-root-name --> py3
;; - python-shell-virtualenv-root --> <repo>/local/py3
;; When this variable is set with the path of the virtualenv to use,
;; `process-environment' and `exec-path' get proper values in order to run
;; shells inside the specified virtualenv, example::
;; (setq python-shell-virtualenv-root "/path/to/env/")
;;
;; To setup such an environment build target 'pyenv' or 'pyenvinstall'::
;;
;; $ make pyenvinstall
;;
;; Alternatively create the virtualenv, source it and install jedi + epc
;; (required by `emacs-jedi <https://tkf.github.io/emacs-jedi>`_)::
;;
;; $ python -m venv ./local/py3
;; ...
;; $ source ./local/py3/bin/activate
;; (py3)$ # now install into the activated 'py3' environment ..
;; (py3)$ pip install jedi epc
;; ...
;;
;; Here is what also I found useful to add to my .emacs::
;;
;; (global-set-key [f6] 'flycheck-mode)
;; (add-hook 'python-mode-hook 'my:python-mode-hook)
;;
;; (defun my:python-mode-hook ()
;; (add-to-list 'company-backends 'company-jedi)
;; (require 'jedi-core)
;; (jedi:setup)
;; (define-key python-mode-map (kbd "C-c C-d") 'jedi:show-doc)
;; (define-key python-mode-map (kbd "M-.") 'jedi:goto-definition)
;; (define-key python-mode-map (kbd "M-,") 'jedi:goto-definition-pop-marker)
;; )
;;
((nil
. ((fill-column . 80)
))
(python-mode
. ((indent-tabs-mode . nil)
;; project root folder is where the `.dir-locals.el' is located
(eval . (setq-local
prj-root (locate-dominating-file default-directory ".dir-locals.el")))
(eval . (setq-local
python-environment-directory (expand-file-name "./local" prj-root)))
;; use 'py3' enviroment as default
(eval . (setq-local
python-environment-default-root-name "py3"))
(eval . (setq-local
python-shell-virtualenv-root
(concat python-environment-directory
"/"
python-environment-default-root-name)))
;; python-shell-virtualenv-path is obsolete, use python-shell-virtualenv-root!
;; (eval . (setq-local
;; python-shell-virtualenv-path python-shell-virtualenv-root))
(eval . (setq-local
python-shell-interpreter
(expand-file-name "bin/python" python-shell-virtualenv-root)))
(eval . (setq-local
python-environment-virtualenv
(list (expand-file-name "bin/virtualenv" python-shell-virtualenv-root)
;;"--system-site-packages"
"--quiet")))
(eval . (setq-local
pylint-command
(expand-file-name "bin/pylint" python-shell-virtualenv-root)))
;; pylint will find the '.pylintrc' file next to the CWD
;; https://pylint.readthedocs.io/en/latest/user_guide/run.html#command-line-options
(eval . (setq-local
flycheck-pylintrc ".pylintrc"))
;; flycheck & other python stuff should use the local py3 environment
(eval . (setq-local
flycheck-python-pylint-executable python-shell-interpreter))
;; use 'M-x jedi:show-setup-info' and 'M-x epc:controller' to inspect jedi server
;; https://tkf.github.io/emacs-jedi/latest/#jedi:environment-root -- You
;; can specify a full path instead of a name (relative path). In that case,
;; python-environment-directory is ignored and Python virtual environment
;; is created at the specified path.
(eval . (setq-local jedi:environment-root python-shell-virtualenv-root))
;; https://tkf.github.io/emacs-jedi/latest/#jedi:server-command
(eval .(setq-local
jedi:server-command
(list python-shell-interpreter
jedi:server-script)
))
;; jedi:environment-virtualenv --> see above 'python-environment-virtualenv'
;; is set buffer local! No need to setup jedi:environment-virtualenv:
;;
;; Virtualenv command to use. A list of string. If it is nil,
;; python-environment-virtualenv is used instead. You must set non-nil
;; value to jedi:environment-root in order to make this setting work.
;;
;; https://tkf.github.io/emacs-jedi/latest/#jedi:environment-virtualenv
;;
;; (eval . (setq-local
;; jedi:environment-virtualenv
;; (list (expand-file-name "bin/virtualenv" python-shell-virtualenv-root)
;; ;;"--python"
;; ;;"/usr/bin/python3.4"
;; )))
;; jedi:server-args
)))

56
src/searx/.dockerignore
View File

@ -1,56 +0,0 @@
*~
*/*~
*/*/*~
*/*/*/*~
*/*/*/*/*~
#
local/
# Git
.git
.gitignore
# CI
.codeclimate.yml
.travis.yml
.taskcluster.yml
# Byte-compiled / optimized / DLL files
__pycache__/
*/__pycache__/
*/*/__pycache__/
*/*/*/__pycache__/
*.py[cod]
*/*.py[cod]
*/*/*.py[cod]
*/*/*/*.py[cod]
# to sync with .gitignore
.coverage
coverage/
.installed.cfg
engines.cfg
env
searx-ve
robot_log.html
robot_output.xml
robot_report.html
test_basic/
setup.cfg
# node_modules
node_modules/
*/node_modules/
*/*/node_modules/
*/*/*/node_modules/
*/*/*/*/node_modules/
.tx/
#
build/
dist/
local/
gh-pages/
searx.egg-info/

10
src/searx/.gitattributes vendored
View File

@ -1,10 +0,0 @@
searx/static/css/bootstrap.min.css -diff
searx/static/js/bootstrap.min.js -diff
searx/static/themes/oscar/css/logicodev.min.css -diff
searx/static/themes/oscar/css/leaflet.min.css -diff
searx/static/themes/oscar/css/pointhi.min.css -diff
searx/static/themes/oscar/js/searx.min.js -diff
searx/static/themes/simple/css/searx.min.css -diff
searx/static/themes/simple/css/searx-rtl.min.css -diff
searx/static/themes/simple/js/searx.min.js -diff
searx/static/themes/simple/js/searx.min.js.map -diff

39
src/searx/.github/ISSUE_TEMPLATE/bug-report.md vendored
View File

@ -1,39 +0,0 @@
---
name: Bug report
about: Report a bug in Searx
title: ''
labels: bug
assignees: ''
---
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SEARX -->
**Version of Searx, commit number if you are using on master branch and stipulate if you forked Searx**
<!-- If you are running on master branch using git execute this command
in order to fetch the latest commit ID:
```
git log -1
```
If you are using searx-docker then look at the bottom of the Searx page
and check for the version after "Powered by searx"
Please also stipulate if you are using a forked version of Searx and
include a link to the fork source code.
-->
**How did you install Searx?**
<!-- Did you install Searx using the official wiki or using searx-docker
or manually by executing the searx/webapp.py file? -->
**What happened?**
<!-- A clear and concise description of what the bug is. -->
**How To Reproduce**
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->
**Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->
**Screenshots & Logs**
<!-- If applicable, add screenshots, logs to help explain your problem. -->
**Additional context**
<!-- Add any other context about the problem here. -->

14
src/searx/.github/ISSUE_TEMPLATE/config.yml vendored
View File

@ -1,14 +0,0 @@
blank_issues_enabled: true
contact_links:
- name: Installation issue using the official wiki
url: https://github.com/searx/searx/discussions/new?category_id=32001257
about: Ask for help if you are having some issue with installing Searx using the official wiki.
- name: Installation issue using searx-docker
url: https://github.com/searx/searx/discussions/new?category_id=32001259
about: Ask for help if you are having some issue with installing Searx using searx-docker.
- name: Installation issue without using the official wiki nor searx-docker
url: https://github.com/searx/searx/discussions/new?category_id=32001260
about: Ask for help if you are having some issue with installing Searx manually without using the wiki nor searx-docker.
- name: Report a bug on a public Searx instance
url: https://github.com/searx/searx/discussions/new?category_id=32001412
about: Report a bug that you discovered on a public Searx instance.

31
src/searx/.github/ISSUE_TEMPLATE/engine-request.md vendored
View File

@ -1,31 +0,0 @@
---
name: Engine request
about: Request a new engine in Searx
title: ''
labels: enhancement, engine request
assignees: ''
---
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SEARX -->
**Working URL to the engine**
<!-- Please check if the engine is responding correctly before submitting it. -->
**Why do you want to add this engine?**
<!-- What's special about this engine? Is it open source or libre? -->
**Features of this engine**
<!-- Features of this engine: Doesn't track its users, fast, easy to integrate, ... -->
**How can Searx fetch the information from this engine?**
<!-- List API URL, example code (using the correct markdown) and more
that could be useful for the developers in order to implement this engine.
If you don't know what to write, let this part blank.>
**Applicable category of this engine**
<!-- Where should this new engine fit in Searx? Current categories in Searx:
general, files, images, it, map, music, news, science, social media and videos.
You can add multiple categories at the same time. -->
**Additional context**
<!-- Add any other context about this engine here. -->

21
src/searx/.github/ISSUE_TEMPLATE/feature-request.md vendored
View File

@ -1,21 +0,0 @@
---
name: Feature request
about: Request a new feature in Searx
title: ''
labels: enhancement
assignees: ''
---
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SEARX -->
**Is your feature request related to a problem? Please describe.**
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
**Describe the solution you'd like**
<!-- A clear and concise description of what you want to happen. -->
**Describe alternatives you've considered**
<!-- A clear and concise description of any alternative solutions or features you've considered. -->
**Additional context**
<!-- Add any other context or screenshots about the feature request here. -->

10
src/searx/.github/dependabot.yml vendored
View File

@ -1,10 +0,0 @@
# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
version: 2
updates:
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
day: "friday"
open-pull-requests-limit: 5
target-branch: "master"

66
src/searx/.github/workflows/data-update.yml vendored
View File

@ -1,66 +0,0 @@
name: "Update searx.data"
on:
schedule:
- cron: "05 06 1 * *"
workflow_dispatch:
jobs:
updateData:
name: Update data - ${{ matrix.fetch }}
runs-on: ubuntu-20.04
if: ${{ github.repository_owner == 'searx'}}
strategy:
matrix:
fetch:
- update_ahmia_blacklist.py
- update_currencies.py
- update_external_bangs.py
- update_firefox_version.py
- update_languages.py
- update_wikidata_units.py
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Install Ubuntu packages
run: |
sudo ./utils/searx.sh install packages
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- name: Install Python dependencies
run: |
make V=1 install
- name: Fetch data
env:
FETCH_SCRIPT: ./searx_extra/update/${{ matrix.fetch }}
run: |
V=1 ./manage pyenv.cmd python "$FETCH_SCRIPT"
- name: Create Pull Request
id: cpr
uses: peter-evans/create-pull-request@v3
with:
token: ${{ secrets.DATA_PR_TOKEN }}
commit-message: Update searx.data - ${{ matrix.fetch }}
committer: searx-bot <noreply@github.com>
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
signoff: false
branch: update_data_${{ matrix.fetch }}
delete-branch: true
draft: false
title: 'Update searx.data - ${{ matrix.fetch }}'
body: |
Update searx.data - ${{ matrix.fetch }}
labels: |
data
- name: Check outputs
run: |
echo "Pull Request Number - ${{ steps.cpr.outputs.pull-request-number }}"
echo "Pull Request URL - ${{ steps.cpr.outputs.pull-request-url }}"

122
src/searx/.github/workflows/integration.yml vendored
View File

@ -1,122 +0,0 @@
name: Integration
on:
push:
branches: ["master"]
pull_request:
branches: ["master"]
jobs:
python:
name: Python ${{ matrix.python-version }}
runs-on: ubuntu-20.04
strategy:
matrix:
os: [ubuntu-20.04]
python-version: [3.6, 3.7, 3.8, 3.9]
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Install Ubuntu packages
run: |
sudo ./utils/searx.sh install packages
sudo apt install firefox
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
architecture: 'x64'
- name: Cache Python dependencies
id: cache-python
uses: actions/cache@v2
with:
path: ./local
key: python-${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('requirements*.txt', 'setup.py') }}
- name: Install Python dependencies
if: steps.cache-python.outputs.cache-hit != 'true'
run: |
make V=1 install
make V=1 gecko.driver
- name: Run tests
run: make V=1 ci.test
- name: Test coverage
run: make V=1 test.coverage
- name: Store coverage result
uses: actions/upload-artifact@v2
with:
name: coverage-${{ matrix.python-version }}
path: coverage/
retention-days: 60
themes:
name: Themes
runs-on: ubuntu-20.04
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Install Ubuntu packages
run: sudo ./utils/searx.sh install packages
- name: Install node dependencies
run: make V=1 node.env
- name: Build themes
run: make V=1 themes.all
documentation:
name: Documentation
runs-on: ubuntu-20.04
steps:
- name: Checkout
uses: actions/checkout@v2
with:
persist-credentials: false
- name: Install Ubuntu packages
run: sudo ./utils/searx.sh install buildhost
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- name: Build documentation
run: |
make V=1 docs.clean docs.html
- name: Deploy
if: github.ref == 'refs/heads/master'
uses: JamesIves/github-pages-deploy-action@3.7.1
with:
GITHUB_TOKEN: ${{ github.token }}
BRANCH: gh-pages
FOLDER: dist/docs
CLEAN: true # Automatically remove deleted files from the deploy branch
dockers:
name: Docker
if: github.ref == 'refs/heads/master'
needs:
- python
- themes
- documentation
env:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
runs-on: ubuntu-18.04
steps:
- name: Checkout
if: env.DOCKERHUB_USERNAME != null
uses: actions/checkout@v2
with:
# make sure "make docker.push" can get the git history
fetch-depth: '0'
- name: Set up QEMU
if: env.DOCKERHUB_USERNAME != null
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
if: env.DOCKERHUB_USERNAME != null
uses: docker/setup-buildx-action@v1
- name: Login to DockerHub
if: env.DOCKERHUB_USERNAME != null
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and push
if: env.DOCKERHUB_USERNAME != null
run: make -e GIT_URL=$(git remote get-url origin) docker.push

28
src/searx/.gitignore vendored
View File

@ -1,28 +0,0 @@
# to sync with .dockerignore
.coverage
coverage/
cache/
.installed.cfg
engines.cfg
env
searx-ve
robot_log.html
robot_output.xml
robot_report.html
test_basic/
setup.cfg
*.pyc
*/*.pyc
*~
*.swp
/node_modules
.tx/
build/
dist/
local/
gh-pages/
searx.egg-info/

3
src/searx/.landscape.yaml
View File

@ -1,3 +0,0 @@
strictness: high
ignore-paths:
- bootstrap.py

444
src/searx/.pylintrc
View File

@ -1,444 +0,0 @@
# -*- coding: utf-8; mode: conf -*-
# lint Python modules using external checkers.
#
# This is the main checker controlling the other ones and the reports
# generation. It is itself both a raw checker and an astng checker in order
# to:
# * handle message activation / deactivation at the module level
# * handle some basic but necessary stats'data (number of classes, methods...)
#
[MASTER]
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code
extension-pkg-whitelist=lxml.etree
# Add files or directories to the blacklist. They should be base names, not
# paths.
ignore=CVS, .git, .svn
# Add files or directories matching the regex patterns to the blacklist. The
# regex matches against base names, not paths.
ignore-patterns=
# Python code to execute, usually for sys.path manipulation such as
# pygtk.require().
#init-hook=
# Use multiple processes to speed up Pylint.
jobs=1
# List of plugins (as comma separated values of python modules names) to load,
# usually to register additional checkers.
load-plugins=
# Pickle collected data for later comparisons.
persistent=yes
# Specify a configuration file.
#rcfile=
# Allow loading of arbitrary C extensions. Extensions are imported into the
# active Python interpreter and may run arbitrary code.
unsafe-load-any-extension=no
[MESSAGES CONTROL]
# Only show warnings with the listed confidence levels. Leave empty to show
# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED
confidence=
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifiers separated by comma (,) or put this
# option multiple times (only on the command line, not in the configuration
# file where it should appear only once).You can also use "--disable=all" to
# disable everything first and then reenable specific checks. For example, if
# you want to run only the similarities checker, you can use "--disable=all
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use"--disable=all --enable=classes
# --disable=W"
disable=bad-whitespace, duplicate-code
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once). See also the "--disable" option for examples.
enable=
[REPORTS]
# Python expression which should return a note less than 10 (10 is the highest
# note). You have access to the variables errors warning, statement which
# respectively contain the number of errors / warnings messages and the total
# number of statements analyzed. This is used by the global evaluation report
# (RP0004).
evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)
# Template used to display messages. This is a python new-style format string
# used to format the message information. See doc for all details
# HINT: do not set this here, use argument --msg-template=...
#msg-template={path}:{line}: [{msg_id}({symbol}),{obj}] {msg}
# Set the output format. Available formats are text, parseable, colorized, json
# and msvs (visual studio).You can also give a reporter class, eg
# mypackage.mymodule.MyReporterClass.
# HINT: do not set this here, use argument --output-format=...
#output-format=text
# Tells whether to display a full report or only the messages
reports=no
# Activate the evaluation score.
score=yes
[REFACTORING]
# Maximum number of nested blocks for function / method body
max-nested-blocks=5
[BASIC]
# List of builtins function names that should not be used, separated by a comma
bad-functions=map,filter,apply,input
# Naming hint for argument names
argument-name-hint=(([a-z][a-z0-9_]{2,30})|(_[a-z0-9_]*))$
# Regular expression matching correct argument names
argument-rgx=(([a-z][a-zA-Z0-9_]{2,30})|(_[a-z0-9_]*))$
# Naming hint for attribute names
attr-name-hint=(([a-z][a-z0-9_]{2,30})|(_[a-z0-9_]*))$
# Regular expression matching correct attribute names
attr-rgx=(([a-z][a-z0-9_]{2,30})|(_[a-z0-9_]*)|([A-Z0-9_]*))$
# Bad variable names which should always be refused, separated by a comma
bad-names=foo,bar,baz,toto,tutu,tata
# Naming hint for class attribute names
class-attribute-name-hint=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$
# Regular expression matching correct class attribute names
class-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$
# Naming hint for class names
class-name-hint=[A-Z_][a-zA-Z0-9]+$
# Regular expression matching correct class names
class-rgx=[A-Z_][a-zA-Z0-9]+$
# Naming hint for constant names
const-name-hint=(([A-Z_][A-Z0-9_]*)|(__.*__))$
# Regular expression matching correct constant names
const-rgx=(([a-zA-Z_][a-zA-Z0-9_]*)|(__.*__))$
#const-rgx=[f]?[A-Z_][a-zA-Z0-9_]{2,30}$
# Minimum line length for functions/classes that require docstrings, shorter
# ones are exempt.
docstring-min-length=-1
# Naming hint for function names
function-name-hint=(([a-z][a-z0-9_]{2,30})|(_[a-z0-9_]*))$
# Regular expression matching correct function names
function-rgx=(([a-z][a-zA-Z0-9_]{2,30})|(_[a-z0-9_]*))$
# Good variable names which should always be accepted, separated by a comma
good-names=i,j,k,ex,Run,_,log,cfg,id
# Include a hint for the correct naming format with invalid-name
include-naming-hint=no
# Naming hint for inline iteration names
inlinevar-name-hint=[A-Za-z_][A-Za-z0-9_]*$
# Regular expression matching correct inline iteration names
inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$
# Naming hint for method names
method-name-hint=(([a-z][a-z0-9_]{2,30})|(_[a-z0-9_]*))$
# Regular expression matching correct method names
method-rgx=(([a-z][a-zA-Z0-9_]{2,30})|(_[a-z0-9_]*))$
# Naming hint for module names
module-name-hint=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$
# Regular expression matching correct module names
#module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$
module-rgx=([a-z_][a-z0-9_]*)$
# Colon-delimited sets of names that determine each other's naming style when
# the name regexes allow several styles.
name-group=
# Regular expression which should only match function or class names that do
# not require a docstring.
no-docstring-rgx=^_
# List of decorators that produce properties, such as abc.abstractproperty. Add
# to this list to register other decorators that produce valid properties.
property-classes=abc.abstractproperty
# Naming hint for variable names
variable-name-hint=(([a-z][a-z0-9_]{2,30})|(_[a-z0-9_]*))$
# Regular expression matching correct variable names
variable-rgx=(([a-z][a-zA-Z0-9_]{2,30})|(_[a-z0-9_]*)|([a-z]))$
[FORMAT]
# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.
expected-line-ending-format=
# Regexp for a line that is allowed to be longer than the limit.
ignore-long-lines=^\s*(# )?<?https?://\S+>?$
# Number of spaces of indent required inside a hanging or continued line.
indent-after-paren=4
# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
# tab).
indent-string=' '
# Maximum number of characters on a single line.
max-line-length=120
# Maximum number of lines in a module
max-module-lines=2000
# List of optional constructs for which whitespace checking is disabled. `dict-
# separator` is used to allow tabulation in dicts, etc.: {1 : 1,\n222: 2}.
# `trailing-comma` allows a space between comma and closing bracket: (a, ).
# `empty-line` allows space-only lines.
no-space-check=trailing-comma,dict-separator
# Allow the body of a class to be on the same line as the declaration if body
# contains single statement.No config file found, using default configuration
single-line-class-stmt=no
# Allow the body of an if to be on the same line as the test if there is no
# else.
single-line-if-stmt=no
[LOGGING]
# Logging modules to check that the string format arguments are in logging
# function parameter format
logging-modules=logging
[MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
notes=FIXME,XXX,TODO
[SIMILARITIES]
# Ignore comments when computing similarities.
ignore-comments=yes
# Ignore docstrings when computing similarities.
ignore-docstrings=yes
# Ignore imports when computing similarities.
ignore-imports=no
# Minimum lines number of a similarity.
min-similarity-lines=4
[SPELLING]
# Spelling dictionary name. Available dictionaries: none. To make it working
# install python-enchant package.
spelling-dict=
# List of comma separated words that should not be checked.
spelling-ignore-words=
# A path to a file that contains private dictionary; one word per line.
spelling-private-dict-file=
# Tells whether to store unknown words to indicated private dictionary in
# --spelling-private-dict-file option instead of raising a message.
spelling-store-unknown-words=no
[TYPECHECK]
# List of decorators that produce context managers, such as
# contextlib.contextmanager. Add to this list to register other decorators that
# produce valid context managers.
contextmanager-decorators=contextlib.contextmanager
# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=
# Tells whether missing members accessed in mixin class should be ignored. A
# mixin class is detected if its name ends with "mixin" (case insensitive).
ignore-mixin-members=yes
# This flag controls whether pylint should warn about no-member and similar
# checks whenever an opaque object is returned when inferring. The inference
# can return multiple potential results while evaluating a Python object, but
# some branches might not be evaluated, which results in partial inference. In
# that case, it might be useful to still emit no-member and other checks for
# the rest of the inferred objects.
ignore-on-opaque-inference=yes
# List of class names for which member attributes should not be checked (useful
# for classes with dynamically set attributes). This supports the use of
# qualified names.
ignored-classes=optparse.Values,thread._local,_thread._local
# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis. It
# supports qualified module names, as well as Unix pattern matching.
ignored-modules=
# Show a hint with possible names when a member name was not found. The aspect
# of finding the hint is based on edit distance.
missing-member-hint=yes
# The minimum edit distance a name should have in order to be considered a
# similar match for a missing member name.
missing-member-hint-distance=1
# The total number of similar names that should be taken in consideration when
# showing a hint for a missing member.
missing-member-max-choices=1
[VARIABLES]
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid to define new builtins when possible.
additional-builtins=
# Tells whether unused global variables should be treated as a violation.
allow-global-unused-variables=yes
# List of strings which can identify a callback function by name. A callback
# name must start or end with one of those strings.
callbacks=cb_,_cb
# A regular expression matching the name of dummy variables (i.e. expectedly
# not used).
dummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_
# Argument names that match this expression will be ignored. Default to name
# with leading underscore
ignored-argument-names=_.*|^ignored_|^unused_
# Tells whether we should check for unused import in __init__ files.
init-import=no
# List of qualified module names which can have objects that can redefine
# builtins.
redefining-builtins-modules=six.moves,future.builtins
[CLASSES]
# List of method names used to declare (i.e. assign) instance attributes.
defining-attr-methods=__init__,__new__,setUp
# List of member names, which should be excluded from the protected access
# warning.
exclude-protected=_asdict,_fields,_replace,_source,_make
# List of valid names for the first argument in a class method.
valid-classmethod-first-arg=cls
# List of valid names for the first argument in a metaclass class method.
valid-metaclass-classmethod-first-arg=mcs
[DESIGN]
# Maximum number of arguments for function / method
max-args=8
# Maximum number of attributes for a class (see R0902).
max-attributes=20
# Maximum number of boolean expressions in a if statement
max-bool-expr=5
# Maximum number of branch for function / method body
max-branches=12
# Maximum number of locals for function / method body
max-locals=20
# Maximum number of parents for a class (see R0901).
max-parents=7
# Maximum number of public methods for a class (see R0904).
max-public-methods=20
# Maximum number of return / yield for function / method body
max-returns=6
# Maximum number of statements in function / method body
max-statements=50
# Minimum number of public methods for a class (see R0903).
min-public-methods=2
[IMPORTS]
# Allow wildcard imports from modules that define __all__.
allow-wildcard-with-all=no
# Analyse import fallback blocks. This can be used to support both Python 2 and
# 3 compatible code, which means that the block might have code that exists
# only in one or another interpreter, leading to false positives when analysed.
analyse-fallback-blocks=no
# Deprecated modules which should not be used, separated by a comma
deprecated-modules=optparse,tkinter.tix
# Create a graph of external dependencies in the given file (report RP0402 must
# not be disabled)
ext-import-graph=
# Create a graph of every (i.e. internal and external) dependencies in the
# given file (report RP0402 must not be disabled)
import-graph=
# Create a graph of internal dependencies in the given file (report RP0402 must
# not be disabled)
int-import-graph=
# Force import order to recognize a module as part of the standard
# compatibility libraries.
known-standard-library=
# Force import order to recognize a module as part of a third party library.
known-third-party=enchant
[EXCEPTIONS]
# Exceptions that will emit a warning when being caught. Defaults to
# "Exception"
overgeneral-exceptions=Exception

168
src/searx/AUTHORS.rst
View File

@ -1,168 +0,0 @@
Searx was created by Adam Tauber and is maintained by Adam Tauber, Noémi Ványi, @pofilo, Gaspard d'Hautefeuille and Émilien Devos.
Major contributing authors:
- Adam Tauber <asciimoo@gmail.com> `@asciimoo <https://github.com/asciimoo>`_
- Matej Cotman
- Thomas Pointhuber
- Alexandre Flament `@dalf <https://github.com/dalf>`_
- @Cqoicebordel
- Noémi Ványi
- Marc Abonce Seguin @a01200356
- @pofilo
- Markus Heiser @return42
- Émilien Devos @unixfox
- Alexandre Flament
People who have submitted patches/translations, reported bugs, consulted features or
generally made searx better:
- Laszlo Hammerl
- Stefan Marsiske
- Gabor Nagy
- @pw3t
- @rhapsodhy
- András Veres-Szentkirályi
- Benjamin Sonntag
- @HLFH
- @TheRadialActive
- @Okhin
- André Koot
- Alejandro León Aznar
- rike
- dp
- Martin Zimmermann
- @courgette
- @kernc
- @Reventl0v
- Caner Başaran
- Benjamin Sonntag
- @opi
- @dimqua
- Giorgos Logiotatidis
- Luc Didry
- Niklas Haas
- @underr
- Emmanuel Benazera
- @GreenLunar
- Kang-min Liu
- Kirill Isakov
- Guilhem Bonnefille
- @jibe-b
- Christian Pietsch @pietsch
- @Maxqia
- Ashutosh Das @pyprism
- YuLun Shih @imZack
- Dmitry Mikhirev @mikhirev
- David A Roberts `@davidar <https://github.com/davidar>`_
- Jan Verbeek @blyxxyz
- Ammar Najjar @ammarnajjar
- @stepshal
- François Revol @mmuman
- Harry Wood @harry-wood
- Thomas Renard @threnard
- Pydo `<https://github.com/pydo>`_
- Athemis `<https://github.com/Athemis>`_
- Stefan Antoni `<http://stefan.antoni.io>`
- @firebovine
- Lorenzo J. Lucchini @luccoj
- @eig8phei
- @maxigas
- Jannik Winkel @kiney
- @juanitobananas
- Vache Asatryan @vachi
- Luca CPZ @lcpz
- @nikaiw
- Thirnearez
- Hypolite Petovan @MrPetovan
- @woorst
- @Apply55gx
- @pyrrh0n1c
- @cclauss
- QGW @moon2l
- Pierre-Alain Toret @daftaupe
- Matthew Olmsted @icegiant
- Michael Tran @trankmichael
- Joseph Nuthalapati @josephkiranbabu
- @maiki
- Richard Didier @zeph33
- Michael Vieria @Themimitoof
- Richard Nespithal @rndevfx
- Stanislas @angristan
- @rinpatch
- g. s. @usernameisntallowed
- Léo Bourrel @bourrel
- @cy8aer
- @Popolon
- Alice Ferrazzi @aliceinwire
- @LiquidLemon
- @dadosch
- Václav Zouzalík @Venca24
- @ZEROF
- Ivan Skytte Jørgensen @isj-privacore
- @miicha
- Étienne Deparis @milouse
- @pelag0s
- Denis Wernert @d-tux
- Robin Hallabro-Kokko @hallabro
- Jonas Zohren @jfowl
- Elias Ojala @theel0ja
- @brunob
- Nick Espig @nachtalb
- Rachmadani Haryono @rachmadaniHaryono
- Frank de Lange @yetangitu
- Nicolas Gelot @nfk
- @volth
- Mathieu Brunot @madmath03
- @lorddavidiii
- @x250
- Robby O'Connor @robbyoconnor
- Finn @0xhtml
- @tmikaeld
- @hobbestigrou
- Vipul @finn0
- @CaffeinatedTech
- Robin Schneider @ypid
- @splintah
- Lukas van den Berk @lukasvdberk
- @piplongrun
- Jason Kaltsikis @jjasonkal
- Sion Kazama @KazamaSion
- @resynth1943
- Mostafa Ahangarha @ahangarha
- @gordon-quad
- Sophie Tauchert @999eagle
- @bauruine
- Michael Ilsaas `<https://mikeri.net>`_
- @renyhp
- rachmadani haryono @rachmadaniHaryono
- Mohamad Safadieh @msafadieh
- @gardouille
- @resynth1943
- @Eliesemoule
- @gardouille
- @GazoilKerozen
- Lukáš Kucharczyk @KucharczykL
- Lynda Lopez @lyndalopez544
- M. Efe Çetin @efectn
- Nícholas Kegler @nicholasks
- @pierrechtux
- Scott Wallace @scottwallacesh
- @Singustromo
- @TheEvilSkeleton
- @Wonderfall
- @mrwormo
- Xiaoyu WEI @xywei
- @joshu9h
- Daniel Hones
- @cyclaero
- @thezeroalpha
- @Tobi823
- @archiecodes
- @BBaoVanC
- @datagram1
- @lucky13820
- @jhigginbotham
- @xenrox
- @OliveiraHermogenes
- Paul Alcock @Guilvareux

706
src/searx/CHANGELOG.rst
View File

@ -1,706 +0,0 @@
1.0.0 2021.03.27
================
Core
~~~~
- drop support for Python 3.5 ( #2459 )
- add support for Python 3.9 ( #2397 #2459 )
- update Python dependencies ( #2428 #2459 #2206 ) ⚠️ pyopenssl is not longer required
- automatic update of searx.data ( #2555 #2585 #2595 #2592 #2600 )
- update searx.data ( #2604 #2605 #2606 #2607 #2415 )
- add ability to send engine data to subsequent requests ( #2615 )
- add checker ( #2419 #2476 #2481 #2682 #2682 #2657 )
- by default allow only HTTPS, not HTTP ( #2641 #2659 )
- replace /translations.js with an embedded JSON ( #2660 )
- activate raise_for_error by default ( #2557 )
- don't dump traceback of SearxEngineResponseException on init ( #2635 )
Documentation
~~~~~~~~~~~~~
- update nginx configuration ( #2618 )
- document workaround for using 2 languages simultaneously ( #2479 )
- improve admin-docs about result proxy (morty) configuration ( #2509 )
- fixed typo ( #2457 )
New settings.yml
~~~~~~~~~~~~~~~~
- `general.contact_url` : add link to contact instance maintainer to footer of each page ( #2391 14c7cc0e118f1d0873b32b34793cdec2c5c9c13e #2412 )
- `brand` : move brand options from Makefile to settings.yml ( #2408 #2473 )
Themes
~~~~~~
- oscar: Hide links panel in mobile screens ( #2458 )
- oscar: upgrade dependencies ( #2346 #2673 #2662 )
- remove legacy, courgette and pix-art themes ( #2344 )
- add hyperlink to searx instances list in error message ( #2387 )
- preferences: a tooltip is shown when the mouse is over the engine names ( #2661 )
- Ignore double-quotes when highlighting query parts ( #2553 )
- update autocomplete ( #2593 )
New engines
~~~~~~~~~~~
- ccengine ( #2533 )
- mediathekviewweb ( #2541 )
- solidtorrents ( #2626 )
- solr ( #2652 )
- rumble ( #2588 )
- wiby.me ( #2452 )
Fixed engines
~~~~~~~~~~~~~
- apk_mirror ( #2556 #2642 )
- bing ( #2602 )
- duckduckgo ( #2560 #2559 )
- library genesis ( #2448 )
- ina ( 0ba71c3644c4d20f70528c10eed1385399ec1c82 )
- invidious ( #2451 )
- json_engine ( #2562 )
- google ( #2482 )
- google images ( #2482 )
- google play apps ( 88657fe9c2a41b9be38ee5146e5870672416db12 )
- google play movies ( 50ba2b9e87ef61e96da124f906d3aff4c7870e3f )
- google news ( #2483 #2498 )
- google scholar ( #2611 )
- google video ( #2482 )
- hoogle ( 6255b33c9dcf0d28f0a3307af988565f69259ce2 )
- naver ( #2542 )
- semantic schollar ( f596f5767bed915a5c3bed59ae26283e53f975ca f596f5767bed915a5c3bed59ae26283e53f975ca )
- startpage ( #2396 )
- seznam ( #2564 28286cf3f2308113bf440fb6e7cf326c6ed07889 )
- wikipedia ( #2554 #2565 #2681 #2681 )
- yacy ( #2669 )
- yahoo news ( #2640 #2655 )
Updated engines
~~~~~~~~~~~~~~~
- duckduckgo ( 5f450fda74e80bf350eb1493f66cfa61deaf5cea )
- geektimes ( 45f0e1a859fa12ce2ae0c24dc356922fcad50c8d )
- lobste.rs ( 06b754ad67aa6066aed6df77b5ffb74aabebb040 )
- soundcloud ( #2671 )
- peertube ( #2570 )
- recoll ( #2539 )
- yggtorrent ( #2573 )
Removed engines
~~~~~~~~~~~~~~~
- acgsou ( #2654 )
- google_play_music ( #2558 )
- metager ( #2538 )
- voat ( #2445 )
- yandex ( #2566 )
Bug fixes
~~~~~~~~~
- Fix empty colon in query from selecting Chinese ( #2454 )
- Get correct locale with country from browser ( #2531 )
Code refactoring / reduce the technical debt
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- refactor searx.search.SearchQuery and searx.search.EngineRef ( #2398 )
- dynamically set language_support variable ( #2499 )
- engines: add about variable ( #2460 )
- processors ( #2225 5c6a5407a0b124c3323e73c33b81ec1fbd7d2fce )
- remove Fabric file ( #2494 )
- use unittest from py3, remove unittest2 from py2 ( #2608 )
Github
~~~~~~
- add notice for the issue templates ( #2447 )
- every Sunday, call utils/fetch_*.py scripts and create a PR automatically ( #2500 728e09676400221a064627509a31470d8f6e33bf )
- minor change: replace "travis" by "CI" ( #2528 )
Build scripts
~~~~~~~~~~~~~
- update secret key check ( #2411 )
- fix makefile targets `books/{name}.*` and `books/user.pdf` ( #2420 #2530 )
- upload-pypi-test & linuxdoc has been released on PyPi ( #2456 )
- fix makefile target `gh-pages` : flatten history of branch gh.pages ( #2514 )
- optimize creation of the virtualenv & pyenvinstall targets ( #2421 )
- update pyenv pyenvinstall Make targets ( #2517 )
- makefile.python: remove duplicate pyenv-(un)install targets ( #2418 )
- [fix] make targets engines.languages and useragents.update ( #2643 )
- [fix] utils/serax.sh create_pyenv() - drop duplicate 'pip install .' ( #2621 )
Install scripts
~~~~~~~~~~~~~~~
- drop Ubuntu 16.04 (Xenial Xerus) support ( #2619 )
- replace ubu1910 image by ubu2010 image ( #2435 )
- LXC switch to Fedora 33 / Fedora 31 reached its EOL #2634 ( #2634 )
- add package which to CentOS-7 boilerplate ( #2623 )
- use SEARX_SETTINGS_TEMPLATE from .config environment ( #2417 )
- determine path to makefile.lxc in a LXC ( #2399 )
- remove unused code ( #2401 #2497 )
- support git versions <v2.22 ( #2620 )
Announcement
~~~~~~~~~~~~
We, the searx maintainer team, would like to say a huge thank you for everybody who had been involved in the development of searx or supported us in the past 7 years - making our first stable release available. Special thanks to [NLNet](https://nlnet.nl) for sponsoring multiple features of this release.
0.18.0 2020.12.14
=================
Core
~~~~
- drop Python 2 support ( #2137 #2244 )
- separate index and search routes ( #1681 ). ⚠️ add & remove your searx instance(s) from your browser.
- add external_bang ( #2027 #2043 #2059 )
- add external plugins supports ( #2074 )
- add plugin converting strings into hash digests ( #1246 )
- new category: Onions ( #565 )
- allow searx query parts anywhere in the query ( commit aa3c18dda9329fff875328f6ba97483c417b149a 2aef38c3b9d1fe93e9d665a49b10151d63d92392 )
- preferences: use base_url for prefix of sharing 'currenly saved preferences' (#1249 )
- upgrade to request 2.24.0, pyopenssl is optional ( #2199 )
- force admins to set secret_key if debug mode is disabled ( #2256 )
- standalone searx update ( #1591 )
- architecture clean up ( #2140 #2185 #2195 #2196 #2198 #2189 #2208 #2239 #2241 #2246 #2248 )
- record detail about engine error ( #2332 #2375 #2350 ). Add a new API endpoint: ``/stats/errors``.
- display if an engine does not support HTTPS requests ( #2373 )
New settings.yml
~~~~~~~~~~~~~~~~
- ``use_default_settings``: user settings can relied on the default settings ( #2291 #2362 #2349 )
- ``ui.results_on_new_tab: False`` - for opening result links in a new tab ( #2167 )
- ``ui.advanced_search`` - add preference for displaying advanced settings ( #2327 )
- ``server.method: "POST"`` - Make default query submission method configurable ( #2130 )
- ``server.default_http_headers`` - add default http headers ( #2295 )
- ``engines.*.proxies`` - Using proxy only for specific engines ( #1827 #2319 ), see https://searx.github.io/searx/dev/engine_overview.html#settings-yml
- ``enabled_plugins`` - Enabled plugins ( a05c660e3036ad8d02072fc6731af54c2ed6151c )
- ``preferences.lock`` - Let admins lock user preferences ( #2270 )
Oscar theme
~~~~~~~~~~~
- update infobox ( #2131 )
- Make infoboxes shorter by default.
- Hide the main image by default as well and set a maximum height even when expanded.
- Add a toggle at the bottom of the infobox to expand it or to shrink it again.
- Fix pointhi style
- query suggestion does not keep the language tag of the original query ( #1314 )
- fix the clear button ( #2306 )
Simple theme
~~~~~~~~~~~~
- Fix autocomplete ( #2205 )
New engines
~~~~~~~~~~~
- ahmia, not_evil ( #565 )
- codeberg ( #2104 )
- command line engines: git grep, find, etc. ( #2128 #2250 )
- elasticsearch ( #2292 )
- metager ( #2139 )
- naver ( #1912 )
- opensemanticsearch ( #2271 )
- peertube ( #2109 )
- recoll (#2325)
- sepiasearch ( #2227 )
Updated engines
~~~~~~~~~~~~~~~
- digg ( #2285 )
- dbpedia ( #2352 )
- duckduckgo_definitions ( #2224 #2356 )
- duden ( #2359 )
- invidious ( #2116 )
- libgen ( #2360 )
- photon ( #2336 )
- soundclound ( #2365 )
- wikipedia ( #2178 #2363 #2354 )
- wikidata ( #2151 #2224 #2353 ) - faster response time
- yaCy ( #2255 ) - support HTTP digest authentication.
- youtube_noapi ( #2364 )
Fixed engines
~~~~~~~~~~~~~
- 1x ( #2361 )
- answer 'random sha256' ( #2121 )
- bing image ( #1496 )
- duckduckgo ( #2254 )
- genius ( #2371 )
- google ( #2236 )
- google image ( #2115 )
- lobste.rs ( #2253 )
- piratebay ( #2133 )
- startpage ( #2385 )
- torrentz ( #2101 )
Removed engines
~~~~~~~~~~~~~~~
- filecrop ( #2352 )
- searchcode_doc ( #2372 )
- seedpeer ( #2366 )
- twitter ( #2372 )
- yggtorrent ( #2099 #2375 )
Install scripts & documentation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- install script & documentation ( #2384 #2380 #2362 #2287 #2283 #2277 #2223 #2211 #2118 #2117 #2063 )
Docker image
~~~~~~~~~~~~
- use Alpine 3.12 ( #1983 )
- uwsgi serves the static files directly. ( #1865 )
- fix k8s support ( #2099 )
- make docker produces clean tag version ( #2182 )
Bug fixes
~~~~~~~~~
- searx.utils.HTMLTextExtractor: invalid HTML don't raise an Exception ( #2190 )
- Fix static URL ( commit da8b227044f45127f705f6ea94a72d368eea73bb )
- Fix autocomplete ( #2127 )
- Fix opensearch.xml ( #2132 #2247 )
- Fix documentation build ( #2237 )
- Some fixes in the fetch languages script ( #2212 )
Special thanks to `NLNet <https://nlnet.nl>`__ for sponsoring multiple features of this release.
0.17.0 2020.07.09
=================
- New engines
- eTools
- Wikibooks
- Wikinews
- Wikiquote
- Wikisource
- Wiktionary
- Wikiversity
- Wikivoyage
- Rubygems
- Engine fixes (google, google images, startpage, gigablast, yacy)
- Private engines introduced - more details: https://searx.github.io/searx/blog/private-engines.html
- Greatly improved documentation - check it at https://searx.github.io/searx
- Added autofocus to all search inputs
- CSP friendly oscar theme
- Added option to hide engine errors with `display_error_messages` engine option (true/false values, default is true)
- Tons of accessibility fixes - see https://github.com/searx/searx/issues/350 for details
- More flexible branding options: configurable vcs/issue tracker links
- Added "disable all" & "allow all" options to preferences engine select
- Autocomplete keyboard navigation fixes
- Configurable category order
- Wrap long lines in infoboxes
- Added RSS subscribtion link
- Added routing directions to OSM results
- Added author and length attributes to youtube videos
- Fixed image stretch with mobile viewport in oscar theme
- Added translatable JS strings
- Better HTML annotations - engine names and endpoints are available as classes
- RTL text fixes in oscar theme
- Handle weights in accept-language HTTP headers
- Added answerer results to rss/csv output
- Added new autocomplete backends to settings.yml
- Updated opensearch.xml
- Fixed custom locale setting from settings.yml
- Translation updates
- Removed engines: faroo
Special thanks to `NLNet <https://nlnet.nl>`__ for sponsoring multiple features of this release.
Special thanks to https://www.accessibility.nl/english for making accessibilty audit.
News
~~~~
- @HLFH joined the maintainer team
- Dropped Python2 support
0.16.0 2020.01.30
=================
- New engines
- Splash
- Apkmirror
- NPM search
- Invidious
- Seedpeer
- New languages
- Estonian
- Interlingua
- Lithuanian
- Tibetian
- Occitan
- Tamil
- Engine fixes (wolframalpha, google scholar, youtube, google images, seznam, google, soundcloud, google cloud, duden, btdigg, google play, bing images, flickr noapi, wikidata, dailymotion, google videos, arxiv, dictzone, fdroid, etymonline, bing, duckduckgo, startpage, voat, 1x, deviantart, digg, gigablast, mojeek, duckduckgo definitions, spotify, libgen, qwant, openstreetmap, wikipedia, ina, microsoft academic, scanr structures)
- Dependency updates
- Speed optimizations
- Initial support for offline engines
- Image format display
- Inline js scripts removed
- Infinite scroll plugin fix
- Simple theme bugfixes
- Docker image updates
- Bang expression fixes
- Result merging fixes
- New environment variable added: SEARX_BIND_ADDRESS
News
~~~~
- @return42 joined the maintainer team
- This is the last release with Python2 support
0.15.0 2019.01.06
=================
- New engines
- Acgsou (files, images, videos, music)
- Duden.de (general)
- Seznam (general)
- Mojeek (general)
- New languages
- Catalan
- Welsh
- Basque
- Persian (Iran)
- Galician
- Dutch (Belgium)
- Telugu
- Vietnamese
- New random answerers
- sha256
- uuidv4
- New DOI resolsvers
- sci-hub.tw
- Fix Vim mode on Firefox
- Fix custom select in Oscar theme
- Engine fixes (duckduckgo, google news, currency convert, gigablast, google scholar, wikidata image, etymonline, google videos, startpage, bing image)
- Minor simple theme fixes
- New Youtube icon in Oscar theme
- Get DOI rewriters from settings.yml
- Hide page buttons when infinite scrolling is enabled
- Update user agent versions
- Make Oscar style configurable
- Make suspend times of errored engines configurable
0.14.0 2018.02.19
=================
- New theme: oscar-logicodev dark
- New engines
- AskSteem (general)
- Autocompleter fix for py3
- Engine fixes (pdbe, pubmed, gigablast, google, yacy, bing videos, microsoft academic)
- "All" option is removed from languages
- Minor UI changes
- Translation updates
0.13.1 2017.11.23
=================
- Bug fixes
- https://github.com/searx/searx/issues/1088
- https://github.com/searx/searx/issues/1089
- Dependency updates
0.13.0 2017.11.21
=================
- New theme: simple
- New engines
- Google videos (video)
- Bing videos (video)
- Arxiv (science)
- OpenAIRE (science)
- Pubmed (science)
- Genius (music/lyrics)
- Display engine errors
- Faster startup
- Lots of engine fixes (google images, dictzone, duckduckgo, duckduckgo images, torrentz, faroo, digg, tokyotoshokan, nyaa.si, google news, gitlab, gigablast, geektimes.ru, habrahabr.ru, voat.co, base, json engine, currency convert, google)
- Shorter saved preferences URL
- Fix engine duplications in results
- Py3 compatibility fixes
- Translation updates
0.12.0 2017.06.04
=================
- Python3 compatibility
- New engines
- 1337x.to (files, music, video)
- Semantic Scholar (science)
- Library Genesis (general)
- Framalibre (IT)
- Free Software Directory (IT)
- More compact result UI (oscar theme)
- Configurable static content and template path
- Spelling suggestions
- Multiple engine fixes (duckduckgo, bing, swisscows, yahoo news, bing news, twitter, bing images)
- Reduced static image size
- Docker updates
- Translation updates
Special thanks to `NLNet <https://nlnet.nl>`__ for sponsoring multiple features of this release.
0.11.0 2017.01.10
=================
- New engines
- Protein Data Bank Europe (science)
- Voat.co (general, social media)
- Online Etimology Dictionary (science)
- CCC tv (video, it)
- Searx (all categories - can rotate multiple other instances)
- Answerer functionality (see answerer section on /preferences)
- Local answerers
- Statistical functions
- Random value generator
- Result proxy support (with `morty <https://github.com/asciimoo/morty>`__)
- Extended time range filter
- Improved search language support
- Multiple engine fixes (digbt, 500px, google news, ixquick, bing, kickass, google play movies, habrahabr, yandex)
- Minor UI improvements
- Suggestion support for JSON engine
- Result and query escaping fixes
- Configurable HTTP server version
- More robust search error handling
- Faster webapp initialization in debug mode
- Search module refactor
- Translation updates
0.10.0 2016.09.06
=================
- New engines
- Archive.is (general)
- INA (videos)
- Scanr (science)
- Google Scholar (science)
- Crossref (science)
- Openrepos (files)
- Microsoft Academic Search Engine (science)
- Hoogle (it)
- Diggbt (files)
- Dictzone (general - dictionary)
- Translated (general - translation)
- New Plugins
- Infinite scroll on results page
- DOAI rewrite
- Full theme redesign
- Display the number of results
- Filter searches by date range
- Instance config API endpoint
- Dependency version updates
- Socks proxy support for outgoing requests
- 404 page
News
~~~~
@kvch joined the maintainer team
0.9.0 2016.05.24
================
- New search category: science
- New engines
- Wolframalpha (science)
- Frinkiac (images)
- Arch Linux (it)
- BASE - Bielefeld Academic Search Engine (science)
- Dokuwiki (general)
- Nyaa.se (files, images, music, video)
- Reddit (general, images, news, social media)
- Torrentz.eu (files, music, video)
- Tokyo Toshokan (files, music, video)
- F-Droid (files)
- Erowid (general)
- Bitbucket (it)
- GitLab (it)
- Geektimes (it)
- Habrahabr (it)
- New plugins
- Open links in new tab
- Vim hotkeys for better navigation
- Wikipedia/Mediawiki engine improvements
- Configurable instance name
- Configurable connection pool size
- Fixed broken google engine
- Better docker image
- Images in standard results
- Fixed and refactored user settings (Warning: backward incompatibility - you have to reset your custom engine preferences)
- Suspending engines on errors
- Simplified development/deployment tooling
- Translation updates
- Multilingual autocompleter
- Qwant autocompleter backend
0.8.1 2015.12.22
================
- More efficient result parsing
- Rewritten google engine to prevent app crashes
- Other engine fixes/tweaks
- Bing news
- Btdigg
- Gigablast
- Google images
- Startpage
News
~~~~
New documentation page is available: https://searx.github.io/searx
0.8.0 2015.09.08
================
- New engines
- Blekko (image)
- Gigablast (general)
- Spotify (music)
- Swisscows (general, images)
- Qwant (general, images, news, social media)
- Plugin system
- New plugins
- HTTPS rewrite
- Search on cagetory select
- User information
- Tracker url part remover
- Multiple outgoing IP and HTTP/HTTPS proxy support
- New autocompleter: startpage
- New theme: pix-art
- Settings file structure change
- Fabfile, docker deployment
- Optional safesearch result filter
- Force HTTPS in engines if possible
- Disabled HTTP referrer on outgoing links
- Display cookie information
- Prettier search URLs
- Right-to-left text handling in themes
- Translation updates (New locales: Chinese, Hebrew, Portuguese, Romanian)
New dependencies
~~~~~~~~~~~~~~~~
- pyopenssl
- ndg-httpsclient
- pyasn1
- pyasn1-modules
- certifi
News
~~~~
@dalf joined the maintainer "team"
0.7.0 2015.02.03
================
- New engines
- Digg
- Google Play Store
- Deezer
- Btdigg
- Mixcloud
- 1px
- Image proxy
- Search speed improvements
- Autocompletition of engines, shortcuts and supported languages
- Translation updates (New locales: Turkish, Russian)
- Default theme changed to oscar
- Settings option to disable engines by default
- UI code cleanup and restructure
- Engine tests
- Multiple engine bug fixes and tweaks
- Config option to set default interface locale
- Flexible result template handling
- Application logging and sophisticated engine exception tracebacks
- Kickass torrent size display (oscar theme)
New dependencies
~~~~~~~~~~~~~~~~
- pygments - http://pygments.org/
0.6.0 - 2014.12.25
==================
- Changelog added
- New engines
- Flickr (api)
- Subtitleseeker
- photon
- 500px
- Searchcode
- Searchcode doc
- Kickass torrent
- Precise search request timeout handling
- Better favicon support
- Stricter config parsing
- Translation updates
- Multiple ui fixes
- Flickr (noapi) engine fix
- Pep8 fixes
News
~~~~
Health status of searx instances and engines: http://stats.searx.oe5tpo.com
(source: https://github.com/pointhi/searx_stats)

49
src/searx/CONTRIBUTING.md
View File

@ -1,49 +0,0 @@
# How to contribute
## Resources in the documentation
* [Development quickstart](https://searx.github.io/searx/dev/contribution_guide.html)
* [Contribution guide](https://searx.github.io/searx/dev/contribution_guide.html)
## Submitting PRs
Please follow the provided PR template when writing a description for your changes.
Do not take criticism personally. When you get feedback, it is about your work,
not your character, personality, etc. Keep in mind we all want to make the project better.
When something is not clear, please ask questions to clear things up.
If you would like to introduce a big architectural changes or do a refactoring
either in the codebase or the development tools, please open an issue with a proposal
first. This way we can think together about the problem and probably come up
with a better solution.
## Coding conventions and guidelines
### Commit messages
* Always write descriptive commit messages ("fix bug" is not acceptable).
* Use the present tense ("Add feature" not "Added feature").
* Use the imperative mood ("Move cursor to..." not "Moves cursor to...").
* Limit the first line to 72 characters or less.
* Include the number of the issue you are fixing.
### Coding guidelines
As a Python project, we must follow [PEP 8](https://www.python.org/dev/peps/pep-0008/) and [PEP 20](https://www.python.org/dev/peps/pep-0020/) guidelines.
Furthermore, follow the Clean code conventions. The most important
in this project are the following rules:
* Simpler is better. [KISS principle](https://en.wikipedia.org/wiki/KISS_principle)
* Be consistent.
* Every function must do one thing.
* Use descriptive names for functions and variables.
* Always look for the root cause.
* Keep configurable data high level.
* Avoid negative conditionals.
* Prefer fewer arguments.
* Do not add obvious comment to code.
* Do not comment out code, just delete lines.

93
src/searx/Dockerfile
View File

@ -1,93 +0,0 @@
FROM alpine:3.12
ENTRYPOINT ["/sbin/tini","--","/usr/local/searx/dockerfiles/docker-entrypoint.sh"]
EXPOSE 8080
VOLUME /etc/searx
VOLUME /var/log/uwsgi
ARG SEARX_GID=977
ARG SEARX_UID=977
RUN addgroup -g ${SEARX_GID} searx && \
adduser -u ${SEARX_UID} -D -h /usr/local/searx -s /bin/sh -G searx searx
ENV INSTANCE_NAME=searx \
AUTOCOMPLETE= \
BASE_URL= \
MORTY_KEY= \
MORTY_URL= \
SEARX_SETTINGS_PATH=/etc/searx/settings.yml \
UWSGI_SETTINGS_PATH=/etc/searx/uwsgi.ini
WORKDIR /usr/local/searx
COPY requirements.txt ./requirements.txt
RUN apk upgrade --no-cache \
&& apk add --no-cache -t build-dependencies \
build-base \
py3-setuptools \
python3-dev \
libffi-dev \
libxslt-dev \
libxml2-dev \
openssl-dev \
tar \
git \
&& apk add --no-cache \
ca-certificates \
su-exec \
python3 \
py3-pip \
libxml2 \
libxslt \
openssl \
tini \
uwsgi \
uwsgi-python3 \
brotli \
&& pip3 install --upgrade pip \
&& pip3 install --no-cache -r requirements.txt \
&& apk del build-dependencies \
&& rm -rf /root/.cache
COPY --chown=searx:searx . .
ARG TIMESTAMP_SETTINGS=0
ARG TIMESTAMP_UWSGI=0
ARG VERSION_GITCOMMIT=unknown
RUN su searx -c "/usr/bin/python3 -m compileall -q searx"; \
touch -c --date=@${TIMESTAMP_SETTINGS} searx/settings.yml; \
touch -c --date=@${TIMESTAMP_UWSGI} dockerfiles/uwsgi.ini; \
if [ ! -z $VERSION_GITCOMMIT ]; then\
echo "VERSION_STRING = VERSION_STRING + \"-$VERSION_GITCOMMIT\"" >> /usr/local/searx/searx/version.py; \
fi; \
find /usr/local/searx/searx/static -a \( -name '*.html' -o -name '*.css' -o -name '*.js' \
-o -name '*.svg' -o -name '*.ttf' -o -name '*.eot' \) \
-type f -exec gzip -9 -k {} \+ -exec brotli --best {} \+
# Keep these arguments at the end to prevent redundant layer rebuilds
ARG LABEL_DATE=
ARG GIT_URL=unknown
ARG SEARX_GIT_VERSION=unknown
ARG LABEL_VCS_REF=
ARG LABEL_VCS_URL=
LABEL maintainer="searx <${GIT_URL}>" \
description="A privacy-respecting, hackable metasearch engine." \
version="${SEARX_GIT_VERSION}" \
org.label-schema.schema-version="1.0" \
org.label-schema.name="searx" \
org.label-schema.version="${SEARX_GIT_VERSION}" \
org.label-schema.url="${LABEL_VCS_URL}" \
org.label-schema.vcs-ref=${LABEL_VCS_REF} \
org.label-schema.vcs-url=${LABEL_VCS_URL} \
org.label-schema.build-date="${LABEL_DATE}" \
org.label-schema.usage="https://github.com/searx/searx-docker" \
org.opencontainers.image.title="searx" \
org.opencontainers.image.version="${SEARX_GIT_VERSION}" \
org.opencontainers.image.url="${LABEL_VCS_URL}" \
org.opencontainers.image.revision=${LABEL_VCS_REF} \
org.opencontainers.image.source=${LABEL_VCS_URL} \
org.opencontainers.image.created="${LABEL_DATE}" \
org.opencontainers.image.documentation="https://github.com/searx/searx-docker"

107
src/searx/Makefile
View File

@ -1,107 +0,0 @@
# -*- coding: utf-8; mode: makefile-gmake -*-
# SPDX-License-Identifier: AGPL-3.0-or-later
.DEFAULT_GOAL=help
export MTOOLS=./manage
include utils/makefile.include
all: clean install
PHONY += help
help:
@./manage --help
@echo '----'
@echo 'run - run developer instance'
@echo 'install - developer install of searx into virtualenv'
@echo 'uninstall - uninstall developer installation'
@echo 'clean - clean up working tree'
@echo 'search.checker - check search engines'
@echo 'test - run shell & CI tests'
@echo 'test.sh - test shell scripts'
@echo 'ci.test - run CI tests'
PHONY += run
run: install
$(Q) ( \
sleep 2 ; \
xdg-open http://127.0.0.1:8888/ ; \
) &
SEARX_DEBUG=1 ./manage pyenv.cmd python ./searx/webapp.py
PHONY += install uninstall
install uninstall:
$(Q)./manage pyenv.$@
PHONY += clean
clean: py.clean docs.clean node.clean test.clean
$(Q)./manage build_msg CLEAN "common files"
$(Q)find . -name '*.orig' -exec rm -f {} +
$(Q)find . -name '*.rej' -exec rm -f {} +
$(Q)find . -name '*~' -exec rm -f {} +
$(Q)find . -name '*.bak' -exec rm -f {} +
PHONY += search.checker search.checker.%
search.checker: install
$(Q)./manage pyenv.cmd searx-checker -v
search.checker.%: install
$(Q)./manage pyenv.cmd searx-checker -v "$(subst _, ,$(patsubst search.checker.%,%,$@))"
PHONY += ci.test test test.sh
ci.test: test.pep8 test.pylint test.unit test.robot
test: ci.test
test.sh:
$(Q)shellcheck -x -s bash \
utils/brand.env \
./manage \
utils/lib.sh \
utils/filtron.sh \
utils/searx.sh \
utils/morty.sh \
utils/lxc.sh \
utils/lxc-searx.env \
.config.sh
$(Q)./manage build_msg TEST "$@ OK"
# wrap ./manage script
MANAGE += buildenv
MANAGE += babel.compile
MANAGE += data.all data.languages data.useragents
MANAGE += docs.html docs.live docs.gh-pages docs.prebuild docs.clean
MANAGE += docker.build docker.push
MANAGE += gecko.driver
MANAGE += node.env node.clean
MANAGE += py.build py.clean
MANAGE += pyenv pyenv.install pyenv.uninstall
MANAGE += pypi.upload pypi.upload.test
MANAGE += test.pylint test.pep8 test.unit test.coverage test.robot test.clean
MANAGE += themes.all themes.oscar themes.simple themes.bootstrap
PHONY += $(MANAGE)
$(MANAGE):
$(Q)$(MTOOLS) $@
# deprecated
PHONY += docs docs-clean docs-live docker themes
docs: docs.html
$(Q)./manage build_msg WARN $@ is deprecated use docs.html
docs-clean: docs.clean
$(Q)./manage build_msg WARN $@ is deprecated use docs.clean
docs-live: docs.live
$(Q)./manage build_msg WARN $@ is deprecated use docs.live
docker: docker.build
$(Q)./manage build_msg WARN $@ is deprecated use docker.build
themes: themes.all
$(Q)./manage build_msg WARN $@ is deprecated use themes.all

73
src/searx/README.rst
View File

@ -1,73 +0,0 @@
.. SPDX-License-Identifier: AGPL-3.0-or-later
.. figure:: https://raw.githubusercontent.com/searx/searx/master/searx/static/themes/oscar/img/logo_searx_a.png
:target: https://searx.github.io/searx/
:alt: searX
:width: 100%
:align: center
-------
|searx install|
|searx homepage|
|searx wiki|
|AGPL License|
|Issues|
|commits|
|OpenCollective searx backers|
|OpenCollective searx sponsors|
Privacy-respecting, hackable `metasearch engine`_ / *pronunciation* **sɜːks**.
.. _metasearch engine: https://en.wikipedia.org/wiki/Metasearch_engine
.. |searx install| image:: https://img.shields.io/badge/-install-blue
:target: https://searx.github.io/searx/admin/installation.html
.. |searx homepage| image:: https://img.shields.io/badge/-homepage-blue
:target: https://searx.github.io/searx
.. |searx wiki| image:: https://img.shields.io/badge/-wiki-blue
:target: https://github.com/searx/searx/wiki
.. |AGPL License| image:: https://img.shields.io/badge/license-AGPL-blue.svg
:target: https://github.com/searx/searx/blob/master/LICENSE
.. |Issues| image:: https://img.shields.io/github/issues/searx/searx?color=yellow&label=issues
:target: https://github.com/searx/searx/issues
.. |PR| image:: https://img.shields.io/github/issues-pr-raw/searx/searx?color=yellow&label=PR
:target: https://github.com/searx/searx/pulls
.. |commits| image:: https://img.shields.io/github/commit-activity/y/searx/searx?color=yellow&label=commits
:target: https://github.com/searx/searx/commits/master
.. |OpenCollective searx backers| image:: https://opencollective.com/searx/backers/badge.svg
:target: https://opencollective.com/searx#backer
.. |OpenCollective searx sponsors| image:: https://opencollective.com/searx/sponsors/badge.svg
:target: https://opencollective.com/searx#sponsor
If you are looking for running instances, ready to use, then visit searx.space_.
Otherwise jump to the user_, admin_ and developer_ handbooks you will find on
our homepage_.
.. _searx.space: https://searx.space
.. _user: https://searx.github.io/searx/user
.. _admin: https://searx.github.io/searx/admin
.. _developer: https://searx.github.io/searx/dev
.. _homepage: https://searx.github.io/searx
contact:
openhub_ // twitter_ // IRC: #searx @ freenode
.. _openhub: https://www.openhub.net/p/searx
.. _twitter: https://twitter.com/Searx_engine
-------
|gluten free|
.. |gluten free| image:: https://forthebadge.com/images/featured/featured-gluten-free.svg

3
src/searx/babel.cfg
View File

@ -1,3 +0,0 @@
[python: **.py]
[jinja2: **/templates/**.html]
extensions=jinja2.ext.autoescape,jinja2.ext.with_

145
src/searx/dockerfiles/docker-entrypoint.sh
View File

@ -1,145 +0,0 @@
#!/bin/sh
help() {
printf "Command line:\n\n"
printf " -h Display this help\n"
printf " -d Dry run to update the configuration files.\n"
printf " -f Always update on the configuration files (existing files are renamed with the .old suffix)\n"
printf " Without this option, the new configuration files are copied with the .new suffix\n"
printf "\nEnvironment variables:\n\n"
printf " INSTANCE_NAME settings.yml : general.instance_name\n"
printf " AUTOCOMPLETE settings.yml : search.autocomplete\n"
printf " BASE_URL settings.yml : server.base_url\n"
printf " MORTY_URL settings.yml : result_proxy.url\n"
printf " MORTY_KEY settings.yml : result_proxy.key\n"
printf " BIND_ADDRESS uwsgi bind to the specified TCP socket using HTTP protocol. Default value: \"${DEFAULT_BIND_ADDRESS}\"\n"
printf "\nVolume:\n\n"
printf " /etc/searx the docker entry point copies settings.yml and uwsgi.ini in this directory (see the -f command line option)\n"
echo
exit 0
}
export DEFAULT_BIND_ADDRESS="0.0.0.0:8080"
if [ -z "${BIND_ADDRESS}" ]; then
export BIND_ADDRESS="${DEFAULT_BIND_ADDRESS}"
fi
# Parse special command line
# see docs/admin/installation-docker.rst
# display the help message without the version
if [ "$1" = "help" ]; then
help
fi
# Version
export SEARX_VERSION=$(su searx -c 'python3 -c "import six; import searx.version; six.print_(searx.version.VERSION_STRING)"' 2>/dev/null)
printf 'searx version %s\n\n' "${SEARX_VERSION}"
# Parse command line
FORCE_CONF_UPDATE=0
DRY_RUN=0
while getopts "fdh" option
do
case $option in
f)
FORCE_CONF_UPDATE=1
;;
d)
DRY_RUN=1
;;
h)
help
esac
done
# helpers to update the configuration files
patch_uwsgi_settings() {
CONF="$1"
# Nothing
}
patch_searx_settings() {
CONF="$1"
# Make sure that there is trailing slash at the end of BASE_URL
# see https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Shell-Parameter-Expansion
export BASE_URL="${BASE_URL%/}/"
# update settings.yml
sed -i -e "s|base_url : False|base_url : ${BASE_URL}|g" \
-e "s/instance_name : \"searx\"/instance_name : \"${INSTANCE_NAME}\"/g" \
-e "s/autocomplete : \"\"/autocomplete : \"${AUTOCOMPLETE}\"/g" \
-e "s/ultrasecretkey/$(openssl rand -hex 32)/g" \
"${CONF}"
# Morty configuration
if [ ! -z "${MORTY_KEY}" -a ! -z "${MORTY_URL}" ]; then
sed -i -e "s/image_proxy : False/image_proxy : True/g" \
"${CONF}"
cat >> "${CONF}" <<-EOF
# Morty configuration
result_proxy:
url : ${MORTY_URL}
key : !!binary "${MORTY_KEY}"
EOF
fi
}
update_conf() {
FORCE_CONF_UPDATE=$1
CONF="$2"
NEW_CONF="${2}.new"
OLD_CONF="${2}.old"
REF_CONF="$3"
PATCH_REF_CONF="$4"
if [ -f "${CONF}" ]; then
if [ "${REF_CONF}" -nt "${CONF}" ]; then
# There is a new version
if [ $FORCE_CONF_UPDATE -ne 0 ]; then
# Replace the current configuration
printf '⚠️ Automaticaly update %s to the new version\n' "${CONF}"
if [ ! -f "${OLD_CONF}" ]; then
printf 'The previous configuration is saved to %s\n' "${OLD_CONF}"
mv "${CONF}" "${OLD_CONF}"
fi
cp "${REF_CONF}" "${CONF}"
$PATCH_REF_CONF "${CONF}"
else
# Keep the current configuration
printf '⚠️ Check new version %s to make sure searx is working properly\n' "${NEW_CONF}"
cp "${REF_CONF}" "${NEW_CONF}"
$PATCH_REF_CONF "${NEW_CONF}"
fi
else
printf 'Use existing %s\n' "${CONF}"
fi
else
printf 'Create %s\n' "${CONF}"
cp "${REF_CONF}" "${CONF}"
$PATCH_REF_CONF "${CONF}"
fi
}
# make sure there are uwsgi settings
update_conf ${FORCE_CONF_UPDATE} "${UWSGI_SETTINGS_PATH}" "/usr/local/searx/dockerfiles/uwsgi.ini" "patch_uwsgi_settings"
# make sure there are searx settings
update_conf "${FORCE_CONF_UPDATE}" "${SEARX_SETTINGS_PATH}" "/usr/local/searx/searx/settings.yml" "patch_searx_settings"
# dry run (to update configuration files, then inspect them)
if [ $DRY_RUN -eq 1 ]; then
printf 'Dry run\n'
exit
fi
#
touch /var/run/uwsgi-logrotate
chown -R searx:searx /var/log/uwsgi /var/run/uwsgi-logrotate
unset MORTY_KEY
# Start uwsgi
printf 'Listen on %s\n' "${BIND_ADDRESS}"
exec su-exec searx:searx uwsgi --master --http-socket "${BIND_ADDRESS}" "${UWSGI_SETTINGS_PATH}"

47
src/searx/dockerfiles/uwsgi.ini
View File

@ -1,47 +0,0 @@
[uwsgi]
# Who will run the code
uid = searx
gid = searx
# Number of workers (usually CPU count)
workers = 4
# The right granted on the created socket
chmod-socket = 666
# Plugin to use and interpretor config
single-interpreter = true
master = true
plugin = python3
lazy-apps = true
enable-threads = true
# Module to import
module = searx.webapp
# Virtualenv and python path
pythonpath = /usr/local/searx/
chdir = /usr/local/searx/searx/
# Disable logging for privacy
disable-logging=True
# But keep errors for 2 days
touch-logrotate = /run/uwsgi-logrotate
unique-cron = 15 0 -1 -1 -1 { touch /run/uwsgi-logrotate }
log-backupname = /var/log/uwsgi/uwsgi.log.1
logto = /var/log/uwsgi/uwsgi.log
# No keep alive
# See https://github.com/searx/searx-docker/issues/24
add-header = Connection: close
# uwsgi serves the static files
# expires set to one day as Flask does
static-map = /static=/usr/local/searx/searx/static
static-expires = /* 864000
static-gzip-all = True
offload-threads = %k
# Cache
cache2 = name=searxcache,items=2000,blocks=2000,blocksize=4096,bitmap=1

140
src/searx/docs/_themes/searx/static/searx.css vendored
View File

@ -1,140 +0,0 @@
@import url("pocoo.css");
a, a.reference, a.footnote-reference {
color: #004b6b;
border-color: #004b6b;
}
a:hover {
color: #6d4100;
border-color: #6d4100;
}
p.version-warning {
background-color: #004b6b;
}
div.sidebar {
background-color: whitesmoke;
border-color: lightsteelblue;
border-radius: 3pt;
}
p.sidebar-title, .sidebar p {
margin: 6pt;
}
.sidebar li,
.hlist li {
list-style-type: disclosure-closed;
}
/* admonitions
*/
div.admonition, div.topic, div.toctree-wrapper {
background-color: #fafafa;
margin: 8px 0px;
padding: 1em;
border-radius: 3pt 0 0 3pt;
border-top: none;
border-right: none;
border-bottom: none;
border-left: 5pt solid #ccc;
list-style-type: disclosure-closed;
}
div.toctree-wrapper p.caption {
font-weight: normal;
font-size: 24px;
margin: 0 0 10px 0;
padding: 0;
line-height: 1;
display: inline;
}
p.admonition-title:after {
content: none;
}
.admonition.hint { border-color: #416dc0b0; }
.admonition.note { border-color: #6c856cb0; }
.admonition.tip { border-color: #85c5c2b0; }
.admonition.attention { border-color: #ecec97b0; }
.admonition.caution { border-color: #a6c677b0; }
.admonition.danger { border-color: #d46262b0; }
.admonition.important { border-color: #dfa3a3b0; }
.admonition.error { border-color: red; }
.admonition.warning { border-color: darkred; }
.admonition.admonition-generic-admonition-title {
border-color: #416dc0b0;
}
/* admonitions with (rendered) reST markup examples (:class: rst-example)
*
* .. admonition:: title of the example
* :class: rst-example
* ....
*/
div.rst-example {
background-color: inherit;
margin: 0;
border-top: none;
border-right: 1px solid #ccc;
border-bottom: none;
border-left: none;
border-radius: none;
padding: 0;
}
div.rst-example > p.admonition-title {
font-family: Sans Serif;
font-style: italic;
font-size: 0.8em;
display: block;
border-bottom: 1px solid #ccc;
padding: 0.5em 1em;
text-align: right;
}
/* code block in figures
*/
div.highlight pre {
text-align: left;
}
/* Table theme
*/
thead, tfoot {
background-color: #fff;
}
th:hover, td:hover {
background-color: #ffc;
}
thead th, tfoot th, tfoot td, tbody th {
background-color: #fffaef;
}
tbody tr:nth-child(odd) {
background-color: #fff;
}
tbody tr:nth-child(even) {
background-color: #fafafa;
}
caption {
font-family: Sans Serif;
padding: 0.5em;
margin: 0.5em 0 0.5em 0;
caption-side: top;
text-align: left;
}

96
src/searx/docs/admin/api.rst
View File

@ -1,96 +0,0 @@
.. _adminapi:
==================
Administration API
==================
Get configuration data
======================
.. code:: http
GET /config HTTP/1.1
Sample response
---------------
.. code:: json
{
"autocomplete": "",
"categories": [
"map",
"it",
"images",
],
"default_locale": "",
"default_theme": "oscar",
"engines": [
{
"categories": [
"map"
],
"enabled": true,
"name": "openstreetmap",
"shortcut": "osm"
},
{
"categories": [
"it"
],
"enabled": true,
"name": "arch linux wiki",
"shortcut": "al"
},
{
"categories": [
"images"
],
"enabled": true,
"name": "google images",
"shortcut": "goi"
},
{
"categories": [
"it"
],
"enabled": false,
"name": "bitbucket",
"shortcut": "bb"
},
],
"instance_name": "searx",
"locales": {
"de": "Deutsch (German)",
"en": "English",
"eo": "Esperanto (Esperanto)",
},
"plugins": [
{
"enabled": true,
"name": "HTTPS rewrite"
},
{
"enabled": false,
"name": "Vim-like hotkeys"
}
],
"safe_search": 0
}
Embed search bar
================
The search bar can be embedded into websites. Just paste the example into the
HTML of the site. URL of the searx instance and values are customizable.
.. code:: html
<form method="post" action="https://searx.me/">
<!-- search --> <input type="text" name="q" />
<!-- categories --> <input type="hidden" name="categories" value="general,social media" />
<!-- language --> <input type="hidden" name="lang" value="all" />
<!-- locale --> <input type="hidden" name="locale" value="en" />
<!-- date filter --> <input type="hidden" name="time_range" value="month" />
</form>

33
src/searx/docs/admin/arch_public.dot
View File

@ -1,33 +0,0 @@
digraph G {
node [style=filled, shape=box, fillcolor="#ffffcc", fontname="Sans"];
edge [fontname="Sans"];
browser [label="Browser", shape=Mdiamond];
rp [label="Reverse Proxy", href="https://searx.github.io/searx/utils/filtron.sh.html#public-reverse-proxy"];
filtron [label="Filtron", href="https://searx.github.io/searx/utils/filtron.sh.html"];
morty [label="Morty", href="https://searx.github.io/searx/utils/morty.sh.html"];
static [label="Static files", href="url to configure static files"];
uwsgi [label="uwsgi", href="https://searx.github.io/searx/utils/searx.sh.html"]
searx1 [label="Searx #1"];
searx2 [label="Searx #2"];
searx3 [label="Searx #3"];
searx4 [label="Searx #4"];
browser -> rp [label="HTTPS"]
subgraph cluster_searx {
label = "Searx instance" fontname="Sans";
bgcolor="#fafafa";
{ rank=same; static rp };
rp -> morty [label="optional: images and HTML pages proxy"];
rp -> static [label="optional: reverse proxy serves directly static files"];
rp -> filtron [label="HTTP"];
filtron -> uwsgi [label="HTTP"];
uwsgi -> searx1;
uwsgi -> searx2;
uwsgi -> searx3;
uwsgi -> searx4;
}
}

28
src/searx/docs/admin/architecture.rst
View File

@ -1,28 +0,0 @@
.. _architecture:
============
Architecture
============
.. sidebar:: Further reading
- Reverse Proxy: :ref:`Apache <apache searx site>` & :ref:`nginx <nginx searx
site>`
- Filtron: :ref:`searx filtron`
- Morty: :ref:`searx morty`
- uWSGI: :ref:`searx uwsgi`
- Searx: :ref:`installation basic`
Herein you will find some hints and suggestions about typical architectures of
searx infrastructures.
We start with a contribution from :pull:`@dalf <1776#issuecomment-567917320>`.
It shows a *reference* setup for public searx instances which can build up and
maintained by the scripts from our :ref:`toolboxing`.
.. _arch public:
.. kernel-figure:: arch_public.dot
:alt: arch_public.dot
Reference architecture of a public searx setup.

88
src/searx/docs/admin/engines.rst
View File

@ -1,88 +0,0 @@
=======
Engines
=======
Special Engine Settings
=======================
.. sidebar:: Further reading ..
- :ref:`settings engine`
- :ref:`engine settings` & :ref:`engine file`
.. toctree::
:maxdepth: 1
engines/recoll.rst
.. _engines generic:
General Engine Settings
=======================
Explanation of the :ref:`general engine configuration` shown in the table
:ref:`configured engines`.
============= =========== ==================== ============
:ref:`engine settings` :ref:`engine file`
------------------------- ---------------------------------
Name (cfg) Categories
------------------------- ---------------------------------
Engine .. Paging support **P**
------------------------- -------------------- ------------
Shortcut **S** Language support **L**
Timeout **TO** Time range support **TR**
Disabled **D** Engine type **ET**
------------- ----------- -------------------- ------------
Safe search **SS**
------------- ----------- ---------------------------------
Weigth **W**
------------- ----------- ---------------------------------
Disabled **D**
------------- ----------- ---------------------------------
Show errors **DE**
============= =========== =================================
.. _configured engines:
.. jinja:: searx
.. flat-table:: Engines configured at built time (defaults)
:header-rows: 1
:stub-columns: 2
* - Name (cfg)
- S
- Engine
- TO
- Categories
- P
- L
- SS
- D
- TR
- ET
- W
- D
- DE
{% for name, mod in engines.items() %}
* - {{name}}
- !{{mod.shortcut}}
- {{mod.__name__}}
- {{mod.timeout}}
- {{", ".join(mod.categories)}}
- {{(mod.paging and "y") or ""}}
- {{(mod.language_support and "y") or ""}}
- {{(mod.safesearch and "y") or ""}}
- {{(mod.disabled and "y") or ""}}
- {{(mod.time_range_support and "y") or ""}}
- {{mod.engine_type or ""}}
- {{mod.weight or 1 }}
- {{(mod.disabled and "y") or ""}}
- {{(mod.display_error_messages and "y") or ""}}
{% endfor %}

50
src/searx/docs/admin/engines/recoll.rst
View File

@ -1,50 +0,0 @@
.. _engine recoll:
======
Recoll
======
.. sidebar:: info
- `Recoll <https://www.lesbonscomptes.com/recoll/>`_
- `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_
Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_
does not offer web or API access, this can be achieved using recoll-webui_
Configuration
=============
You must configure the following settings:
``base_url``:
Location where recoll-webui can be reached.
``mount_prefix``:
Location where the file hierarchy is mounted on your *local* filesystem.
``dl_prefix``:
Location where the file hierarchy as indexed by recoll can be reached.
``search_dir``:
Part of the indexed file hierarchy to be search, if empty the full domain is
searched.
Example
=======
Scenario:
#. Recoll indexes a local filesystem mounted in ``/export/documents/reference``,
#. the Recoll search inteface can be reached at https://recoll.example.org/ and
#. the contents of this filesystem can be reached though https://download.example.org/reference
.. code:: yaml
base_url: https://recoll.example.org/
mount_prefix: /export/documents
dl_prefix: https://download.example.org
search_dir: ''

191
src/searx/docs/admin/filtron.rst
View File

@ -1,191 +0,0 @@
.. _searx filtron:
==========================
How to protect an instance
==========================
.. sidebar:: further reading
- :ref:`filtron.sh`
- :ref:`nginx searx site`
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. _filtron: https://github.com/asciimoo/filtron
Searx depends on external search services. To avoid the abuse of these services
it is advised to limit the number of requests processed by searx.
An application firewall, filtron_ solves exactly this problem. Filtron is just
a middleware between your web server (nginx, apache, ...) and searx, we describe
such infratructures in chapter: :ref:`architecture`.
filtron & go
============
.. _Go: https://golang.org/
.. _filtron README: https://github.com/asciimoo/filtron/blob/master/README.md
Filtron needs Go_ installed. If Go_ is preinstalled, filtron_ is simply
installed by ``go get`` package management (see `filtron README`_). If you use
filtron as middleware, a more isolated setup is recommended. To simplify such
an installation and the maintenance of, use our script :ref:`filtron.sh`.
.. _Sample configuration of filtron:
Sample configuration of filtron
===============================
.. sidebar:: Tooling box
- :origin:`/etc/filtron/rules.json <utils/templates/etc/filtron/rules.json>`
An example configuration can be find below. This configuration limits the access
of:
- scripts or applications (roboagent limit)
- webcrawlers (botlimit)
- IPs which send too many requests (IP limit)
- too many json, csv, etc. requests (rss/json limit)
- the same UserAgent of if too many requests (useragent limit)
.. code:: json
[
{
"name": "search request",
"filters": [
"Param:q",
"Path=^(/|/search)$"
],
"interval": "<time-interval-in-sec (int)>",
"limit": "<max-request-number-in-interval (int)>",
"subrules": [
{
"name": "missing Accept-Language",
"filters": ["!Header:Accept-Language"],
"limit": "<max-request-number-in-interval (int)>",
"stop": true,
"actions": [
{"name":"log"},
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
},
{
"name": "suspiciously Connection=close header",
"filters": ["Header:Connection=close"],
"limit": "<max-request-number-in-interval (int)>",
"stop": true,
"actions": [
{"name":"log"},
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
},
{
"name": "IP limit",
"interval": "<time-interval-in-sec (int)>",
"limit": "<max-request-number-in-interval (int)>",
"stop": true,
"aggregations": [
"Header:X-Forwarded-For"
],
"actions": [
{ "name": "log"},
{ "name": "block",
"params": {
"message": "Rate limit exceeded"
}
}
]
},
{
"name": "rss/json limit",
"filters": [
"Param:format=(csv|json|rss)"
],
"interval": "<time-interval-in-sec (int)>",
"limit": "<max-request-number-in-interval (int)>",
"stop": true,
"actions": [
{ "name": "log"},
{ "name": "block",
"params": {
"message": "Rate limit exceeded"
}
}
]
},
{
"name": "useragent limit",
"interval": "<time-interval-in-sec (int)>",
"limit": "<max-request-number-in-interval (int)>",
"aggregations": [
"Header:User-Agent"
],
"actions": [
{ "name": "log"},
{ "name": "block",
"params": {
"message": "Rate limit exceeded"
}
}
]
}
]
}
]
.. _filtron route request:
Route request through filtron
=============================
.. sidebar:: further reading
- :ref:`filtron.sh overview`
- :ref:`installation nginx`
- :ref:`installation apache`
Filtron can be started using the following command:
.. code:: sh
$ filtron -rules rules.json
It listens on ``127.0.0.1:4004`` and forwards filtered requests to
``127.0.0.1:8888`` by default.
Use it along with ``nginx`` with the following example configuration.
.. code:: nginx
# https://example.org/searx
location /searx {
proxy_pass http://127.0.0.1:4004/;
proxy_set_header Host $host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Script-Name /searx;
}
location /searx/static {
/usr/local/searx/searx-src/searx/static;
}
Requests are coming from port 4004 going through filtron and then forwarded to
port 8888 where a searx is being run. For a complete setup see: :ref:`nginx
searx site`.

23
src/searx/docs/admin/index.rst
View File

@ -1,23 +0,0 @@
===========================
Administrator documentation
===========================
.. toctree::
:maxdepth: 2
:caption: Contents
installation
installation-searx
installation-uwsgi
installation-nginx
installation-apache
installation-docker
update-searx
settings
api
architecture
filtron
morty
engines
plugins
buildhosts

515
src/searx/docs/admin/installation-apache.rst
View File

@ -1,515 +0,0 @@
.. _installation apache:
===================
Install with apache
===================
.. _Apache: https://httpd.apache.org/
.. _Apache Debian:
https://cwiki.apache.org/confluence/display/HTTPD/DistrosDefaultLayout#DistrosDefaultLayout-Debian,Ubuntu(Apachehttpd2.x):
.. _README.Debian:
https://salsa.debian.org/apache-team/apache2/raw/master/debian/apache2.README.Debian
.. _Apache Arch Linux:
https://wiki.archlinux.org/index.php/Apache_HTTP_Server
.. _Apache Fedora:
https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-apache-http-server/index.html
.. _Apache directives:
https://httpd.apache.org/docs/trunk/mod/directives.html
.. _Getting Started:
https://httpd.apache.org/docs/current/en/getting-started.html
.. _Terms Used to Describe Directives:
https://httpd.apache.org/docs/current/en/mod/directive-dict.html
.. _Configuration Files:
https://httpd.apache.org/docs/current/en/configuring.html
.. _ProxyPreserveHost: https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypreservehost
.. _LoadModule:
https://httpd.apache.org/docs/2.4/mod/mod_so.html#loadmodule
.. _DocumentRoot:
https://httpd.apache.org/docs/trunk/mod/core.html#documentroot
.. _Location:
https://httpd.apache.org/docs/trunk/mod/core.html#location
.. _uWSGI Apache support:
https://uwsgi-docs.readthedocs.io/en/latest/Apache.html
.. _mod_proxy_uwsgi:
https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-proxy-uwsgi
.. sidebar:: further read
- `Apache Arch Linux`_
- `Apache Debian`_ and `README.Debian`_
- `Apache Fedora`_
- `Apache directives`_
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
----
**Install** :ref:`apache searx site` using :ref:`filtron.sh <filtron.sh overview>`
.. code:: bash
$ sudo -H ./utils/filtron.sh apache install
**Install** :ref:`apache searx site` using :ref:`morty.sh <morty.sh overview>`
.. code:: bash
$ sudo -H ./utils/morty.sh apache install
----
The apache HTTP server
======================
If Apache_ is not installed, install it now. If apache_ is new to you, the
`Getting Started`_, `Configuration Files`_ and `Terms Used to Describe
Directives`_ documentation gives first orientation. There is also a list of
`Apache directives`_ *to keep in the pocket*.
.. tabs::
.. group-tab:: Ubuntu / debian
.. code:: sh
sudo -H apt-get install apache2
.. group-tab:: Arch Linux
.. code:: sh
sudo -H pacman -S apache
sudo -H systemctl enable httpd
sudo -H systemctl start http
.. group-tab:: Fedora / RHEL
.. code:: sh
sudo -H dnf install httpd
sudo -H systemctl enable httpd
sudo -H systemctl start httpd
Now at http://localhost you should see any kind of *Welcome* or *Test* page.
How this default intro site is configured, depends on the linux distribution
(compare `Apache directives`_).
.. tabs::
.. group-tab:: Ubuntu / debian
.. code:: sh
less /etc/apache2/sites-enabled/000-default.conf
In this file, there is a line setting the `DocumentRoot`_ directive:
.. code:: apache
DocumentRoot /var/www/html
And the *welcome* page is the HTML file at ``/var/www/html/index.html``.
.. group-tab:: Arch Linux
.. code:: sh
less /etc/httpd/conf/httpd.conf
In this file, there is a line setting the `DocumentRoot`_ directive:
.. code:: apache
DocumentRoot "/srv/http"
<Directory "/srv/http">
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>
The *welcome* page of Arch Linux is a page showing directory located at
``DocumentRoot``. This is *directory* page is generated by the Module
`mod_autoindex <https://httpd.apache.org/docs/2.4/mod/mod_autoindex.html>`_:
.. code:: apache
LoadModule autoindex_module modules/mod_autoindex.so
...
Include conf/extra/httpd-autoindex.conf
.. group-tab:: Fedora / RHEL
.. code:: sh
less /etc/httpd/conf/httpd.conf
In this file, there is a line setting the ``DocumentRoot`` directive:
.. code:: apache
DocumentRoot "/var/www/html"
...
<Directory "/var/www">
AllowOverride None
# Allow open access:
Require all granted
</Directory>
On fresh installations, the ``/var/www`` is empty and the *default
welcome page* is shown, the configuration is located at::
less /etc/httpd/conf.d/welcome.conf
.. _apache searx site:
Apache Reverse Proxy
====================
.. sidebar:: public to the internet?
If your searx instance is public, stop here and first install :ref:`filtron
reverse proxy <filtron.sh>` and :ref:`result proxy morty <morty.sh>`, see
:ref:`installation scripts`. If already done, follow setup: *searx via
filtron plus morty*.
To setup a Apache revers proxy you have to enable the *headers* and *proxy*
modules and create a `Location`_ configuration for the searx site. In most
distributions you have to un-comment the lines in the main configuration file,
except in :ref:`The Debian Layout`.
.. tabs::
.. group-tab:: Ubuntu / debian
In the Apache setup, enable headers and proxy modules:
.. code:: sh
sudo -H a2enmod headers
sudo -H a2enmod proxy
sudo -H a2enmod proxy_http
In :ref:`The Debian Layout` you create a ``searx.conf`` with the
``<Location /searx >`` directive and save this file in the *sites
available* folder at ``/etc/apache2/sites-available``. To enable the
``searx.conf`` use :man:`a2ensite`:
.. code:: sh
sudo -H a2ensite searx.conf
.. group-tab:: Arch Linux
In the ``/etc/httpd/conf/httpd.conf`` file, activate headers and proxy
modules (LoadModule_):
.. code:: apache
FIXME needs test
LoadModule headers_module modules/mod_headers.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
.. group-tab:: Fedora / RHEL
In the ``/etc/httpd/conf/httpd.conf`` file, activate headers and proxy
modules (LoadModule_):
.. code:: apache
FIXME needs test
LoadModule headers_module modules/mod_headers.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
With ProxyPreserveHost_ the incoming Host HTTP request header is passed to the
proxied host.
.. _apache searx via filtron plus morty:
.. tabs::
.. group-tab:: searx via filtron plus morty
Use this setup, if your instance is public to the internet, compare
figure: :ref:`architecture <arch public>` and :ref:`installation scripts`.
1. Configure a reverse proxy for :ref:`filtron <filtron.sh>`, listening on
*localhost 4004* (:ref:`filtron route request`):
.. code:: apache
<Location /searx >
# SetEnvIf Request_URI "/searx" dontlog
# CustomLog /dev/null combined env=dontlog
Require all granted
Order deny,allow
Deny from all
#Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1
Allow from all
ProxyPreserveHost On
ProxyPass http://127.0.0.1:4004
RequestHeader set X-Script-Name /searx
</Location>
2. Configure reverse proxy for :ref:`morty <searx morty>`, listening on
*localhost 3000*
.. code:: apache
ProxyPreserveHost On
<Location /morty >
# SetEnvIf Request_URI "/morty" dontlog
# CustomLog /dev/null combined env=dontlog
Require all granted
Order deny,allow
Deny from all
#Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1
Allow from all
ProxyPass http://127.0.0.1:3000
RequestHeader set X-Script-Name /morty
</Location>
For a fully result proxification add :ref:`morty's <searx morty>` **public
URL** to your :origin:`searx/settings.yml`:
.. code:: yaml
result_proxy:
# replace example.org with your server's public name
url : https://example.org/morty
key : !!binary "insert_your_morty_proxy_key_here"
server:
image_proxy : True
uWSGI support
=============
Be warned, with this setup, your instance isn't :ref:`protected <searx
filtron>`, nevertheless it is good enough for intranet usage. In modern Linux
distributions, the `mod_proxy_uwsgi`_ is compiled into the *normal* apache
package and you need to install only the :ref:`uWSGI <searx uwsgi>` package:
.. tabs::
.. group-tab:: Ubuntu / debian
.. code:: sh
sudo -H apt-get install uwsgi
# Ubuntu =< 18.04
sudo -H apt-get install libapache2-mod-proxy-uwsgi
.. group-tab:: Arch Linux
.. code:: sh
sudo -H pacman -S uwsgi
.. group-tab:: Fedora / RHEL
.. code:: sh
sudo -H dnf install uwsgi
The next example shows a configuration using the `uWSGI Apache support`_ via
unix sockets and `mod_proxy_uwsgi`_.
For socket communication, you have to activate ``socket =
/run/uwsgi/app/searx/socket`` and comment out the ``http = 127.0.0.1:8888``
configuration in your :ref:`uwsgi ini file <uwsgi configuration>`. If not
already exists, create a folder for the unix sockets, which can be used by the
searx account (see :ref:`create searx user`):
.. code:: bash
sudo -H mkdir -p /run/uwsgi/app/searx/
sudo -H chown -R searx:searx /run/uwsgi/app/searx/
If the server is public; to limit access to your intranet replace ``Allow from
all`` directive and replace ``192.168.0.0/16`` with your subnet IP/class.
.. tabs::
.. group-tab:: Ubuntu / debian
.. code:: apache
LoadModule headers_module /usr/lib/apache2/mod_headers.so
LoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.so
LoadModule proxy_uwsgi_module /usr/lib/apache2/modules/mod_proxy_uwsgi.so
# SetEnvIf Request_URI /searx dontlog
# CustomLog /dev/null combined env=dontlog
<Location /searx>
Require all granted
Order deny,allow
Deny from all
# Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1
Allow from all
ProxyPreserveHost On
ProxyPass unix:/run/uwsgi/app/searx/socket|uwsgi://uwsgi-uds-searx/
</Location>
.. group-tab:: Arch Linux
.. code:: apache
FIXME needs test
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so
# SetEnvIf Request_URI /searx dontlog
# CustomLog /dev/null combined env=dontlog
<Location /searx>
Require all granted
Order deny,allow
Deny from all
# Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1
Allow from all
ProxyPreserveHost On
ProxyPass unix:/run/uwsgi/app/searx/socket|uwsgi://uwsgi-uds-searx/
</Location>
.. group-tab:: Fedora / RHEL
.. code:: apache
FIXME needs test
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so
<IfModule proxy_uwsgi_module>
# SetEnvIf Request_URI /searx dontlog
# CustomLog /dev/null combined env=dontlog
<Location /searx>
Require all granted
Order deny,allow
Deny from all
# Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1
Allow from all
ProxyPreserveHost On
ProxyPass unix:/run/uwsgi/app/searx/socket|uwsgi://uwsgi-uds-searx/
</Location>
</IfModule>
.. group-tab:: old mod_wsgi
We show this only for historical reasons, DON'T USE `mod_uwsgi
<https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-uwsgi>`_.
ANYMORE!
.. code:: apache
<IfModule mod_uwsgi.c>
# SetEnvIf Request_URI "/searx" dontlog
# CustomLog /dev/null combined env=dontlog
<Location /searx >
Require all granted
Options FollowSymLinks Indexes
SetHandler uwsgi-handler
uWSGISocket /run/uwsgi/app/searx/socket
Order deny,allow
Deny from all
# Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1
Allow from all
</Location>
</IfModule>
.. _restart apache:
Restart service
===============
.. tabs::
.. group-tab:: Ubuntu / debian
.. code:: sh
sudo -H systemctl restart apache2
sudo -H service uwsgi restart searx
.. group-tab:: Arch Linux
.. code:: sh
sudo -H systemctl restart httpd
sudo -H systemctl restart uwsgi@searx
.. group-tab:: Fedora / RHEL
.. code:: sh
sudo -H systemctl restart httpd
sudo -H touch /etc/uwsgi.d/searx.ini
disable logs
============
For better privacy you can disable Apache logs. In the examples above activate
one of the lines and `restart apache`_::
# SetEnvIf Request_URI "/searx" dontlog
# CustomLog /dev/null combined env=dontlog
The ``CustomLog`` directive disable logs for the whole (virtual) server, use it
when the URL of the service does not have a path component (``/searx``) / is
located at root (``/``).
.. _The Debian Layout:
The Debian Layout
=================
Be aware that the Debian layout is quite different from the standard Apache
configuration. For details look at the README.Debian_
(``/usr/share/doc/apache2/README.Debian.gz``). Some commands you should know on
Debian:
* :man:`apache2ctl`: Apache HTTP server control interface
* :man:`a2enmod`, :man:`a2dismod`: switch on/off modules
* :man:`a2enconf`, :man:`a2disconf`: switch on/off configurations
* :man:`a2ensite`, :man:`a2dissite`: switch on/off sites

60
src/searx/docs/admin/installation-docker.rst
View File

@ -1,60 +0,0 @@
.. _installation docker:
===================
Docker installation
===================
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
----
Docker image searx/searx
========================
The docker image is `searx/searx <https://hub.docker.com/r/searx/searx>`_ (based on `github.com/searx/searx <https://github.com/searx/searx>`_).
Make sure you have `installed Docker <https://docs.docker.com/get-docker/>`_. For instance, you can deploy a local instance:
.. code:: sh
export PORT=80
docker pull searx/searx
docker run --rm -d -v ${PWD}/searx:/etc/searx -p $PORT:8080 -e BASE_URL=http://localhost:$PORT/ searx/searx
Go to ``http://localhost:$PORT``.
Inside ``${PWD}/searx``, you will find ``settings.yml`` and ``uwsgi.ini``.
You can modify these files according to your needs and restart the Docker image.
Command line
------------
.. code:: sh
docker run --rm -it searx/searx -h
.. program-output:: ../dockerfiles/docker-entrypoint.sh help
Build the image
---------------
It's also possible to build searx from the embedded Dockerfile.
.. code:: sh
git clone https://github.com/searx/searx.git
cd searx
make docker.build
Public instance
===============
If you intend to create a public instance using Docker, see https://github.com/searx/searx-docker

383
src/searx/docs/admin/installation-nginx.rst
View File

@ -1,383 +0,0 @@
.. _installation nginx:
==================
Install with nginx
==================
.. _nginx:
https://docs.nginx.com/nginx/admin-guide/
.. _nginx server configuration:
https://docs.nginx.com/nginx/admin-guide/web-server/web-server/#setting-up-virtual-servers
.. _nginx beginners guide:
https://nginx.org/en/docs/beginners_guide.html
.. _Getting Started wiki:
https://www.nginx.com/resources/wiki/start/
.. _uWSGI support from nginx:
https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html
.. _uwsgi_params:
https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html#configuring-nginx
.. _SCRIPT_NAME:
https://werkzeug.palletsprojects.com/en/1.0.x/wsgi/#werkzeug.wsgi.get_script_name
.. sidebar:: further reading
- nginx_
- `nginx beginners guide`_
- `nginx server configuration`_
- `Getting Started wiki`_
- `uWSGI support from nginx`_
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
----
**Install** :ref:`nginx searx site` using :ref:`filtron.sh <filtron.sh overview>`
.. code:: bash
$ sudo -H ./utils/filtron.sh nginx install
**Install** :ref:`nginx searx site` using :ref:`morty.sh <morty.sh overview>`
.. code:: bash
$ sudo -H ./utils/morty.sh nginx install
----
The nginx HTTP server
=====================
If nginx_ is not installed (uwsgi will not work with the package nginx-light),
install it now.
.. tabs::
.. group-tab:: Ubuntu / debian
.. code:: sh
sudo -H apt-get install nginx
.. group-tab:: Arch Linux
.. code-block:: sh
sudo -H pacman -S nginx-mainline
sudo -H systemctl enable nginx
sudo -H systemctl start nginx
.. group-tab:: Fedora / RHEL
.. code-block:: sh
sudo -H dnf install nginx
sudo -H systemctl enable nginx
sudo -H systemctl start nginx
Now at http://localhost you should see a *Welcome to nginx!* page, on Fedora you
see a *Fedora Webserver - Test Page*. The test page comes from the default
`nginx server configuration`_. How this default intro site is configured,
depends on the linux distribution:
.. tabs::
.. group-tab:: Ubuntu / debian
.. code:: sh
less /etc/nginx/nginx.conf
there is a line including site configurations from:
.. code:: nginx
include /etc/nginx/sites-enabled/*;
.. group-tab:: Arch Linux
.. code-block:: sh
less /etc/nginx/nginx.conf
in there is a configuration section named ``server``:
.. code-block:: nginx
server {
listen 80;
server_name localhost;
# ...
}
.. group-tab:: Fedora / RHEL
.. code-block:: sh
less /etc/nginx/nginx.conf
there is a line including site configurations from:
.. code:: nginx
include /etc/nginx/conf.d/*.conf;
.. _nginx searx site:
A nginx searx site
==================
.. sidebar:: public to the internet?
If your searx instance is public, stop here and first install :ref:`filtron
reverse proxy <filtron.sh>` and :ref:`result proxy morty <morty.sh>`, see
:ref:`installation scripts`. If already done, follow setup: *searx via
filtron plus morty*.
Now you have to create a configuration for the searx site. If nginx_ is new to
you, the `nginx beginners guide`_ is a good starting point and the `Getting
Started wiki`_ is always a good resource *to keep in the pocket*.
.. tabs::
.. group-tab:: Ubuntu / debian
Create configuration at ``/etc/nginx/sites-available/searx`` and place a
symlink to sites-enabled:
.. code:: sh
sudo -H ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx
.. group-tab:: Arch Linux
In the ``/etc/nginx/nginx.conf`` file, replace the configuration section
named ``server``.
.. group-tab:: Fedora / RHEL
Create configuration at ``/etc/nginx/conf.d/searx`` and place a
symlink to sites-enabled:
.. _nginx searx via filtron plus morty:
.. tabs::
.. group-tab:: searx via filtron plus morty
Use this setup, if your instance is public to the internet, compare
figure: :ref:`architecture <arch public>` and :ref:`installation scripts`.
1. Configure a reverse proxy for :ref:`filtron <filtron.sh>`, listening on
*localhost 4004* (:ref:`filtron route request`):
.. code:: nginx
# https://example.org/searx
location /searx {
proxy_pass http://127.0.0.1:4004/;
proxy_set_header Host $host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Script-Name /searx;
}
location /searx/static/ {
alias /usr/local/searx/searx-src/searx/static/;
}
2. Configure reverse proxy for :ref:`morty <searx morty>`, listening on
*localhost 3000*:
.. code:: nginx
# https://example.org/morty
location /morty {
proxy_pass http://127.0.0.1:3000/;
proxy_set_header Host $host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
}
For a fully result proxification add :ref:`morty's <searx morty>` **public
URL** to your :origin:`searx/settings.yml`:
.. code:: yaml
result_proxy:
# replace example.org with your server's public name
url : https://example.org/morty
key : !!binary "insert_your_morty_proxy_key_here"
server:
image_proxy : True
.. group-tab:: proxy or uWSGI
Be warned, with this setup, your instance isn't :ref:`protected <searx
filtron>`. Nevertheless it is good enough for intranet usage and it is a
excellent example of; *how different services can be set up*. The next
example shows a reverse proxy configuration wrapping the :ref:`searx-uWSGI
application <uwsgi configuration>`, listening on ``http =
127.0.0.1:8888``.
.. code:: nginx
# https://hostname.local/
location / {
proxy_pass http://127.0.0.1:8888;
proxy_set_header Host $host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_buffering off;
}
Alternatively you can use the `uWSGI support from nginx`_ via unix
sockets. For socket communication, you have to activate ``socket =
/run/uwsgi/app/searx/socket`` and comment out the ``http =
127.0.0.1:8888`` configuration in your :ref:`uwsgi ini file <uwsgi
configuration>`.
The example shows a nginx virtual ``server`` configuration, listening on
port 80 (IPv4 and IPv6 http://[::]:80). The uWSGI app is configured at
location ``/`` by importing the `uwsgi_params`_ and passing requests to
the uWSGI socket (``uwsgi_pass``). The ``server``\'s root points to the
:ref:`searx-src clone <searx-src>` and wraps directly the
:origin:`searx/static/` content at ``location /static``.
.. code:: nginx
server {
# replace hostname.local with your server's name
server_name hostname.local;
listen 80;
listen [::]:80;
location / {
include uwsgi_params;
uwsgi_pass unix:/run/uwsgi/app/searx/socket;
}
root /usr/local/searx/searx-src/searx;
location /static { }
}
If not already exists, create a folder for the unix sockets, which can be
used by the searx account:
.. code:: bash
mkdir -p /run/uwsgi/app/searx/
sudo -H chown -R searx:searx /run/uwsgi/app/searx/
.. group-tab:: \.\. at subdir URL
Be warned, with these setups, your instance isn't :ref:`protected <searx
filtron>`. The examples are just here to demonstrate how to export the
searx application from a subdirectory URL ``https://example.org/searx/``.
.. code:: nginx
# https://hostname.local/searx
location /searx {
proxy_pass http://127.0.0.1:8888;
proxy_set_header Host $host;
proxy_set_header Connection $http_connection;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Script-Name /searx;
proxy_buffering off;
}
location /searx/static/ {
alias /usr/local/searx/searx-src/searx/static/;
}
The ``X-Script-Name /searx`` is needed by the searx implementation to
calculate relative URLs correct. The next example shows a uWSGI
configuration. Since there are no HTTP headers in a (u)WSGI protocol, the
value is shipped via the SCRIPT_NAME_ in the WSGI environment.
.. code:: nginx
# https://hostname.local/searx
location /searx {
uwsgi_param SCRIPT_NAME /searx;
include uwsgi_params;
uwsgi_pass unix:/run/uwsgi/app/searx/socket;
}
location /searx/static/ {
alias /usr/local/searx/searx-src/searx/;
}
For searx to work correctly the ``base_url`` must be set in the
:origin:`searx/settings.yml`.
.. code:: yaml
server:
# replace example.org with your server's public name
base_url : https://example.org/searx/
Restart service:
.. tabs::
.. group-tab:: Ubuntu / debian
.. code:: sh
sudo -H systemctl restart nginx
sudo -H service uwsgi restart searx
.. group-tab:: Arch Linux
.. code:: sh
sudo -H systemctl restart nginx
sudo -H systemctl restart uwsgi@searx
.. group-tab:: Fedora
.. code:: sh
sudo -H systemctl restart nginx
sudo -H touch /etc/uwsgi.d/searx.ini
Disable logs
============
For better privacy you can disable nginx logs in ``/etc/nginx/nginx.conf``.
.. code:: nginx
http {
# ...
access_log /dev/null;
error_log /dev/null;
# ...
}

120
src/searx/docs/admin/installation-searx.rst
View File

@ -1,120 +0,0 @@
.. _installation basic:
=========================
Step by step installation
=========================
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
Step by step installation with virtualenv. For Ubuntu, be sure to have enable
universe repository.
.. _install packages:
Install packages
================
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START distro-packages
:end-before: END distro-packages
.. hint::
This installs also the packages needed by :ref:`searx uwsgi`
.. _create searx user:
Create user
===========
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START create user
:end-before: END create user
.. _searx-src:
install searx & dependencies
============================
Start a interactive shell from new created user and clone searx:
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START clone searx
:end-before: END clone searx
In the same shell create *virtualenv*:
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START create virtualenv
:end-before: END create virtualenv
To install searx's dependencies, exit the searx *bash* session you opened above
and restart a new. Before install, first check if your *virtualenv* was sourced
from the login (*~/.profile*):
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START manage.sh update_packages
:end-before: END manage.sh update_packages
.. tip::
Open a second terminal for the configuration tasks and leave the ``(searx)$``
terminal open for the tasks below.
.. _use_default_settings.yml:
Configuration
=============
.. sidebar:: ``use_default_settings: True``
- :ref:`settings global`
- :ref:`settings location`
- :ref:`settings use_default_settings`
- :origin:`/etc/searx/settings.yml <utils/templates/etc/searx/use_default_settings.yml>`
To create a initial ``/etc/searx/settings.yml`` you can start with a copy of the
file :origin:`utils/templates/etc/searx/use_default_settings.yml`. This setup
:ref:`use default settings <settings use_default_settings>` from
:origin:`searx/settings.yml` and is recommended since :pull:`2291` is merged.
For a *minimal setup*, configure like shown below – replace ``searx@$(uname
-n)`` with a name of your choice, set ``ultrasecretkey`` -- *and/or* edit
``/etc/searx/settings.yml`` to your needs.
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START searx config
:end-before: END searx config
.. tabs::
.. group-tab:: Use default settings
.. literalinclude:: ../../utils/templates/etc/searx/use_default_settings.yml
:language: yaml
.. group-tab:: searx/settings.yml
.. literalinclude:: ../../searx/settings.yml
:language: yaml
Check
=====
To check your searx setup, optional enable debugging and start the *webapp*.
Searx looks at the exported environment ``$SEARX_SETTINGS_PATH`` for a
configuration file.
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START check searx installation
:end-before: END check searx installation
If everything works fine, hit ``[CTRL-C]`` to stop the *webapp* and disable the
debug option in ``settings.yml``. You can now exit searx user bash (enter exit
command twice). At this point searx is not demonized; uwsgi allows this.

150
src/searx/docs/admin/installation-uwsgi.rst
View File

@ -1,150 +0,0 @@
.. _searx uwsgi:
=====
uwsgi
=====
.. sidebar:: further reading
- `systemd.unit`_
- `uWSGI Emperor`_
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. _systemd.unit: https://www.freedesktop.org/software/systemd/man/systemd.unit.html
.. _One service per app in systemd:
https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html#one-service-per-app-in-systemd
.. _uWSGI Emperor:
https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html
.. _uwsgi ini file:
https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#ini-files
.. _systemd unit template:
http://0pointer.de/blog/projects/instances.html
Origin uWSGI
============
How uWSGI is implemented by distributors is different. uWSGI itself
recommend two methods
`systemd.unit`_ template files as described here `One service per app in systemd`_.
There is one `systemd unit template`_ and one `uwsgi ini file`_ per uWSGI-app
placed at dedicated locations. Take archlinux and a searx.ini as example::
unit template --> /usr/lib/systemd/system/uwsgi@.service
uwsgi ini files --> /etc/uwsgi/searx.ini
The searx app can be maintained as know from common systemd units::
systemctl enable uwsgi@searx
systemctl start uwsgi@searx
systemctl restart uwsgi@searx
systemctl stop uwsgi@searx
The `uWSGI Emperor`_ mode which fits for maintaining a large range of uwsgi apps.
The Emperor mode is a special uWSGI instance that will monitor specific
events. The Emperor mode (service) is started by a (common, not template)
systemd unit. The Emperor service will scan specific directories for `uwsgi
ini file`_\s (also know as *vassals*). If a *vassal* is added, removed or the
timestamp is modified, a corresponding action takes place: a new uWSGI
instance is started, reload or stopped. Take Fedora and a searx.ini as
example::
to start a new searx instance create --> /etc/uwsgi.d/searx.ini
to reload the instance edit timestamp --> touch /etc/uwsgi.d/searx.ini
to stop instance remove ini --> rm /etc/uwsgi.d/searx.ini
Distributors
============
The `uWSGI Emperor`_ mode and `systemd unit template`_ is what the distributors
mostly offer their users, even if they differ in the way they implement both
modes and their defaults. Another point they might differ is the packaging of
plugins (if so, compare :ref:`install packages`) and what the default python
interpreter is (python2 vs. python3).
Fedora starts a Emperor by default, while archlinux does not start any uwsgi
service by default. Worth to know; debian (ubuntu) follow a complete different
approach. *debian*: your are familiar with the apache infrastructure? .. they
do similar for the uWSGI infrastructure (with less comfort), the folders are::
/etc/uwsgi/apps-available/
/etc/uwsgi/apps-enabled/
The `uwsgi ini file`_ is enabled by a symbolic link::
ln -s /etc/uwsgi/apps-available/searx.ini /etc/uwsgi/apps-enabled/
From debian's documentation (``/usr/share/doc/uwsgi/README.Debian.gz``): You
could control specific instance(s) by issuing::
service uwsgi <command> <confname> <confname> ...
sudo -H service uwsgi start searx
sudo -H service uwsgi stop searx
My experience is, that this command is a bit buggy.
.. _uwsgi configuration:
Alltogether
===========
Create the configuration ini-file according to your distribution (see below) and
restart the uwsgi application.
.. tabs::
.. group-tab:: Ubuntu / debian
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START searx uwsgi-description ubuntu-20.04
:end-before: END searx uwsgi-description ubuntu-20.04
.. hotfix: a bug group-tab need this comment
.. group-tab:: Arch Linux
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START searx uwsgi-description arch
:end-before: END searx uwsgi-description arch
.. hotfix: a bug group-tab need this comment
.. group-tab:: Fedora / RHEL
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START searx uwsgi-description fedora
:end-before: END searx uwsgi-description fedora
.. tabs::
.. group-tab:: Ubuntu / debian
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START searx uwsgi-appini ubuntu-20.04
:end-before: END searx uwsgi-appini ubuntu-20.04
.. hotfix: a bug group-tab need this comment
.. group-tab:: Arch Linux
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START searx uwsgi-appini arch
:end-before: END searx uwsgi-appini arch
.. hotfix: a bug group-tab need this comment
.. group-tab:: Fedora / RHEL
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START searx uwsgi-appini fedora
:end-before: END searx uwsgi-appini fedora

86
src/searx/docs/admin/installation.rst
View File

@ -1,86 +0,0 @@
.. _installation:
============
Installation
============
*You're spoilt for choice*, choose your preferred method of installation.
- :ref:`installation docker`
- :ref:`installation scripts`
- :ref:`installation basic`
The :ref:`installation basic` is good enough for intranet usage and it is a
excellent illustration of *how a searx instance is build up*. If you place your
instance public to the internet you should really consider to install a
:ref:`filtron reverse proxy <filtron.sh>` and for privacy a :ref:`result proxy
<morty.sh>` is mandatory.
Therefore, if you do not have any special preferences, its recommend to use the
:ref:`installation docker` or the `Installation scripts`_ from our :ref:`tooling
box <toolboxing>` as described below.
.. _installation scripts:
Installation scripts
====================
.. sidebar:: Update OS first!
To avoid unwanted side effects, update your OS before installing searx.
The following will install a setup as shown in :ref:`architecture`. First you
need to get a clone. The clone is only needed for the installation procedure
and some maintenance tasks (alternatively you can create your own fork).
For the installation procedure, use a *sudoer* login to run the scripts. If you
install from ``root``, take into account that the scripts are creating a
``searx``, a ``filtron`` and a ``morty`` user. In the installation procedure
these new created users do need read access to the clone of searx, which is not
the case if you clone into a folder below ``/root``.
.. code:: bash
$ cd ~/Downloads
$ git clone https://github.com/searx/searx searx
$ cd searx
.. sidebar:: further read
- :ref:`toolboxing`
- :ref:`update searx`
- :ref:`inspect searx`
**Install** :ref:`searx service <searx.sh>`
This installs searx as described in :ref:`installation basic`.
.. code:: bash
$ sudo -H ./utils/searx.sh install all
**Install** :ref:`filtron reverse proxy <filtron.sh>`
.. code:: bash
$ sudo -H ./utils/filtron.sh install all
**Install** :ref:`result proxy <morty.sh>`
.. code:: bash
$ sudo -H ./utils/morty.sh install all
If all services are running fine, you can add it to your HTTP server:
- :ref:`installation apache`
- :ref:`installation nginx`
.. _git stash: https://git-scm.com/docs/git-stash
.. tip::
About script's installation options have a look at chapter :ref:`toolboxing
setup`. How to brand your instance see chapter :ref:`settings global`. To
*stash* your instance's setup, `git stash`_ your clone's :origin:`Makefile`
and :origin:`.config.sh` file .

40
src/searx/docs/admin/morty.rst
View File

@ -1,40 +0,0 @@
.. _searx morty:
=========================
How to setup result proxy
=========================
.. sidebar:: further reading
- :ref:`morty.sh`
.. _morty: https://github.com/asciimoo/morty
.. _morty's README: https://github.com/asciimoo/morty
By default searx can only act as an image proxy for result images, but it is
possible to proxify all the result URLs with an external service, morty_.
To use this feature, morty has to be installed and activated in searx's
``settings.yml``. Add the following snippet to your ``settings.yml`` and
restart searx:
.. code:: yaml
result_proxy:
url : http://127.0.0.1:3000/
key : !!binary "insert_your_morty_proxy_key_here"
Note that the example above (``http://127.0.0.1:3000``) is only for single-user
instances without a HTTP proxy. If your morty service is public, the url is the
address of the reverse proxy (e.g ``https://example.org/morty``).
For more information about *result proxy* have a look at *"searx via filtron
plus morty"* in the :ref:`nginx <nginx searx via filtron plus morty>` and
:ref:`apache <apache searx via filtron plus morty>` sections.
``url``
Is the address of the running morty service.
``key``
Is an optional argument, see `morty's README`_ for more information.

394
src/searx/docs/admin/settings.rst
View File

@ -1,394 +0,0 @@
.. _settings.yml:
================
``settings.yml``
================
This page describe the options possibilities of the :origin:`searx/settings.yml`
file.
.. sidebar:: Further reading ..
- :ref:`use_default_settings.yml`
- :ref:`search API`
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. _settings location:
settings.yml location
=====================
First, searx will try to load settings.yml from these locations:
1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable.
2. ``/etc/searx/settings.yml``
If these files don't exist (or are empty or can't be read), searx uses the
:origin:`searx/settings.yml` file.
.. _settings global:
Global Settings
===============
``general:``
------------
.. code:: yaml
general:
debug : False # Debug mode, only for development
instance_name : "searx" # displayed name
git_url: https://github.com/searx/searx
git_branch: master
issue_url: https://github.com/searx/searx/issues
docs_url: https://searx.github.io/searx
public_instances: https://searx.space
contact_url: False # mailto:contact@example.com
wiki_url: https://github.com/searx/searx/wiki
twitter_url: https://twitter.com/Searx_engine
``debug`` :
Allow a more detailed log if you run searx directly. Display *detailed* error
messages in the browser too, so this must be deactivated in production.
``contact_url``:
Contact ``mailto:`` address or WEB form.
``git_url`` and ``git_branch``:
Changes this, to point to your searx fork (branch).
``docs_url``
If you host your own documentation, change this URL.
``wiki_url``:
Link to your wiki (or ``False``)
``twitter_url``:
Link to your tweets (or ``False``)
``server:``
-----------
.. code:: yaml
server:
port : 8888
bind_address : "127.0.0.1" # address to listen on
secret_key : "ultrasecretkey" # change this!
base_url : False # set custom base_url (or False)
image_proxy : False # proxying image results through searx
default_locale : "" # default interface locale
default_theme : oscar # ui theme
default_http_headers:
X-Content-Type-Options : nosniff
X-XSS-Protection : 1; mode=block
X-Download-Options : noopen
X-Robots-Tag : noindex, nofollow
Referrer-Policy : no-referrer
``port`` & ``bind_address``:
Port number and *bind address* of the searx web application if you run it
directly using ``python searx/webapp.py``. Doesn't apply to searx running on
Apache or Nginx.
``secret_key`` :
Used for cryptography purpose.
``base_url`` :
The base URL where searx is deployed. Used to create correct inbound links.
``image_proxy`` :
Allow your instance of searx of being able to proxy images. Uses memory space.
``default_locale`` :
Searx interface language. If blank, the locale is detected by using the
browser language. If it doesn't work, or you are deploying a language
specific instance of searx, a locale can be defined using an ISO language
code, like ``fr``, ``en``, ``de``.
``default_theme`` :
Name of the theme you want to use by default on your searx instance.
.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
``default_http_headers``:
Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__
``outgoing:``
-------------
.. code:: yaml
outgoing: # communication with search engines
request_timeout : 2.0 # default timeout in seconds, can be override by engine
# max_request_timeout: 10.0 # the maximum timeout in seconds
useragent_suffix : "" # informations like an email address to the administrator
pool_connections : 100 # Maximum number of allowable connections, or None for no limits. The default is 100.
pool_maxsize : 10 # Number of allowable keep-alive connections, or None to always allow. The default is 10.
enable_http2: True # See https://www.python-httpx.org/http2/
# uncomment below section if you want to use a proxy
# proxies:
# all://:
# - http://proxy1:8080
# - http://proxy2:8080
# uncomment below section only if you have more than one network interface
# which can be the source of outgoing search requests
# source_ips:
# - 1.1.1.1
# - 1.1.1.2
# - fe80::/126
``request_timeout`` :
Global timeout of the requests made to others engines in seconds. A bigger
timeout will allow to wait for answers from slow engines, but in consequence
will slow searx reactivity (the result page may take the time specified in the
timeout to load). Can be override by :ref:`settings engine`
``useragent_suffix`` :
Suffix to the user-agent searx uses to send requests to others engines. If an
engine wish to block you, a contact info here may be useful to avoid that.
``keepalive_expiry``:
Number of seconds to keep a connection in the pool. By default 5.0 seconds.
.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying
``proxies`` :
Define one or more proxies you wish to use, see `httpx proxies`_.
If there are more than one proxy for one protocol (http, https),
requests to the engines are distributed in a round-robin fashion.
``source_ips`` :
If you use multiple network interfaces, define from which IP the requests must
be made. Example:
* ``0.0.0.0`` any local IPv4 address.
* ``::`` any local IPv6 address.
* ``192.168.0.1``
* ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses
* ``fe80::60a2:1691:e5a2:ee1f``
* ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.
* ``[ 192.168.0.1, fe80::/126 ]``
``retries`` :
Number of retry in case of an HTTP error.
On each retry, searx uses an different proxy and source ip.
``retry_on_http_error`` :
Retry request on some HTTP status code.
Example:
* ``true`` : on HTTP status code between 400 and 599.
* ``403`` : on HTTP status code 403.
* ``[403, 429]``: on HTTP status code 403 and 429.
``enable_http2`` :
Enable by default. Set to ``False`` to disable HTTP/2.
``max_redirects`` :
30 by default. Maximum redirect before it is an error.
``locales:``
------------
.. code:: yaml
locales:
en : English
de : Deutsch
he : Hebrew
hu : Magyar
fr : Français
es : Español
it : Italiano
nl : Nederlands
ja : 日本語 (Japanese)
tr : Türkçe
ru : Russian
ro : Romanian
``locales`` :
Locales codes and their names. Available translations of searx interface.
.. _settings engine:
Engine settings
===============
.. sidebar:: Further reading ..
- :ref:`engines-dev`
.. code:: yaml
- name : bing
engine : bing
shortcut : bi
base_url : 'https://{language}.wikipedia.org/'
categories : general
timeout : 3.0
api_key : 'apikey'
disabled : True
language : en_US
#enable_http: False
#enable_http2: False
#retries: 1
#retry_on_http_error: True # or 403 or [404, 429]
#max_connections: 100
#max_keepalive_connections: 10
#keepalive_expiry: 5.0
#proxies:
# http:
# - http://proxy1:8080
# - http://proxy2:8080
# https:
# - http://proxy1:8080
# - http://proxy2:8080
# - socks5://user:password@proxy3:1080
# - socks5h://user:password@proxy4:1080
``name`` :
Name that will be used across searx to define this engine. In settings, on
the result page...
``engine`` :
Name of the python file used to handle requests and responses to and from this
search engine.
``shortcut`` :
Code used to execute bang requests (in this case using ``!bi`` or ``?bi``)
``base_url`` : optional
Part of the URL that should be stable across every request. Can be useful to
use multiple sites using only one engine, or updating the site URL without
touching at the code.
``categories`` : optional
Define in which categories this engine will be active. Most of the time, it is
defined in the code of the engine, but in a few cases it is useful, like when
describing multiple search engine using the same code.
``timeout`` : optional
Timeout of the search with the current search engine. **Be careful, it will
modify the global timeout of searx.**
``api_key`` : optional
In a few cases, using an API needs the use of a secret key. How to obtain them
is described in the file.
``disabled`` : optional
To disable by default the engine, but not deleting it. It will allow the user
to manually activate it in the settings.
``language`` : optional
If you want to use another language for a specific engine, you can define it
by using the full ISO code of language and country, like ``fr_FR``, ``en_US``,
``de_DE``.
``weigth`` : default ``1``
Weighting of the results of this engine.
``display_error_messages`` : default ``True``
When an engine returns an error, the message is displayed on the user interface.
``network``: optional
Use the network configuration from another engine.
In addition, there are two default networks:
* ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)
* ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)
.. note::
A few more options are possible, but they are pretty specific to some
engines, and so won't be described here.
.. _settings use_default_settings:
use_default_settings
====================
.. sidebar:: ``use_default_settings: True``
- :ref:`settings location`
- :ref:`use_default_settings.yml`
- :origin:`/etc/searx/settings.yml <utils/templates/etc/searx/use_default_settings.yml>`
The user defined ``settings.yml`` is loaded from the :ref:`settings location`
and can relied on the default configuration :origin:`searx/settings.yml` using:
``use_default_settings: True``
``server:``
In the following example, the actual settings are the default settings defined
in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and
the ``bind_address``:
.. code-block:: yaml
use_default_settings: True
server:
secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
bind_address: "0.0.0.0"
``engines:``
With ``use_default_settings: True``, each settings can be override in a
similar way, the ``engines`` section is merged according to the engine
``name``. In this example, searx will load all the engine and the arch linux
wiki engine has a :ref:`token<private engines>`:
.. code-block:: yaml
use_default_settings: True
server:
secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
``engines:`` / ``remove:``
It is possible to remove some engines from the default settings. The following
example is similar to the above one, but searx doesn't load the the google
engine:
.. code-block:: yaml
use_default_settings:
engines:
remove:
- google
server:
secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
``engines:`` / ``keep_only:``
As an alternative, it is possible to specify the engines to keep. In the
following example, searx has only two engines:
.. code-block:: yaml
use_default_settings:
engines:
keep_only:
- google
- duckduckgo
server:
secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
engines:
- name: google
tokens: ['$ecretValue']
- name: duckduckgo
tokens: ['$ecretValue']

59
src/searx/docs/admin/update-searx.rst
View File

@ -1,59 +0,0 @@
.. _update searx:
=============
How to update
=============
How to update depends on the :ref:`installation` method. If you have used the
:ref:`installation scripts`, use ``update`` command from the scripts.
**Update** :ref:`searx service <searx.sh>`
.. code:: sh
sudo -H ./utils/searx.sh update searx
**Update** :ref:`filtron reverse proxy <filtron.sh>`
.. code:: sh
sudo -H ./utils/filtron.sh update filtron
**Update** :ref:`result proxy <morty.sh>`
.. code:: bash
sudo -H ./utils/morty.sh update morty
.. _inspect searx:
======================
How to inspect & debug
======================
.. sidebar:: further read
- :ref:`toolboxing`
- :ref:`Makefile`
How to debug depends on the :ref:`installation` method. If you have used the
:ref:`installation scripts`, use ``inspect`` command from the scripts.
**Inspect** :ref:`searx service <searx.sh>`
.. code:: sh
sudo -H ./utils/searx.sh inspect service
**Inspect** :ref:`filtron reverse proxy <filtron.sh>`
.. code:: sh
sudo -H ./utils/filtron.sh inspect service
**Inspect** :ref:`result proxy <morty.sh>`
.. code:: bash
sudo -H ./utils/morty.sh inspect service

43
src/searx/docs/blog/admin.rst
View File

@ -1,43 +0,0 @@
=============================================================
Searx admin interface
=============================================================
.. _searx-admin: https://github.com/kvch/searx-admin#searx-admin
.. _NLnet Foundation: https://nlnet.nl/
manage your instance from your browser
.. sidebar:: Installation
Installation guide can be found in the repository of searx-admin_.
One of the biggest advantages of searx is being extremely customizable. But at
first it can be daunting to newcomers. A barrier of taking advantage of this
feature is our ugly settings file which is sometimes hard to understand and
edit.
To make self-hosting searx more accessible a new tool is introduced, called
``searx-admin``. It is a web application which is capable of managing your
instance and manipulating its settings via a web UI. It aims to replace editing
of ``settings.yml`` for less experienced administrators or people who prefer
graphical admin interfaces.
.. figure:: searx-admin-engines.png
:alt: Screenshot of engine list
Configuration page of engines
Since ``searx-admin`` acts as a supervisor for searx, we have decided to
implement it as a standalone tool instead of part of searx. Another reason for
making it a standalone tool is that the codebase and dependencies of searx
should not grow because of a fully optional feature, which does not affect
existing instances.
Acknowledgements
================
This development was sponsored by `NLnet Foundation`_.
| Happy hacking.
| kvch // 2017.08.22 21:25

65
src/searx/docs/blog/command-line-engines.rst
View File

@ -1,65 +0,0 @@
========================================
Running shell commands to fetch results
========================================
Previously, with searx you could search over the Internet on other people's
computers. Now it is possible to fetch results from your local machine without
connecting to any networks from the same graphical user interface.
Command line engines
====================
In :pull:`2128` a new type of engine has been introduced called ``command``.
This engine lets administrators add engines which run arbitrary shell commands
and show its output on the web UI of searx.
When creating and enabling a ``command`` engine on a public searx instance,
you must be careful to avoid leaking private data. The easiest solution
is to add tokens to the engine. Thus, only those who have the appropriate token
can retrieve results from the it.
The engine base is flexible. Only your imagination can limit the power of this engine. (And
maybe security concerns.) The following options are available:
* ``command``: A comma separated list of the elements of the command. A special token {{QUERY}} tells searx where to put the search terms of the user. Example: ``['ls', '-l', '-h', '{{QUERY}}']``
* ``delimiter``: A dict containing a delimiter char and the "titles" of each element in keys.
* ``parse_regex``: A dict containing the regular expressions for each result key.
* ``query_type``: The expected type of user search terms. Possible values: ``path`` and ``enum``. ``path`` checks if the uesr provided path is inside the working directory. If not the query is not executed. ``enum`` is a list of allowed search terms. If the user submits something which is not included in the list, the query returns an error.
* ``query_enum``: A list containing allowed search terms if ``query_type`` is set to ``enum``.
* ``working_dir``: The directory where the command has to be executed. Default: ``.``
* ``result_separator``: The character that separates results. Default: ``\n``
The example engine below can be used to find files with a specific name in the configured
working directory.
.. code:: yaml
- name: find
engine: command
command: ['find', '.', '-name', '{{QUERY}}']
query_type: path
shortcut: fnd
delimiter:
chars: ' '
keys: ['line']
Next steps
==========
In the next milestone, support for local search engines and indexers (e.g. Elasticsearch)
are going to be added. This way, you will be able to query your own databases/indexers.
Acknowledgement
===============
This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ .
.. _Search and Discovery Fund: https://nlnet.nl/discovery
.. _NLnet Foundation: https://nlnet.nl/
| Happy hacking.
| kvch // 2020.09.28 21:26

17
src/searx/docs/blog/index.rst
View File

@ -1,17 +0,0 @@
====
Blog
====
.. toctree::
:titlesonly:
:reversed:
python3
admin
intro-offline
private-engines
lxcdev-202006
command-line-engines
search-indexer-engines
sql-engines
search-database-engines

77
src/searx/docs/blog/intro-offline.rst
View File

@ -1,77 +0,0 @@
===============================
Preparation for offline engines
===============================
Offline engines
===============
To extend the functionality of searx, offline engines are going to be
introduced. An offline engine is an engine which does not need Internet
connection to perform a search and does not use HTTP to communicate.
Offline engines can be configured as online engines, by adding those to the
`engines` list of :origin:`settings.yml <searx/settings.yml>`. Thus, searx
finds the engine file and imports it.
Example skeleton for the new engines:
.. code:: python
from subprocess import PIPE, Popen
categories = ['general']
offline = True
def init(settings):
pass
def search(query, params):
process = Popen(['ls', query], stdout=PIPE)
return_code = process.wait()
if return_code != 0:
raise RuntimeError('non-zero return code', return_code)
results = []
line = process.stdout.readline()
while line:
result = parse_line(line)
results.append(results)
line = process.stdout.readline()
return results
Development progress
====================
First, a proposal has been created as a Github issue. Then it was moved to the
wiki as a design document. You can read it here: :wiki:`Offline-engines`.
In this development step, searx core was prepared to accept and perform offline
searches. Offline search requests are scheduled together with regular offline
requests.
As offline searches can return arbitrary results depending on the engine, the
current result templates were insufficient to present such results. Thus, a new
template is introduced which is caplable of presenting arbitrary key value pairs
as a table. You can check out the pull request for more details see
:pull:`1700`.
Next steps
==========
Today, it is possible to create/run an offline engine. However, it is going to be publicly available for everyone who knows the searx instance. So the next step is to introduce token based access for engines. This way administrators are able to limit the access to private engines.
Acknowledgement
===============
This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ .
.. _Search and Discovery Fund: https://nlnet.nl/discovery
.. _NLnet Foundation: https://nlnet.nl/
| Happy hacking.
| kvch // 2019.10.21 17:03

413
src/searx/docs/blog/lxcdev-202006.rst
View File

@ -1,413 +0,0 @@
.. _blog-lxcdev-202006:
=======================================
Developing in Linux containers [202006]
=======================================
.. _LXC: https://linuxcontainers.org/lxc/introduction/
.. sidebar:: Audience
This blog post is written for experienced admins and developers / readers
should have a serious meaning about: *distributed*, *merge* and *linux
container*.
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
In PR :PR:`1803` we added a lot of scripts to Searx's boilerplate. In this blog
post I will show you, how you can make use of them in *distributed and
heterogeneous development cycles* (TL;DR; jump to the :ref:`blog-lxcdev-202006
abstract`).
Motivation
==========
Normally in our development cycle, we edit the sources and run some test and/or
builds by using ``make`` before we commit. This cycle is simple and perfect but
might fail in some aspects we should not overlook.
The environment in which we run all our development processes matters!
The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but they
do not have access to all prerequisites. For example, there may have
dependencies on packages that are installed on the developer's desktop, but
usually are not preinstalled on a server or client system. Another examples
are; settings have been made to the software on the developer's host that would
never be set on a *production* system.
*Linux Containers* (LXC_) are isolate environments and not to mix up on
developer's all the prerequisites of all the projects he contribute to, is
always a good choice.
The scripts from PR :PR:`1803` can divide in those to install and maintain
software:
- :ref:`searx.sh`
- :ref:`filtron.sh`
- :ref:`morty.sh`
and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or
even development tasks over a stack of containers, what we call: *Searx's lxc
suite*.
Gentlemen, start your engines!
==============================
.. _LXD: https://linuxcontainers.org/lxd/introduction/
.. _archlinux: https://www.archlinux.org/
Before you can start with containers, you need to install and initiate LXD_
once:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ snap install lxd
$ lxd init --auto
And you need to clone from origin or if you have your own fork, clone from your
fork:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ cd ~/Downloads
$ git clone https://github.com/searx/searx.git
$ cd searx
The :ref:`lxc-searx.env` consists of several images, see ``export
LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post
we exercise on a archlinux_ image. The container of this image is named
``searx-archlinux``. Lets build the container, but be sure that this container
does not already exists, so first lets remove possible old one:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh remove searx-archlinux
$ sudo -H ./utils/lxc.sh build searx-archlinux
.. sidebar:: The ``searx-archlinux`` container
is the base of all our exercises here.
In this container we install all services :ref:`including searx, morty & filtron
<lxc.sh install suite>` in once:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh install suite searx-archlinux
To proxy HTTP from filtron and morty in the container to the outside of the
container, install nginx into the container. Once for the bot blocker filtron:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
./utils/filtron.sh nginx install
...
INFO: got 429 from http://10.174.184.156/searx
and once for the content sanitizer (content proxy morty):
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
./utils/morty.sh nginx install
...
INFO: got 200 from http://10.174.184.156/morty/
.. sidebar:: Fully functional searx suite
From here on you have a fully functional searx suite running with bot blocker
(filtron) and Web content sanitizer (content proxy morty) needed for a
*privacy protecting* search engine.
On your system, the IP of your ``searx-archlinux`` container differs from
http://10.174.184.156/searx, just open the URL reported in your installation
protocol in your WEB browser from the desktop to test the instance from outside
of the container.
In such a searx suite admins can maintain and access the debug log of the
different services quite easy.
.. _working in containers:
In containers, work as usual
============================
Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers
open the root-bash in the container using ``./utils/lxc.sh cmd
searx-archlinux``:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux bash
INFO: [searx-archlinux] bash
[root@searx-archlinux searx]# pwd
/share/searx
The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in
the searx-container. To debug the running searx instance use:
.. tabs::
.. group-tab:: root@searx-archlinux
.. code:: sh
$ ./utils/searx.sh inspect service
...
use [CTRL-C] to stop monitoring the log
...
Back in the browser on your desktop open the service http://10.174.184.156/searx
and run your application tests while the debug log is shown in the terminal from
above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug
option"* in searx's settings file and restarts the searx uwsgi application. To
debug services from filtron and morty analogous use:
.. tabs::
.. group-tab:: root@searx-archlinux
.. code:: sh
$ ./utils/filtron.sh inspect service
$ ./utils/morty.sh inspect service
Another point we have to notice is that each service (:ref:`searx <searx.sh>`,
:ref:`filtron <filtron.sh>` and :ref:`morty <morty.sh>`) runs under dedicated
system user account with the same name (compare :ref:`create searx user`). To
get a shell from theses accounts, simply call one of the scripts:
.. tabs::
.. group-tab:: root@searx-archlinux
.. code:: sh
$ ./utils/searx.sh shell
$ ./utils/filtron.sh shell
$ ./utils/morty.sh shell
To get in touch, open a shell from the service user (searx@searx-archlinux):
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
./utils/searx.sh shell
// exit with [CTRL-D]
(searx-pyenv) [searx@searx-archlinux ~]$ ...
The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system
user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv*
``(searx-pyenv)`` environment is activated.
.. tabs::
.. group-tab:: searx@searx-archlinux
.. code:: sh
(searx-pyenv) [searx@searx-archlinux ~]$ pwd
/usr/local/searx
Wrap production into developer suite
====================================
In this section we will see how to change the *"Fully functional searx suite"*
from a LXC container (which is quite ready for production) into a developer
suite. For this, we have to keep an eye on the :ref:`installation basic`:
- searx setup in: ``/etc/searx/settings.yml``
- searx user's home: ``/usr/local/searx``
- virtualenv in: ``/usr/local/searx/searx-pyenv``
- searx software in: ``/usr/local/searx/searx-src``
The searx software is a clone of the ``git_url`` (see :ref:`settings global`) and
the working tree is checked out from the ``git_branch``. With the use of the
:ref:`searx.sh` the searx service was installed as :ref:`uWSGI application
<searx uwsgi>`. To maintain this service, we can use ``systemctl`` (compare
:ref:`service architectures on distributions <uwsgi configuration>`).
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
systemctl stop uwsgi@searx
With the command above, we stopped the searx uWSGI-App in the archlinux
container.
The uWSGI-App for the archlinux dsitros is configured in
:origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at
least you should attend the settings of ``uid``, ``chdir``, ``env`` and
``http``::
env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml
http = 127.0.0.1:8888
chdir = /usr/local/searx/searx-src/searx
virtualenv = /usr/local/searx/searx-pyenv
pythonpath = /usr/local/searx/searx-src
If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that
each container shares the root folder of the repository and the command
``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the
searx installation into a developer one, we simple have to create a smylink to
the **transparent** reposetory from the desktop. Now lets replace the
repository at ``searx-src`` in the container with the working tree from outside
of the container:
.. tabs::
.. group-tab:: container becomes a developer suite
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
ln -s /share/searx/ /usr/local/searx/searx-src
Now we can develop as usual in the working tree of our desktop system. Every
time the software was changed, you have to restart the searx service (in the
conatiner):
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
systemctl restart uwsgi@searx
Remember: :ref:`working in containers` .. here are just some examples from my
daily usage:
.. tabs::
.. group-tab:: desktop
To *inspect* the searx instance (already described above):
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
./utils/searx.sh inspect service
Run :ref:`makefile`, e.g. to test inside the container:
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
make test
To install all prerequisites needed for a :ref:`buildhosts`:
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
./utils/searx.sh install buildhost
To build the docs on a buildhost :ref:`buildhosts`:
.. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
make docs.html
.. _blog-lxcdev-202006 abstract:
Abstract
========
We build up a fully functional searx suite in a archlinux container:
.. code:: sh
$ sudo -H ./utils/lxc.sh install suite searx-archlinux
To access HTTP from the desktop we installed nginx for the services inside the
conatiner:
.. tabs::
.. group-tab:: [root@searx-archlinux]
.. code:: sh
$ ./utils/filtron.sh nginx install
$ ./utils/morty.sh nginx install
To wrap the suite into a developer one, we created a symbolic link to the
repository which is shared **transparent** from the desktop's file system into
the container :
.. tabs::
.. group-tab:: [root@searx-archlinux]
.. code:: sh
$ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old
$ ln -s /share/searx/ /usr/local/searx/searx-src
$ systemctl restart uwsgi@searx
To get remarks from the suite of the archlinux container we can use:
.. tabs::
.. group-tab:: desktop
.. code:: sh
$ sudo -H ./utils/lxc.sh show suite searx-archlinux
...
[searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx
[searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/
[searx-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/
[searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65]
...

65
src/searx/docs/blog/private-engines.rst
View File

@ -1,65 +0,0 @@
==================================
Limit access to your searx engines
==================================
Administrators might find themselves wanting to limit access to some of the
enabled engines on their instances. It might be because they do not want to
expose some private information through an offline engine. Or they
would rather share engines only with their trusted friends or colleagues.
.. _private engines:
Private engines
===============
To solve this issue private engines were introduced in :pull:`1823`.
A new option was added to engines named `tokens`. It expects a list
of strings. If the user making a request presents one of the tokens
of an engine, they can access information about the engine
and make search requests.
Example configuration to restrict access to the Arch Linux Wiki engine:
.. code:: yaml
- name : arch linux wiki
engine : archlinux
shortcut : al
tokens : [ 'my-secret-token' ]
Unless a user has configured the right token, the engine is going
to be hidden from him/her. It is not going to be included in the
list of engines on the Preferences page and in the output of
`/config` REST API call.
Tokens can be added to one's configuration on the Preferences page
under "Engine tokens". The input expects a comma separated list of
strings.
The distribution of the tokens from the administrator to the users
is not carved in stone. As providing access to such engines
implies that the admin knows and trusts the user, we do not see
necessary to come up with a strict process. Instead,
we would like to add guidelines to the documentation of the feature.
Next steps
==========
Now that searx has support for both offline engines and private engines,
it is possible to add concrete engines which benefit from these features.
For example engines which search on the local host running the instance.
Be it searching your file system or querying a private database. Be creative
and come up with new solutions which fit your use case.
Acknowledgement
===============
This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ .
.. _Search and Discovery Fund: https://nlnet.nl/discovery
.. _NLnet Foundation: https://nlnet.nl/
| Happy hacking.
| kvch // 2020.02.28 22:26

68
src/searx/docs/blog/python3.rst
View File

@ -1,68 +0,0 @@
============================
Introducing Python 3 support
============================
.. _Python 2.7 clock: https://pythonclock.org/
.. sidebar:: Python 2.7 to 3 upgrade
This chapter exists of historical reasons. Python 2.7 release schedule ends
(`Python 2.7 clock`_) after 11 years Python 3 exists
As most operation systems are coming with Python3 installed by default. So it is
time for searx to support Python3. But don't worry support of Python2.7 won't be
dropped.
.. image:: searxpy3.png
:scale: 50 %
:alt: hurray
:align: center
How to run searx using Python 3
===============================
Please make sure that you run at least Python 3.5.
To run searx, first a Python3 virtualenv should be created. After entering the
virtualenv, dependencies must be installed. Then run searx with python3 instead
of the usual python command.
.. code:: sh
virtualenv -p python3 venv3
source venv3/bin/activate
pip3 install -r requirements.txt
python3 searx/webapp.py
If you want to run searx using Python2.7, you don't have to do anything
differently as before.
Fun facts
=========
- 115 files were changed when implementing the support for both Python versions.
- All of the dependencies was compatible except for the robotframework used for
browser tests. Thus, these tests were migrated to splinter. So from now on
both versions are being tested on Travis and can be tested locally.
If you found bugs
=================
Please open an issue on `GitHub`_. Make sure that you mention your Python
version in your issue, so we can investigate it properly.
.. _GitHub: https://github.com/searx/searx/issues
Acknowledgment
==============
This development was sponsored by `NLnet Foundation`_.
.. _NLnet Foundation: https://nlnet.nl/
| Happy hacking.
| kvch // 2017.05.13 22:57

95
src/searx/docs/blog/search-database-engines.rst
View File

@ -1,95 +0,0 @@
===============================
Query more of your NoSQL stores
===============================
From now on, searx lets you to query your NoSQL data stores:
* `Redis`_
* `MongoDB`_
The reference configuration of the engines are included ``settings.yml`` just commented out,
as you have to set various options and install dependencies before using them.
By default, the engines use ``key-value`` template for displaying results.
If you are not satisfied with the original result layout,
you can use your owm template by placing the template under
``searx/templates/{theme_name}/result_templates/{template_name}`` and setting
``result_template`` attribute to ``{template_name}``.
Futhermore, if you do not want to expose these engines on a public instance, you can
still add them and limit the access by setting ``tokens`` as described in the `blog post about
private engines`_.
Configuring searx to use the stores
===================================
NoSQL data stores are used for storing arbitrary data without first defining their
structure.
Redis
-----
Reqired package: ``redis``
Redis is a key value based data store usually stored in memory.
Select a database to search in and set its index in the option ``db``. You can
either look for exact matches or use partial keywords to find what you are looking for
by configuring ``exact_match_only``.
In this example you can search for exact matches in your first database:
.. code:: yaml
- name : mystore
engine : redis_server
exact_match_only : True
host : 127.0.0.1
port : 6379
password : secret-password
db : 0
shortcut : rds
enable_http : True
MongoDB
-------
Required package: ``pymongo``
MongoDB is a document based database program that handles JSON like data.
In order to query MongoDB, you have to select a ``database`` and a ``collection``. Furthermore,
you have to select a ``key`` that is going to be searched. MongoDB also supports the option ``exact_match_only``, so configure it
as you wish.
Above is an example configuration for using a MongoDB collection:
.. code:: yaml
- name : mymongo
engine : mongodb
shortcut : md
host : '127.0.0.1'
port : 27017
database : personal
collection : income
key : month
enable_http: True
Acknowledgement
===============
This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ .
.. _Redis: https://redis.io/
.. _MongoDB: https://mongodb.com/
.. _blog post about private engines: private-engines.html#private-engines
.. _Search and Discovery Fund: https://nlnet.nl/discovery
.. _NLnet Foundation: https://nlnet.nl/
| Happy hacking.
| kvch // 2021.07.13 23:16

114
src/searx/docs/blog/search-indexer-engines.rst
View File

@ -1,114 +0,0 @@
===============================
Query your local search engines
===============================
From now on, searx lets you to query your locally running search engines. The following
ones are supported now:
* `Elasticsearch`_
* `Meilisearch`_
* `Solr`_
All of the engines above are added to ``settings.yml`` just commented out, as you have to
``base_url`` for all them.
Please note that if you are not using HTTPS to access these engines, you have to enable
HTTP requests by setting ``enable_http`` to ``True``.
Futhermore, if you do not want to expose these engines on a public instance, you can
still add them and limit the access by setting ``tokens`` as described in the `blog post about
private engines`_.
Configuring searx for search engines
====================================
Each search engine is powerful, capable of full-text search.
Elasticsearch
-------------
Elasticsearch supports numerous ways to query the data it is storing. At the moment
the engine supports the most popular search methods: ``match``, ``simple_query_string``, ``term`` and ``terms``.
If none of the methods fit your use case, you can select ``custom`` query type and provide the JSON payload
searx has to submit to Elasticsearch in ``custom_query_json``.
The following is an example configuration for an Elasticsearch instance with authentication
configured to read from ``my-index`` index.
.. code:: yaml
- name : elasticsearch
shortcut : es
engine : elasticsearch
base_url : http://localhost:9200
username : elastic
password : changeme
index : my-index
query_type : match
enable_http : True
Meilisearch
-----------
This search engine is aimed at individuals and small companies. It is designed for
small-scale (less than 10 million documents) data collections. E.g. it is great for storing
web pages you have visited and searching in the contents later.
The engine supports faceted search, so you can search in a subset of documents of the collection.
Futhermore, you can search in Meilisearch instances that require authentication by setting ``auth_token``.
Here is a simple example to query a Meilisearch instance:
.. code:: yaml
- name : meilisearch
engine : meilisearch
shortcut: mes
base_url : http://localhost:7700
index : my-index
enable_http: True
Solr
----
Solr is a popular search engine based on Lucene, just like Elasticsearch.
But instead of searching in indices, you can search in collections.
This is an example configuration for searching in the collection ``my-collection`` and get
the results in ascending order.
.. code:: yaml
- name : solr
engine : solr
shortcut : slr
base_url : http://localhost:8983
collection : my-collection
sort : asc
enable_http : True
Next steps
==========
The next step is to add support for various SQL databases.
Acknowledgement
===============
This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ .
.. _blog post about private engines: private-engines.html#private-engines
.. _Elasticsearch: https://www.elastic.co/elasticsearch/
.. _Meilisearch: https://www.meilisearch.com/
.. _Solr: https://solr.apache.org/
.. _Search and Discovery Fund: https://nlnet.nl/discovery
.. _NLnet Foundation: https://nlnet.nl/
| Happy hacking.
| kvch // 2021.04.07 23:16

BIN
src/searx/docs/blog/searx-admin-engines.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 50 KiB

BIN
src/searx/docs/blog/searxpy3.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 30 KiB

117
src/searx/docs/blog/sql-engines.rst
View File

@ -1,117 +0,0 @@
=================
Query SQL servers
=================
Now you can query SQL servers using searx. The following ones are supported:
* `PostgreSQL`_
* `MySQL`_
* `SQLite`_
All of the engines above are added to ``settings.yml`` just commented out, as you have to
set the required attributes for the engines, e.g. ``database``. By default, the engines use
``key-value`` template for displaying results. If you are not satisfied with the original result layout,
you can use your owm template by placing the template under
``searx/templates/{theme_name}/result_templates/{template_name}`` and setting
``result_template`` attribute to ``{template_name}``.
As mentioned in previous blog posts, if you do not wish to expose these engines on a
public instance, you can still add them and limit the access by setting ``tokens``
as described in the `blog post about private engines`_.
Configure the engines
=====================
The configuration of the new database engines are similar. You must put a valid
SELECT SQL query in ``query_str``. At the moment you can only bind at most
one parameter in your query. By setting the attribute ``limit`` you can
define how many results you want from the SQL server. Basically, it
is the same as the LIMIT keyword in SQL.
Please, do not include LIMIT or OFFSET in your SQL query as the engines
rely on these keywords during paging. If you want to configure the number of returned results
use the option ``limit``.
PostgreSQL
----------
PostgreSQL is a powerful and robust open source database.
Before configuring the PostgreSQL engine, you must install the dependency ``psychopg2``.
You can find an example configuration below:
.. code:: yaml
- name : postgresql
engine : postgresql
database : my_database
username : searx
password : password
query_str : 'SELECT * from my_table WHERE my_column = %(query)s'
shortcut : psql
MySQL
-----
MySQL is said to be the most popular open source database.
Before enabling MySQL engine, you must install the package ``mysql-connector-python``.
The authentication plugin is configurable by setting ``auth_plugin`` in the attributes.
By default it is set to ``caching_sha2_password``.
This is an example configuration for quering a MySQL server:
.. code:: yaml
- name : mysql
engine : mysql_server
database : my_database
username : searx
password : password
limit : 5
query_str : 'SELECT * from my_table WHERE my_column=%(query)s'
shortcut : mysql
SQLite
------
SQLite is a small, fast and reliable SQL database engine. It does not require
any extra dependency.
You can read from your database ``my_database`` using this example configuration:
.. code:: yaml
- name : sqlite
engine : sqlite
shortcut: sq
database : my_database
query_str : 'SELECT * FROM my_table WHERE my_column=:query'
Next steps
==========
The next step is to add support for more data stores, e.g. Redis and MongoDB.
Acknowledgement
===============
This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ .
.. _PostgreSQL: https://www.postgresql.org/
.. _MySQL: https://www.mysql.com/
.. _SQLite: https://www.sqlite.org/index.html
.. _blog post about private engines: private-engines.html#private-engines
.. _Search and Discovery Fund: https://nlnet.nl/discovery
.. _NLnet Foundation: https://nlnet.nl/
| Happy hacking.
| kvch // 2021.05.23 23:16

210
src/searx/docs/build-templates/searx.rst
View File

@ -1,210 +0,0 @@
.. template evaluated by: ./utils/searx.sh docs
.. hint: all dollar-names are variables, dollar sign itself is quoted by: \\$
.. START distro-packages
.. tabs::
.. group-tab:: Ubuntu / debian
.. code-block:: sh
$ sudo -H apt-get install -y \\
${debian}
.. group-tab:: Arch Linux
.. code-block:: sh
$ sudo -H pacman -S --noconfirm \\
${arch}
.. group-tab:: Fedora / RHEL
.. code-block:: sh
$ sudo -H dnf install -y \\
${fedora}
.. END distro-packages
.. START build-packages
.. tabs::
.. group-tab:: Ubuntu / debian
.. code-block:: sh
$ sudo -H apt-get install -y \\
${debian_build}
.. group-tab:: Arch Linux
.. code-block:: sh
$ sudo -H pacman -S --noconfirm \\
${arch_build}
.. group-tab:: Fedora / RHEL
.. code-block:: sh
$ sudo -H dnf install -y \\
${fedora_build}
.. END build-packages
.. START create user
.. tabs::
.. group-tab:: bash
.. code-block:: sh
$ sudo -H useradd --shell /bin/bash --system \\
--home-dir \"$SERVICE_HOME\" \\
--comment 'Privacy-respecting metasearch engine' $SERVICE_USER
$ sudo -H mkdir \"$SERVICE_HOME\"
$ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\"
.. END create user
.. START clone searx
.. tabs::
.. group-tab:: bash
.. code-block:: sh
$ sudo -H -u ${SERVICE_USER} -i
(${SERVICE_USER})$ git clone \"https://github.com/searx/searx.git\" \"$SEARX_SRC\"
.. END clone searx
.. START create virtualenv
.. tabs::
.. group-tab:: bash
.. code-block:: sh
(${SERVICE_USER})$ python3 -m venv \"${SEARX_PYENV}\"
(${SERVICE_USER})$ echo \". ${SEARX_PYENV}/bin/activate\" >> \"$SERVICE_HOME/.profile\"
.. END create virtualenv
.. START manage.sh update_packages
.. tabs::
.. group-tab:: bash
.. code-block:: sh
$ sudo -H -u ${SERVICE_USER} -i
(${SERVICE_USER})$ command -v python && python --version
$SEARX_PYENV/bin/python
Python 3.8.1
# update pip's boilerplate ..
pip install -U pip
pip install -U setuptools
pip install -U wheel
pip install -U pyyaml
# jump to searx's working tree and install searx into virtualenv
(${SERVICE_USER})$ cd \"$SEARX_SRC\"
(${SERVICE_USER})$ pip install -e .
.. END manage.sh update_packages
.. START searx config
.. tabs::
.. group-tab:: Use default settings
.. code-block:: sh
$ sudo -H mkdir -p \"$(dirname ${SEARX_SETTINGS_PATH})\"
$ sudo -H cp \"$SEARX_SRC/utils/templates/etc/searx/use_default_settings.yml\" \\
\"${SEARX_SETTINGS_PATH}\"
.. group-tab:: searx/settings.yml
.. code-block:: sh
$ sudo -H mkdir -p \"$(dirname ${SEARX_SETTINGS_PATH})\"
$ sudo -H cp \"$SEARX_SRC/searx/settings.yml\" \\
\"${SEARX_SETTINGS_PATH}\"
.. tabs::
.. group-tab:: minimal setup
.. code-block:: sh
$ sudo -H sed -i -e \"s/ultrasecretkey/\$(openssl rand -hex 16)/g\" \"$SEARX_SETTINGS_PATH\"
$ sudo -H sed -i -e \"s/{instance_name}/searx@\$(uname -n)/g\" \"$SEARX_SETTINGS_PATH\"
.. END searx config
.. START check searx installation
.. tabs::
.. group-tab:: bash
.. code-block:: sh
# enable debug ..
$ sudo -H sed -i -e \"s/debug : False/debug : True/g\" \"$SEARX_SETTINGS_PATH\"
# start webapp
$ sudo -H -u ${SERVICE_USER} -i
(${SERVICE_USER})$ cd ${SEARX_SRC}
(${SERVICE_USER})$ export SEARX_SETTINGS_PATH=\"${SEARX_SETTINGS_PATH}\"
(${SERVICE_USER})$ python searx/webapp.py
# disable debug
$ sudo -H sed -i -e \"s/debug : True/debug : False/g\" \"$SEARX_SETTINGS_PATH\"
Open WEB browser and visit http://$SEARX_INTERNAL_URL . If you are inside a
container or in a script, test with curl:
.. tabs::
.. group-tab:: WEB browser
.. code-block:: sh
$ xdg-open http://$SEARX_INTERNAL_URL
.. group-tab:: curl
.. code-block:: none
$ curl --location --verbose --head --insecure $SEARX_INTERNAL_URL
* Trying 127.0.0.1:8888...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> HEAD / HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
HTTP/1.0 200 OK
...
.. END check searx installation

131
src/searx/docs/conf.py
View File

@ -1,131 +0,0 @@
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: AGPL-3.0-or-later
import sys, os
from pallets_sphinx_themes import ProjectLink
from searx import brand
from searx.version import VERSION_STRING
# Project --------------------------------------------------------------
project = u'searx'
copyright = u'2015-2021, Adam Tauber, Noémi Ványi'
author = u'Adam Tauber'
release, version = VERSION_STRING, VERSION_STRING
highlight_language = 'none'
# General --------------------------------------------------------------
master_doc = "index"
source_suffix = '.rst'
numfig = True
exclude_patterns = ['build-templates/*.rst']
import searx.search
import searx.engines
import searx.plugins
searx.search.initialize()
jinja_contexts = {
'searx': {
'engines': searx.engines.engines,
'plugins': searx.plugins.plugins
},
}
# usage:: lorem :patch:`f373169` ipsum
extlinks = {}
# upstream links
extlinks['wiki'] = ('https://github.com/searx/searx/wiki/%s', ' ')
extlinks['pull'] = ('https://github.com/searx/searx/pull/%s', 'PR ')
# links to custom brand
extlinks['origin'] = (brand.GIT_URL + '/blob/' + brand.GIT_BRANCH + '/%s', 'git://')
extlinks['patch'] = (brand.GIT_URL + '/commit/%s', '#')
extlinks['search'] = (brand.SEARX_URL + '/%s', '#')
extlinks['docs'] = (brand.DOCS_URL + '/%s', 'docs: ')
extlinks['pypi'] = ('https://pypi.org/project/%s', 'PyPi: ')
extlinks['man'] = ('https://manpages.debian.org/jump?q=%s', '')
#extlinks['role'] = (
# 'https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-%s', '')
extlinks['duref'] = (
'https://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#%s', '')
extlinks['durole'] = (
'https://docutils.sourceforge.net/docs/ref/rst/roles.html#%s', '')
extlinks['dudir'] = (
'https://docutils.sourceforge.net/docs/ref/rst/directives.html#%s', '')
extlinks['ctan'] = (
'https://ctan.org/pkg/%s', 'CTAN: ')
extensions = [
'sphinx.ext.imgmath',
'sphinx.ext.extlinks',
'sphinx.ext.viewcode',
"sphinx.ext.autodoc",
"sphinx.ext.intersphinx",
"pallets_sphinx_themes",
"sphinx_issues", # https://github.com/sloria/sphinx-issues/blob/master/README.rst
"sphinxcontrib.jinja", # https://github.com/tardyp/sphinx-jinja
"sphinxcontrib.programoutput", # https://github.com/NextThought/sphinxcontrib-programoutput
'linuxdoc.kernel_include', # Implementation of the 'kernel-include' reST-directive.
'linuxdoc.rstFlatTable', # Implementation of the 'flat-table' reST-directive.
'linuxdoc.kfigure', # Sphinx extension which implements scalable image handling.
"sphinx_tabs.tabs", # https://github.com/djungelorm/sphinx-tabs
]
intersphinx_mapping = {
"python": ("https://docs.python.org/3/", None),
"flask": ("https://flask.palletsprojects.com/", None),
# "werkzeug": ("https://werkzeug.palletsprojects.com/", None),
"jinja": ("https://jinja.palletsprojects.com/", None),
"linuxdoc" : ("https://return42.github.io/linuxdoc/", None),
"sphinx" : ("https://www.sphinx-doc.org/en/master/", None),
}
issues_github_path = "searx/searx"
# HTML -----------------------------------------------------------------
sys.path.append(os.path.abspath('_themes'))
sys.path.insert(0, os.path.abspath("../utils/"))
html_theme_path = ['_themes']
html_theme = "searx"
# sphinx.ext.imgmath setup
html_math_renderer = 'imgmath'
imgmath_image_format = 'svg'
imgmath_font_size = 14
# sphinx.ext.imgmath setup END
html_theme_options = {"index_sidebar_logo": True}
html_context = {"project_links": [] }
html_context["project_links"].append(ProjectLink("Blog", "blog/index.html"))
if brand.GIT_URL:
html_context["project_links"].append(ProjectLink("Source", brand.GIT_URL))
if brand.WIKI_URL:
html_context["project_links"].append(ProjectLink("Wiki", brand.WIKI_URL))
if brand.PUBLIC_INSTANCES:
html_context["project_links"].append(ProjectLink("Public instances", brand.PUBLIC_INSTANCES))
if brand.TWITTER_URL:
html_context["project_links"].append(ProjectLink("Twitter", brand.TWITTER_URL))
if brand.ISSUE_URL:
html_context["project_links"].append(ProjectLink("Issue Tracker", brand.ISSUE_URL))
if brand.CONTACT_URL:
html_context["project_links"].append(ProjectLink("Contact", brand.CONTACT_URL))
html_sidebars = {
"**": ["project.html", "relations.html", "searchbox.html"],
}
singlehtml_sidebars = {"index": ["project.html", "localtoc.html"]}
html_static_path = ["static"]
html_logo = "static/img/searx_logo_small.png"
html_title = "Searx Documentation ({})".format("Searx-{}.tex".format(VERSION_STRING))
html_show_sourcelink = False
# LaTeX ----------------------------------------------------------------
latex_documents = [
(master_doc, "searx-{}.tex".format(VERSION_STRING), html_title, author, "manual")
]

189
src/searx/docs/dev/contribution_guide.rst
View File

@ -1,189 +0,0 @@
.. _how to contribute:
=================
How to contribute
=================
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
Prime directives: Privacy, Hackability
======================================
Searx has two prime directives, **privacy-by-design and hackability** . The
hackability comes in three levels:
- support of search engines
- plugins to alter search behaviour
- hacking searx itself
Note the lack of "world domination" among the directives. Searx has no
intention of wide mass-adoption, rounded corners, etc. The prime directive
"privacy" deserves a separate chapter, as it's quite uncommon unfortunately.
Privacy-by-design
-----------------
Searx was born out of the need for a **privacy-respecting** search tool which
can be extended easily to maximize both, its search and its privacy protecting
capabilities.
A few widely used features work differently or turned off by default or not
implemented at all **as a consequence of privacy-by-design**.
If a feature reduces the privacy preserving aspects of searx, it should be
switched off by default or should not implemented at all. There are plenty of
search engines already providing such features. If a feature reduces the
protection of searx, users must be informed about the effect of choosing to
enable it. Features that protect privacy but differ from the expectations of
the user should also be explained.
Also, if you think that something works weird with searx, it's might be because
of the tool you use is designed in a way to interfere with the privacy respect.
Submitting a bugreport to the vendor of the tool that misbehaves might be a good
feedback to reconsider the disrespect to its customers (e.g. ``GET`` vs ``POST``
requests in various browsers).
Remember the other prime directive of searx is to be hackable, so if the above
privacy concerns do not fancy you, simply fork it.
*Happy hacking.*
Code
====
.. _PEP8: https://www.python.org/dev/peps/pep-0008/
.. _Conventional Commits: https://www.conventionalcommits.org/
.. _Git Commit Good Practice: https://wiki.openstack.org/wiki/GitCommitMessages
.. _Structural split of changes:
https://wiki.openstack.org/wiki/GitCommitMessages#Structural_split_of_changes
.. _gitmoji: https://gitmoji.carloscuesta.me/
.. _Semantic PR: https://github.com/zeke/semantic-pull-requests
.. sidebar:: Create good commits!
- `Structural split of changes`_
- `Conventional Commits`_
- `Git Commit Good Practice`_
- some like to use: gitmoji_
- not yet active: `Semantic PR`_
In order to submit a patch, please follow the steps below:
- Follow coding conventions.
- PEP8_ standards apply, except the convention of line length
- Maximum line length is 120 characters
- The cardinal rule for creating good commits is to ensure there is only one
*logical change* per commit / read `Structural split of changes`_
- Check if your code breaks existing tests. If so, update the tests or fix your
code.
- If your code can be unit-tested, add unit tests.
- Add yourself to the :origin:`AUTHORS.rst` file.
- Choose meaningful commit messages, read `Conventional Commits`_
.. code::
<type>[optional scope]: <description>
[optional body]
[optional footer(s)]
- Create a pull request.
For more help on getting started with searx development, see :ref:`devquickstart`.
Translation
===========
Translation currently takes place on :ref:`transifex <translation>`.
.. caution::
Please, do not update translation files in the repo.
.. _contrib docs:
Documentation
=============
.. _Sphinx: https://www.sphinx-doc.org
.. _reST: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
.. sidebar:: The reST sources
has been moved from ``gh-branch`` into ``master`` (:origin:`docs`).
The documentation is built using Sphinx_. So in order to be able to generate
the required files, you have to install it on your system. Much easier, use
our :ref:`makefile`.
Here is an example which makes a complete rebuild:
.. code:: sh
$ make docs.clean docs.html
...
The HTML pages are in dist/docs.
.. _make docs.live:
live build
----------
.. _sphinx-autobuild:
https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md
.. sidebar:: docs.clean
It is recommended to assert a complete rebuild before deploying (use
``docs.clean``).
Live build is like WYSIWYG. If you want to edit the documentation, its
recommended to use. The Makefile target ``docs.live`` builds the docs, opens
URL in your favorite browser and rebuilds every time a reST file has been
changed.
.. code:: sh
$ make docs.live
...
The HTML pages are in dist/docs.
... Serving on http://0.0.0.0:8000
... Start watching changes
Live builds are implemented by sphinx-autobuild_. Use environment
``$(SPHINXOPTS)`` to pass arguments to the sphinx-autobuild_ command. Except
option ``--host`` (which is always set to ``0.0.0.0``) you can pass any
argument. E.g to find and use a free port, use:
.. code:: sh
$ SPHINXOPTS="--port 0" make docs.live
...
... Serving on http://0.0.0.0:50593
...
.. _deploy on github.io:
deploy on github.io
-------------------
To deploy documentation at :docs:`github.io <.>` use Makefile target :ref:`make
docs.gh-pages`, which builds the documentation and runs all the needed git add,
commit and push:
.. code:: sh
$ make docs.clean docs.gh-pages

306
src/searx/docs/dev/engine_overview.rst
View File

@ -1,306 +0,0 @@
.. _engines-dev:
===============
Engine overview
===============
.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
searx is a metasearch-engine_, so it uses different search engines to provide
better results.
Because there is no general search API which could be used for every search
engine, an adapter has to be built between searx and the external search
engines. Adapters are stored under the folder :origin:`searx/engines`.
.. contents::
:depth: 3
:backlinks: entry
.. _general engine configuration:
general engine configuration
============================
It is required to tell searx the type of results the engine provides. The
arguments can be set in the engine file or in the settings file
(normally ``settings.yml``). The arguments in the settings file override
the ones in the engine file.
It does not matter if an option is stored in the engine file or in the
settings. However, the standard way is the following:
.. _engine file:
engine file
-----------
======================= =========== ========================================================
argument type information
======================= =========== ========================================================
categories list pages, in which the engine is working
paging boolean support multible pages
time_range_support boolean support search time range
engine_type str ``online`` by default, other possibles values are
``offline``, ``online_dictionnary``, ``online_currency``
======================= =========== ========================================================
.. _engine settings:
settings.yml
------------
======================= =========== =============================================
argument type information
======================= =========== =============================================
name string name of search-engine
engine string name of searx-engine
(filename without ``.py``)
enable_http bool enable HTTP
(by default only HTTPS is enabled).
shortcut string shortcut of search-engine
timeout string specific timeout for search-engine
display_error_messages boolean display error messages on the web UI
proxies dict set proxies for a specific engine
(e.g. ``proxies : {http: socks5://proxy:port,
https: socks5://proxy:port}``)
======================= =========== =============================================
overrides
---------
A few of the options have default values in the engine, but are often
overwritten by the settings. If ``None`` is assigned to an option in the engine
file, it has to be redefined in the settings, otherwise searx will not start
with that engine.
The naming of overrides is arbitrary. But the recommended overrides are the
following:
======================= =========== ===========================================
argument type information
======================= =========== ===========================================
base_url string base-url, can be overwritten to use same
engine on other URL
number_of_results int maximum number of results per request
language string ISO code of language and country like en_US
api_key string api-key if required by engine
======================= =========== ===========================================
example code
------------
.. code:: python
# engine dependent config
categories = ['general']
paging = True
making a request
================
To perform a search an URL have to be specified. In addition to specifying an
URL, arguments can be passed to the query.
passed arguments
----------------
These arguments can be used to construct the search query. Furthermore,
parameters with default value can be redefined for special purposes.
If the ``engine_type`` is ``online```:
====================== ============== ========================================================================
argument type default-value, information
====================== ============== ========================================================================
url str ``''``
method str ``'GET'``
headers set ``{}``
data set ``{}``
cookies set ``{}``
verify bool ``True``
headers.User-Agent str a random User-Agent
category str current category, like ``'general'``
safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict)
time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year``
pageno int current pagenumber
language str specific language code like ``'en_US'``, or ``'all'`` if unspecified
====================== ============== ========================================================================
If the ``engine_type`` is ``online_dictionnary```, in addition to the ``online`` arguments:
====================== ============ ========================================================================
argument type default-value, information
====================== ============ ========================================================================
from_lang str specific language code like ``'en_US'``
to_lang str specific language code like ``'en_US'``
query str the text query without the languages
====================== ============ ========================================================================
If the ``engine_type`` is ``online_currency```, in addition to the ``online`` arguments:
====================== ============ ========================================================================
argument type default-value, information
====================== ============ ========================================================================
amount float the amount to convert
from str ISO 4217 code
to str ISO 4217 code
from_name str currency name
to_name str currency name
====================== ============ ========================================================================
parsed arguments
----------------
The function ``def request(query, params):`` always returns the ``params``
variable. Inside searx, the following paramters can be used to specify a search
request:
=================== =========== ==========================================================================
argument type information
=================== =========== ==========================================================================
url str requested url
method str HTTP request method
headers set HTTP header information
data set HTTP data information
cookies set HTTP cookies
verify bool Performing SSL-Validity check
allow_redirects bool Follow redirects
max_redirects int maximum redirects, hard limit
soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine
raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300
=================== =========== ==========================================================================
example code
------------
.. code:: python
# search-url
base_url = 'https://example.com/'
search_string = 'search?{query}&page={page}'
# do search-request
def request(query, params):
search_path = search_string.format(
query=urlencode({'q': query}),
page=params['pageno'])
params['url'] = base_url + search_path
return params
returned results
================
Searx is able to return results of different media-types. Currently the
following media-types are supported:
- default_
- images_
- videos_
- torrent_
- map_
To set another media-type as default, the parameter ``template`` must be set to
the desired type.
default
-------
========================= =====================================================
result-parameter information
========================= =====================================================
url string, url of the result
title string, title of the result
content string, general result-text
publishedDate :py:class:`datetime.datetime`, time of publish
========================= =====================================================
images
------
To use this template, the parameter:
========================= =====================================================
result-parameter information
========================= =====================================================
template is set to ``images.html``
url string, url to the result site
title string, title of the result *(partly implemented)*
content *(partly implemented)*
publishedDate :py:class:`datetime.datetime`,
time of publish *(partly implemented)*
img\_src string, url to the result image
thumbnail\_src string, url to a small-preview image
========================= =====================================================
videos
------
========================= =====================================================
result-parameter information
========================= =====================================================
template is set to ``videos.html``
url string, url of the result
title string, title of the result
content *(not implemented yet)*
publishedDate :py:class:`datetime.datetime`, time of publish
thumbnail string, url to a small-preview image
========================= =====================================================
torrent
-------
.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
========================= =====================================================
result-parameter information
========================= =====================================================
template is set to ``torrent.html``
url string, url of the result
title string, title of the result
content string, general result-text
publishedDate :py:class:`datetime.datetime`,
time of publish *(not implemented yet)*
seed int, number of seeder
leech int, number of leecher
filesize int, size of file in bytes
files int, number of files
magnetlink string, magnetlink_ of the result
torrentfile string, torrentfile of the result
========================= =====================================================
map
---
========================= =====================================================
result-parameter information
========================= =====================================================
url string, url of the result
title string, title of the result
content string, general result-text
publishedDate :py:class:`datetime.datetime`, time of publish
latitude latitude of result (in decimal format)
longitude longitude of result (in decimal format)
boundingbox boundingbox of result (array of 4. values
``[lat-min, lat-max, lon-min, lon-max]``)
geojson geojson of result (https://geojson.org/)
osm.type type of osm-object (if OSM-Result)
osm.id id of osm-object (if OSM-Result)
address.name name of object
address.road street name of object
address.house_number house number of object
address.locality city, place of object
address.postcode postcode of object
address.country country of object
========================= =====================================================

16
src/searx/docs/dev/index.rst
View File

@ -1,16 +0,0 @@
=======================
Developer documentation
=======================
.. toctree::
:maxdepth: 2
:caption: Contents
quickstart
contribution_guide
engine_overview
search_api
plugins
translation
makefile
reST

222
src/searx/docs/dev/makefile.rst
View File

@ -1,222 +0,0 @@
.. _makefile:
========
Makefile
========
.. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction
.. sidebar:: build environment
Before looking deeper at the targets, first read about :ref:`make
install`.
To install system requirements follow :ref:`buildhosts`.
All relevant build tasks are implemented in :origin:`manage.sh` and for CI or
IDE integration a small ``Makefile`` wrapper is available. If you are not
familiar with Makefiles, we recommend to read gnu-make_ introduction.
The usage is simple, just type ``make {target-name}`` to *build* a target.
Calling the ``help`` target gives a first overview (``make help``):
.. program-output:: bash -c "cd ..; make --no-print-directory help"
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. _make install:
Python environment
==================
.. sidebar:: activate environment
``source ./local/py3/bin/activate``
We do no longer need to build up the virtualenv manually. Jump into your git
working tree and release a ``make install`` to get a virtualenv with a
*developer install* of searx (:origin:`setup.py`). ::
$ cd ~/searx-clone
$ make install
PYENV [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV OK
PYENV [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
If you release ``make install`` multiple times the installation will only
rebuild if the sha256 sum of the *requirement files* fails. With other words:
the check fails if you edit the requirements listed in
:origin:`requirements-dev.txt` and :origin:`requirements.txt`). ::
$ make install
PYENV OK
PYENV [virtualenv] requirements.sha256 failed
[virtualenv] - 6cea6eb6def9e14a18bf32f8a3e... ./requirements-dev.txt
[virtualenv] - 471efef6c73558e391c3adb35f4... ./requirements.txt
...
PYENV [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV OK
PYENV [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
.. sidebar:: drop environment
To get rid of the existing environment before re-build use :ref:`clean target
<make clean>` first.
If you think, something goes wrong with your ./local environment or you change
the :origin:`setup.py` file, you have to call :ref:`make clean`.
.. _make run:
``make run``
============
To get up a running a developer instance simply call ``make run``. This enables
*debug* option in :origin:`searx/settings.yml`, starts a ``./searx/webapp.py``
instance, disables *debug* option again and opens the URL in your favorite WEB
browser (:man:`xdg-open`)::
$ make run
PYENV OK
SEARX_DEBUG=1 ./manage.sh pyenv.cmd python ./searx/webapp.py
...
INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
.. _make clean:
``make clean``
==============
Drop all intermediate files, all builds, but keep sources untouched. Before
calling ``make clean`` stop all processes using :ref:`make install`. ::
$ make clean
CLEAN pyenv
PYENV [virtualenv] drop ./local/py3
CLEAN docs -- ./build/docs ./dist/docs
CLEAN locally installed npm dependencies
CLEAN test stuff
CLEAN common files
.. _make docs:
``make docs docs.autobuild docs.clean``
=======================================
We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute /
Documentation <contrib docs>` section. If you want to edit the documentation
read our :ref:`make docs.live` section. If you are working in your own brand,
adjust your :ref:`settings global`.
.. _make docs.gh-pages:
``make docs.gh-pages``
======================
To deploy on github.io first adjust your :ref:`settings global`. For any
further read :ref:`deploy on github.io`.
.. _make test:
``make test``
=============
Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit``
and ``test.robot``. You can run tests selective, e.g.::
$ make test.pep8 test.unit test.sh
TEST test.pep8 OK
...
TEST test.unit OK
...
TEST test.sh OK
.. _make test.sh:
``make test.sh``
================
:ref:`sh lint` / if you have changed some bash scripting run this test before
commit.
.. _make test.pylint:
``make test.pylint``
====================
.. _Pylint: https://www.pylint.org/
Pylint_ is known as one of the best source-code, bug and quality checker for the
Python programming language. The pylint profile we use at searx project is
found in project's root folder :origin:`.pylintrc`.
.. _make search.checker:
``search.checker.{engine name}``
================================
To check all engines::
make search.checker
To check a engine with whitespace in the name like *google news* replace space
by underline::
make search.checker.google_news
To see HTTP requests and more use SEARX_DEBUG::
make SEARX_DEBUG=1 search.checker.google_news
.. _3xx: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_redirection
To filter out HTTP redirects (3xx_)::
make SEARX_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]"
...
Engine google news Checking
https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
``make pybuild``
================
.. _PyPi: https://pypi.org/
.. _twine: https://twine.readthedocs.io/en/latest/
Build Python packages in ``./dist/py``::
$ make pybuild
...
BUILD pybuild
running sdist
running egg_info
...
running bdist_wheel
$ ls ./dist
searx-0.18.0-py3-none-any.whl searx-0.18.0.tar.gz
To upload packages to PyPi_, there is also a ``pypi.upload`` target (to test use
``pypi.upload.test``). Since you are not the owner of :pypi:`searx` you will
never need to upload.

69
src/searx/docs/dev/plugins.rst
View File

@ -1,69 +0,0 @@
.. _dev plugin:
=======
Plugins
=======
.. sidebar:: Further reading ..
- :ref:`plugins generic`
Plugins can extend or replace functionality of various components of searx.
Example plugin
==============
.. code:: python
name = 'Example plugin'
description = 'This plugin extends the suggestions with the word "example"'
default_on = False # disabled by default
js_dependencies = tuple() # optional, list of static js files
css_dependencies = tuple() # optional, list of static css files
# attach callback to the post search hook
# request: flask request object
# ctx: the whole local context of the post search hook
def post_search(request, ctx):
ctx['search'].suggestions.add('example')
return True
External plugins
================
External plugins are standard python modules implementing all the requirements of the standard plugins.
Plugins can be enabled by adding them to :ref:`settings.yml`'s ``plugins`` section.
Example external plugin can be found `here <https://github.com/asciimoo/searx_external_plugin_example>`_.
Register your plugin
====================
To enable your plugin register your plugin in
searx > plugin > __init__.py.
And at the bottom of the file add your plugin like.
``plugins.register(name_of_python_file)``
Plugin entry points
===================
Entry points (hooks) define when a plugin runs. Right now only three hooks are
implemented. So feel free to implement a hook if it fits the behaviour of your
plugin.
Pre search hook
---------------
Runs BEFORE the search request. Function to implement: ``pre_search``
Post search hook
----------------
Runs AFTER the search request. Function to implement: ``post_search``
Result hook
-----------
Runs when a new result is added to the result list. Function to implement:
``on_result``

55
src/searx/docs/dev/quickstart.rst
View File

@ -1,55 +0,0 @@
.. _devquickstart:
======================
Development Quickstart
======================
.. _npm: https://www.npmjs.com/
Searx loves developers, just clone and start hacking. All the rest is done for
you simply by using :ref:`make <makefile>`.
.. code:: sh
git clone https://github.com/searx/searx.git
Here is how a minimal workflow looks like:
1. *start* hacking
2. *run* your code: :ref:`make run`
3. *test* your code: :ref:`make test`
If you think at some point something fails, go back to *start*. Otherwise,
choose a meaningful commit message and we are happy to receive your pull
request. To not end in *wild west* we have some directives, please pay attention
to our ":ref:`how to contribute`" guideline.
If you implement themes, you will need to compile styles and JavaScript before
*run*.
.. code:: sh
make themes
Don't forget to install npm_ first.
.. tabs::
.. group-tab:: Ubuntu / debian
.. code:: sh
sudo -H apt-get install npm
.. group-tab:: Arch Linux
.. code-block:: sh
sudo -H pacman -S npm
.. group-tab:: Fedora / RHEL
.. code-block:: sh
sudo -H dnf install npm

1436
src/searx/docs/dev/reST.rst
View File

File diff suppressed because it is too large Load Diff

120
src/searx/docs/dev/search_api.rst
View File

@ -1,120 +0,0 @@
.. _search API:
==========
Search API
==========
The search supports both ``GET`` and ``POST``.
Furthermore, two endpoints ``/`` and ``/search`` are available for querying.
``GET /``
``GET /search``
Parameters
==========
.. sidebar:: Further reading ..
- :ref:`engines-dev`
- :ref:`settings.yml`
- :ref:`engines generic`
``q`` : required
The search query. This string is passed to external search services. Thus,
searx supports syntax of each search service. For example, ``site:github.com
searx`` is a valid query for Google. However, if simply the query above is
passed to any search engine which does not filter its results based on this
syntax, you might not get the results you wanted.
See more at :ref:`search-syntax`
``categories`` : optional
Comma separated list, specifies the active search categories
``engines`` : optional
Comma separated list, specifies the active search engines.
``lang`` : default ``all``
Code of the language.
``pageno`` : default ``1``
Search page number.
``time_range`` : optional
[ ``day``, ``month``, ``year`` ]
Time range of search for engines which support it. See if an engine supports
time range search in the preferences page of an instance.
``format`` : optional
[ ``json``, ``csv``, ``rss`` ]
Output format of results.
``results_on_new_tab`` : default ``0``
[ ``0``, ``1`` ]
Open search results on new tab.
``image_proxy`` : default ``False``
[ ``True``, ``False`` ]
Proxy image results through searx.
``autocomplete`` : default *empty*
[ ``google``, ``dbpedia``, ``duckduckgo``, ``startpage``, ``wikipedia`` ]
Service which completes words as you type.
``safesearch`` : default ``None``
[ ``0``, ``1``, ``None`` ]
Filter search results of engines which support safe search. See if an engine
supports safe search in the preferences page of an instance.
``theme`` : default ``oscar``
[ ``oscar``, ``simple`` ]
Theme of instance.
Please note, available themes depend on an instance. It is possible that an
instance administrator deleted, created or renamed themes on their instance.
See the available options in the preferences page of the instance.
``oscar-style`` : default ``logicodev``
[ ``pointhi``, ``logicodev`` ]
Style of Oscar theme. It is only parsed if the theme of an instance is
``oscar``.
Please note, available styles depend on an instance. It is possible that an
instance administrator deleted, created or renamed styles on their
instance. See the available options in the preferences page of the instance.
``enabled_plugins`` : optional
List of enabled plugins.
:default: ``HTTPS_rewrite``, ``Self_Informations``,
``Search_on_category_select``, ``Tracker_URL_remover``
:values: [ ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``,
``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``,
``Search_on_category_select`` ]
``disabled_plugins``: optional
List of disabled plugins.
:default: ``DOAI_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys``
:values: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``,
``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``,
``Search_on_category_select``
``enabled_engines`` : optional : *all* :origin:`engines <searx/engines>`
List of enabled engines.
``disabled_engines`` : optional : *all* :origin:`engines <searx/engines>`
List of disabled engines.

71
src/searx/docs/dev/translation.rst
View File

@ -1,71 +0,0 @@
.. _translation:
===========
Translation
===========
.. _searx@transifex: https://www.transifex.com/asciimoo/searx/
Translation currently takes place on `searx@transifex`_
Requirements
============
* Transifex account
* Installed CLI tool of Transifex
Init Transifex project
======================
After installing ``transifex`` using pip, run the following command to
initialize the project.
.. code:: sh
tx init # Transifex instance: https://www.transifex.com/asciimoo/searx/
After ``$HOME/.transifexrc`` is created, get a Transifex API key and insert it
into the configuration file.
Create a configuration file for ``tx`` named ``$HOME/.tx/config``.
.. code:: ini
[main]
host = https://www.transifex.com
[searx.messagespo]
file_filter = searx/translations/<lang>/LC_MESSAGES/messages.po
source_file = messages.pot
source_lang = en
type = PO
Then run ``tx set``:
.. code:: shell
tx set --auto-local -r searx.messagespo 'searx/translations/<lang>/LC_MESSAGES/messages.po' \
--source-lang en --type PO --source-file messages.pot --execute
Update translations
===================
To retrieve the latest translations, pull it from Transifex.
.. code:: sh
tx pull -a
Then check the new languages. If strings translated are not enough, delete those
folders, because those should not be compiled. Call the command below to compile
the ``.po`` files.
.. code:: shell
pybabel compile -d searx/translations
After the compilation is finished commit the ``.po`` and ``.mo`` files and
create a PR.

37
src/searx/docs/index.rst
View File

@ -1,37 +0,0 @@
================
Welcome to searx
================
*Search without being tracked.*
Searx is a free internet metasearch engine which aggregates results from more
than 70 search services. Users are neither tracked nor profiled. Additionally,
searx can be used over Tor for online anonymity.
Get started with searx by using one of the Searx-instances_. If you don't trust
anyone, you can set up your own, see :ref:`installation`.
.. sidebar:: Features
- Self hosted
- No user tracking
- No user profiling
- About 70 supported search engines
- Easy integration with any search engine
- Cookies are not used by default
- Secure, encrypted connections (HTTPS/SSL)
- Hosted by organizations, such as *La Quadrature du Net*, which promote
digital rights
.. toctree::
:maxdepth: 2
:caption: Contents
user/index
admin/index
dev/index
searx_extra/index
utils/index
blog/index
.. _Searx-instances: https://searx.space

14
src/searx/docs/searx_extra/index.rst
View File

@ -1,14 +0,0 @@
.. _searx_extra:
======================================================
Tooling box ``searx_extra`` for developers and users
======================================================
In the folder :origin:`searx_extra/` we maintain some tools useful for
developers and users.
.. toctree::
:maxdepth: 2
:caption: Contents
standalone_searx.py

9
src/searx/docs/searx_extra/standalone_searx.py.rst
View File

@ -1,9 +0,0 @@
.. _standalone_searx.py:
===================================
``searx_extra/standalone_searx.py``
===================================
.. automodule:: searx_extra.standalone_searx
:members:

79
src/searx/docs/user/own-instance.rst
View File

@ -1,79 +0,0 @@
===========================
Why use a private instance?
===========================
*"Is it worth to run my own instance?"*
\.\. is a common question among searx users. Before answering this question,
see what options a searx user has.
Public instances are open to everyone who has access to its URL. Usually, these
are operated by unknown parties (from the users' point of view). Private
instances can be used by a select group of people. It is for example a searx of
group of friends or a company which can be accessed through VPN. Also it can be
single user one which runs on the user's laptop.
To gain more insight on how these instances work let's dive into how searx
protects its users.
How does searx protect privacy?
===============================
Searx protects the privacy of its users in multiple ways regardless of the type
of the instance (private, public). Removal of private data from search requests
comes in three forms:
1. removal of private data from requests going to search services
2. not forwarding anything from a third party services through search services
(e.g. advertisement)
3. removal of private data from requests going to the result pages
Removing private data means not sending cookies to external search engines and
generating a random browser profile for every request. Thus, it does not matter
if a public or private instance handles the request, because it is anonymized in
both cases. IP addresses will be the IP of the instance. But searx can be
configured to use proxy or Tor. `Result proxy
<https://github.com/asciimoo/morty>`__ is supported, too.
Searx does not serve ads or tracking content unlike most search services. So
private data is not forwarded to third parties who might monetize it. Besides
protecting users from search services, both referring page and search query are
hidden from visited result pages.
What are the consequences of using public instances?
----------------------------------------------------
If someone uses a public instance, they have to trust the administrator of that
instance. This means that the user of the public instance does not know whether
their requests are logged, aggregated and sent or sold to a third party.
Also, public instances without proper protection are more vulnerable to abusing
the search service, In this case the external service in exchange returns
CAPTCHAs or bans the IP of the instance. Thus, search requests return less
results.
I see. What about private instances?
------------------------------------
If users run their :ref:`own instances <installation>`, everything is in their
control: the source code, logging settings and private data. Unknown instance
administrators do not have to be trusted.
Furthermore, as the default settings of their instance is editable, there is no
need to use cookies to tailor searx to their needs. So preferences will not be
reset to defaults when clearing browser cookies. As settings are stored on
their computer, it will not be accessible to others as long as their computer is
not compromised.
Conclusion
==========
Always use an instance which is operated by people you trust. The privacy
features of searx are available to users no matter what kind of instance they
use.
If someone is on the go or just wants to try searx for the first time public
instances are the best choices. Additionally, public instance are making a
world a better place, because those who cannot or do not want to run an
instance, have access to a privacy respecting search service.

66
src/searx/docs/user/search_syntax.rst
View File

@ -1,66 +0,0 @@
.. _search-syntax:
=============
Search syntax
=============
Searx allows you to modify the default categories, engines and search language
via the search query.
Prefix ``!``
to set Category/engine
Prefix: ``:``
to set language
Prefix: ``?``
to add engines and categories to the currently selected categories
Abbrevations of the engines and languages are also accepted. Engine/category
modifiers are chainable and inclusive (e.g. with :search:`!it !ddg !wp qwer
<?q=%21it%20%21ddg%20%21wp%20qwer>` search in IT category **and** duckduckgo
**and** wikipedia for ``qwer``).
See the :search:`/preferences page <preferences>` for the list of engines,
categories and languages.
Examples
========
Search in wikipedia for ``qwer``:
- :search:`!wp qwer <?q=%21wp%20qwer>` or
- :search:`!wikipedia qwer :search:<?q=%21wikipedia%20qwer>`
Image search:
- :search:`!images Cthulhu <?q=%21images%20Cthulhu>`
Custom language in wikipedia:
- :search:`:hu !wp hackerspace <?q=%3Ahu%20%21wp%20hackerspace>`
Multilingual Search
===================
Searx does not support true multilingual search.
You have to use the language prefix in your search query when searching in a different language.
But there is a workaround:
By adding a new search engine with a different language, Searx will search in your default and other language.
Example configuration in settings.yml for a German and English speaker:
.. code-block:: yaml
search:
language : "de"
...
engines:
- name : google english
engine : google
language : english
...
When searching, the default google engine will return German results and "google english" will return English results.

52
src/searx/docs/utils/index.rst
View File

@ -1,52 +0,0 @@
.. _searx_utils:
.. _toolboxing:
===================
Admin's tooling box
===================
In the folder :origin:`utils/` we maintain some tools useful for administrators.
.. toctree::
:maxdepth: 2
:caption: Contents
searx.sh
filtron.sh
morty.sh
lxc.sh
.. _toolboxing common:
Common commands & environment
=============================
Scripts to maintain services often dispose of common commands and environments.
``shell`` : command
Opens a shell from the service user ``${SERVICE_USSR}``, very helpful for
troubleshooting.
``inspect service`` : command
Shows status and log of the service, most often you have a option to enable
more verbose debug logs. Very helpful for debugging, but be careful not to
enable debugging in a production environment!
``FORCE_TIMEOUT`` : environment
Sets timeout for interactive prompts. If you want to run a script in batch
job, with defaults choices, set ``FORCE_TIMEOUT=0``. By example; to install a
reverse proxy for filtron on all containers of the :ref:`searx suite
<lxc-searx.env>` use ::
sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/filtron.sh apache install
.. _toolboxing setup:
Tooling box setup
=================
The main setup is done in the :origin:`.config.sh` (read also :ref:`settings
global`).
.. literalinclude:: ../../.config.sh
:language: bash

149
src/searx/docs/utils/lxc.sh.rst
View File

@ -1,149 +0,0 @@
.. _snap: https://snapcraft.io
.. _snapcraft LXD: https://snapcraft.io/lxd
.. _LXC/LXD Image Server: https://uk.images.linuxcontainers.org/
.. _LXC: https://linuxcontainers.org/lxc/introduction/
.. _LXD: https://linuxcontainers.org/lxd/introduction/
.. _`LXD@github`: https://github.com/lxc/lxd
.. _archlinux: https://www.archlinux.org/
.. _lxc.sh:
================
``utils/lxc.sh``
================
.. sidebar:: further reading
- snap_, `snapcraft LXD`_
- LXC_, LXD_
- `LXC/LXD Image Server`_
- `LXD@github`_
With the use of *Linux Containers* (LXC_) we can scale our tasks over a stack of
containers, what we call the: *lxc suite*. The *searx suite*
(:origin:`lxc-searx.env <utils/lxc-searx.env>`) is loaded by default, every time
you start the ``lxc.sh`` script (*you do not need to care about*).
Before you can start with containers, you need to install and initiate LXD_
once::
$ snap install lxd
$ lxd init --auto
To make use of the containers from the *searx suite*, you have to build the
:ref:`LXC suite containers <lxc.sh help>` initial. But be warned, **this might
take some time**::
$ sudo -H ./utils/lxc.sh build
A cup of coffee later, your LXC suite is build up and you can run whatever task
you want / in a selected or even in all :ref:`LXC suite containers <lxc.sh
help>`. If you do not want to build all containers, **you can build just
one**::
$ sudo -H ./utils/lxc.sh build searx-ubu1804
*Good to know ...*
Each container shares the root folder of the repository and the command
``utils/lxc.sh cmd`` **handles relative path names transparent**, compare output
of::
$ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile
...
In the containers, you can run what ever you want, e.g. to start a bash use::
$ sudo -H ./utils/lxc.sh cmd searx-ubu1804 bash
INFO: [searx-ubu1804] bash
root@searx-ubu1804:/share/searx#
If there comes the time you want to **get rid off all** the containers and
**clean up local images** just type::
$ sudo -H ./utils/lxc.sh remove
$ sudo -H ./utils/lxc.sh remove images
.. _lxc.sh install suite:
Install suite
=============
To install the complete :ref:`searx suite (includes searx, morty & filtron)
<lxc-searx.env>` into all LXC_ use::
$ sudo -H ./utils/lxc.sh install suite
The command above installs a searx suite (see :ref:`installation scripts`). To
get the IP (URL) of the filtron service in the containers use ``show suite``
command. To test instances from containers just open the URLs in your
WEB-Browser::
$ sudo ./utils/lxc.sh show suite | grep filtron
[searx-ubu1604] INFO: (eth0) filtron: http://n.n.n.246:4004/ http://n.n.n.246/searx
[searx-ubu1804] INFO: (eth0) filtron: http://n.n.n.147:4004/ http://n.n.n.147/searx
[searx-ubu1910] INFO: (eth0) filtron: http://n.n.n.140:4004/ http://n.n.n.140/searx
[searx-ubu2004] INFO: (eth0) filtron: http://n.n.n.18:4004/ http://n.n.n.18/searx
[searx-fedora31] INFO: (eth0) filtron: http://n.n.n.46:4004/ http://n.n.n.46/searx
[searx-archlinux] INFO: (eth0) filtron: http://n.n.n.32:4004/ http://n.n.n.32/searx
To :ref:`install a nginx <installation nginx>` reverse proxy for filtron and
morty use (or alternatively use :ref:`apache <installation apache>`)::
sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/filtron.sh nginx install
sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/morty.sh nginx install
Running commands
================
**Inside containers, you can use make or run scripts** from the
:ref:`toolboxing`. By example: to setup a :ref:`buildhosts` and run the
Makefile target ``test`` in the archlinux_ container::
sudo -H ./utils/lxc.sh cmd searx-archlinux ./utils/searx.sh install buildhost
sudo -H ./utils/lxc.sh cmd searx-archlinux make test
Setup searx buildhost
=====================
You can **install the searx buildhost environment** into one or all containers.
The installation procedure to set up a :ref:`build host<buildhosts>` takes its
time. Installation in all containers will take more time (time for another cup
of coffee).::
sudo -H ./utils/lxc.sh cmd -- ./utils/searx.sh install buildhost
To build (live) documentation inside a archlinux_ container::
sudo -H ./utils/lxc.sh cmd searx-archlinux make docs.clean docs.live
...
[I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080
To get IP of the container and the port number *live docs* is listening::
$ sudo ./utils/lxc.sh show suite | grep docs.live
...
[searx-archlinux] INFO: (eth0) docs.live: http://n.n.n.12:8080/
.. _lxc.sh help:
Overview
========
The ``--help`` output of the script is largely self-explanatory:
.. program-output:: ../utils/lxc.sh --help
.. _lxc-searx.env:
searx suite
===========
.. literalinclude:: ../../utils/lxc-searx.env
:language: bash

80
src/searx/docs/utils/morty.sh.rst
View File

@ -1,80 +0,0 @@
.. _morty: https://github.com/asciimoo/morty
.. _morty's README: https://github.com/asciimoo/morty
.. _Go: https://golang.org/
.. _morty.sh:
==================
``utils/morty.sh``
==================
.. sidebar:: further reading
- :ref:`architecture`
- :ref:`installation` (:ref:`nginx <installation nginx>` & :ref:`apache
<installation apache>`)
- :ref:`searx morty`
To simplify installation and maintenance of a morty_ instance you can use the
script :origin:`utils/morty.sh`. In most cases you will install morty_ simply by
running the command:
.. code:: bash
sudo -H ./utils/morty.sh install all
The script adds a ``${SERVICE_USER}`` (default:``morty``) and installs morty_
into this user account:
#. Create a separated user account (``morty``).
#. Download and install Go_ binary in user's $HOME (``~morty``).
#. Install morty_ with the package management from Go_ (``go get -v -u
github.com/asciimoo/morty``)
#. Setup a systemd service unit :origin:`[ref]
<utils/templates/lib/systemd/system/morty.service>`
(``/lib/systemd/system/morty.service``).
.. hint::
To add morty to your searx instance read chapter :ref:`searx morty`.
Create user
===========
.. kernel-include:: $DOCS_BUILD/includes/morty.rst
:start-after: START create user
:end-before: END create user
Install go
==========
.. kernel-include:: $DOCS_BUILD/includes/morty.rst
:start-after: START install go
:end-before: END install go
Install morty
=============
Install morty software and systemd unit:
.. kernel-include:: $DOCS_BUILD/includes/morty.rst
:start-after: START install morty
:end-before: END install morty
.. kernel-include:: $DOCS_BUILD/includes/morty.rst
:start-after: START install systemd unit
:end-before: END install systemd unit
.. _morty.sh overview:
Overview
========
The ``--help`` output of the script is largely self-explanatory
(:ref:`toolboxing common`):
.. program-output:: ../utils/morty.sh --help

39
src/searx/docs/utils/searx.sh.rst
View File

@ -1,39 +0,0 @@
.. _searx.sh:
==================
``utils/searx.sh``
==================
.. sidebar:: further reading
- :ref:`architecture`
- :ref:`installation`
- :ref:`installation nginx`
- :ref:`installation apache`
To simplify installation and maintenance of a searx instance you can use the
script :origin:`utils/searx.sh`.
Install
=======
In most cases you will install searx simply by running the command:
.. code:: bash
sudo -H ./utils/searx.sh install all
The script adds a ``${SERVICE_USER}`` (default:``searx``) and installs searx
into this user account. The installation is described in chapter
:ref:`installation basic`.
.. _intranet reverse proxy:
Overview
========
The ``--help`` output of the script is largely self-explanatory
(:ref:`toolboxing common`):
.. program-output:: ../utils/searx.sh --help

504
src/searx/manage
View File

@ -1,504 +0,0 @@
#!/usr/bin/env bash
# -*- coding: utf-8; mode: sh indent-tabs-mode: nil -*-
# SPDX-License-Identifier: AGPL-3.0-or-later
# shellcheck disable=SC2031
# shellcheck source=utils/lib.sh
source "$(dirname "${BASH_SOURCE[0]}")/utils/lib.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
source_dot_config
# config
PY_SETUP_EXTRAS='[test]'
NPM_PACKAGES="less@2.7 less-plugin-clean-css grunt-cli"
GECKODRIVER_VERSION="v0.28.0"
# SPHINXOPTS=
# These py files are linted by test.pylint(), all other files are linted by
# test.pep8()
PYLINT_FILES=(
searx/preferences.py
searx/testing.py
searx/engines/gigablast.py
searx/engines/deviantart.py
searx/engines/digg.py
searx/engines/google.py
searx/engines/google_news.py
searx/engines/google_videos.py
searx/engines/google_images.py
searx/engines/mediathekviewweb.py
searx/engines/meilisearch.py
searx/engines/solidtorrents.py
searx/engines/solr.py
searx/engines/sqlite.py
searx/engines/springer.py
searx/engines/google_scholar.py
searx/engines/yahoo_news.py
searx/engines/apkmirror.py
searx/engines/core.py
searx_extra/update/update_external_bangs.py
)
PYLINT_SEARX_DISABLE_OPTION="\
I,C,R,\
W0105,W0212,W0511,W0603,W0613,W0621,W0702,W0703,W1401,\
E1136"
PYLINT_ADDITIONAL_BUILTINS_FOR_ENGINES="supported_languages,language_aliases"
PYLINT_OPTIONS="-m pylint -j 0 --rcfile .pylintrc"
help() {
cat <<EOF
buildenv
rebuild ./utils/brand.env
babel.compile
pybabel compile ./searx/translations
data.*
all : update searx/languages.py and ./data/*
languages : update searx/data/engines_languages.json & searx/languages.py
useragents: update searx/data/useragents.json with the most recent versions of Firefox.
docs.*
html : build HTML documentation
live : autobuild HTML documentation while editing
gh-pages : deploy on gh-pages branch
prebuild : build reST include files (./${DOCS_BUILD}/includes)
clean : clean documentation build
docker
build : build docker image
push : build and push docker image
gecko.driver
download & install geckodriver if not already installed (required for
robot_tests)
node.*
env : download & install npm dependencies locally
clean : drop npm installations
py.*
build : Build python packages at ./${PYDIST}
clean : delete virtualenv and intermediate py files
pyenv.* :
install : developer install of searx into virtualenv
uninstall : uninstall developer installation
cmd ... : run command ... in virtualenv
OK : test if virtualenv is OK
pypi.upload:
Upload python packages to PyPi (to test use pypi.upload.test)
test.* :
pylint : lint PYLINT_FILES, searx/engines, searx & tests
pep8 : pycodestyle (pep8) for all files except PYLINT_FILES
unit : run unit tests
coverage : run unit tests with coverage
robot : run robot test
clean : clean intermediate test stuff
themes.* :
all : build all themes
oscar : build oscar theme
simple : build simple theme
EOF
}
if [ "$VERBOSE" = "1" ]; then
SPHINX_VERBOSE="-v"
PYLINT_VERBOSE="-v"
fi
# needed by sphinx-docs
export DOCS_BUILD
buildenv() {
SEARX_DEBUG=1 pyenv.cmd python utils/build_env.py 2>&1 \
| prefix_stdout "${_Blue}BUILDENV${_creset} "
return "${PIPESTATUS[0]}"
}
babel.compile() {
build_msg BABEL compile
pyenv.cmd pybabel compile -d "${REPO_ROOT}/searx/translations"
dump_return $?
}
data.all() {
data.languages
data.useragents
build_msg DATA "update searx/data/ahmia_blacklist.txt"
pyenv.cmd python searx_extra/update/update_ahmia_blacklist.py
build_msg DATA "update searx/data/wikidata_units.json"
pyenv.cmd python searx_extra/update/update_wikidata_units.py
build_msg DATA "update searx/data/currencies.json"
pyenv.cmd python searx_extra/update/update_currencies.py
}
data.languages() {
( set -e
build_msg ENGINES "fetch languages .."
pyenv.cmd python searx_extra/update/update_languages.py
build_msg ENGINES "update update searx/languages.py"
build_msg DATA "update searx/data/engines_languages.json"
)
dump_return $?
}
data.useragents() {
build_msg DATA "update searx/data/useragents.json"
pyenv.cmd python searx_extra/update/update_firefox_version.py
dump_return $?
}
docs.prebuild() {
build_msg DOCS "build ${DOCS_BUILD}/includes"
(
set -e
[ "$VERBOSE" = "1" ] && set -x
mkdir -p "${DOCS_BUILD}/includes"
./utils/searx.sh doc | cat > "${DOCS_BUILD}/includes/searx.rst"
./utils/filtron.sh doc | cat > "${DOCS_BUILD}/includes/filtron.rst"
./utils/morty.sh doc | cat > "${DOCS_BUILD}/includes/morty.rst"
)
dump_return $?
}
docker.push() {
docker.build push
}
# shellcheck disable=SC2119
docker.build() {
pyenv.install
build_msg DOCKER build
# run installation in a subprocess and activate pyenv
# See https://www.shellcheck.net/wiki/SC1001 and others ..
# shellcheck disable=SC2031,SC2230,SC2002,SC2236,SC2143,SC1001
( set -e
# shellcheck source=/dev/null
source "${PY_ENV_BIN}/activate"
# Check if it is a git repository
if [ ! -d .git ]; then
die 1 "This is not Git repository"
fi
if [ ! -x "$(which git)" ]; then
die 1 "git is not installed"
fi
if ! git remote get-url origin 2> /dev/null; then
die 1 "there is no remote origin"
fi
# This is a git repository
# "git describe" to get the Docker version (for example : v0.15.0-89-g0585788e)
# awk to remove the "v" and the "g"
SEARX_GIT_VERSION=$(git describe --match "v[0-9]*\.[0-9]*\.[0-9]*" HEAD 2>/dev/null | awk -F'-' '{OFS="-"; $1=substr($1, 2); if ($3) { $3=substr($3, 2); } print}')
# add the suffix "-dirty" if the repository has uncommited change
# /!\ HACK for searx/searx: ignore utils/brand.env
git update-index -q --refresh
if [ ! -z "$(git diff-index --name-only HEAD -- | grep -v 'utils/brand.env')" ]; then
SEARX_GIT_VERSION="${SEARX_GIT_VERSION}-dirty"
fi
# Get the last git commit id, will be added to the Searx version (see Dockerfile)
VERSION_GITCOMMIT=$(echo "$SEARX_GIT_VERSION" | cut -d- -f2-4)
build_msg DOCKER "Last commit : $VERSION_GITCOMMIT"
# Check consistency between the git tag and the searx/version.py file
# /! HACK : parse Python file with bash /!
# otherwise it is not possible build the docker image without all Python
# dependencies ( version.py loads __init__.py )
# SEARX_PYTHON_VERSION=$(python3 -c "import six; import searx.version; six.print_(searx.version.VERSION_STRING)")
SEARX_PYTHON_VERSION=$(cat searx/version.py | grep "\(VERSION_MAJOR\|VERSION_MINOR\|VERSION_BUILD\) =" | cut -d\= -f2 | sed -e 's/^[[:space:]]*//' | paste -sd "." -)
if [ "$(echo "$SEARX_GIT_VERSION" | cut -d- -f1)" != "$SEARX_PYTHON_VERSION" ]; then
err_msg "git tag: $SEARX_GIT_VERSION"
err_msg "searx/version.py: $SEARX_PYTHON_VERSION"
die 1 "Inconsistency between the last git tag and the searx/version.py file"
fi
# define the docker image name
GITHUB_USER=$(echo "${GIT_URL}" | sed 's/.*github\.com\/\([^\/]*\).*/\1/')
SEARX_IMAGE_NAME="${SEARX_IMAGE_NAME:-${GITHUB_USER:-searx}/searx}"
# build Docker image
build_msg DOCKER "Building image ${SEARX_IMAGE_NAME}:${SEARX_GIT_VERSION}"
sudo docker build \
--build-arg GIT_URL="${GIT_URL}" \
--build-arg SEARX_GIT_VERSION="${SEARX_GIT_VERSION}" \
--build-arg VERSION_GITCOMMIT="${VERSION_GITCOMMIT}" \
--build-arg LABEL_DATE="$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
--build-arg LABEL_VCS_REF="$(git rev-parse HEAD)" \
--build-arg LABEL_VCS_URL="${GIT_URL}" \
--build-arg TIMESTAMP_SETTINGS="$(git log -1 --format="%cd" --date=unix -- searx/settings.yml)" \
--build-arg TIMESTAMP_UWSGI="$(git log -1 --format="%cd" --date=unix -- dockerfiles/uwsgi.ini)" \
-t "${SEARX_IMAGE_NAME}:latest" -t "${SEARX_IMAGE_NAME}:${SEARX_GIT_VERSION}" .
if [ "$1" = "push" ]; then
sudo docker push "${SEARX_IMAGE_NAME}:latest"
sudo docker push "${SEARX_IMAGE_NAME}:${SEARX_GIT_VERSION}"
fi
)
dump_return $?
}
# shellcheck disable=SC2119
gecko.driver() {
pyenv.install
build_msg INSTALL "gecko.driver"
# run installation in a subprocess and activate pyenv
( set -e
# shellcheck source=/dev/null
source "${PY_ENV_BIN}/activate"
# TODO : check the current geckodriver version
geckodriver -V > /dev/null 2>&1 || NOTFOUND=1
set +e
if [ -z "$NOTFOUND" ]; then
build_msg INSTALL "geckodriver already installed"
return
fi
PLATFORM="$(python3 -c 'import platform; print(platform.system().lower(), platform.architecture()[0])')"
case "$PLATFORM" in
"linux 32bit" | "linux2 32bit") ARCH="linux32";;
"linux 64bit" | "linux2 64bit") ARCH="linux64";;
"windows 32 bit") ARCH="win32";;
"windows 64 bit") ARCH="win64";;
"mac 64bit") ARCH="macos";;
esac
GECKODRIVER_URL="https://github.com/mozilla/geckodriver/releases/download/$GECKODRIVER_VERSION/geckodriver-$GECKODRIVER_VERSION-$ARCH.tar.gz";
build_msg GECKO "Installing ${PY_ENV_BIN}/geckodriver from $GECKODRIVER_URL"
FILE="$(mktemp)"
wget -qO "$FILE" -- "$GECKODRIVER_URL" && tar xz -C "${PY_ENV_BIN}" -f "$FILE" geckodriver
rm -- "$FILE"
chmod 755 -- "${PY_ENV_BIN}/geckodriver"
)
dump_return $?
}
node.env() {
local err=0
pushd "${REPO_ROOT}" &> /dev/null
# shellcheck disable=SC2230
which npm &> /dev/null || die 1 'node.env - npm is not found!'
( set -e
# shellcheck disable=SC2030
PATH="$(npm bin):$PATH"
export PATH
build_msg INSTALL "npm install $NPM_PACKAGES"
# shellcheck disable=SC2086
npm install $NPM_PACKAGES
cd "${REPO_ROOT}/searx/static/themes/oscar"
build_msg INSTALL "($(pwd)) npm install"
npm install
build_msg INSTALL "($(pwd)) npm install"
cd "${REPO_ROOT}/searx/static/themes/simple"
npm install
)
err=$?
popd &> /dev/null
dump_return "$err"
}
node.clean() {
build_msg CLEAN "locally installed npm dependencies"
rm -rf \
./node_modules \
./package-lock.json \
./searx/static/themes/oscar/package-lock.json \
./searx/static/themes/oscar/node_modules \
./searx/static/themes/simple/package-lock.json \
./searx/static/themes/simple/node_modules
dump_return $?
}
py.build() {
build_msg BUILD "[pylint] python package ${PYDIST}"
pyenv.cmd python setup.py \
sdist -d "${PYDIST}" \
bdist_wheel --bdist-dir "${PYBUILD}" -d "${PYDIST}"
}
py.clean() {
build_msg CLEAN pyenv
( set -e
pyenv.drop
[ "$VERBOSE" = "1" ] && set -x
rm -rf "${PYDIST}" "${PYBUILD}" "${PY_ENV}" ./.tox ./*.egg-info
find . -name '*.pyc' -exec rm -f {} +
find . -name '*.pyo' -exec rm -f {} +
find . -name __pycache__ -exec rm -rf {} +
)
}
pyenv.check() {
cat <<EOF
import yaml
print('import yaml --> OK')
EOF
}
pyenv.install() {
if ! pyenv.OK; then
py.clean > /dev/null
fi
if pyenv.install.OK > /dev/null; then
return 0
fi
( set -e
pyenv
build_msg PYENV "[install] pip install -e 'searx${PY_SETUP_EXTRAS}'"
"${PY_ENV_BIN}/python" -m pip install -e ".${PY_SETUP_EXTRAS}"
buildenv
) || die 42 "error while build & install pyenv (${PY_ENV_BIN})"
}
pyenv.uninstall() {
build_msg PYENV "[pyenv.uninstall] uninstall packages: ${PYOBJECTS}"
pyenv.cmd python setup.py develop --uninstall 2>&1 \
| prefix_stdout "${_Blue}PYENV ${_creset}[pyenv.uninstall] "
}
pypi.upload() {
py.clean
py.build
# https://github.com/pypa/twine
pyenv.cmd twine upload "${PYDIST}"/*
}
pypi.upload.test() {
py.clean
py.build
pyenv.cmd twine upload -r testpypi "${PYDIST}"/*
}
test.pylint() {
# shellcheck disable=SC2086
( set -e
build_msg TEST "[pylint] \$PYLINT_FILES"
pyenv.cmd python ${PYLINT_OPTIONS} ${PYLINT_VERBOSE} \
"${PYLINT_FILES[@]}"
build_msg TEST "[pylint] searx/engines"
pyenv.cmd python ${PYLINT_OPTIONS} ${PYLINT_VERBOSE} \
--disable="${PYLINT_SEARX_DISABLE_OPTION}" \
--additional-builtins="${PYLINT_ADDITIONAL_BUILTINS_FOR_ENGINES}" \
searx/engines
build_msg TEST "[pylint] searx tests"
pyenv.cmd python ${PYLINT_OPTIONS} ${PYLINT_VERBOSE} \
--disable="${PYLINT_SEARX_DISABLE_OPTION}" \
--ignore=searx/engines \
tests
)
dump_return $?
}
test.pep8() {
build_msg TEST 'pycodestyle (formerly pep8)'
local _exclude=""
printf -v _exclude '%s, ' "${PYLINT_FILES[@]}"
pyenv.cmd pycodestyle \
--exclude="searx/static, searx/languages.py, $_exclude " \
--max-line-length=120 \
--ignore "E117,E252,E402,E722,E741,W503,W504,W605" \
searx tests
dump_return $?
}
test.unit() {
build_msg TEST 'tests/unit'
pyenv.cmd python -m nose2 -s tests/unit
dump_return $?
}
test.coverage() {
build_msg TEST 'unit test coverage'
( set -e
pyenv.cmd python -m nose2 -C --log-capture --with-coverage --coverage searx -s tests/unit
pyenv.cmd coverage report
pyenv.cmd coverage html
)
dump_return $?
}
test.robot() {
build_msg TEST 'robot'
gecko.driver
PYTHONPATH=. pyenv.cmd python searx/testing.py robot
dump_return $?
}
test.clean() {
build_msg CLEAN "test stuff"
rm -rf geckodriver.log .coverage coverage/
dump_return $?
}
themes.all() {
( set -e
node.env
themes.oscar
themes.simple
)
dump_return $?
}
themes.oscar() {
local gruntfile=searx/static/themes/oscar/gruntfile.js
build_msg GRUNT "${gruntfile}"
PATH="$(npm bin):$PATH" grunt --gruntfile "${gruntfile}"
dump_return $?
}
themes.simple() {
local gruntfile=searx/static/themes/simple/gruntfile.js
build_msg GRUNT "${gruntfile}"
PATH="$(npm bin):$PATH" grunt --gruntfile "${gruntfile}"
dump_return $?
}
# shellcheck disable=SC2119
main() {
local _type
local cmd="$1"; shift
if [ "$cmd" == "" ]; then
help
err_msg "missing command"
return 42
fi
case "$cmd" in
--getenv) var="$1"; echo "${!var}";;
--help) help;;
--*)
help
err_msg "unknown option $cmd"
return 42
;;
*)
_type="$(type -t "$cmd")"
if [ "$_type" != 'function' ]; then
err_msg "unknown command $1 / use --help"
return 42
else
"$cmd" "$@"
fi
;;
esac
}
main "$@"

19
src/searx/requirements-dev.txt
View File

@ -1,19 +0,0 @@
mock==4.0.3
nose2[coverage_plugin]==0.10.0
cov-core==1.15.0
pycodestyle==2.7.0
pylint==2.10.2
splinter==0.15.0
transifex-client==0.14.3
selenium==3.141.0
twine==3.4.2
Pallets-Sphinx-Themes==2.0.1
docutils==0.16
Sphinx==4.2.0
sphinx-issues==1.2.0
sphinx-jinja==1.1.1
sphinx-tabs==3.2.0
sphinxcontrib-programoutput==0.17
sphinx-autobuild==2021.3.14
linuxdoc==20210324
aiounittest==1.4.0

16
src/searx/requirements.txt
View File

@ -1,16 +0,0 @@
certifi==2021.5.30
babel==2.9.1
flask-babel==2.0.0
flask==1.1.2
jinja2==3.0.2
lxml==4.6.3
pygments==2.8.0
python-dateutil==2.8.2
pyyaml==5.4.1
httpx[http2]==0.19.0
Brotli==1.0.9
uvloop==0.16.0; python_version >= '3.7'
uvloop==0.14.0; python_version < '3.7'
httpx-socks[asyncio]==0.4.1
langdetect==1.0.9
setproctitle==1.2.2

108
src/searx/searx/__init__.py
View File

@ -1,108 +0,0 @@
'''
searx is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
searx is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with searx. If not, see < http://www.gnu.org/licenses/ >.
(C) 2013- by Adam Tauber, <asciimoo@gmail.com>
'''
import logging
import searx.settings_loader
from os import environ
from os.path import realpath, dirname, join, abspath, isfile
searx_dir = abspath(dirname(__file__))
engine_dir = dirname(realpath(__file__))
static_path = abspath(join(dirname(__file__), 'static'))
settings, settings_load_message = searx.settings_loader.load_settings()
if settings['ui']['static_path']:
static_path = settings['ui']['static_path']
'''
enable debug if
the environnement variable SEARX_DEBUG is 1 or true
(whatever the value in settings.yml)
or general.debug=True in settings.yml
disable debug if
the environnement variable SEARX_DEBUG is 0 or false
(whatever the value in settings.yml)
or general.debug=False in settings.yml
'''
searx_debug_env = environ.get('SEARX_DEBUG', '').lower()
if searx_debug_env == 'true' or searx_debug_env == '1':
searx_debug = True
elif searx_debug_env == 'false' or searx_debug_env == '0':
searx_debug = False
else:
searx_debug = settings.get('general', {}).get('debug')
if searx_debug:
logging.basicConfig(level=logging.DEBUG)
else:
logging.basicConfig(level=logging.WARNING)
logger = logging.getLogger('searx')
logger.info(settings_load_message)
logger.info('Initialisation done')
if 'SEARX_SECRET' in environ:
settings['server']['secret_key'] = environ['SEARX_SECRET']
if 'SEARX_BIND_ADDRESS' in environ:
settings['server']['bind_address'] = environ['SEARX_BIND_ADDRESS']
class _brand_namespace:
@classmethod
def get_val(cls, group, name, default=''):
return settings.get(group, {}).get(name) or default
@property
def SEARX_URL(self):
return self.get_val('server', 'base_url')
@property
def CONTACT_URL(self):
return self.get_val('general', 'contact_url')
@property
def GIT_URL(self):
return self.get_val('brand', 'git_url')
@property
def GIT_BRANCH(self):
return self.get_val('brand', 'git_branch')
@property
def ISSUE_URL(self):
return self.get_val('brand', 'issue_url')
@property
def DOCS_URL(self):
return self.get_val('brand', 'docs_url')
@property
def PUBLIC_INSTANCES(self):
return self.get_val('brand', 'public_instances')
@property
def WIKI_URL(self):
return self.get_val('brand', 'wiki_url')
@property
def TWITTER_URL(self):
return self.get_val('brand', 'twitter_url')
brand = _brand_namespace()

50
src/searx/searx/data/__init__.py
View File

@ -1,50 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint
"""This module holds the *data* created by::
make data.all
"""
__all__ = [
'ENGINES_LANGUAGES',
'CURRENCIES',
'USER_AGENTS',
'EXTERNAL_URLS',
'WIKIDATA_UNITS',
'EXTERNAL_BANGS',
'OSM_KEYS_TAGS',
'ahmia_blacklist_loader',
]
import json
from pathlib import Path
data_dir = Path(__file__).parent
def _load(filename):
with open(data_dir / filename, encoding='utf-8') as f:
return json.load(f)
def ahmia_blacklist_loader():
"""Load data from `ahmia_blacklist.txt` and return a list of MD5 values of onion
names. The MD5 values are fetched by::
searx_extra/update/update_ahmia_blacklist.py
This function is used by :py:mod:`searx.plugins.ahmia_filter`.
"""
with open(str(data_dir / 'ahmia_blacklist.txt'), encoding='utf-8') as f:
return f.read().split()
ENGINES_LANGUAGES = _load('engines_languages.json')
CURRENCIES = _load('currencies.json')
USER_AGENTS = _load('useragents.json')
EXTERNAL_URLS = _load('external_urls.json')
WIKIDATA_UNITS = _load('wikidata_units.json')
EXTERNAL_BANGS = _load('external_bangs.json')
OSM_KEYS_TAGS = _load('osm_keys_tags.json')

27245
src/searx/searx/data/ahmia_blacklist.txt
View File

File diff suppressed because it is too large Load Diff

14726
src/searx/searx/data/currencies.json
View File

File diff suppressed because it is too large Load Diff

28588
src/searx/searx/data/engines_languages.json
View File

File diff suppressed because it is too large Load Diff

19067
src/searx/searx/data/external_bangs.json
View File

File diff suppressed because it is too large Load Diff

1195
src/searx/searx/data/wikidata_units.json
View File

File diff suppressed because it is too large Load Diff

53
src/searx/searx/engines/1337x.py
View File

@ -1,53 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""
1337x
"""
from urllib.parse import quote, urljoin
from lxml import html
from searx.utils import extract_text, get_torrent_size, eval_xpath, eval_xpath_list, eval_xpath_getindex
# about
about = {
"website": 'https://1337x.to/',
"wikidata_id": 'Q28134166',
"official_api_documentation": None,
"use_official_api": False,
"require_api_key": False,
"results": 'HTML',
}
url = 'https://1337x.to/'
search_url = url + 'search/{search_term}/{pageno}/'
categories = ['videos']
paging = True
def request(query, params):
params['url'] = search_url.format(search_term=quote(query), pageno=params['pageno'])
return params
def response(resp):
results = []
dom = html.fromstring(resp.text)
for result in eval_xpath_list(dom, '//table[contains(@class, "table-list")]/tbody//tr'):
href = urljoin(url, eval_xpath_getindex(result, './td[contains(@class, "name")]/a[2]/@href', 0))
title = extract_text(eval_xpath(result, './td[contains(@class, "name")]/a[2]'))
seed = extract_text(eval_xpath(result, './/td[contains(@class, "seeds")]'))
leech = extract_text(eval_xpath(result, './/td[contains(@class, "leeches")]'))
filesize_info = extract_text(eval_xpath(result, './/td[contains(@class, "size")]/text()'))
filesize, filesize_multiplier = filesize_info.split()
filesize = get_torrent_size(filesize, filesize_multiplier)
results.append({'url': href,
'title': title,
'seed': seed,
'leech': leech,
'filesize': filesize,
'template': 'torrent.html'})
return results

308
src/searx/searx/engines/__init__.py
View File

@ -1,308 +0,0 @@
'''
searx is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
searx is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with searx. If not, see < http://www.gnu.org/licenses/ >.
(C) 2013- by Adam Tauber, <asciimoo@gmail.com>
'''
import sys
import threading
from os.path import realpath, dirname
from babel.localedata import locale_identifiers
from urllib.parse import urlparse
from flask_babel import gettext
from operator import itemgetter
from searx import settings
from searx import logger
from searx.data import ENGINES_LANGUAGES
from searx.exceptions import SearxEngineResponseException
from searx.network import get, initialize as initialize_network, set_context_network_name
from searx.utils import load_module, match_language, get_engine_from_settings, gen_useragent
logger = logger.getChild('engines')
engine_dir = dirname(realpath(__file__))
engines = {}
categories = {'general': []}
babel_langs = [lang_parts[0] + '-' + lang_parts[-1] if len(lang_parts) > 1 else lang_parts[0]
for lang_parts in (lang_code.split('_') for lang_code in locale_identifiers())]
engine_shortcuts = {}
engine_default_args = {'paging': False,
'categories': ['general'],
'supported_languages': [],
'safesearch': False,
'timeout': settings['outgoing']['request_timeout'],
'shortcut': '-',
'disabled': False,
'enable_http': False,
'suspend_end_time': 0,
'continuous_errors': 0,
'time_range_support': False,
'engine_type': 'online',
'display_error_messages': True,
'tokens': []}
def load_engine(engine_data):
engine_name = engine_data['name']
if '_' in engine_name:
logger.error('Engine name contains underscore: "{}"'.format(engine_name))
sys.exit(1)
if engine_name.lower() != engine_name:
logger.warn('Engine name is not lowercase: "{}", converting to lowercase'.format(engine_name))
engine_name = engine_name.lower()
engine_data['name'] = engine_name
engine_module = engine_data['engine']
try:
engine = load_module(engine_module + '.py', engine_dir)
except (SyntaxError, KeyboardInterrupt, SystemExit, SystemError, ImportError, RuntimeError):
logger.exception('Fatal exception in engine "{}"'.format(engine_module))
sys.exit(1)
except:
logger.exception('Cannot load engine "{}"'.format(engine_module))
return None
for param_name, param_value in engine_data.items():
if param_name == 'engine':
pass
elif param_name == 'categories':
if param_value == 'none':
engine.categories = []
else:
engine.categories = list(map(str.strip, param_value.split(',')))
else:
setattr(engine, param_name, param_value)
for arg_name, arg_value in engine_default_args.items():
if not hasattr(engine, arg_name):
setattr(engine, arg_name, arg_value)
# checking required variables
for engine_attr in dir(engine):
if engine_attr.startswith('_'):
continue
if engine_attr == 'inactive' and getattr(engine, engine_attr) is True:
return None
if getattr(engine, engine_attr) is None:
logger.error('Missing engine config attribute: "{0}.{1}"'
.format(engine.name, engine_attr))
sys.exit(1)
# assign supported languages from json file
if engine_data['engine'] in ENGINES_LANGUAGES:
setattr(engine, 'supported_languages', ENGINES_LANGUAGES[engine_data['engine']])
# find custom aliases for non standard language codes
if hasattr(engine, 'supported_languages'):
if hasattr(engine, 'language_aliases'):
language_aliases = getattr(engine, 'language_aliases')
else:
language_aliases = {}
for engine_lang in getattr(engine, 'supported_languages'):
iso_lang = match_language(engine_lang, babel_langs, fallback=None)
if iso_lang and iso_lang != engine_lang and not engine_lang.startswith(iso_lang) and \
iso_lang not in getattr(engine, 'supported_languages'):
language_aliases[iso_lang] = engine_lang
setattr(engine, 'language_aliases', language_aliases)
# language_support
setattr(engine, 'language_support', len(getattr(engine, 'supported_languages', [])) > 0)
# assign language fetching method if auxiliary method exists
if hasattr(engine, '_fetch_supported_languages'):
headers = {
'User-Agent': gen_useragent(),
'Accept-Language': 'ja-JP,ja;q=0.8,en-US;q=0.5,en;q=0.3', # bing needs a non-English language
}
setattr(engine, 'fetch_supported_languages',
lambda: engine._fetch_supported_languages(get(engine.supported_languages_url, headers=headers)))
engine.stats = {
'sent_search_count': 0, # sent search
'search_count': 0, # succesful search
'result_count': 0,
'engine_time': 0,
'engine_time_count': 0,
'score_count': 0,
'errors': 0
}
engine_type = getattr(engine, 'engine_type', 'online')
if engine_type != 'offline':
engine.stats['page_load_time'] = 0
engine.stats['page_load_count'] = 0
# tor related settings
if settings['outgoing'].get('using_tor_proxy'):
# use onion url if using tor.
if hasattr(engine, 'onion_url'):
engine.search_url = engine.onion_url + getattr(engine, 'search_path', '')
elif 'onions' in engine.categories:
# exclude onion engines if not using tor.
return None
engine.timeout += settings['outgoing'].get('extra_proxy_timeout', 0)
for category_name in engine.categories:
categories.setdefault(category_name, []).append(engine)
if engine.shortcut in engine_shortcuts:
logger.error('Engine config error: ambigious shortcut: {0}'.format(engine.shortcut))
sys.exit(1)
engine_shortcuts[engine.shortcut] = engine.name
return engine
def to_percentage(stats, maxvalue):
for engine_stat in stats:
if maxvalue:
engine_stat['percentage'] = int(engine_stat['avg'] / maxvalue * 100)
else:
engine_stat['percentage'] = 0
return stats
def get_engines_stats(preferences):
# TODO refactor
pageloads = []
engine_times = []
results = []
scores = []
errors = []
scores_per_result = []
max_pageload = max_engine_times = max_results = max_score = max_errors = max_score_per_result = 0 # noqa
for engine in engines.values():
if not preferences.validate_token(engine):
continue
if engine.stats['search_count'] == 0:
continue
results_num = \
engine.stats['result_count'] / float(engine.stats['search_count'])
if engine.stats['engine_time_count'] != 0:
this_engine_time = engine.stats['engine_time'] / float(engine.stats['engine_time_count']) # noqa
else:
this_engine_time = 0
if results_num:
score = engine.stats['score_count'] / float(engine.stats['search_count']) # noqa
score_per_result = score / results_num
else:
score = score_per_result = 0.0
if engine.engine_type != 'offline':
load_times = 0
if engine.stats['page_load_count'] != 0:
load_times = engine.stats['page_load_time'] / float(engine.stats['page_load_count']) # noqa
max_pageload = max(load_times, max_pageload)
pageloads.append({'avg': load_times, 'name': engine.name})
max_engine_times = max(this_engine_time, max_engine_times)
max_results = max(results_num, max_results)
max_score = max(score, max_score)
max_score_per_result = max(score_per_result, max_score_per_result)
max_errors = max(max_errors, engine.stats['errors'])
engine_times.append({'avg': this_engine_time, 'name': engine.name})
results.append({'avg': results_num, 'name': engine.name})
scores.append({'avg': score, 'name': engine.name})
errors.append({'avg': engine.stats['errors'], 'name': engine.name})
scores_per_result.append({
'avg': score_per_result,
'name': engine.name
})
pageloads = to_percentage(pageloads, max_pageload)
engine_times = to_percentage(engine_times, max_engine_times)
results = to_percentage(results, max_results)
scores = to_percentage(scores, max_score)
scores_per_result = to_percentage(scores_per_result, max_score_per_result)
errors = to_percentage(errors, max_errors)
return [
(
gettext('Engine time (sec)'),
sorted(engine_times, key=itemgetter('avg'))
),
(
gettext('Page loads (sec)'),
sorted(pageloads, key=itemgetter('avg'))
),
(
gettext('Number of results'),
sorted(results, key=itemgetter('avg'), reverse=True)
),
(
gettext('Scores'),
sorted(scores, key=itemgetter('avg'), reverse=True)
),
(
gettext('Scores per result'),
sorted(scores_per_result, key=itemgetter('avg'), reverse=True)
),
(
gettext('Errors'),
sorted(errors, key=itemgetter('avg'), reverse=True)
),
]
def load_engines(engine_list):
global engines, engine_shortcuts
engines.clear()
engine_shortcuts.clear()
for engine_data in engine_list:
engine = load_engine(engine_data)
if engine is not None:
engines[engine.name] = engine
return engines
def initialize_engines(engine_list):
load_engines(engine_list)
initialize_network(engine_list, settings['outgoing'])
def engine_init(engine_name, init_fn):
try:
set_context_network_name(engine_name)
init_fn(get_engine_from_settings(engine_name))
except SearxEngineResponseException as exc:
logger.warn('%s engine: Fail to initialize // %s', engine_name, exc)
except Exception:
logger.exception('%s engine: Fail to initialize', engine_name)
else:
logger.debug('%s engine: Initialized', engine_name)
for engine_name, engine in engines.items():
if hasattr(engine, 'init'):
init_fn = getattr(engine, 'init')
if init_fn:
logger.debug('%s engine: Starting background initialization', engine_name)
threading.Thread(target=engine_init, args=(engine_name, init_fn)).start()

68
src/searx/searx/engines/apkmirror.py
View File

@ -1,68 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""APKMirror
"""
# pylint: disable=invalid-name, missing-function-docstring
from urllib.parse import urlencode
from lxml import html
from searx import logger
from searx.utils import (
eval_xpath_list,
eval_xpath_getindex,
extract_text,
)
logger = logger.getChild('APKMirror engine')
about = {
"website": 'https://www.apkmirror.com',
"wikidata_id": None,
"official_api_documentation": None,
"use_official_api": False,
"require_api_key": False,
"results": 'HTML',
}
# engine dependent config
categories = ['files']
paging = True
time_range_support = False
# search-url
base_url = 'https://www.apkmirror.com'
search_url = base_url + '/?post_type=app_release&searchtype=apk&page={pageno}&{query}'
def request(query, params):
params['url'] = search_url.format(
pageno = params['pageno'],
query = urlencode({'s': query}),
)
logger.debug("query_url --> %s", params['url'])
return params
def response(resp):
results = []
dom = html.fromstring(resp.text)
# parse results
for result in eval_xpath_list(dom, "//div[@id='content']//div[@class='listWidget']/div/div[@class='appRow']"):
link = eval_xpath_getindex(result, './/h5/a', 0)
url = base_url + link.attrib.get('href') + '#downloads'
title = extract_text(link)
img_src = base_url + eval_xpath_getindex(result, './/img/@src', 0)
res = {
'url': url,
'title': title,
'img_src': img_src
}
results.append(res)
return results

73
src/searx/searx/engines/bandcamp.py
View File

@ -1,73 +0,0 @@
"""
Bandcamp (Music)
@website https://bandcamp.com/
@provide-api no
@results HTML
@parse url, title, content, publishedDate, embedded, thumbnail
"""
from urllib.parse import urlencode, urlparse, parse_qs
from dateutil.parser import parse as dateparse
from lxml import html
from searx.utils import extract_text
categories = ['music']
paging = True
base_url = "https://bandcamp.com/"
search_string = search_string = 'search?{query}&page={page}'
embedded_url = '''<iframe width="100%" height="166"
scrolling="no" frameborder="no"
data-src="https://bandcamp.com/EmbeddedPlayer/{type}={result_id}/size=large/bgcol=ffffff/linkcol=0687f5/tracklist=false/artwork=small/transparent=true/"
></iframe>'''
def request(query, params):
'''pre-request callback
params<dict>:
method : POST/GET
headers : {}
data : {} # if method == POST
url : ''
category: 'search category'
pageno : 1 # number of the requested page
'''
search_path = search_string.format(
query=urlencode({'q': query}),
page=params['pageno'])
params['url'] = base_url + search_path
return params
def response(resp):
'''post-response callback
resp: requests response object
'''
results = []
tree = html.fromstring(resp.text)
search_results = tree.xpath('//li[contains(@class, "searchresult")]')
for result in search_results:
link = result.xpath('.//div[@class="itemurl"]/a')[0]
result_id = parse_qs(urlparse(link.get('href')).query)["search_item_id"][0]
title = result.xpath('.//div[@class="heading"]/a/text()')
date = dateparse(result.xpath('//div[@class="released"]/text()')[0].replace("released ", ""))
content = result.xpath('.//div[@class="subhead"]/text()')
new_result = {
"url": extract_text(link),
"title": extract_text(title),
"content": extract_text(content),
"publishedDate": date,
}
thumbnail = result.xpath('.//div[@class="art"]/img/@src')
if thumbnail:
new_result['thumbnail'] = thumbnail[0]
if "album" in result.classes:
new_result["embedded"] = embedded_url.format(type='album', result_id=result_id)
elif "track" in result.classes:
new_result["embedded"] = embedded_url.format(type='track', result_id=result_id)
results.append(new_result)
return results

124
src/searx/searx/engines/bing.py
View File

@ -1,124 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""
Bing (Web)
"""
import re
from urllib.parse import urlencode
from lxml import html
from searx import logger
from searx.utils import eval_xpath, extract_text, match_language
logger = logger.getChild('bing engine')
# about
about = {
"website": 'https://www.bing.com',
"wikidata_id": 'Q182496',
"official_api_documentation": 'https://www.microsoft.com/en-us/bing/apis/bing-web-search-api',
"use_official_api": False,
"require_api_key": False,
"results": 'HTML',
}
# engine dependent config
categories = ['general']
paging = True
supported_languages_url = 'https://www.bing.com/account/general'
language_aliases = {'zh-CN': 'zh-CHS', 'zh-TW': 'zh-CHT', 'zh-HK': 'zh-CHT'}
# search-url
base_url = 'https://www.bing.com/'
search_string = 'search?{query}&first={offset}'
def _get_offset_from_pageno(pageno):
return (pageno - 1) * 10 + 1
# do search-request
def request(query, params):
offset = _get_offset_from_pageno(params.get('pageno', 0))
if params['language'] == 'all':
lang = 'EN'
else:
lang = match_language(params['language'], supported_languages, language_aliases)
query = 'language:{} {}'.format(lang.split('-')[0].upper(), query)
search_path = search_string.format(
query=urlencode({'q': query}),
offset=offset)
params['url'] = base_url + search_path
return params
# get response from search-request
def response(resp):
results = []
result_len = 0
dom = html.fromstring(resp.text)
# parse results
for result in eval_xpath(dom, '//div[@class="sa_cc"]'):
link = eval_xpath(result, './/h3/a')[0]
url = link.attrib.get('href')
title = extract_text(link)
content = extract_text(eval_xpath(result, './/p'))
# append result
results.append({'url': url,
'title': title,
'content': content})
# parse results again if nothing is found yet
for result in eval_xpath(dom, '//li[@class="b_algo"]'):
link = eval_xpath(result, './/h2/a')[0]
url = link.attrib.get('href')
title = extract_text(link)
content = extract_text(eval_xpath(result, './/p'))
# append result
results.append({'url': url,
'title': title,
'content': content})
try:
result_len_container = "".join(eval_xpath(dom, '//span[@class="sb_count"]//text()'))
if "-" in result_len_container:
# Remove the part "from-to" for paginated request ...
result_len_container = result_len_container[result_len_container.find("-") * 2 + 2:]
result_len_container = re.sub('[^0-9]', '', result_len_container)
if len(result_len_container) > 0:
result_len = int(result_len_container)
except Exception as e:
logger.debug('result error :\n%s', e)
if result_len and _get_offset_from_pageno(resp.search_params.get("pageno", 0)) > result_len:
return []
results.append({'number_of_results': result_len})
return results
# get supported languages from their site
def _fetch_supported_languages(resp):
lang_tags = set()
setmkt = re.compile('setmkt=([^&]*)')
dom = html.fromstring(resp.text)
lang_links = eval_xpath(dom, "//li/a[contains(@href, 'setmkt')]")
for a in lang_links:
href = eval_xpath(a, './@href')[0]
match = setmkt.search(href)
l_tag = match.groups()[0]
_lang, _nation = l_tag.split('-', 1)
l_tag = _lang.lower() + '-' + _nation.upper()
lang_tags.add(l_tag)
return list(lang_tags)

89
src/searx/searx/engines/btdigg.py
View File

@ -1,89 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""
BTDigg (Videos, Music, Files)
"""
from lxml import html
from urllib.parse import quote, urljoin
from searx.utils import extract_text, get_torrent_size
# about
about = {
"website": 'https://btdig.com',
"wikidata_id": 'Q4836698',
"official_api_documentation": {
'url': 'https://btdig.com/contacts',
'comment': 'on demand'
},
"use_official_api": False,
"require_api_key": False,
"results": 'HTML',
}
# engine dependent config
categories = ['videos', 'music', 'files']
paging = True
# search-url
url = 'https://btdig.com'
search_url = url + '/search?q={search_term}&p={pageno}'
# do search-request
def request(query, params):
params['url'] = search_url.format(search_term=quote(query),
pageno=params['pageno'] - 1)
return params
# get response from search-request
def response(resp):
results = []
dom = html.fromstring(resp.text)
search_res = dom.xpath('//div[@class="one_result"]')
# return empty array if nothing is found
if not search_res:
return []
# parse results
for result in search_res:
link = result.xpath('.//div[@class="torrent_name"]//a')[0]
href = urljoin(url, link.attrib.get('href'))
title = extract_text(link)
excerpt = result.xpath('.//div[@class="torrent_excerpt"]')[0]
content = html.tostring(excerpt, encoding='unicode', method='text', with_tail=False)
# it is better to emit <br/> instead of |, but html tags are verboten
content = content.strip().replace('\n', ' | ')
content = ' '.join(content.split())
filesize = result.xpath('.//span[@class="torrent_size"]/text()')[0].split()[0]
filesize_multiplier = result.xpath('.//span[@class="torrent_size"]/text()')[0].split()[1]
files = (result.xpath('.//span[@class="torrent_files"]/text()') or ['1'])[0]
# convert filesize to byte if possible
filesize = get_torrent_size(filesize, filesize_multiplier)
# convert files to int if possible
try:
files = int(files)
except:
files = None
magnetlink = result.xpath('.//div[@class="torrent_magnet"]//a')[0].attrib['href']
# append result
results.append({'url': href,
'title': title,
'content': content,
'filesize': filesize,
'files': files,
'magnetlink': magnetlink,
'template': 'torrent.html'})
# return results sorted by seeder
return results

172
src/searx/searx/engines/command.py
View File

@ -1,172 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""
Command (offline)
"""
import re
from os.path import expanduser, isabs, realpath, commonprefix
from shlex import split as shlex_split
from subprocess import Popen, PIPE
from threading import Thread
from searx import logger
engine_type = 'offline'
paging = True
command = []
delimiter = {}
parse_regex = {}
query_type = ''
query_enum = []
environment_variables = {}
working_dir = realpath('.')
result_separator = '\n'
result_template = 'key-value.html'
timeout = 4.0
_command_logger = logger.getChild('command')
_compiled_parse_regex = {}
def init(engine_settings):
check_parsing_options(engine_settings)
if 'command' not in engine_settings:
raise ValueError('engine command : missing configuration key: command')
global command, working_dir, result_template, delimiter, parse_regex, timeout, environment_variables
command = engine_settings['command']
if 'working_dir' in engine_settings:
working_dir = engine_settings['working_dir']
if not isabs(engine_settings['working_dir']):
working_dir = realpath(working_dir)
if 'parse_regex' in engine_settings:
parse_regex = engine_settings['parse_regex']
for result_key, regex in parse_regex.items():
_compiled_parse_regex[result_key] = re.compile(regex, flags=re.MULTILINE)
if 'delimiter' in engine_settings:
delimiter = engine_settings['delimiter']
if 'environment_variables' in engine_settings:
environment_variables = engine_settings['environment_variables']
def search(query, params):
cmd = _get_command_to_run(query)
if not cmd:
return []
results = []
reader_thread = Thread(target=_get_results_from_process, args=(results, cmd, params['pageno']))
reader_thread.start()
reader_thread.join(timeout=timeout)
return results
def _get_command_to_run(query):
params = shlex_split(query)
__check_query_params(params)
cmd = []
for c in command:
if c == '{{QUERY}}':
cmd.extend(params)
else:
cmd.append(c)
return cmd
def _get_results_from_process(results, cmd, pageno):
leftover = ''
count = 0
start, end = __get_results_limits(pageno)
with Popen(cmd, stdout=PIPE, stderr=PIPE, env=environment_variables) as process:
line = process.stdout.readline()
while line:
buf = leftover + line.decode('utf-8')
raw_results = buf.split(result_separator)
if raw_results[-1]:
leftover = raw_results[-1]
raw_results = raw_results[:-1]
for raw_result in raw_results:
result = __parse_single_result(raw_result)
if result is None:
_command_logger.debug('skipped result:', raw_result)
continue
if start <= count and count <= end:
result['template'] = result_template
results.append(result)
count += 1
if end < count:
return results
line = process.stdout.readline()
return_code = process.wait(timeout=timeout)
if return_code != 0:
raise RuntimeError('non-zero return code when running command', cmd, return_code)
def __get_results_limits(pageno):
start = (pageno - 1) * 10
end = start + 9
return start, end
def __check_query_params(params):
if not query_type:
return
if query_type == 'path':
query_path = params[-1]
query_path = expanduser(query_path)
if commonprefix([realpath(query_path), working_dir]) != working_dir:
raise ValueError('requested path is outside of configured working directory')
elif query_type == 'enum' and len(query_enum) > 0:
for param in params:
if param not in query_enum:
raise ValueError('submitted query params is not allowed', param, 'allowed params:', query_enum)
def check_parsing_options(engine_settings):
""" Checks if delimiter based parsing or regex parsing is configured correctly """
if 'delimiter' not in engine_settings and 'parse_regex' not in engine_settings:
raise ValueError('failed to init settings for parsing lines: missing delimiter or parse_regex')
if 'delimiter' in engine_settings and 'parse_regex' in engine_settings:
raise ValueError('failed to init settings for parsing lines: too many settings')
if 'delimiter' in engine_settings:
if 'chars' not in engine_settings['delimiter'] or 'keys' not in engine_settings['delimiter']:
raise ValueError
def __parse_single_result(raw_result):
""" Parses command line output based on configuration """
result = {}
if delimiter:
elements = raw_result.split(delimiter['chars'], maxsplit=len(delimiter['keys']) - 1)
if len(elements) != len(delimiter['keys']):
return {}
for i in range(len(elements)):
result[delimiter['keys'][i]] = elements[i]
if parse_regex:
for result_key, regex in _compiled_parse_regex.items():
found = regex.search(raw_result)
if not found:
return {}
result[result_key] = raw_result[found.start():found.end()]
return result

82
src/searx/searx/engines/core.py
View File

@ -1,82 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""CORE (science)
"""
# pylint: disable=missing-function-docstring
from json import loads
from datetime import datetime
from urllib.parse import urlencode
from searx import logger
from searx.exceptions import SearxEngineAPIException
logger = logger.getChild('CORE engine')
about = {
"website": 'https://core.ac.uk',
"wikidata_id": None,
"official_api_documentation": 'https://core.ac.uk/documentation/api/',
"use_official_api": True,
"require_api_key": True,
"results": 'JSON',
}
categories = ['science']
paging = True
nb_per_page = 10
api_key = 'unset'
logger = logger.getChild('CORE engine')
base_url = 'https://core.ac.uk:443/api-v2/search/'
search_string = '{query}?page={page}&pageSize={nb_per_page}&apiKey={apikey}'
def request(query, params):
if api_key == 'unset':
raise SearxEngineAPIException('missing CORE API key')
search_path = search_string.format(
query = urlencode({'q': query}),
nb_per_page = nb_per_page,
page = params['pageno'],
apikey = api_key,
)
params['url'] = base_url + search_path
logger.debug("query_url --> %s", params['url'])
return params
def response(resp):
results = []
json_data = loads(resp.text)
for result in json_data['data']:
source = result['_source']
time = source['publishedDate'] or source['depositedDate']
if time :
date = datetime.fromtimestamp(time / 1000)
else:
date = None
metadata = []
if source['publisher'] and len(source['publisher']) > 3:
metadata.append(source['publisher'])
if source['topics']:
metadata.append(source['topics'][0])
if source['doi']:
metadata.append(source['doi'])
metadata = ' / '.join(metadata)
results.append({
'url': source['urls'][0].replace('http://', 'https://', 1),
'title': source['title'],
'content': source['description'],
'publishedDate': date,
'metadata' : metadata,
})
return results

77
src/searx/searx/engines/deviantart.py
View File

@ -1,77 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""
Deviantart (Images)
"""
# pylint: disable=missing-function-docstring
from urllib.parse import urlencode
from lxml import html
# about
about = {
"website": 'https://www.deviantart.com/',
"wikidata_id": 'Q46523',
"official_api_documentation": 'https://www.deviantart.com/developers/',
"use_official_api": False,
"require_api_key": False,
"results": 'HTML',
}
# engine dependent config
categories = ['images']
paging = True
time_range_support = True
time_range_dict = {
'day': 'popular-24-hours',
'week': 'popular-1-week',
'month': 'popular-1-month',
'year': 'most-recent',
}
# search-url
base_url = 'https://www.deviantart.com'
def request(query, params):
# https://www.deviantart.com/search/deviations?page=5&q=foo
query = {
'page' : params['pageno'],
'q' : query,
}
if params['time_range'] in time_range_dict:
query['order'] = time_range_dict[params['time_range']]
params['url'] = base_url + '/search/deviations?' + urlencode(query)
return params
def response(resp):
results = []
dom = html.fromstring(resp.text)
for row in dom.xpath('//div[contains(@data-hook, "content_row")]'):
for result in row.xpath('./div'):
a_tag = result.xpath('.//a[@data-hook="deviation_link"]')[0]
noscript_tag = a_tag.xpath('.//noscript')
if noscript_tag:
img_tag = noscript_tag[0].xpath('.//img')
else:
img_tag = a_tag.xpath('.//img')
if not img_tag:
continue
img_tag = img_tag[0]
results.append({
'template': 'images.html',
'url': a_tag.attrib.get('href'),
'img_src': img_tag.attrib.get('src'),
'title': img_tag.attrib.get('alt'),
})
return results

60
src/searx/searx/engines/dictzone.py
View File

@ -1,60 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""
Dictzone
"""
from urllib.parse import urljoin
from lxml import html
from searx.utils import eval_xpath
# about
about = {
"website": 'https://dictzone.com/',
"wikidata_id": None,
"official_api_documentation": None,
"use_official_api": False,
"require_api_key": False,
"results": 'HTML',
}
engine_type = 'online_dictionnary'
categories = ['general']
url = 'https://dictzone.com/{from_lang}-{to_lang}-dictionary/{query}'
weight = 100
results_xpath = './/table[@id="r"]/tr'
https_support = True
def request(query, params):
params['url'] = url.format(from_lang=params['from_lang'][2],
to_lang=params['to_lang'][2],
query=params['query'])
return params
def response(resp):
results = []
dom = html.fromstring(resp.text)
for k, result in enumerate(eval_xpath(dom, results_xpath)[1:]):
try:
from_result, to_results_raw = eval_xpath(result, './td')
except:
continue
to_results = []
for to_result in eval_xpath(to_results_raw, './p/a'):
t = to_result.text_content()
if t.strip():
to_results.append(to_result.text_content())
results.append({
'url': urljoin(str(resp.url), '?%d' % k),
'title': from_result.text_content(),
'content': '; '.join(to_results)
})
return results

69
src/searx/searx/engines/digg.py
View File

@ -1,69 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""
Digg (News, Social media)
"""
# pylint: disable=missing-function-docstring
from json import loads
from urllib.parse import urlencode
from datetime import datetime
from lxml import html
# about
about = {
"website": 'https://digg.com',
"wikidata_id": 'Q270478',
"official_api_documentation": None,
"use_official_api": False,
"require_api_key": False,
"results": 'HTML',
}
# engine dependent config
categories = ['news', 'social media']
paging = True
base_url = 'https://digg.com'
# search-url
search_url = base_url + (
'/api/search/'
'?{query}'
'&from={position}'
'&size=20'
'&format=html'
)
def request(query, params):
offset = (params['pageno'] - 1) * 20
params['url'] = search_url.format(
query = urlencode({'q': query}),
position = offset,
)
return params
def response(resp):
results = []
# parse results
for result in loads(resp.text)['mapped']:
# strip html tags and superfluous quotation marks from content
content = html.document_fromstring(
result['excerpt']
).text_content()
# 'created': {'ISO': '2020-10-16T14:09:55Z', ...}
published = datetime.strptime(
result['created']['ISO'], '%Y-%m-%dT%H:%M:%SZ'
)
results.append({
'url': result['url'],
'title': result['title'],
'content' : content,
'template': 'videos.html',
'publishedDate': published,
'thumbnail': result['images']['thumbImage'],
})
return results

186
src/searx/searx/engines/duckduckgo.py
View File

@ -1,186 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint
"""DuckDuckGo Lite
"""
from json import loads
from lxml.html import fromstring
from searx.utils import (
dict_subset,
eval_xpath,
eval_xpath_getindex,
extract_text,
match_language,
)
from searx.network import get
# about
about = {
"website": 'https://lite.duckduckgo.com/lite',
"wikidata_id": 'Q12805',
"official_api_documentation": 'https://duckduckgo.com/api',
"use_official_api": False,
"require_api_key": False,
"results": 'HTML',
}
# engine dependent config
categories = ['general']
paging = True
supported_languages_url = 'https://duckduckgo.com/util/u588.js'
time_range_support = True
language_aliases = {
'ar-SA': 'ar-XA',
'es-419': 'es-XL',
'ja': 'jp-JP',
'ko': 'kr-KR',
'sl-SI': 'sl-SL',
'zh-TW': 'tzh-TW',
'zh-HK': 'tzh-HK'
}
time_range_dict = {
'day': 'd',
'week': 'w',
'month': 'm',
'year': 'y'
}
# search-url
url = 'https://lite.duckduckgo.com/lite'
url_ping = 'https://duckduckgo.com/t/sl_l'
# match query's language to a region code that duckduckgo will accept
def get_region_code(lang, lang_list=None):
if lang == 'all':
return None
lang_code = match_language(lang, lang_list or [], language_aliases, 'wt-WT')
lang_parts = lang_code.split('-')
# country code goes first
return lang_parts[1].lower() + '-' + lang_parts[0].lower()
def request(query, params):
params['url'] = url
params['method'] = 'POST'
params['data']['q'] = query
# The API is not documented, so we do some reverse engineering and emulate
# what https://lite.duckduckgo.com/lite/ does when you press "next Page"
# link again and again ..
params['headers']['Content-Type'] = 'application/x-www-form-urlencoded'
# initial page does not have an offset
if params['pageno'] == 2:
# second page does have an offset of 30
offset = (params['pageno'] - 1) * 30
params['data']['s'] = offset
params['data']['dc'] = offset + 1
elif params['pageno'] > 2:
# third and following pages do have an offset of 30 + n*50
offset = 30 + (params['pageno'] - 2) * 50
params['data']['s'] = offset
params['data']['dc'] = offset + 1
# initial page does not have additional data in the input form
if params['pageno'] > 1:
# request the second page (and more pages) needs 'o' and 'api' arguments
params['data']['o'] = 'json'
params['data']['api'] = 'd.js'
# initial page does not have additional data in the input form
if params['pageno'] > 2:
# request the third page (and more pages) some more arguments
params['data']['nextParams'] = ''
params['data']['v'] = ''
params['data']['vqd'] = ''
region_code = get_region_code(params['language'], supported_languages)
if region_code:
params['data']['kl'] = region_code
params['cookies']['kl'] = region_code
params['data']['df'] = ''
if params['time_range'] in time_range_dict:
params['data']['df'] = time_range_dict[params['time_range']]
params['cookies']['df'] = time_range_dict[params['time_range']]
return params
# get response from search-request
def response(resp):
headers_ping = dict_subset(resp.request.headers, ['User-Agent', 'Accept-Encoding', 'Accept', 'Cookie'])
get(url_ping, headers=headers_ping)
if resp.status_code == 303:
return []
results = []
doc = fromstring(resp.text)
result_table = eval_xpath(doc, '//html/body/form/div[@class="filters"]/table')
if not len(result_table) >= 3:
# no more results
return []
result_table = result_table[2]
tr_rows = eval_xpath(result_table, './/tr')
# In the last <tr> is the form of the 'previous/next page' links
tr_rows = tr_rows[:-1]
len_tr_rows = len(tr_rows)
offset = 0
while len_tr_rows >= offset + 4:
# assemble table rows we need to scrap
tr_title = tr_rows[offset]
tr_content = tr_rows[offset + 1]
offset += 4
# ignore sponsored Adds <tr class="result-sponsored">
if tr_content.get('class') == 'result-sponsored':
continue
a_tag = eval_xpath_getindex(tr_title, './/td//a[@class="result-link"]', 0, None)
if a_tag is None:
continue
td_content = eval_xpath_getindex(tr_content, './/td[@class="result-snippet"]', 0, None)
if td_content is None:
continue
results.append({
'title': a_tag.text_content(),
'content': extract_text(td_content),
'url': a_tag.get('href'),
})
return results
# get supported languages from their site
def _fetch_supported_languages(resp):
# response is a js file with regions as an embedded object
response_page = resp.text
response_page = response_page[response_page.find('regions:{') + 8:]
response_page = response_page[:response_page.find('}') + 1]
regions_json = loads(response_page)
supported_languages = map((lambda x: x[3:] + '-' + x[:2].upper()), regions_json.keys())
return list(supported_languages)

261
src/searx/searx/engines/duckduckgo_definitions.py
View File

@ -1,261 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""
DuckDuckGo (Instant Answer API)
"""
import json
from urllib.parse import urlencode, urlparse, urljoin
from lxml import html
from searx import logger
from searx.data import WIKIDATA_UNITS
from searx.engines.duckduckgo import language_aliases
from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url # NOQA # pylint: disable=unused-import
from searx.utils import extract_text, html_to_text, match_language, get_string_replaces_function
from searx.external_urls import get_external_url, get_earth_coordinates_url, area_to_osm_zoom
logger = logger.getChild('duckduckgo_definitions')
# about
about = {
"website": 'https://duckduckgo.com/',
"wikidata_id": 'Q12805',
"official_api_documentation": 'https://duckduckgo.com/api',
"use_official_api": True,
"require_api_key": False,
"results": 'JSON',
}
URL = 'https://api.duckduckgo.com/'\
+ '?{query}&format=json&pretty=0&no_redirect=1&d=1'
WIKIDATA_PREFIX = [
'http://www.wikidata.org/entity/',
'https://www.wikidata.org/entity/'
]
replace_http_by_https = get_string_replaces_function({'http:': 'https:'})
def is_broken_text(text):
""" duckduckgo may return something like "<a href="xxxx">http://somewhere Related website<a/>"
The href URL is broken, the "Related website" may contains some HTML.
The best solution seems to ignore these results.
"""
return text.startswith('http') and ' ' in text
def result_to_text(text, htmlResult):
# TODO : remove result ending with "Meaning" or "Category"
result = None
dom = html.fromstring(htmlResult)
a = dom.xpath('//a')
if len(a) >= 1:
result = extract_text(a[0])
else:
result = text
if not is_broken_text(result):
return result
return None
def request(query, params):
params['url'] = URL.format(query=urlencode({'q': query}))
language = match_language(params['language'], supported_languages, language_aliases)
language = language.split('-')[0]
params['headers']['Accept-Language'] = language
return params
def response(resp):
results = []
search_res = json.loads(resp.text)
# search_res.get('Entity') possible values (not exhaustive) :
# * continent / country / department / location / waterfall
# * actor / musician / artist
# * book / performing art / film / television / media franchise / concert tour / playwright
# * prepared food
# * website / software / os / programming language / file format / software engineer
# * compagny
content = ''
heading = search_res.get('Heading', '')
attributes = []
urls = []
infobox_id = None
relatedTopics = []
# add answer if there is one
answer = search_res.get('Answer', '')
if answer:
logger.debug('AnswerType="%s" Answer="%s"', search_res.get('AnswerType'), answer)
if search_res.get('AnswerType') not in ['calc', 'ip']:
results.append({'answer': html_to_text(answer)})
# add infobox
if 'Definition' in search_res:
content = content + search_res.get('Definition', '')
if 'Abstract' in search_res:
content = content + search_res.get('Abstract', '')
# image
image = search_res.get('Image')
image = None if image == '' else image
if image is not None and urlparse(image).netloc == '':
image = urljoin('https://duckduckgo.com', image)
# urls
# Official website, Wikipedia page
for ddg_result in search_res.get('Results', []):
firstURL = ddg_result.get('FirstURL')
text = ddg_result.get('Text')
if firstURL is not None and text is not None:
urls.append({'title': text, 'url': firstURL})
results.append({'title': heading, 'url': firstURL})
# related topics
for ddg_result in search_res.get('RelatedTopics', []):
if 'FirstURL' in ddg_result:
firstURL = ddg_result.get('FirstURL')
text = ddg_result.get('Text')
if not is_broken_text(text):
suggestion = result_to_text(text,
ddg_result.get('Result'))
if suggestion != heading and suggestion is not None:
results.append({'suggestion': suggestion})
elif 'Topics' in ddg_result:
suggestions = []
relatedTopics.append({'name': ddg_result.get('Name', ''),
'suggestions': suggestions})
for topic_result in ddg_result.get('Topics', []):
suggestion = result_to_text(topic_result.get('Text'),
topic_result.get('Result'))
if suggestion != heading and suggestion is not None:
suggestions.append(suggestion)
# abstract
abstractURL = search_res.get('AbstractURL', '')
if abstractURL != '':
# add as result ? problem always in english
infobox_id = abstractURL
urls.append({'title': search_res.get('AbstractSource'),
'url': abstractURL,
'official': True})
results.append({'url': abstractURL,
'title': heading})
# definition
definitionURL = search_res.get('DefinitionURL', '')
if definitionURL != '':
# add as result ? as answer ? problem always in english
infobox_id = definitionURL
urls.append({'title': search_res.get('DefinitionSource'),
'url': definitionURL})
# to merge with wikidata's infobox
if infobox_id:
infobox_id = replace_http_by_https(infobox_id)
# attributes
# some will be converted to urls
if 'Infobox' in search_res:
infobox = search_res.get('Infobox')
if 'content' in infobox:
osm_zoom = 17
coordinates = None
for info in infobox.get('content'):
data_type = info.get('data_type')
data_label = info.get('label')
data_value = info.get('value')
# Workaround: ddg may return a double quote
if data_value == '""':
continue
# Is it an external URL ?
# * imdb_id / facebook_profile / youtube_channel / youtube_video / twitter_profile
# * instagram_profile / rotten_tomatoes / spotify_artist_id / itunes_artist_id / soundcloud_id
# * netflix_id
external_url = get_external_url(data_type, data_value)
if external_url is not None:
urls.append({'title': data_label,
'url': external_url})
elif data_type in ['instance', 'wiki_maps_trigger', 'google_play_artist_id']:
# ignore instance: Wikidata value from "Instance Of" (Qxxxx)
# ignore wiki_maps_trigger: reference to a javascript
# ignore google_play_artist_id: service shutdown
pass
elif data_type == 'string' and data_label == 'Website':
# There is already an URL for the website
pass
elif data_type == 'area':
attributes.append({'label': data_label,
'value': area_to_str(data_value),
'entity': 'P2046'})
osm_zoom = area_to_osm_zoom(data_value.get('amount'))
elif data_type == 'coordinates':
if data_value.get('globe') == 'http://www.wikidata.org/entity/Q2':
# coordinate on Earth
# get the zoom information from the area
coordinates = info
else:
# coordinate NOT on Earth
attributes.append({'label': data_label,
'value': data_value,
'entity': 'P625'})
elif data_type == 'string':
attributes.append({'label': data_label,
'value': data_value})
if coordinates:
data_label = coordinates.get('label')
data_value = coordinates.get('value')
latitude = data_value.get('latitude')
longitude = data_value.get('longitude')
url = get_earth_coordinates_url(latitude, longitude, osm_zoom)
urls.append({'title': 'OpenStreetMap',
'url': url,
'entity': 'P625'})
if len(heading) > 0:
# TODO get infobox.meta.value where .label='article_title'
if image is None and len(attributes) == 0 and len(urls) == 1 and\
len(relatedTopics) == 0 and len(content) == 0:
results.append({'url': urls[0]['url'],
'title': heading,
'content': content})
else:
results.append({'infobox': heading,
'id': infobox_id,
'content': content,
'img_src': image,
'attributes': attributes,
'urls': urls,
'relatedTopics': relatedTopics})
return results
def unit_to_str(unit):
for prefix in WIKIDATA_PREFIX:
if unit.startswith(prefix):
wikidata_entity = unit[len(prefix):]
return WIKIDATA_UNITS.get(wikidata_entity, unit)
return unit
def area_to_str(area):
"""parse {'unit': 'http://www.wikidata.org/entity/Q712226', 'amount': '+20.99'}"""
unit = unit_to_str(area.get('unit'))
if unit is not None:
try:
amount = float(area.get('amount'))
return '{} {}'.format(amount, unit)
except ValueError:
pass
return '{} {}'.format(area.get('amount', ''), area.get('unit', ''))

127
src/searx/searx/engines/flickr_noapi.py
View File

@ -1,127 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""
Flickr (Images)
"""
from json import loads
from time import time
import re
from urllib.parse import urlencode
from searx.engines import logger
from searx.utils import ecma_unescape, html_to_text
logger = logger.getChild('flickr-noapi')
# about
about = {
"website": 'https://www.flickr.com',
"wikidata_id": 'Q103204',
"official_api_documentation": 'https://secure.flickr.com/services/api/flickr.photos.search.html',
"use_official_api": False,
"require_api_key": False,
"results": 'HTML',
}
categories = ['images']
url = 'https://www.flickr.com/'
search_url = url + 'search?{query}&page={page}'
time_range_url = '&min_upload_date={start}&max_upload_date={end}'
photo_url = 'https://www.flickr.com/photos/{userid}/{photoid}'
modelexport_re = re.compile(r"^\s*modelExport:\s*({.*}),$", re.M)
image_sizes = ('o', 'k', 'h', 'b', 'c', 'z', 'n', 'm', 't', 'q', 's')
paging = True
time_range_support = True
time_range_dict = {'day': 60 * 60 * 24,
'week': 60 * 60 * 24 * 7,
'month': 60 * 60 * 24 * 7 * 4,
'year': 60 * 60 * 24 * 7 * 52}
def build_flickr_url(user_id, photo_id):
return photo_url.format(userid=user_id, photoid=photo_id)
def _get_time_range_url(time_range):
if time_range in time_range_dict:
return time_range_url.format(start=time(), end=str(int(time()) - time_range_dict[time_range]))
return ''
def request(query, params):
params['url'] = (search_url.format(query=urlencode({'text': query}), page=params['pageno'])
+ _get_time_range_url(params['time_range']))
return params
def response(resp):
results = []
matches = modelexport_re.search(resp.text)
if matches is None:
return results
match = matches.group(1)
model_export = loads(match)
if 'legend' not in model_export:
return results
legend = model_export['legend']
# handle empty page
if not legend or not legend[0]:
return results
for index in legend:
photo = model_export['main'][index[0]][int(index[1])][index[2]][index[3]][int(index[4])]
author = ecma_unescape(photo.get('realname', ''))
source = ecma_unescape(photo.get('username', '')) + ' @ Flickr'
title = ecma_unescape(photo.get('title', ''))
content = html_to_text(ecma_unescape(photo.get('description', '')))
img_src = None
# From the biggest to the lowest format
for image_size in image_sizes:
if image_size in photo['sizes']:
img_src = photo['sizes'][image_size]['url']
img_format = 'jpg ' \
+ str(photo['sizes'][image_size]['width']) \
+ 'x' \
+ str(photo['sizes'][image_size]['height'])
break
if not img_src:
logger.debug('cannot find valid image size: {0}'.format(repr(photo)))
continue
# For a bigger thumbnail, keep only the url_z, not the url_n
if 'n' in photo['sizes']:
thumbnail_src = photo['sizes']['n']['url']
elif 'z' in photo['sizes']:
thumbnail_src = photo['sizes']['z']['url']
else:
thumbnail_src = img_src
if 'ownerNsid' not in photo:
# should not happen, disowned photo? Show it anyway
url = img_src
else:
url = build_flickr_url(photo['ownerNsid'], photo['id'])
result = {
'url': url,
'img_src': img_src,
'thumbnail_src': thumbnail_src,
'source': source,
'img_format': img_format,
'template': 'images.html'
}
result['author'] = author.encode(errors='ignore').decode()
result['source'] = source.encode(errors='ignore').decode()
result['title'] = title.encode(errors='ignore').decode()
result['content'] = content.encode(errors='ignore').decode()
results.append(result)
return results

Some files were not shown because too many files have changed in this diff Show More