Browse Source

Merge branch 'master' into fix/manage.sh

libremiami
Markus Heiser 2 years ago
committed by GitHub
parent
commit
4bae1a9eab
No known key found for this signature in database GPG Key ID: 4AEE18F83AFDEB23
  1. 8
      .dockerignore
  2. 5
      .gitignore
  3. 31
      .travis.yml
  4. 24
      AUTHORS.rst
  5. 38
      CHANGELOG.rst
  6. 12
      Dockerfile
  7. 195
      Makefile
  8. 14
      README.rst
  9. 4
      dockerfiles/uwsgi.ini
  10. 130
      docs/_themes/searx/static/searx.css
  11. 6
      docs/_themes/searx/theme.conf
  12. 96
      docs/admin/api.rst
  13. 33
      docs/admin/arch_public.dot
  14. 24
      docs/admin/architecture.rst
  15. 103
      docs/admin/buildhosts.rst
  16. 71
      docs/admin/engines.rst
  17. 148
      docs/admin/filtron.rst
  18. 16
      docs/admin/index.rst
  19. 349
      docs/admin/installation.rst
  20. 26
      docs/admin/morty.rst
  21. 39
      docs/admin/plugins.rst
  22. 181
      docs/admin/settings.rst
  23. 43
      docs/blog/admin.rst
  24. 11
      docs/blog/index.rst
  25. 77
      docs/blog/intro-offline.rst
  26. 63
      docs/blog/private-engines.rst
  27. 68
      docs/blog/python3.rst
  28. BIN
      docs/blog/searx-admin-engines.png
  29. BIN
      docs/blog/searxpy3.png
  30. 114
      docs/conf.py
  31. 180
      docs/dev/contribution_guide.rst
  32. 6
      docs/dev/csv_table.txt
  33. 267
      docs/dev/engine_overview.rst
  34. 3
      docs/dev/hello.dot
  35. 15
      docs/dev/index.rst
  36. 221
      docs/dev/makefile.rst
  37. 54
      docs/dev/plugins.rst
  38. 132
      docs/dev/quickstart.rst
  39. 1428
      docs/dev/reST.rst
  40. 120
      docs/dev/search_api.rst
  41. 10
      docs/dev/svg_image.svg
  42. 71
      docs/dev/translation.rst
  43. 34
      docs/index.rst
  44. BIN
      docs/static/img/searx_logo_small.png
  45. 9
      docs/user/index.rst
  46. 77
      docs/user/own-instance.rst
  47. 3
      docs/user/public_instances.rst
  48. 42
      docs/user/search_syntax.rst
  49. 100
      manage.sh
  50. 7
      requirements-dev.txt
  51. 18
      requirements.txt
  52. 1
      searx/__init__.py
  53. 9
      searx/autocomplete.py
  54. 5
      searx/brand.py
  55. 55819
      searx/data/engines_languages.json
  56. 13
      searx/data/useragents.json
  57. 9
      searx/engines/__init__.py
  58. 28
      searx/engines/bing.py
  59. 21
      searx/engines/bing_images.py
  60. 4
      searx/engines/bing_news.py
  61. 6
      searx/engines/bing_videos.py
  62. 12
      searx/engines/dummy-offline.py
  63. 54
      searx/engines/etools.py
  64. 96
      searx/engines/faroo.py
  65. 26
      searx/engines/flickr_noapi.py
  66. 1
      searx/engines/genius.py
  67. 41
      searx/engines/gigablast.py
  68. 2
      searx/engines/google_news.py
  69. 9
      searx/engines/ina.py
  70. 2
      searx/engines/microsoft_academic.py
  71. 2
      searx/engines/openstreetmap.py
  72. 1
      searx/engines/qwant.py
  73. 2
      searx/engines/scanr_structures.py
  74. 14
      searx/engines/spotify.py
  75. 7
      searx/engines/startpage.py
  76. 9
      searx/engines/wikipedia.py
  77. 16
      searx/languages.py
  78. 253
      searx/preferences.py
  79. 4
      searx/query.py
  80. 4
      searx/results.py
  81. 22
      searx/search.py
  82. 131
      searx/settings.yml
  83. 8
      searx/settings_robot.yml
  84. 44
      searx/static/plugins/js/infinite_scroll.js
  85. 12
      searx/static/plugins/js/vim_hotkeys.js
  86. 2
      searx/static/themes/oscar/.gitignore
  87. 4
      searx/static/themes/oscar/gruntfile.js
  88. 3
      searx/static/themes/oscar/js/searx.js
  89. 4
      searx/static/themes/oscar/js/searx.min.js
  90. 3
      searx/static/themes/oscar/js/searx_src/autocompleter.js
  91. 1
      searx/static/themes/simple/.gitignore
  92. 6
      searx/static/themes/simple/gruntfile.js
  93. 138
      searx/templates/__common__/about.html
  94. 24
      searx/templates/__common__/opensearch_response_rss.xml
  95. 2
      searx/templates/courgette/github_ribbon.html
  96. 8
      searx/templates/oscar/base.html
  97. 2
      searx/templates/oscar/infobox.html
  98. 6
      searx/templates/oscar/preferences.html
  99. 124
      searx/templates/oscar/results.html
  100. 1
      searx/templates/oscar/search.html

8
.dockerignore

@ -4,6 +4,9 @@
*/*/*/*~
*/*/*/*/*~
#
local/
# Git
.git
.gitignore
@ -36,6 +39,11 @@ robot_report.html
test_basic/
setup.cfg
# node_modules
node_modules/
*/node_modules/
*/*/node_modules/
*/*/*/node_modules/
*/*/*/*/node_modules/
.tx/

5
.gitignore

@ -15,9 +15,12 @@ setup.cfg
*/*.pyc
*~
node_modules/
/node_modules
.tx/
build/
dist/
local/
gh-pages/
searx.egg-info/

31
.travis.yml

@ -1,26 +1,24 @@
os: linux
dist: bionic
language: python
sudo: false
cache:
- pip
- npm
- directories:
- $HOME/.cache/pip
addons:
firefox: "latest"
install:
- ./manage.sh install_geckodriver ~/drivers
- export PATH=~/drivers:$PATH
- ./manage.sh npm_packages
- ./manage.sh update_dev_packages
- pip install codecov
- env
- which python; python --version
- make V=1 install
- make V=1 gecko.driver
- make V=1 node.env
- make V=1 travis.codecov
script:
- ./manage.sh styles
- ./manage.sh grunt_build
- ./manage.sh tests
- make V=1 themes
- make V=1 test
after_success:
- ./manage.sh py_test_coverage
- make V=1 test.coverage
- codecov
stages:
@ -31,10 +29,13 @@ stages:
jobs:
include:
- python: "2.7"
env: PY=2
- python: "3.5"
- python: "3.6"
- python: "3.7"
- python: "3.8"
- stage: docker
python: "3.6"
python: "3.8"
git:
depth: false
services:
@ -44,7 +45,7 @@ jobs:
install: true
script:
- echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin
- ./manage.sh docker_build push
- make -e GIT_URL=$(git remote get-url origin) docker.push
after_success: true
notifications:

24
AUTHORS.rst

@ -100,3 +100,27 @@ generally made searx better:
- @ZEROF
- Ivan Skytte Jørgensen @isj-privacore
- @miicha
- Étienne Deparis @milouse
- @pelag0s
- Denis Wernert @d-tux
- Robin Hallabro-Kokko @hallabro
- Jonas Zohren @jfowl
- Elias Ojala @theel0ja
- @brunob
- Nick Espig @nachtalb
- Rachmadani Haryono @rachmadaniHaryono
- Frank de Lange @yetangitu
- Émilien Devos @unifox
- Nicolas Gelot @nfk
- @volth
- Mathieu Brunot @madmath03
- @lorddavidiii
- @x250
- Robby O'Connor @robbyoconnor
- Finn @0xhtml
- @tmikaeld
- @hobbestigrou
- Vipul @finn0
- @CaffeinatedTech
- Robin Schneider @ypid
- @splintah

38
CHANGELOG.rst

@ -1,3 +1,41 @@
0.16.0 2020.01.30
=================
- New engines
- Splash
- Apkmirror
- NPM search
- Invidious
- Seedpeer
- New languages
- Estonian
- Interlingua
- Lithuanian
- Tibetian
- Occitan
- Tamil
- Engine fixes (wolframalpha, google scholar, youtube, google images, seznam, google, soundcloud, google cloud, duden, btdigg, google play, bing images, flickr noapi, wikidata, dailymotion, google videos, arxiv, dictzone, fdroid, etymonline, bing, duckduckgo, startpage, voat, 1x, deviantart, digg, gigablast, mojeek, duckduckgo definitions, spotify, libgen, qwant, openstreetmap, wikipedia, ina, microsoft academic, scanr structures)
- Dependency updates
- Speed optimizations
- Initial support for offline engines
- Image format display
- Inline js scripts removed
- Infinite scroll plugin fix
- Simple theme bugfixes
- Docker image updates
- Bang expression fixes
- Result merging fixes
- New environment variable added: SEARX_BIND_ADDRESS
News
~~~~
- @return42 joined the maintainer team
- This is the last release with Python2 support
0.15.0 2019.01.06
=================

12
Dockerfile

@ -4,6 +4,7 @@ EXPOSE 8080
VOLUME /etc/searx
VOLUME /var/log/uwsgi
ARG GIT_URL=unknown
ARG VERSION_GITCOMMIT=unknown
ARG SEARX_GIT_VERSION=unknown
@ -66,7 +67,7 @@ RUN su searx -c "/usr/bin/python3 -m compileall -q searx"; \
# Keep this argument at the end since it change each time
ARG LABEL_DATE=
LABEL maintainer="searx <https://github.com/asciimoo/searx>" \
LABEL maintainer="searx <${GIT_URL}>" \
description="A privacy-respecting, hackable metasearch engine." \
version="${SEARX_GIT_VERSION}" \
org.label-schema.schema-version="1.0" \
@ -76,4 +77,11 @@ LABEL maintainer="searx <https://github.com/asciimoo/searx>" \
org.label-schema.vcs-ref=${LABEL_VCS_REF} \
org.label-schema.vcs-url=${LABEL_VCS_URL} \
org.label-schema.build-date="${LABEL_DATE}" \
org.label-schema.usage="https://github.com/searx/searx-docker"
org.label-schema.usage="https://github.com/searx/searx-docker" \
org.opencontainers.image.title="searx" \
org.opencontainers.image.version="${SEARX_GIT_VERSION}" \
org.opencontainers.image.url="${LABEL_VCS_URL}" \
org.opencontainers.image.revision=${LABEL_VCS_REF} \
org.opencontainers.image.source=${LABEL_VCS_URL} \
org.opencontainers.image.created="${LABEL_DATE}" \
org.opencontainers.image.documentation="https://github.com/searx/searx-docker"

195
Makefile

@ -1,36 +1,54 @@
# -*- coding: utf-8; mode: makefile-gmake -*-
export GIT_URL=https://github.com/asciimoo/searx
export SEARX_URL=https://searx.me
export DOCS_URL=https://asciimoo.github.io/searx
PYOBJECTS = searx
DOC = docs
PY_SETUP_EXTRAS ?= \[test\]
PYDIST=./dist/py
PYBUILD=./build/py
include utils/makefile.include
include utils/makefile.python
include utils/makefile.sphinx
all: clean install
PHONY += help
help:
@echo ' test - run developer tests'
@echo ' docs - build documentation'
@echo ' docs-live - autobuild HTML documentation while editing'
@echo ' run - run developer instance'
@echo ' install - developer install (./local)'
@echo ' uninstall - uninstall (./local)'
@echo ' gh-pages - build docs & deploy on gh-pages branch'
@echo ' clean - drop builds and environments'
@echo ' project - re-build generic files of the searx project'
@echo ' buildenv - re-build environment files (aka brand)'
@echo ' themes - re-build build the source of the themes'
@echo ' docker - build Docker image'
@echo ' node.env - download & install npm dependencies locally'
@echo ''
@$(MAKE) -s -f utils/makefile.include make-help
@echo ''
@$(MAKE) -s -f utils/makefile.python python-help
PHONY += install
install: pyenvinstall
install: buildenv pyenvinstall
PHONY += uninstall
uninstall: pyenvuninstall
PHONY += clean
clean: pyclean
clean: pyclean node.clean test.clean
$(call cmd,common_clean)
PHONY += run
run: pyenvinstall
run: buildenv pyenvinstall
$(Q) ( \
sed -i -e "s/debug : False/debug : True/g" ./searx/settings.yml ; \
sleep 2 ; \
@ -40,23 +58,178 @@ run: pyenvinstall
) &
$(PY_ENV)/bin/python ./searx/webapp.py
# docs
# ----
PHONY += docs
docs: buildenv pyenvinstall sphinx-doc
$(call cmd,sphinx,html,docs,docs)
PHONY += docs-live
docs-live: buildenv pyenvinstall sphinx-live
$(call cmd,sphinx_autobuild,html,docs,docs)
$(GH_PAGES)::
@echo "doc available at --> $(DOCS_URL)"
# update project files
# --------------------
PHONY += project engines.languages useragents.update buildenv
project: buildenv useragents.update engines.languages
engines.languages: pyenvinstall
$(Q)echo "fetch languages .."
$(Q)$(PY_ENV_ACT); python utils/fetch_languages.py
$(Q)echo "update searx/data/engines_languages.json"
$(Q)mv engines_languages.json searx/data/engines_languages.json
$(Q)echo "update searx/languages.py"
$(Q)mv languages.py searx/languages.py
useragents.update: pyenvinstall
$(Q)echo "Update searx/data/useragents.json with the most recent versions of Firefox."
$(Q)$(PY_ENV_ACT); python utils/fetch_firefox_version.py
buildenv:
$(Q)echo "build searx/brand.py"
$(Q)echo "GIT_URL = '$(GIT_URL)'" > searx/brand.py
$(Q)echo "ISSUE_URL = 'https://github.com/asciimoo/searx/issues'" >> searx/brand.py
$(Q)echo "SEARX_URL = '$(SEARX_URL)'" >> searx/brand.py
$(Q)echo "DOCS_URL = '$(DOCS_URL)'" >> searx/brand.py
$(Q)echo "PUBLIC_INSTANCES = 'https://searx.space'" >> searx/brand.py
$(Q)echo "build utils/brand.env"
$(Q)echo "export GIT_URL='$(GIT_URL)'" > utils/brand.env
$(Q)echo "export ISSUE_URL='https://github.com/asciimoo/searx/issues'" >> utils/brand.env
$(Q)echo "export SEARX_URL='$(SEARX_URL)'" >> utils/brand.env
$(Q)echo "export DOCS_URL='$(DOCS_URL)'" >> utils/brand.env
$(Q)echo "export PUBLIC_INSTANCES='https://searx.space'" >> utils/brand.env
# node / npm
# ----------
node.env: buildenv
$(Q)./manage.sh npm_packages
node.clean:
$(Q)echo "CLEAN locally installed npm dependencies"
$(Q)rm -rf \
./node_modules \
./package-lock.json \
./searx/static/themes/oscar/package-lock.json \
./searx/static/themes/oscar/node_modules \
./searx/static/themes/simple/package-lock.json \
./searx/static/themes/simple/node_modules
# build themes
# ------------
PHONY += themes.bootstrap themes themes.oscar themes.simple themes.legacy themes.courgette themes.pixart
themes: buildenv themes.bootstrap themes.oscar themes.simple themes.legacy themes.courgette themes.pixart
quiet_cmd_lessc = LESSC $3
cmd_lessc = PATH="$$(npm bin):$$PATH" \
lessc --clean-css="--s1 --advanced --compatibility=ie9" "searx/static/$2" "searx/static/$3"
quiet_cmd_grunt = GRUNT $2
cmd_grunt = PATH="$$(npm bin):$$PATH" \
grunt --gruntfile "$2"
themes.oscar:
$(Q)echo '[!] build oscar theme'
$(call cmd,grunt,searx/static/themes/oscar/gruntfile.js)
themes.simple:
$(Q)echo '[!] build simple theme'
$(call cmd,grunt,searx/static/themes/simple/gruntfile.js)
themes.legacy:
$(Q)echo '[!] build legacy theme'
$(call cmd,lessc,themes/legacy/less/style-rtl.less,themes/legacy/css/style-rtl.css)
$(call cmd,lessc,themes/legacy/less/style.less,themes/legacy/css/style.css)
themes.courgette:
$(Q)echo '[!] build courgette theme'
$(call cmd,lessc,themes/courgette/less/style.less,themes/courgette/css/style.css)
$(call cmd,lessc,themes/courgette/less/style-rtl.less,themes/courgette/css/style-rtl.css)
themes.pixart:
$(Q)echo '[!] build pixart theme'
$(call cmd,lessc,themes/pix-art/less/style.less,themes/pix-art/css/style.css)
themes.bootstrap:
$(call cmd,lessc,less/bootstrap/bootstrap.less,css/bootstrap.min.css)
# docker
# ------
PHONY += docker
docker: buildenv
$(Q)./manage.sh docker_build
docker.push: buildenv
$(Q)./manage.sh docker_build push
# gecko
# -----
PHONY += gecko.driver
gecko.driver:
$(PY_ENV_ACT); ./manage.sh install_geckodriver
# test
# ----
PHONY += test test.pylint test.pep8 test.unit test.robot
PHONY += test test.pylint test.pep8 test.unit test.coverage test.robot
test: buildenv test.pylint test.pep8 test.unit gecko.driver test.robot
ifeq ($(PY),2)
test.pylint:
@echo "LINT skip liniting py2"
else
# TODO: balance linting with pylint
test: test.pep8 test.unit test.robot
- make pylint
test.pylint: pyenvinstall
$(call cmd,pylint,\
searx/preferences.py \
searx/testing.py \
)
endif
# ignored rules:
# E402 module level import not at top of file
# W503 line break before binary operator
test.pep8: pyenvinstall
$(PY_ENV_ACT); ./manage.sh pep8_check
@echo "TEST pep8"
$(Q)$(PY_ENV_ACT); pep8 --exclude=searx/static --max-line-length=120 --ignore "E402,W503" searx tests
test.unit: pyenvinstall
$(PY_ENV_ACT); ./manage.sh unit_tests
@echo "TEST tests/unit"
$(Q)$(PY_ENV_ACT); python -m nose2 -s tests/unit
test.robot: pyenvinstall
$(PY_ENV_ACT); ./manage.sh install_geckodriver
$(PY_ENV_ACT); ./manage.sh robot_tests
test.coverage: pyenvinstall
@echo "TEST unit test coverage"
$(Q)$(PY_ENV_ACT); \
python -m nose2 -C --log-capture --with-coverage --coverage searx -s tests/unit \
&& coverage report \
&& coverage html \
test.robot: pyenvinstall gecko.driver
@echo "TEST robot"
$(Q)$(PY_ENV_ACT); PYTHONPATH=. python searx/testing.py robot
test.clean:
@echo "CLEAN intermediate test stuff"
$(Q)rm -rf geckodriver.log .coverage coverage/
# travis
# ------
travis.codecov:
$(Q)$(PY_ENV_BIN)/python -m pip install codecov
.PHONY: $(PHONY)

14
README.rst

@ -7,7 +7,7 @@ engine <https://en.wikipedia.org/wiki/Metasearch_engine>`__.
Pronunciation: səːks
List of `running
instances <https://github.com/asciimoo/searx/wiki/Searx-instances>`__.
instances <https://searx.space/>`__.
See the `documentation <https://asciimoo.github.io/searx>`__ and the `wiki <https://github.com/asciimoo/searx/wiki>`__ for more information.
@ -18,18 +18,18 @@ Installation
~~~~~~~~~~~~
With Docker
------
-----------
Go to the `searx-docker <https://github.com/searx/searx-docker>`__ project.
Without Docker
------
For all the details, follow this `step by step installation <https://asciimoo.github.io/searx/dev/install/installation.html>`__.
--------------
For all of the details, follow this `step by step installation <https://asciimoo.github.io/searx/admin/installation.html>`__.
Note: the documentation needs to be updated.
If you are in hurry
------
- clone source:
If you are in a hurry
---------------------
- clone the source:
``git clone https://github.com/asciimoo/searx.git && cd searx``
- install dependencies: ``./manage.sh update_packages``
- edit your

4
dockerfiles/uwsgi.ini

@ -31,3 +31,7 @@ touch-logrotate = /run/uwsgi-logrotate
unique-cron = 15 0 -1 -1 -1 { touch /run/uwsgi-logrotate }
log-backupname = /var/log/uwsgi/uwsgi.log.1
logto = /var/log/uwsgi/uwsgi.log
# No keep alive
# See https://github.com/searx/searx-docker/issues/24
add-header = Connection: close

130
docs/_themes/searx/static/searx.css

@ -0,0 +1,130 @@
@import url("pocoo.css");
a, a.reference, a.footnote-reference {
color: #004b6b;
border-color: #004b6b;
}
a:hover {
color: #6d4100;
border-color: #6d4100;
}
p.version-warning {
background-color: #004b6b;
}
div.sidebar {
background-color: whitesmoke;
border-color: lightsteelblue;
border-radius: 3pt;
}
p.sidebar-title, .sidebar p {
margin: 6pt;
}
.sidebar li,
.hlist li {
list-style-type: disclosure-closed;
}
/* admonitions
*/
div.admonition, div.topic {
background-color: #fafafa;
margin: 8px 0px;
padding: 1em;
border-radius: 3pt 0 0 3pt;
border-top: none;
border-right: none;
border-bottom: none;
border-left: 5pt solid #ccc;
}
p.admonition-title:after {
content: none;
}
.admonition.hint { border-color: #416dc0b0; }
.admonition.note { border-color: #6c856cb0; }
.admonition.tip { border-color: #85c5c2b0; }
.admonition.attention { border-color: #ecec97b0; }
.admonition.caution { border-color: #a6c677b0; }
.admonition.danger { border-color: #d46262b0; }
.admonition.important { border-color: #dfa3a3b0; }
.admonition.error { border-color: red; }
.admonition.warning { border-color: darkred; }
.admonition.admonition-generic-admonition-title {
border-color: #416dc0b0;
}
/* admonitions with (rendered) reST markup examples (:class: rst-example)
*
* .. admonition:: title of the example
* :class: rst-example
* ....
*/
div.rst-example {
background-color: inherit;
margin: 0;
border-top: none;
border-right: 1px solid #ccc;
border-bottom: none;
border-left: none;
border-radius: none;
padding: 0;
}
div.rst-example > p.admonition-title {
font-family: Sans Serif;
font-style: italic;
font-size: 0.8em;
display: block;
border-bottom: 1px solid #ccc;
padding: 0.5em 1em;
text-align: right;
}
/* code block in figures
*/
div.highlight pre {
text-align: left;
}
/* Table theme
*/
thead, tfoot {
background-color: #fff;
}
th:hover, td:hover {
background-color: #ffc;
}
thead th, tfoot th, tfoot td, tbody th {
background-color: #fffaef;
}
tbody tr:nth-child(odd) {
background-color: #fff;
}
tbody tr:nth-child(even) {
background-color: #fafafa;
}
caption {
font-family: Sans Serif;
padding: 0.5em;
margin: 0.5em 0 0.5em 0;
caption-side: top;
text-align: left;
}

6
docs/_themes/searx/theme.conf

@ -0,0 +1,6 @@
[theme]
inherit = pocoo
stylesheet = searx.css
[options]
touch_icon =

96
docs/admin/api.rst

@ -0,0 +1,96 @@
.. _adminapi:
==================
Administration API
==================
Get configuration data
======================
.. code:: http
GET /config HTTP/1.1
Sample response
---------------
.. code:: json
{
"autocomplete": "",
"categories": [
"map",
"it",
"images",
],
"default_locale": "",
"default_theme": "oscar",
"engines": [
{
"categories": [
"map"
],
"enabled": true,
"name": "openstreetmap",
"shortcut": "osm"
},
{
"categories": [
"it"
],
"enabled": true,
"name": "arch linux wiki",
"shortcut": "al"
},
{
"categories": [
"images"
],
"enabled": true,
"name": "google images",
"shortcut": "goi"
},
{
"categories": [
"it"
],
"enabled": false,
"name": "bitbucket",
"shortcut": "bb"
},
],
"instance_name": "searx",
"locales": {
"de": "Deutsch (German)",
"en": "English",
"eo": "Esperanto (Esperanto)",
},
"plugins": [
{
"enabled": true,
"name": "HTTPS rewrite"
},
{
"enabled": false,
"name": "Vim-like hotkeys"
}
],
"safe_search": 0
}
Embed search bar
================
The search bar can be embedded into websites. Just paste the example into the
HTML of the site. URL of the searx instance and values are customizable.
.. code:: html
<form method="post" action="https://searx.me/">
<!-- search --> <input type="text" name="q" />
<!-- categories --> <input type="hidden" name="categories" value="general,social media" />
<!-- language --> <input type="hidden" name="lang" value="all" />
<!-- locale --> <input type="hidden" name="locale" value="en" />
<!-- date filter --> <input type="hidden" name="time_range" value="month" />
</form>

33
docs/admin/arch_public.dot

@ -0,0 +1,33 @@
digraph G {
node [style=filled, shape=box, fillcolor="#ffffcc", fontname="Sans"];
edge [fontname="Sans"];
browser [label="Browser", shape=Mdiamond];
rp [label="Reverse Proxy", href="url to configure reverse proxy"];
filtron [label="Filtron", href="https://github.com/asciimoo/filtron"];
morty [label="Morty", href="https://github.com/asciimoo/morty"];
static [label="Static files", href="url to configure static files"];
uwsgi [label="uwsgi", href="url to configure uwsgi"]
searx1 [label="Searx #1"];
searx2 [label="Searx #2"];
searx3 [label="Searx #3"];
searx4 [label="Searx #4"];
browser -> rp [label="HTTPS"]
subgraph cluster_searx {
label = "Searx instance" fontname="Sans";
bgcolor="#fafafa";
{ rank=same; static rp };
rp -> morty [label="optional: images and HTML pages proxy"];
rp -> static [label="optional: reverse proxy serves directly static files"];
rp -> filtron [label="HTTP"];
filtron -> uwsgi [label="HTTP"];
uwsgi -> searx1;
uwsgi -> searx2;
uwsgi -> searx3;
uwsgi -> searx4;
}
}

24
docs/admin/architecture.rst

@ -0,0 +1,24 @@
.. _architecture:
============
Architecture
============
.. sidebar:: Needs work!
This article needs some work / Searx is a collaborative effort. If you have
any contribution, feel welcome to send us your :pull:`PR <../pulls>`, see
:ref:`how to contribute`.
Herein you will find some hints and suggestions about typical architectures of
searx infrastructures.
We start with a contribution from :pull:`@dalf <1776#issuecomment-567917320>`.
It shows a *reference* setup for public searx instances.
.. _arch public:
.. kernel-figure:: arch_public.dot
:alt: arch_public.dot
Reference architecture of a public searx setup.

103
docs/admin/buildhosts.rst

@ -0,0 +1,103 @@
.. _buildhosts:
==========
Buildhosts
==========
.. sidebar:: This article needs some work
If you have any contribution send us your :pull:`PR <../pulls>`, see
:ref:`how to contribute`.
To get best results from build, its recommend to install additional packages
on build hosts.
.. _docs build:
Build docs
==========
.. _Graphviz: https://graphviz.gitlab.io
.. _ImageMagick: https://www.imagemagick.org
.. _XeTeX: https://tug.org/xetex/
.. _dvisvgm: https://dvisvgm.de/
.. sidebar:: Sphinx build needs
- ImageMagick_
- Graphviz_
- XeTeX_
- dvisvgm_
Most of the sphinx requirements are installed from :origin:`setup.py` and the
docs can be build from scratch with ``make docs``. For better math and image
processing additional packages are needed. The XeTeX_ needed not only for PDF
creation, its also needed for :ref:`math` when HTML output is build.
To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered
as images (``sphinx.ext.imgmath`` extension). If your docs build (``make
docs``) shows warnings like this::
WARNING: dot(1) not found, for better output quality install \
graphviz from http://www.graphviz.org
..
WARNING: LaTeX command 'latex' cannot be run (needed for math \
display), check the imgmath_latex setting
you need to install additional packages on your build host, to get better HTML
output.
.. _system requirements:
.. tabs::
.. group-tab:: Ubuntu / debian
.. code-block:: sh
$ sudo apt install graphviz imagemagick texlive-xetex librsvg2-bin
.. group-tab:: Arch Linux
.. code-block:: sh
$ sudo pacman -S graphviz imagemagick texlive-bin extra/librsvg
.. group-tab:: Fedora / RHEL
.. code-block:: sh
$ sudo dnf install graphviz graphviz-gd texlive-xetex-bin librsvg2-tools
For PDF output you also need:
.. tabs::
.. group-tab:: Ubuntu / debian
.. code:: sh
$ sudo apt texlive-latex-recommended texlive-extra-utils ttf-dejavu
.. group-tab:: Arch Linux
.. code:: sh
$ sudo pacman -S texlive-core texlive-latexextra ttf-dejavu
.. group-tab:: Fedora / RHEL
.. code:: sh
$ sudo dnf install \
texlive-collection-fontsrecommended texlive-collection-latex \
dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts
.. _system requirements END:
.. literalinclude:: ../conf.py
:language: python
:start-after: # sphinx.ext.imgmath setup
:end-before: # sphinx.ext.imgmath setup END

71
docs/admin/engines.rst

@ -0,0 +1,71 @@
.. _engines generic:
=======
Engines
=======
.. sidebar:: Further reading ..
- :ref:`settings engine`
- :ref:`engine settings`
- :ref:`engine file`
============= =========== ==================== ============
:ref:`engine settings` :ref:`engine file`
------------------------- ---------------------------------
Name (cfg) Categories
------------------------- ---------------------------------
Engine .. Paging support **P**
------------------------- -------------------- ------------
Shortcut **S** Language support **L**
Timeout **TO** Time range support **TR**
Disabled **D** Offline **O**
------------- ----------- -------------------- ------------
Safe search **SS**
------------- ----------- ---------------------------------
Weigth **W**
------------- ----------- ---------------------------------
Disabled **D**
============= =========== =================================
Configuration defaults (at built time):
.. _configured engines:
.. jinja:: webapp
.. flat-table:: Engines configured at built time (defaults)
:header-rows: 1
:stub-columns: 2
* - Name (cfg)
- S
- Engine
- TO
- Categories
- P
- L
- SS
- D
- TR
- O
- W
- D
{% for name, mod in engines.items() %}
* - {{name}}
- !{{mod.shortcut}}
- {{mod.__name__}}
- {{mod.timeout}}
- {{", ".join(mod.categories)}}
- {{(mod.paging and "y") or ""}}
- {{(mod.language_support and "y") or ""}}
- {{(mod.safesearch and "y") or ""}}
- {{(mod.disabled and "y") or ""}}
- {{(mod.time_range_support and "y") or ""}}
- {{(mod.offline and "y") or ""}}
- {{mod.weight or 1 }}
- {{(mod.disabled and "y") or ""}}
{% endfor %}

148
docs/admin/filtron.rst

@ -0,0 +1,148 @@
==========================
How to protect an instance
==========================
Searx depens on external search services. To avoid the abuse of these services
it is advised to limit the number of requests processed by searx.
An application firewall, ``filtron`` solves exactly this problem. Information
on how to install it can be found at the `project page of filtron
<https://github.com/asciimoo/filtron>`__.
Sample configuration of filtron
===============================
An example configuration can be find below. This configuration limits the access
of:
- scripts or applications (roboagent limit)
- webcrawlers (botlimit)
- IPs which send too many requests (IP limit)
- too many json, csv, etc. requests (rss/json limit)
- the same UserAgent of if too many requests (useragent limit)
.. code:: json
[{
"name":"search request",
"filters":[
"Param:q",
"Path=^(/|/search)$"
],
"interval":"<time-interval-in-sec (int)>",
"limit":"<max-request-number-in-interval (int)>",
"subrules":[
{
"name":"roboagent limit",
"interval":"<time-interval-in-sec (int)>",
"limit":"<max-request-number-in-interval (int)>",
"filters":[
"Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"
],
"actions":[
{
"name":"block",
"params":{
"message":"Rate limit exceeded"
}
}
]
},
{
"name":"botlimit",
"limit":0,
"stop":true,
"filters":[
"Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"
],
"actions":[
{
"name":"block",
"params":{
"message":"Rate limit exceeded"
}
}
]
},
{
"name":"IP limit",
"interval":"<time-interval-in-sec (int)>",
"limit":"<max-request-number-in-interval (int)>",
"stop":true,
"aggregations":[
"Header:X-Forwarded-For"
],
"actions":[
{
"name":"block",
"params":{
"message":"Rate limit exceeded"
}
}
]
},
{
"name":"rss/json limit",
"interval":"<time-interval-in-sec (int)>",
"limit":"<max-request-number-in-interval (int)>",
"stop":true,
"filters":[
"Param:format=(csv|json|rss)"
],
"actions":[
{
"name":"block",
"params":{
"message":"Rate limit exceeded"
}
}
]
},
{
"name":"useragent limit",
"interval":"<time-interval-in-sec (int)>",
"limit":"<max-request-number-in-interval (int)>",
"aggregations":[
"Header:User-Agent"
],
"actions":[
{
"name":"block",
"params":{
"message":"Rate limit exceeded"
}
}
]
}
]
}]
Route request through filtron
=============================
Filtron can be started using the following command:
.. code:: sh
$ filtron -rules rules.json
It listens on ``127.0.0.1:4004`` and forwards filtered requests to
``127.0.0.1:8888`` by default.
Use it along with ``nginx`` with the following example configuration.
.. code:: nginx
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_pass http://127.0.0.1:4004/;
}
Requests are coming from port 4004 going through filtron and then forwarded to
port 8888 where a searx is being run.

16
docs/admin/index.rst

@ -0,0 +1,16 @@
===========================
Administrator documentation
===========================
.. toctree::
:maxdepth: 1
installation
settings
api
architecture
filtron
morty
engines
plugins
buildhosts

349
docs/admin/installation.rst

@ -0,0 +1,349 @@
.. _installation:
============
Installation
============
.. contents::
:depth: 3
Basic installation
==================
Step by step installation for Debian/Ubuntu with virtualenv. For Ubuntu, be sure
to have enable universe repository.
Install packages:
.. code:: sh
$ sudo -H apt-get install \
git build-essential libxslt-dev \
python-dev python-virtualenv python-babel \
zlib1g-dev libffi-dev libssl-dev
Install searx:
.. code:: sh
cd /usr/local
sudo -H git clone https://github.com/asciimoo/searx.git
sudo -H useradd searx -d /usr/local/searx
sudo -H chown searx:searx -R /usr/local/searx
Install dependencies in a virtualenv:
.. code:: sh
cd /usr/local/searx
sudo -H -u searx -i
.. code:: sh
(searx)$ virtualenv searx-ve
(searx)$ . ./searx-ve/bin/activate
(searx)$ ./manage.sh update_packages
Configuration
==============
.. code:: sh
sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml
Edit searx/settings.yml if necessary.
Check
=====
Start searx:
.. code:: sh
python searx/webapp.py
Go to http://localhost:8888
If everything works fine, disable the debug option in settings.yml:
.. code:: sh
sed -i -e "s/debug : True/debug : False/g" searx/settings.yml
At this point searx is not demonized ; uwsgi allows this.
You can exit the virtualenv and the searx user bash (enter exit command
twice).
uwsgi
=====
Install packages:
.. code:: sh
sudo -H apt-get install \
uwsgi uwsgi-plugin-python
Create the configuration file ``/etc/uwsgi/apps-available/searx.ini`` with this
content:
.. code:: ini
[uwsgi]
# Who will run the code
uid = searx
gid = searx
# disable logging for privacy
disable-logging = true
# Number of workers (usually CPU count)
workers = 4
# The right granted on the created socket
chmod-socket = 666
# Plugin to use and interpretor config
single-interpreter = true
master = true
plugin = python
lazy-apps = true
enable-threads = true
# Module to import
module = searx.webapp
# Support running the module from a webserver subdirectory.
route-run = fixpathinfo:
# Virtualenv and python path
virtualenv = /usr/local/searx/searx-ve/
pythonpath = /usr/local/searx/
chdir = /usr/local/searx/searx/
Activate the uwsgi application and restart:
.. code:: sh
cd /etc/uwsgi/apps-enabled
ln -s ../apps-available/searx.ini
/etc/init.d/uwsgi restart
Web server
==========
with nginx
----------
If nginx is not installed (uwsgi will not work with the package
nginx-light):
.. code:: sh
sudo -H apt-get install nginx
Hosted at /
~~~~~~~~~~~
Create the configuration file ``/etc/nginx/sites-available/searx`` with this
content:
.. code:: nginx
server {
listen 80;
server_name searx.example.com;
root /usr/local/searx/searx;
location /static {
}
location / {
include uwsgi_params;
uwsgi_pass unix:/run/uwsgi/app/searx/socket;
}
}
Create a symlink to sites-enabled:
.. code:: sh
sudo -H ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx
Restart service:
.. code:: sh
sudo -H service nginx restart
sudo -H service uwsgi restart
from subdirectory URL (/searx)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Add this configuration in the server config file
``/etc/nginx/sites-enabled/default``:
.. code:: nginx
location /searx/static {
alias /usr/local/searx/searx/static;
}
location /searx {
uwsgi_param SCRIPT_NAME /searx;
include uwsgi_params;
uwsgi_pass unix:/run/uwsgi/app/searx/socket;
}
**OR** using reverse proxy (Please, note that reverse proxy advised to be used
in case of single-user or low-traffic instances.)
.. code:: nginx
location /searx/static {
alias /usr/local/searx/searx/static;
}
location /searx {
proxy_pass http://127.0.0.1:8888;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Script-Name /searx;
proxy_buffering off;
}
Enable ``base_url`` in ``searx/settings.yml``
.. code:: yaml
base_url : http://your.domain.tld/searx/
Restart service:
.. code:: sh
sudo -H service nginx restart
sudo -H service uwsgi restart
disable logs
^^^^^^^^^^^^
for better privacy you can disable nginx logs about searx.
how to proceed: below ``uwsgi_pass`` in ``/etc/nginx/sites-available/default``
add:
.. code:: nginx
access_log /dev/null;
error_log /dev/null;
Restart service:
.. code:: sh
sudo -H service nginx restart
with apache
-----------
Add wsgi mod:
.. code:: sh
sudo -H apt-get install libapache2-mod-uwsgi
sudo -H a2enmod uwsgi
Add this configuration in the file ``/etc/apache2/apache2.conf``:
.. code:: apache
<Location />
Options FollowSymLinks Indexes
SetHandler uwsgi-handler
uWSGISocket /run/uwsgi/app/searx/socket
</Location>
Note that if your instance of searx is not at the root, you should change
``<Location />`` by the location of your instance, like ``<Location /searx>``.
Restart Apache:
.. code:: sh
sudo -H /etc/init.d/apache2 restart
disable logs
~~~~~~~~~~~~
For better privacy you can disable Apache logs.
.. warning::
You can only disable logs for the whole (virtual) server not for a specific
path.
Go back to ``/etc/apache2/apache2.conf`` and above ``<Location />`` add:
.. code:: apache
CustomLog /dev/null combined
Restart Apache:
.. code:: sh
sudo -H /etc/init.d/apache2 restart
How to update
=============
.. code:: sh
cd /usr/local/searx
sudo -H -u searx -i
.. code:: sh
(searx)$ . ./searx-ve/bin/activate
(searx)$ git stash
(searx)$ git pull origin master
(searx)$ git stash apply
(searx)$ ./manage.sh update_packages
.. code:: sh
sudo -H service uwsgi restart
Docker
======
Make sure you have installed Docker. For instance, you can deploy searx like this:
.. code:: sh
docker pull wonderfall/searx
docker run -d --name searx -p $PORT:8888 wonderfall/searx
Go to ``http://localhost:$PORT``.
See https://hub.docker.com/r/wonderfall/searx/ for more informations. It's also
possible to build searx from the embedded Dockerfile.
.. code:: sh
git clone https://github.com/asciimoo/searx.git
cd searx
docker build -t whatever/searx .
References
==========
* https://about.okhin.fr/posts/Searx/ with some additions
* How to: `Setup searx in a couple of hours with a free SSL certificate
<https://www.reddit.com/r/privacytoolsIO/comments/366kvn/how_to_setup_your_own_privacy_respecting_search/>`__

26
docs/admin/morty.rst

@ -0,0 +1,26 @@
=========================
How to setup result proxy
=========================
.. _morty: https://github.com/asciimoo/morty
.. _morty's README: https://github.com/asciimoo/morty
By default searx can only act as an image proxy for result images, but it is
possible to proxify all the result URLs with an external service, morty_.
To use this feature, morty has to be installed and activated in searx's
``settings.yml``.
Add the following snippet to your ``settings.yml`` and restart searx:
.. code:: yaml
result_proxy:
url : http://127.0.0.1:3000/
key : your_morty_proxy_key
``url``
Is the address of the running morty service.
``key``
Is an optional argument, see `morty's README`_ for more information.

39
docs/admin/plugins.rst

@ -0,0 +1,39 @@
.. _plugins generic:
===============
Plugins builtin
===============
.. sidebar:: Further reading ..
- :ref:`dev plugin`
Configuration defaults (at built time):
:DO: Default on
.. _configured plugins:
.. jinja:: webapp
.. flat-table:: Plugins configured at built time (defaults)
:header-rows: 1
:stub-columns: 1
:widths: 3 1 9
* - Name
- DO
- Description
JS & CSS dependencies
{% for plgin in plugins %}
* - {{plgin.name}}
- {{(plgin.default_on and "y") or ""}}
- {{plgin.description}}
{% for dep in (plgin.js_dependencies + plgin.css_dependencies) %}
| ``{{dep}}`` {% endfor %}
{% endfor %}

181
docs/admin/settings.rst

@ -0,0 +1,181 @@
.. _settings.yml:
================
``settings.yml``
================
.. sidebar:: Further reading ..
- :ref:`search API`
This page describe the options possibilities of the settings.yml file.
.. _settings global:
Global Settings
===============
.. code:: yaml
server:
port : 8888
secret_key : "ultrasecretkey" # change this!
debug : False # debug mode, only for development
request_timeout : 2.0 # seconds
base_url : False # set custom base_url (or False)
themes_path : "" # custom ui themes path
default_theme : oscar # ui theme
useragent_suffix : "" # suffix of searx_useragent, could contain
# informations like admins email address
image_proxy : False # proxying image results through searx
default_locale : "" # default interface locale
# uncomment below section if you want to use a proxy
#outgoing_proxies :
# http : http://127.0.0.1:8080
# https: http://127.0.0.1:8080
# uncomment below section only if you have more than one network interface
# which can be the source of outgoing search requests
#source_ips:
# - 1.1.1.1
# - 1.1.1.2
locales:
en : English
de : Deutsch
he : Hebrew
hu : Magyar
fr : Français
es : Español
it : Italiano
nl : Nederlands
ja : 日本語 (Japanese)
tr : Türkçe
ru : Russian
ro : Romanian
``port`` :
Port number of the searx web application if you run it directly using ``python
searx/webapp.py``. Doesn't apply to searx running on Apache or Nginx.
``secret_key`` :
Used for cryptography purpose.
``debug`` :
Allow a more detailed log if you run searx directly. Display *detailed* error
messages in the browser too, so this must be deactivated in production.
``request_timeout`` :
Global timeout of the requests made to others engines in seconds. A bigger
timeout will allow to wait for answers from slow engines, but in consequence
will slow searx reactivity (the result page may take the time specified in the
timeout to load)
``base_url`` :
The base URL where searx is deployed. Used to create correct inbound links.
``themes_path`` :
Path to where the themes are located. If you didn't develop anything, leave it
blank.
``default_theme`` :
Name of the theme you want to use by default on you searx instance.
``useragent_suffix`` :
Suffix to the user-agent searx uses to send requests to others engines. If an
engine wish to block you, a contact info here may be useful to avoid that.
``image_proxy`` :
Allow your instance of searx of being able to proxy images. Uses memory space.
``default_locale`` :
Aearx interface language. If blank, the locale is detected by using the
browser language. If it doesn't work, or you are deploying a language
specific instance of searx, a locale can be defined using an ISO language
code, like ``fr``, ``en``, ``de``.
.. _requests proxies: http://docs.python-requests.org/en/latest/user/advanced/#proxies
.. _PR SOCKS support: https://github.com/kennethreitz/requests/pull/478
``outgoing_proxies`` :
Define a proxy you wish to use, see `requests proxies`_. SOCKS proxies are
not supported / see `PR SOCKS support`.
``source_ips`` :
If you use multiple nework interfaces, define from which IP the requests must
be made.
``locales`` :
Locales codes and their names. Available translations of searx interface.
.. _settings engine:
Engine settings
===============
.. sidebar:: Further reading ..
- :ref:`engines-dev`
.. code:: yaml
- name : bing
engine : bing
shortcut : bi
base_url : 'https://{language}.wikipedia.org/'
categories : general
timeout : 3.0
api_key : 'apikey'
disabled : True
language : en_US
``name`` :
Name that will be used accross searx to define this engine. In settings, on
the result page...
``engine`` :
Name of the python file used to handle requests and responses to and from this
search engine.
``shortcut`` :
Code used to execute bang requests (in this case using ``!bi`` or ``?bi``)
``base_url`` : optional
Part of the URL that should be stable accross every request. Can be useful to
use multiple sites using only one engine, or updating the site URL without
touching at the code.
``categories`` : optional
Define in which categories this engine will be active. Most of the time, it is
defined in the code of the engine, but in a few cases it is useful, like when
describing multiple search engine using the same code.
``timeout`` : optional
Timeout of the search with the current search engine. **Be careful, it will
modify the global timeout of searx.**
``api_key`` : optional
In a few cases, using an API needs the use of a secret key. How to obtain them
is described in the file.
``disabled`` : optional
To disable by default the engine, but not deleting it. It will allow the user
to manually activate it in the settings.
``language`` : optional
If you want to use another language for a specific engine, you can define it
by using the full ISO code of language and country, like ``fr_FR``, ``en_US``,
``de_DE``.
``weigth`` : default ``1``
Weighting of the results of this engine.
.. note::
A few more options are possible, but they are pretty specific to some
engines, and so won't be described here.

43
docs/blog/admin.rst

@ -0,0 +1,43 @@
=============================================================
Searx admin interface