Browse Source

doc: proofread of the all reST sources (no content change)

Normalize reST sources with best practice and KISS in mind.

to name a few points:

- simplify reST tables
- make use of ``literal`` markup for monospace rendering
- fix code-blocks for better rendering in HTML
- normalize section header markup
- limit all lines to a maximum of 79 characters
- add option -H to the sudo command used in code blocks
- drop useless indentation of lists
- ...

[1] https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
libremiami
Markus Heiser 3 years ago
parent
commit
e9fff4fde6
  1. 160
      docs/admin/api.rst
  2. 55
      docs/admin/filtron.rst
  3. 19
      docs/admin/morty.rst
  4. 53
      docs/blog/admin.rst
  5. 1
      docs/blog/blog.rst
  6. 64
      docs/blog/intro-offline.rst
  7. 48
      docs/blog/python3.rst
  8. 8
      docs/conf.py
  9. 103
      docs/dev/contribution_guide.rst
  10. 453
      docs/dev/engine_overview.rst
  11. 45
      docs/dev/plugins.rst
  12. 92
      docs/dev/quickstart.rst
  13. 226
      docs/dev/search_api.rst
  14. 44
      docs/dev/translation.rst
  15. 34
      docs/index.rst
  16. 70
      docs/user/own-instance.rst
  17. 10
      docs/user/search_syntax.rst

160
docs/admin/api.rst

@ -1,94 +1,96 @@
.. _adminapi:
==================
Administration API
------------------
==================
Get configuration data
~~~~~~~~~~~~~~~~~~~~~~
======================
.. code:: sh
.. code:: http
GET /config
GET /config HTTP/1.1
Sample response
```````````````
---------------
.. code:: json
{
"autocomplete": "",
"categories": [
"map",
"it",
"images",
],
"default_locale": "",
"default_theme": "oscar",
"engines": [
{
"categories": [
"map"
],
"enabled": true,
"name": "openstreetmap",
"shortcut": "osm"
},
{
"categories": [
"it"
],
"enabled": true,
"name": "arch linux wiki",
"shortcut": "al"
},
{
"categories": [
"images"
],
"enabled": true,
"name": "google images",
"shortcut": "goi"
},
{
"categories": [
"it"
],
"enabled": false,
"name": "bitbucket",
"shortcut": "bb"
},
],
"instance_name": "searx",
"locales": {
"de": "Deutsch (German)",
"en": "English",
"eo": "Esperanto (Esperanto)",
},
"plugins": [
{
"enabled": true,
"name": "HTTPS rewrite"
},
{
"enabled": false,
"name": "Vim-like hotkeys"
}
],
"safe_search": 0
}
.. code:: sh
{
"autocomplete": "",
"categories": [
"map",
"it",
"images",
],
"default_locale": "",
"default_theme": "oscar",
"engines": [
{
"categories": [
"map"
],
"enabled": true,
"name": "openstreetmap",
"shortcut": "osm"
},
{
"categories": [
"it"
],
"enabled": true,
"name": "arch linux wiki",
"shortcut": "al"
},
{
"categories": [
"images"
],
"enabled": true,
"name": "google images",
"shortcut": "goi"
},
{
"categories": [
"it"
],
"enabled": false,
"name": "bitbucket",
"shortcut": "bb"
},
],
"instance_name": "searx",
"locales": {
"de": "Deutsch (German)",
"en": "English",
"eo": "Esperanto (Esperanto)",
},
"plugins": [
{
"enabled": true,
"name": "HTTPS rewrite"
},
{
"enabled": false,
"name": "Vim-like hotkeys"
}
],
"safe_search": 0
}
Embed search bar
----------------
================
The search bar can be embedded into websites. Just paste the example into the HTML of the site.
URL of the searx instance and values are customizable.
The search bar can be embedded into websites. Just paste the example into the
HTML of the site. URL of the searx instance and values are customizable.
.. code:: html
<form method="post" action="https://searx.me/">
<!-- search query --> <input type="text" name="q" />
<!-- categories --> <input type="hidden" name="categories" value="general,social media" />
<!-- language --> <input type="hidden" name="lang" value="all" />
<!-- locale --> <input type="hidden" name="locale" value="en" />
<!-- date filter --> <input type="hidden" name="time_range" value="month" />
</form>
<form method="post" action="https://searx.me/">
<!-- search --> <input type="text" name="q" />
<!-- categories --> <input type="hidden" name="categories" value="general,social media" />
<!-- language --> <input type="hidden" name="lang" value="all" />
<!-- locale --> <input type="hidden" name="locale" value="en" />
<!-- date filter --> <input type="hidden" name="time_range" value="month" />
</form>

55
docs/admin/filtron.rst

@ -1,25 +1,26 @@
==========================
How to protect an instance
==========================
Searx depens on external search services. To avoid the abuse of these services it is advised to limit the number of requests processed by searx.
An application firewall, ``filtron`` solves exactly this problem. Information on how to install it can be found at the `project page of filtron <https://github.com/asciimoo/filtron>`__.
Sample configuration of filtron
-------------------------------
Searx depens on external search services. To avoid the abuse of these services
it is advised to limit the number of requests processed by searx.
An example configuration can be find below. This configuration limits the access of
An application firewall, ``filtron`` solves exactly this problem. Information
on how to install it can be found at the `project page of filtron
<https://github.com/asciimoo/filtron>`__.
* scripts or applications (roboagent limit)
* webcrawlers (botlimit)
* IPs which send too many requests (IP limit)
* too many json, csv, etc. requests (rss/json limit)
Sample configuration of filtron
===============================
* the same UserAgent of if too many requests (useragent limit)
An example configuration can be find below. This configuration limits the access
of:
- scripts or applications (roboagent limit)
- webcrawlers (botlimit)
- IPs which send too many requests (IP limit)
- too many json, csv, etc. requests (rss/json limit)
- the same UserAgent of if too many requests (useragent limit)
.. code:: json
@ -120,26 +121,28 @@ An example configuration can be find below. This configuration limits the access
Route request through filtron
-----------------------------
=============================
Filtron can be started using the following command:
.. code:: bash
.. code:: sh
$ filtron -rules rules.json
$ filtron -rules rules.json
It listens on 127.0.0.1:4004 and forwards filtered requests to 127.0.0.1:8888 by default.
It listens on ``127.0.0.1:4004`` and forwards filtered requests to
``127.0.0.1:8888`` by default.
Use it along with ``nginx`` with the following example configuration.
.. code:: nginx
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_pass http://127.0.0.1:4004/;
}
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_pass http://127.0.0.1:4004/;
}
Requests are coming from port 4004 going through filtron and then forwarded to port 8888 where a searx is being run.
Requests are coming from port 4004 going through filtron and then forwarded to
port 8888 where a searx is being run.

19
docs/admin/morty.rst

@ -1,14 +1,17 @@
=========================
How to setup result proxy
=========================
By default searx can only act as an image proxy for result images,
but it is possible to proxify all the result URLs with an external service,
`morty <https://github.com/asciimoo/morty>`__.
.. _morty: https://github.com/asciimoo/morty
.. _morty's README: https://github.com/asciimoo/morty
To use this feature, morty has to be installed and activated in searx's ``settings.yml``.
By default searx can only act as an image proxy for result images, but it is
possible to proxify all the result URLs with an external service, morty_.
Add the following snippet to your ``settings.yml`` and restart searx:
To use this feature, morty has to be installed and activated in searx's
``settings.yml``.
Add the following snippet to your ``settings.yml`` and restart searx:
.. code:: yaml
@ -16,6 +19,8 @@ Add the following snippet to your ``settings.yml`` and restart searx:
url : http://127.0.0.1:3000/
key : your_morty_proxy_key
``url`` is the address of the running morty service
``url``
Is the address of the running morty service.
``key`` is an optional argument, see `morty's README <https://github.com/asciimoo/morty>`__ for more information.
``key``
Is an optional argument, see `morty's README`_ for more information.

53
docs/blog/admin.rst

@ -1,42 +1,43 @@
Searx admin interface: manage your instance from your browser
=============================================================
Searx admin interface
=============================================================
One of the biggest advantages of searx is being extremely customizable. But at first it can be daunting to newcomers.
A barrier of taking advantage of this feature is our ugly settings file which is sometimes hard to understand and edit.
.. _searx-admin: https://github.com/kvch/searx-admin#searx-admin
.. _NLnet Foundation: https://nlnet.nl/
To make self-hosting searx more accessible a new tool is introduced, called ``searx-admin``.
It is a web application which is capable of managing your instance and manipulating its settings via a web UI.
It aims to replace editing of ``settings.yml`` for less experienced administrators or people
who prefer graphical admin interfaces.
manage your instance from your browser
.. figure:: searx-admin-engines.png
:scale: 50 %
:alt: Screenshot of engine list
:align: center
:figclass: align-center
Configuration page of engines
.. sidebar:: Installation
Installation guide can be found in the repository of searx-admin_.
Since ``searx-admin`` acts as a supervisor for searx, we have decided to implement it
as a standalone tool instead of part of searx. Another reason for making it a standalone
tool is that the codebase and dependencies of searx should not grow because of a fully optional feature,
which does not affect existing instances.
One of the biggest advantages of searx is being extremely customizable. But at
first it can be daunting to newcomers. A barrier of taking advantage of this
feature is our ugly settings file which is sometimes hard to understand and
edit.
To make self-hosting searx more accessible a new tool is introduced, called
``searx-admin``. It is a web application which is capable of managing your
instance and manipulating its settings via a web UI. It aims to replace editing
of ``settings.yml`` for less experienced administrators or people who prefer
graphical admin interfaces.
Installation
------------
.. figure:: searx-admin-engines.png
:alt: Screenshot of engine list
Installation guide can be found in the repository of searx-admin:
https://github.com/kvch/searx-admin#installation--usage
Configuration page of engines
Acknowledgements
----------------
Since ``searx-admin`` acts as a supervisor for searx, we have decided to
implement it as a standalone tool instead of part of searx. Another reason for
making it a standalone tool is that the codebase and dependencies of searx
should not grow because of a fully optional feature, which does not affect
existing instances.
This development was sponsored by `NLnet Foundation`_.
.. _NLnet Foundation: https://nlnet.nl/
Acknowledgements
================
This development was sponsored by `NLnet Foundation`_.
| Happy hacking.
| kvch // 2017.08.22 21:25

1
docs/blog/blog.rst

@ -1,3 +1,4 @@
====
Blog
====

64
docs/blog/intro-offline.rst

@ -1,58 +1,70 @@
===============================
Preparation for offline engines
===============================
Offline engines
---------------
===============
To extend the functionality of searx, offline engines are going to be introduced. An offline engine is an engine which does not need Internet connection to perform a search and does not use HTTP to communicate.
To extend the functionality of searx, offline engines are going to be
introduced. An offline engine is an engine which does not need Internet
connection to perform a search and does not use HTTP to communicate.
Offline engines can be configured as online engines, by adding those to the `engines` list of `settings.yml`. Thus, searx finds the engine file and imports it.
Offline engines can be configured as online engines, by adding those to the
`engines` list of :origin:`settings.yml <searx/settings.yml>`. Thus, searx
finds the engine file and imports it.
Example skeleton for the new engines:
.. code:: python
from subprocess import PIPE, Popen
from subprocess import PIPE, Popen
categories = ['general']
offline = True
categories = ['general']
offline = True
def init(settings):
pass
def init(settings):
pass
def search(query, params):
process = Popen(['ls', query], stdout=PIPE)
return_code = process.wait()
if return_code != 0:
raise RuntimeError('non-zero return code', return_code)
def search(query, params):
process = Popen(['ls', query], stdout=PIPE)
return_code = process.wait()
if return_code != 0:
raise RuntimeError('non-zero return code', return_code)
results = []
line = process.stdout.readline()
while line:
result = parse_line(line)
results.append(results)
results = []
line = process.stdout.readline()
while line:
result = parse_line(line)
results.append(results)
line = process.stdout.readline()
line = process.stdout.readline()
return results
return results
Development progress
--------------------
====================
First, a proposal has been created as a Github issue. Then it was moved to the wiki as a design document. You can read it here: :wiki:`Offline-engines`.
First, a proposal has been created as a Github issue. Then it was moved to the
wiki as a design document. You can read it here: :wiki:`Offline-engines`.
In this development step, searx core was prepared to accept and perform offline searches. Offline search requests are scheduled together with regular offline requests.
In this development step, searx core was prepared to accept and perform offline
searches. Offline search requests are scheduled together with regular offline
requests.
As offline searches can return arbitrary results depending on the engine, the current result templates were insufficient to present such results. Thus, a new template is introduced which is caplable of presenting arbitrary key value pairs as a table. You can check out the pull request for more details: https://github.com/asciimoo/searx/pull/1700
As offline searches can return arbitrary results depending on the engine, the
current result templates were insufficient to present such results. Thus, a new
template is introduced which is caplable of presenting arbitrary key value pairs
as a table. You can check out the pull request for more details see
:pull:`1700`.
Next steps
----------
==========
Today, it is possible to create/run an offline engine. However, it is going to be publicly available for everyone who knows the searx instance. So the next step is to introduce token based access for engines. This way administrators are able to limit the access to private engines.
Acknowledgement
---------------
===============
This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ .

48
docs/blog/python3.rst

@ -1,7 +1,17 @@
Introducing Python3 support
===========================
============================
Introducing Python 3 support
============================
As most operation systems are coming with Python3 installed by default. So it is time for searx to support Python3. But don't worry support of Python2.7 won't be dropped.
.. _Python 2.7 clock: https://pythonclock.org/
.. sidebar:: Python 2.7 to 3 upgrade
This chapter exists of historical reasons. Python 2.7 release schedule ends
(`Python 2.7 clock`_) after 11 years Python 3 exists
As most operation systems are coming with Python3 installed by default. So it is
time for searx to support Python3. But don't worry support of Python2.7 won't be
dropped.
.. image:: searxpy3.png
:scale: 50 %
@ -9,13 +19,14 @@ As most operation systems are coming with Python3 installed by default. So it is
:align: center
How to run searx using Python3
------------------------------
How to run searx using Python 3
===============================
Please make sure that you run at least Python3.5.
Please make sure that you run at least Python 3.5.
To run searx, first a Python3 virtualenv should be created. After entering the virtualenv,
dependencies must be installed. Then run searx with python3 instead of the usual python command.
To run searx, first a Python3 virtualenv should be created. After entering the
virtualenv, dependencies must be installed. Then run searx with python3 instead
of the usual python command.
.. code:: sh
@ -25,25 +36,28 @@ dependencies must be installed. Then run searx with python3 instead of the usual
python3 searx/webapp.py
If you want to run searx using Python2.7, you don't have to do anything differently as before.
If you want to run searx using Python2.7, you don't have to do anything
differently as before.
Fun facts
---------
=========
- 115 files were changed when implementing the support for both Python versions.
- All of the dependencies was compatible except for the robotframework used for browser tests. Thus, these tests were migrated to splinter. So from now on both versions are being tested on Travis and can be tested locally.
- All of the dependencies was compatible except for the robotframework used for
browser tests. Thus, these tests were migrated to splinter. So from now on
both versions are being tested on Travis and can be tested locally.
If you found bugs...
--------------------
If you found bugs
=================
...please open an issue on `GitHub`_. Make sure that you mention your Python version in your issue,
so we can investigate it properly.
Please open an issue on `GitHub`_. Make sure that you mention your Python
version in your issue, so we can investigate it properly.
.. _GitHub: https://github.com/asciimoo/searx/issues
Acknowledgement
---------------
Acknowledgment
==============
This development was sponsored by `NLnet Foundation`_.

8
docs/conf.py

@ -20,12 +20,16 @@ master_doc = "index"
source_suffix = '.rst'
# usage:: lorem :patch:`f373169` ipsum
extlinks = {}
# upstream links
extlinks['wiki'] = ('https://github.com/asciimoo/searx/wiki/%s', ' ')
extlinks['pull'] = ('https://github.com/asciimoo/searx/pull/%s', 'PR ')
# links to custom brand
extlinks['origin'] = (GIT_URL + '/blob/master/%s', 'git://')
extlinks['patch'] = (GIT_URL + '/commit/%s', '#')
extlinks['search'] = (SEARX_URL + '/%s', '#')
extlinks['wiki'] = ('https://github.com/asciimoo/searx/wiki/%s', ' ')
extensions = [
'sphinx.ext.extlinks',

103
docs/dev/contribution_guide.rst

@ -1,66 +1,68 @@
=================
How to contribute
-----------------
=================
Prime directives: Privacy, Hackability
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
======================================
Searx has two prime directives, privacy-by-design and hackability. The
Searx has two prime directives, **privacy-by-design and hackability** . The
hackability comes in three levels:
- support of search engines
- plugins to alter search behaviour
- hacking searx itself
- support of search engines
- plugins to alter search behaviour
- hacking searx itself
Note the lack of "world domination" among the directives.
Searx has no intention of wide mass-adoption, rounded
corners, etc. The prime directive "privacy" deserves a separate
chapter, as it's quite uncommon unfortunately.
Note the lack of "world domination" among the directives. Searx has no
intention of wide mass-adoption, rounded corners, etc. The prime directive
"privacy" deserves a separate chapter, as it's quite uncommon unfortunately.
Privacy-by-design
^^^^^^^^^^^^^^^^^
-----------------
Searx was born out of the need for a privacy-respecting search tool
which can be extended easily to maximize both its search and its
privacy protecting capabilities.
Searx was born out of the need for a **privacy-respecting** search tool which
can be extended easily to maximize both, its search and its privacy protecting
capabilities.
A few widely used features work differently or turned off by default or not implemented
at all as a consequence of privacy-by-design.
A few widely used features work differently or turned off by default or not
implemented at all **as a consequence of privacy-by-design**.
If a feature reduces the privacy preserving aspects of searx, it
should be switched off by default or should not implemented at all.
There are plenty of search engines already providing such features.
If a feature reduces the protection of searx, users must be
informed about the effect of choosing to enable it. Features
that protect privacy but differ from the expectations of the
user should also be explained.
If a feature reduces the privacy preserving aspects of searx, it should be
switched off by default or should not implemented at all. There are plenty of
search engines already providing such features. If a feature reduces the
protection of searx, users must be informed about the effect of choosing to
enable it. Features that protect privacy but differ from the expectations of
the user should also be explained.
Also, if you think that something works weird with searx,
it's might be because of the tool you use is designed in a way to interfere with
the privacy respect. Submitting a bugreport to the vendor of the tool that
misbehaves might be a good feedback to reconsider the disrespect to
its customers (e.g. GET vs POST requests in various browsers).
Also, if you think that something works weird with searx, it's might be because
of the tool you use is designed in a way to interfere with the privacy respect.
Submitting a bugreport to the vendor of the tool that misbehaves might be a good
feedback to reconsider the disrespect to its customers (e.g. ``GET`` vs ``POST``
requests in various browsers).
Remember the other prime directive of searx is to be hackable, so if the
above privacy concerns do not fancy you, simply fork it.
Remember the other prime directive of searx is to be hackable, so if the above
privacy concerns do not fancy you, simply fork it.
Happy hacking.
*Happy hacking.*
Code
~~~~
====
.. _PEP8: https://www.python.org/dev/peps/pep-0008/
In order to submit a patch, please follow the steps below:
- Follow coding conventions.
- PEP8 standards apply, except the convention of line length
- PEP8_ standards apply, except the convention of line length
- Maximum line length is 120 characters
- Check if your code breaks existing tests. If so, update the tests or fix your code.
- Check if your code breaks existing tests. If so, update the tests or fix your
code.
- If your code can be unit-tested, add unit tests.
- Add yourself to the AUTHORS file.
- Add yourself to the :origin:`AUTHORS.rst` file.
- Create a pull request.
@ -68,31 +70,20 @@ For more help on getting started with searx development, see :ref:`devquickstart
Translation
~~~~~~~~~~~
===========
Translation currently takes place on
`transifex <https://transifex.com/projects/p/searx>`__.
Translation currently takes place on :ref:`transifex <translation>`.
**Please, do not update translation files in the repo.**
.. caution::
Documentation
~~~~~~~~~~~~~
Please, do not update translation files in the repo.
The documentation is built using Sphinx. So in order to be able to generate the required
files, you have to install it on your system. (It can be installed easily using pip.)
1. Checkout the gh-pages branch.
2. Edit the rst file you wish to update. Or create a new rst file and place it under the appropriate folder.
3. Build the documentation using Sphinx.
4. Add the updated and created files of these extension:
- .rst
Documentation
=============
- .html
.. admonition:: ToDo
- .txt
docs are no longer in gh-branch
6. Create a pull request.
Update this chapter and document the /docs workflow!!

453
docs/dev/engine_overview.rst

@ -1,21 +1,25 @@
.. _engines-dev:
===============
Engine overview
===============
.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
searx is a `metasearch-engine <https://en.wikipedia.org/wiki/Metasearch_engine>`__,
so it uses different search engines to provide better results.
Because there is no general search API which could be used for every
search engine, an adapter has to be built between searx and the
external search engines. Adapters are stored under the folder
:origin:`searx/engines`.
searx is a metasearch-engine_, so it uses different search engines to provide
better results.
Because there is no general search API which could be used for every search
engine, an adapter has to be built between searx and the external search
engines. Adapters are stored under the folder :origin:`searx/engines`.
.. contents::
:depth: 3
:backlinks: entry
general engine configuration
----------------------------
============================
It is required to tell searx the type of results the engine provides. The
arguments can be set in the engine file or in the settings file
@ -23,292 +27,239 @@ arguments can be set in the engine file or in the settings file
the ones in the engine file.
It does not matter if an option is stored in the engine file or in the
settings. However, the standard way is the following:
settings. However, the standard way is the following:
engine file
~~~~~~~~~~~
+----------------------+-----------+-----------------------------------------+
| argument | type | information |
+======================+===========+=========================================+
| categories | list | pages, in which the engine is working |
+----------------------+-----------+-----------------------------------------+
| paging | boolean | support multible pages |
+----------------------+-----------+-----------------------------------------+
| language\_support | boolean | support language choosing |
+----------------------+-----------+-----------------------------------------+
| time\_range\_support | boolean | support search time range |
+----------------------+-----------+-----------------------------------------+
| offline | boolean | engine runs offline |
+----------------------+-----------+-----------------------------------------+
-----------
======================= =========== ===========================================
argument type information
======================= =========== ===========================================
categories list pages, in which the engine is working
paging boolean support multible pages
language_support boolean support language choosing
time_range_support boolean support search time range
offline boolean engine runs offline
======================= =========== ===========================================
settings.yml
~~~~~~~~~~~~
+------------+----------+-----------------------------------------------+
| argument | type | information |
+============+==========+===============================================+
| name | string | name of search-engine |
+------------+----------+-----------------------------------------------+
| engine | string | name of searx-engine (filename without .py) |
+------------+----------+-----------------------------------------------+
| shortcut | string | shortcut of search-engine |
+------------+----------+-----------------------------------------------+
| timeout | string | specific timeout for search-engine |
+------------+----------+-----------------------------------------------+
------------
======================= =========== ===========================================
argument type information
======================= =========== ===========================================
name string name of search-engine
engine string name of searx-engine
(filename without ``.py``)
shortcut string shortcut of search-engine
timeout string specific timeout for search-engine
======================= =========== ===========================================
overrides
~~~~~~~~~
A few of the options have default values in the engine, but are
often overwritten by the settings. If ``None`` is assigned to an option
in the engine file, it has to be redefined in the settings,
otherwise searx will not start with that engine.
The naming of overrides is arbitrary. But the recommended
overrides are the following:
+-----------------------+----------+----------------------------------------------------------------+
| argument | type | information |
+=======================+==========+================================================================+
| base\_url | string | base-url, can be overwritten to use same engine on other URL |
+-----------------------+----------+----------------------------------------------------------------+
| number\_of\_results | int | maximum number of results per request |
+-----------------------+----------+----------------------------------------------------------------+
| language | string | ISO code of language and country like en\_US |
+-----------------------+----------+----------------------------------------------------------------+
| api\_key | string | api-key if required by engine |
+-----------------------+----------+----------------------------------------------------------------+
---------
A few of the options have default values in the engine, but are often
overwritten by the settings. If ``None`` is assigned to an option in the engine
file, it has to be redefined in the settings, otherwise searx will not start
with that engine.
The naming of overrides is arbitrary. But the recommended overrides are the
following:
======================= =========== ===========================================
argument type information
======================= =========== ===========================================
base_url string base-url, can be overwritten to use same
engine on other URL
number_of_results int maximum number of results per request
language string ISO code of language and country like en_US
api_key string api-key if required by engine
======================= =========== ===========================================
example code
~~~~~~~~~~~~
------------
.. code:: python
# engine dependent config
categories = ['general']
paging = True
language_support = True
# engine dependent config
categories = ['general']
paging = True
language_support = True
making a request
----------------
================
To perform a search an URL have to be specified. In addition to
specifying an URL, arguments can be passed to the query.
To perform a search an URL have to be specified. In addition to specifying an
URL, arguments can be passed to the query.
passed arguments
~~~~~~~~~~~~~~~~
----------------
These arguments can be used to construct the search query. Furthermore,
These arguments can be used to construct the search query. Furthermore,
parameters with default value can be redefined for special purposes.
+----------------------+------------+------------------------------------------------------------------------+
| argument | type | default-value, information |
+======================+============+========================================================================+
| url | string | ``''`` |
+----------------------+------------+------------------------------------------------------------------------+
| method | string | ``'GET'`` |
+----------------------+------------+------------------------------------------------------------------------+
| headers | set | ``{}`` |
+----------------------+------------+------------------------------------------------------------------------+
| data | set | ``{}`` |
+----------------------+------------+------------------------------------------------------------------------+
| cookies | set | ``{}`` |
+----------------------+------------+------------------------------------------------------------------------+
| verify | boolean | ``True`` |
+----------------------+------------+------------------------------------------------------------------------+
| headers.User-Agent | string | a random User-Agent |
+----------------------+------------+------------------------------------------------------------------------+
| category | string | current category, like ``'general'`` |
+----------------------+------------+------------------------------------------------------------------------+
| started | datetime | current date-time |
+----------------------+------------+------------------------------------------------------------------------+
| pageno | int | current pagenumber |
+----------------------+------------+------------------------------------------------------------------------+
| language | string | specific language code like ``'en_US'``, or ``'all'`` if unspecified |
+----------------------+------------+------------------------------------------------------------------------+
====================== ============ ========================================================================
argument type default-value, information
====================== ============ ========================================================================
url string ``''``
method string ``'GET'``
headers set ``{}``
data set ``{}``
cookies set ``{}``
verify boolean ``True``
headers.User-Agent string a random User-Agent
category string current category, like ``'general'``
started datetime current date-time
pageno int current pagenumber
language string specific language code like ``'en_US'``, or ``'all'`` if unspecified
====================== ============ ========================================================================
parsed arguments
~~~~~~~~~~~~~~~~
The function ``def request(query, params):`` always returns the
``params`` variable. Inside searx, the following paramters can be
used to specify a search request:
+------------+-----------+---------------------------------------------------------+
| argument | type | information |
+============+===========+=========================================================+
| url | string | requested url |
+------------+-----------+---------------------------------------------------------+
| method | string | HTTP request method |
+------------+-----------+---------------------------------------------------------+
| headers | set | HTTP header information |
+------------+-----------+---------------------------------------------------------+
| data | set | HTTP data information (parsed if ``method != 'GET'``) |
+------------+-----------+---------------------------------------------------------+
| cookies | set | HTTP cookies |
+------------+-----------+---------------------------------------------------------+
| verify | boolean | Performing SSL-Validity check |
+------------+-----------+---------------------------------------------------------+
----------------
The function ``def request(query, params):`` always returns the ``params``
variable. Inside searx, the following paramters can be used to specify a search
request:
============ =========== =========================================================
argument type information
============ =========== =========================================================
url string requested url
method string HTTP request method
headers set HTTP header information
data set HTTP data information (parsed if ``method != 'GET'``)
cookies set HTTP cookies
verify boolean Performing SSL-Validity check
============ =========== =========================================================
example code
~~~~~~~~~~~~
------------
.. code:: python
# search-url
base_url = 'https://example.com/'
search_string = 'search?{query}&page={page}'
# search-url
base_url = 'https://example.com/'
search_string = 'search?{query}&page={page}'
# do search-request
def request(query, params):
search_path = search_string.format(
query=urlencode({'q': query}),
page=params['pageno'])
# do search-request
def request(query, params):
search_path = search_string.format(
query=urlencode({'q': query}),
page=params['pageno'])
params['url'] = base_url + search_path
params['url'] = base_url + search_path
return params
return params
returned results
----------------
================
Searx is able to return results of different media-types.
Currently the following media-types are supported:
Searx is able to return results of different media-types. Currently the
following media-types are supported:
- default
- images
- videos
- torrent
- map
- default_
- images_
- videos_
- torrent_
- map_
To set another media-type as default, the parameter
``template`` must be set to the desired type.
To set another media-type as default, the parameter ``template`` must be set to
the desired type.
default
~~~~~~~
+--------------------+---------------------------------------------------------------------------------------------------------------+
| result-parameter | information |
+====================+===============================================================================================================+
| url | string, url of the result |
+--------------------+---------------------------------------------------------------------------------------------------------------+
| title | string, title of the result |
+--------------------+---------------------------------------------------------------------------------------------------------------+
| content | string, general result-text |
+--------------------+---------------------------------------------------------------------------------------------------------------+
| publishedDate | :py:class:`datetime.datetime`, time of publish |
+--------------------+---------------------------------------------------------------------------------------------------------------+
-------
========================= =====================================================
result-parameter information
========================= =====================================================
url string, url of the result
title string, title of the result
content string, general result-text
publishedDate :py:class:`datetime.datetime`, time of publish
========================= =====================================================
images
~~~~~~
to use this template, the parameter
+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| result-parameter | information |
+====================+=======================================================================================================================================+
| template | is set to ``images.html`` |
+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| url | string, url to the result site |
+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| title | string, title of the result *(partly implemented)* |
+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| content | *(partly implemented)* |
+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| publishedDate | :py:class:`datetime.datetime`, time of publish *(partly implemented)* |
+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| img\_src | string, url to the result image |
+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| thumbnail\_src | string, url to a small-preview image |
+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
------
To use this template, the parameter:
========================= =====================================================
result-parameter information
========================= =====================================================
template is set to ``images.html``
url string, url to the result site
title string, title of the result *(partly implemented)*
content *(partly implemented)*
publishedDate :py:class:`datetime.datetime`,
time of publish *(partly implemented)*
img\_src string, url to the result image
thumbnail\_src string, url to a small-preview image
========================= =====================================================
videos
~~~~~~
+--------------------+--------------------------------------------------------------------------------------------------------------+
| result-parameter | information |
+====================+==============================================================================================================+
| template | is set to ``videos.html`` |
+--------------------+--------------------------------------------------------------------------------------------------------------+
| url | string, url of the result |
+--------------------+--------------------------------------------------------------------------------------------------------------+
| title | string, title of the result |
+--------------------+--------------------------------------------------------------------------------------------------------------+
| content | *(not implemented yet)* |
+--------------------+--------------------------------------------------------------------------------------------------------------+
| publishedDate | :py:class:`datetime.datetime`, time of publish |
+--------------------+--------------------------------------------------------------------------------------------------------------+
| thumbnail | string, url to a small-preview image |
+--------------------+--------------------------------------------------------------------------------------------------------------+
------
========================= =====================================================
result-parameter information
========================= =====================================================
template is set to ``videos.html``
url string, url of the result
title string, title of the result
content *(not implemented yet)*
publishedDate :py:class:`datetime.datetime`, time of publish
thumbnail string, url to a small-preview image
========================= =====================================================
torrent
~~~~~~~
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| result-parameter | information |
+==================+=======================================================================================================================================+
| template | is set to ``torrent.html`` |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| url | string, url of the result |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| title | string, title of the result |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| content | string, general result-text |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| publishedDate | :py:class:`datetime.datetime`, time of publish *(not implemented yet)* |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| seed | int, number of seeder |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| leech | int, number of leecher |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| filesize | int, size of file in bytes |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| files | int, number of files |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| magnetlink | string, `magnetlink <https://en.wikipedia.org/wiki/Magnet_URI_scheme>`__ of the result |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| torrentfile | string, torrentfile of the result |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
-------
.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
========================= =====================================================
result-parameter information
========================= =====================================================
template is set to ``torrent.html``
url string, url of the result
title string, title of the result
content string, general result-text
publishedDate :py:class:`datetime.datetime`,
time of publish *(not implemented yet)*
seed int, number of seeder
leech int, number of leecher
filesize int, size of file in bytes
files int, number of files
magnetlink string, magnetlink_ of the result
torrentfile string, torrentfile of the result
========================= =====================================================
map
~~~
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| result-parameter | information |
+=========================+==============================================================================================================+
| url | string, url of the result |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| title | string, title of the result |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| content | string, general result-text |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| publishedDate | :py:class:`datetime.datetime`, time of publish |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| latitude | latitude of result (in decimal format) |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| longitude | longitude of result (in decimal format) |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| boundingbox | boundingbox of result (array of 4. values ``[lat-min, lat-max, lon-min, lon-max]``) |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| geojson | geojson of result (http://geojson.org) |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| osm.type | type of osm-object (if OSM-Result) |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| osm.id | id of osm-object (if OSM-Result) |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| address.name | name of object |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| address.road | street name of object |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| address.house\_number | house number of object |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| address.locality | city, place of object |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| address.postcode | postcode of object |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| address.country | country of object |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
---
========================= =====================================================
result-parameter information
========================= =====================================================
url string, url of the result
title string, title of the result
content string, general result-text
publishedDate :py:class:`datetime.datetime`, time of publish
latitude latitude of result (in decimal format)
longitude longitude of result (in decimal format)
boundingbox boundingbox of result (array of 4. values
``[lat-min, lat-max, lon-min, lon-max]``)
geojson geojson of result (http://geojson.org)
osm.type type of osm-object (if OSM-Result)
osm.id id of osm-object (if OSM-Result)
address.name name of object
address.road street name of object
address.house_number house number of object
address.locality city, place of object
address.postcode postcode of object
address.country country of object
========================= =====================================================

45
docs/dev/plugins.rst

@ -1,45 +1,48 @@
=======
Plugins
-------
=======
Plugins can extend or replace functionality of various components of
searx.
Plugins can extend or replace functionality of various components of searx.
Example plugin
~~~~~~~~~~~~~~
==============
.. code:: python
name = 'Example plugin'
description = 'This plugin extends the suggestions with the word "example"'
default_on = False # disabled by default
name = 'Example plugin'
description = 'This plugin extends the suggestions with the word "example"'
default_on = False # disabled by default
js_dependencies = tuple() # optional, list of static js files
css_dependencies = tuple() # optional, list of static css files
js_dependencies = tuple() # optional, list of static js files
css_dependencies = tuple() # optional, list of static css files
# attach callback to the post search hook
# request: flask request object
# ctx: the whole local context of the post search hook
def post_search(request, ctx):
ctx['search'].suggestions.add('example')
return True
# attach callback to the post search hook
# request: flask request object
# ctx: the whole local context of the post search hook
def post_search(request, ctx):
ctx['search'].suggestions.add('example')
return True
Plugin entry points
~~~~~~~~~~~~~~~~~~~
===================
Entry points (hooks) define when a plugin runs. Right now only three hooks are implemented. So feel free to implement a hook if it fits the behaviour of your plugin.
Entry points (hooks) define when a plugin runs. Right now only three hooks are
implemented. So feel free to implement a hook if it fits the behaviour of your
plugin.
Pre search hook
```````````````
---------------
Runs BEFORE the search request. Function to implement: ``pre_search``
Post search hook
````````````````
----------------
Runs AFTER the search request. Function to implement: ``post_search``
Result hook
```````````
-----------
Runs when a new result is added to the result list. Function to implement: ``on_result``
Runs when a new result is added to the result list. Function to implement:
``on_result``

92
docs/dev/quickstart.rst

@ -1,17 +1,18 @@
.. _devquickstart:
======================
Development Quickstart
----------------------
======================
This quickstart guide gets your environment set up with searx. Furthermore, it gives a
short introduction to the new manage.sh script.
This quickstart guide gets your environment set up with searx. Furthermore, it
gives a short introduction to the ``manage.sh`` script.
How to setup your development environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=========================================
First, clone the source code of searx to the desired folder. In this case the source
is cloned to ~/myprojects/searx. Then create and activate the searx-ve
virtualenv and install the required packages using manage.sh.
First, clone the source code of searx to the desired folder. In this case the
source is cloned to ``~/myprojects/searx``. Then create and activate the
searx-ve virtualenv and install the required packages using manage.sh.
.. code:: sh
@ -24,85 +25,86 @@ virtualenv and install the required packages using manage.sh.
How to run tests
~~~~~~~~~~~~~~~~
================
Tests can be run using the manage.sh script.
Following tests and checks are available:
Tests can be run using the ``manage.sh`` script. Following tests and checks are
available:
- Unit tests
- Selenium tests
- PEP8 validation
- Unit test coverage check
For example unit tests are run with the command below:
.. code:: sh
./manage.sh unit_tests
./manage.sh unit_tests
For further test options, please consult the help of the manage.sh script.
For further test options, please consult the help of the ``manage.sh`` script.
How to compile styles and javascript
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
====================================
.. _less: http://lesscss.org/
.. _NodeJS: https://nodejs.org
How to build styles
^^^^^^^^^^^^^^^^^^^
-------------------
Less is required to build the styles of searx. Less can be installed using either NodeJS or Apt.
Less_ is required to build the styles of searx. Less_ can be installed using
either NodeJS_ or Apt.
.. code:: sh
sudo apt-get install nodejs
sudo npm install -g less
sudo -H apt-get install nodejs
sudo -H npm install -g less
OR
.. code:: sh
sudo apt-get install node-less
sudo -H apt-get install node-less
After satisfying the requirements styles can be build using manage.sh
After satisfying the requirements styles can be build using ``manage.sh``
.. code:: sh
./manage.sh styles
./manage.sh styles
How to build the source of the oscar theme
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
==========================================
Grunt must be installed in order to build the javascript sources. It depends on NodeJS, so first
Node has to be installed.
.. _grunt: https://gruntjs.com/
Grunt_ must be installed in order to build the javascript sources. It depends on
NodeJS, so first Node has to be installed.
.. code:: sh
sudo apt-get install nodejs
sudo npm install -g grunt-cli
sudo -H apt-get install nodejs
sudo -H npm install -g grunt-cli
After installing grunt, the files can be built using the following command:
After installing grunt, the files can be built using the following command:
.. code:: sh
./manage.sh grunt_build
./manage.sh grunt_build
Tips for debugging/development
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Turn on debug logging
Whether you are working on a new engine or trying to eliminate a bug, it is always a good idea
to turn on debug logging. When debug logging is enabled a stack trace appears,
instead of the cryptic ``Internal Server Error`` message. It can be turned on by setting
``debug: False`` to ``debug: True`` in settings.yml.
2. Run ``./manage.sh tests`` before creating a PR.
Failing build on Travis is common because of PEP8 checks. So a new commit must be created
containing these format fixes. This phase can be skipped if ``./manage.sh tests`` is run
locally before creating a PR.
==============================
Turn on debug logging
Whether you are working on a new engine or trying to eliminate a bug, it is
always a good idea to turn on debug logging. When debug logging is enabled a
stack trace appears, instead of the cryptic ``Internal Server Error``
message. It can be turned on by setting ``debug: False`` to ``debug: True`` in
:origin:`settings.yml <searx/settings.yml>`.
Run ``./manage.sh tests`` before creating a PR.
Failing build on Travis is common because of PEP8 checks. So a new commit
must be created containing these format fixes. This phase can be skipped if
``./manage.sh tests`` is run locally before creating a PR.

226
docs/dev/search_api.rst

@ -1,3 +1,4 @@
==========
Search API
==========
@ -11,194 +12,101 @@ Furthermore, two enpoints ``/`` and ``/search`` are available for querying.
``GET /search``
Parameters
~~~~~~~~~~
.. code:: sh
q
The search query. This string is passed to external search services.
Thus, searx supports syntax of each search service. For example, ``site:github.com searx`` is a valid
query for Google. However, if simply the query above is passed to any search engine which does not filter its
results based on this syntax, you might not get the results you wanted.
See more at :doc:`/user/search_syntax`
Required.
.. code:: sh
categories
Comma separated list, specifies the active search categories
Optional.
.. code:: sh
engines
Comma separated list, specifies the active search engines.
Optional.
.. code:: sh
lang
Code of the language.
Optional.
Default: ``all``
.. code:: sh
pageno
Search page number.
Optional.
Default: ``1``
.. code:: sh
time_range
Time range of search for engines which support it. See if an engine supports time range search in the preferences page of an instance.
Optional.
Possible: ``day``, ``month``, ``year``
.. code:: sh
format
Output format of results.
Optional.
Possible: ``json``, ``csv``, ``rss``
.. code:: sh
results_on_new_tab
Open search results on new tab.
Optional.
Default: ``0``
Possible: ``0``, ``1``
.. code:: sh
image_proxy
Proxy image results through searx.
Optional.
Default: ``False``
Possible: ``True``, ``False``
.. code:: sh
autocomplete