Cyber
/
archmanweb
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Codebase for the Arch manual pages repository (read-only mirror)
269 Commits 2 Branches 4 Tags 409 KiB
Python 73.9%
HTML 13.8%
CSS 12.3%
Go to file
Jakub Klinkovský ca5e7b2d8d Merge branch 'dev' into 'master' 
Refactor the anchorencode function

Closes #31

See merge request archlinux/archmanweb!29
 		2021-08-30 17:34:41 +00:00
archlinux-common-style@a6008aeb65 Update archlinux-common-style submodule 2021-06-05 08:32:28 +02:00
archmanweb Refactor the anchorencode function 2021-08-30 19:27:52 +02:00
.gitignore reorganize .gitignore files 2020-10-22 21:16:53 +02:00
.gitmodules replace sitestatic with the archlinux-common-style repository 2020-12-19 19:53:40 +01:00
LICENSE add LICENSE file 2020-07-19 10:20:19 +02:00
README.md Add instructions to build the archlinux-common-style submodule to README 2021-06-05 08:33:18 +02:00
local_settings.py.example get rid of the mysite subdirectory and refactor settings 2020-12-19 19:53:40 +01:00
manage.py get rid of the mysite subdirectory and refactor settings 2020-12-19 19:53:40 +01:00
settings.py Add Content Security Policy headers 2021-05-30 20:52:00 +02:00
urls.py get rid of the mysite subdirectory and refactor settings 2020-12-19 19:53:40 +01:00
wsgi.py get rid of the mysite subdirectory and refactor settings 2020-12-19 19:53:40 +01:00

README.md

Arch manual pages

Git submodules

Make sure that git submodules are initialized after cloning the repository:

git submodule update --init --recursive

Or initialize the submodules while cloning:

git clone --recurse-submodules ssh://git@gitlab.archlinux.org:222/archlinux/archmanweb.git

Dependencies

pacman -S pyalpm python-chardet python-django python-django-csp python-psycopg2 python-requests python-xtarfile

Installation

  1. Copy local_settings.py.example to local_settings.py and edit DEBUG = True and the SECRET_KEY variable.

  2. Configure a connection to a PostgreSQL database in the Django database settings in the local_settings.py file.

  3. Make sure that the pg_trgm extension is created in the database. For example:

     psql --username=<username> --dbname=<dbname> --command "create extension if not exists pg_trgm;"

  4. Make migrations.

     ./manage.py makemigrations

  5. Migrate changes.

     ./manage.py migrate

  6. Build the archlinux-common-style submodule.

    A SASS compiler is needed. For example, install sassc and run

     cd archlinux-common-style
 make SASS=sassc

  7. Start the development web server with ./manage.py runserver. The site should be available at http://localhost:8000, saying that there are 0 man pages and 0 packages (because they were not imported yet). The server will automatically reload when you make changes to the webapp code or templates.

  8. Run the update.py script to import some man pages. However, note that the full import requires to download about 7.5 GiB of packages from a mirror of the Arch repos and then the extraction takes about 20-30 minutes. (The volume of all man pages is less than 300 MiB though.) If you won't need all man pages for the development, you can run e.g. update.py --only-repos core to import only man pages from the core repository (the smallest one, download size is about 160 MiB) or even update.py --only-packages coreutils man-pages.

About

This website was created for the man template on the Arch wiki. Originally, the template replaced plain text, unclickable references to man pages with links to man7.org, which contains a handful of manuals taken directly from upstream. Later, we considered switching to another site providing more manuals. Since we did not find a suitable external site, we decided to build a new service to satisfy all our requirements:

  1. All man pages from official Arch packages are available. Old versions and permalinks are not necessary.
  2. Functionality does not require Javascript.
  3. Pages are addressable by their name and section, both occurring exactly once in the URL to avoid problems with pages such as ar(1) and ar(1p).
  4. The URLs used by the man template should not redirect to permalinks, otherwise users would start copy-pasting them to the wiki and it would be hard to check if they are the same as the canonical URLs.
  5. Human-readable subsection anchors.
  6. The page should clearly indicate the Arch package version containing the page.

See the original discussion for details.

We used a dynamic approach instead of building a website consisting of completely static pages. The main building blocks are the Django web framework, the PostgreSQL database server, the mandoc tool from the mandoc toolset for the conversion to HTML and the pyalpm library for data extraction from the Arch repositories. The code is available in the archmanweb repository at GitHub.

Overall, this approach allows us to provide the following features without rebuilding the whole website from scratch:

  • Listings with custom filters and orderings.
  • Links to other versions of the same manual provided by different packages.
  • Links to similar manuals available in other sections or languages.
  • Searching in the names and descriptions of packages and manuals, similarly to apropos(1).

Similar projects

Some similar projects, each using a different approach, are:

Test cases

These links serve as test cases to ensure that all features still work, they are not useful to regular users.

URLs with dots

Best match lookup

Ambiguous cases are ordered by section, package repository and package version, then the first manual is selected.

Language fallback

Package filter

.so macros

There is a groff(1) extension for the man(7) and mdoc(7) languages to include contents of other files using the .so macro. In normal operation where manuals are stored as files on a file system, the soelim(1) pre-processor handles the inclusion. Our system is based on a database rather than a file system, so we need a custom soelim as well.

Some pages which contain the .so macro: