177 lines
7.8 KiB
TeX
177 lines
7.8 KiB
TeX
\documentclass[a4paper,12pt]{article}
|
||
\usepackage[english,vietnamese]{babel}
|
||
\usepackage{amsmath}
|
||
\usepackage{booktabs}
|
||
\usepackage{lmodern}
|
||
\usepackage[utf8]{inputenc}
|
||
\usepackage{graphicx}
|
||
\usepackage{hyperref}
|
||
\usepackage{lmodern}
|
||
\usepackage[nottoc,numbib]{tocbibind}
|
||
\renewcommand{\thefootnote}{\fnsymbol{footnote}}
|
||
|
||
\begin{document}
|
||
\setcounter{page}{0}
|
||
\thispagestyle{empty}
|
||
\vspace*{\stretch{1}}
|
||
\begin{flushright}
|
||
\setlength{\baselineskip}{1.4\baselineskip}
|
||
\textbf{\Huge Python Package\\Metadata Management}
|
||
\noindent\rule{\textwidth}{5pt}
|
||
\emph{\Large Basic Databases}
|
||
\vspace{\stretch{1}}
|
||
|
||
\textbf{by Nguyễn Gia Phong, Nguyễn Quốc Thông,\\
|
||
Nguyễn Văn Tùng and Trần Minh Vương\\}
|
||
\selectlanguage{english}
|
||
\today
|
||
\end{flushright}
|
||
\vspace*{\stretch{2}}
|
||
\pagebreak
|
||
|
||
\selectlanguage{english}
|
||
\tableofcontents
|
||
\pagebreak
|
||
|
||
\section{Introduction}
|
||
\subsection{Brief Description}
|
||
In traditional Unix-like operating systems like GNU/Linux distributions
|
||
and BSD-based OSes, package managers tries to synchronize the packages metadata
|
||
(such as available versions and dependencies) with that of central repositories.
|
||
While this proves to be reliable and efficient, language-specific
|
||
package managers do not usually have such synchronized databases,
|
||
since they focus on development libraries which have more flexible contraints.
|
||
|
||
Within the Python packaging ecosystem, this is the case, where package managers
|
||
like \verb|pip| needs to fetch metadata of each package to be installed
|
||
to find out dependencies and other information. This turns out to have heavy
|
||
performance penalty on the dependency resolution process alone, which is
|
||
already a NP-hard problem. This project explores ways to store these metadata
|
||
in an efficient in a database, to be used in practice either locally or in a
|
||
local/regional network, to avoid Python package managers from having to
|
||
fetch (and potentially build) entire packages just to find out if it conflicts
|
||
with others.
|
||
|
||
\selectlanguage{vietnamese}
|
||
\subsection{Authors and Credits}
|
||
The work has been undertaken by group number 8, whose members are listed
|
||
in the following table.
|
||
\begin{center}
|
||
\begin{tabular}{c c}
|
||
\toprule
|
||
Full name & Student ID\\
|
||
\midrule
|
||
Nguyễn Gia Phong & BI9-184\\
|
||
Nguyễn Quốc Thông & BI9-214\\
|
||
Nguyễn Văn Tùng & BI9-229\\
|
||
Trần Minh Vương & BI9-239\\
|
||
\bottomrule
|
||
\end{tabular}
|
||
\end{center}
|
||
|
||
This report is licensed under a CC BY-SA 4.0 license, while the source code is
|
||
available on GitHub\footnote{\url{https://github.com/McSinyx/cheese-shop}}
|
||
under AGPLv3+.
|
||
|
||
We would like to express our special thanks to Dr. Nguyễn Hoàng Hà,
|
||
whose lectures gave us basic understanding on the key principles of
|
||
relational databases. In addition, we also recieved a lot of help from
|
||
the Python packaging community over \#pypa on Freenode on understanding
|
||
the structure of the metadata as well as finding a way to fetch these
|
||
data from package indices.
|
||
|
||
\newpage
|
||
\selectlanguage{english}
|
||
\section{User Requirements}
|
||
This project aims to provide a database for metadata queries and Python packages
|
||
exploration. We try to replicate the PyPI's XML-RPC API~\cite{xmlrpc},
|
||
which supports queris similar to the following:
|
||
\begin{itemize}
|
||
\item \verb|list_projects()|: Retrieve a list of registered project names.
|
||
\item \verb|project_releases(project)|: Retrieve a list of releases for
|
||
the given \verb|project|, ordered by version.
|
||
\item \verb|project_release_latest()|: Retrieve the latest release
|
||
of the given \verb|project|.
|
||
\item \verb|belong_to(name)|: Retrieve a list of projects whose author
|
||
is \verb|name|.
|
||
\item \verb|browse(classifier)|: Retrieve a list of (\verb|project|,
|
||
\verb|version|) of all releases classified with all of the given classifier.
|
||
\item \verb|release_data(project, version)|: Retrieve the following metadata
|
||
matching the given release: project, version, homepage, author,
|
||
author's email, summary, license, keywords, classifiers and dependencies
|
||
\item \verb|search_name(pattern)|: Retrieve a list of (\verb|project|,
|
||
\verb|version|, \verb|summary|) where the project name matches the pattern.
|
||
\item \verb|search_summary(pattern)|: Retrieve a list of (\verb|project|,
|
||
\verb|version|, \verb|summary|) where the summary matches the pattern.
|
||
\end{itemize}
|
||
|
||
\section{Data Definition}
|
||
\subsection{Entity Relationship Diagram}
|
||
The entity relationship diagram represents the relationship between each of
|
||
its entity set of data extracted from projects:
|
||
\begin{itemize}
|
||
\item Author(Releases-Contact: Many-One): Within each release, there could be
|
||
one author, due to data extraction method doesn't support multi-author.
|
||
Yet an author could have multiple releases under per name.
|
||
\item Require(Releases-Dependencies: Many-Many): Every release would require
|
||
a number of dependencies, and many dependencies can each be used by
|
||
multiple releases.
|
||
\item Classify(Releases-Trove: Many-Many): This relationship indicates the
|
||
relationship between trove classifier and each releases, with many release
|
||
could be classified under one trove classifier, and a release could be
|
||
classified by many classifiers.
|
||
\item Contain(Releases-Keyword: Many-Many): A release has many keywords,
|
||
and also a keyword can also be in many different releases.
|
||
\item Release(Releases-Distribution: One-Many): Within each releases,
|
||
a number of distribution(s) would be released. A distribution could
|
||
relate to only one releases, but many distributions could be released
|
||
in the same releases.
|
||
\end{itemize}
|
||
\includegraphics[width=\textwidth]{erd.jpg}
|
||
|
||
\subsection{Database Schema}
|
||
\begin{center}
|
||
\includegraphics[width=\textwidth]{schema.png}
|
||
\end{center}
|
||
|
||
\subsubsection{releases}
|
||
This entity set represents each releases of the project,include the name of the project and its version in addition to summary,homepage and author's email. The ID of each releases is the primary key to represent each one of them.
|
||
This release ID is also the foreign key of many primary key in other entity set.
|
||
\subsubsection{keywords}
|
||
Containing both the ID of the releases and the terminology as primary key,this entity represent the keywords of a specific release.
|
||
\subsubsection{contact}
|
||
Containing contact information of the author,including email (primary key) and name
|
||
\subsubsection{information}
|
||
Specific information of each releases. Containing release ID,summary,homepage and author's email of the releases.
|
||
\subsubsection{trove}
|
||
This entity set represent Trove classifiers,identified by its ID.
|
||
\subsubsection{classifiers}
|
||
Containing the release ID and Trove classifiers ID,this table has the role of representing the relationship of trove and releases
|
||
\subsubsection{Distribution}
|
||
This entity set represents the distribution of each releases. With its primary key its release ID along with its filename,each distribution contains the url,python version and the python version it requires,the distribtions it requires and its digests (a dictionary) sha256 and md5
|
||
|
||
|
||
\section{Data Query}
|
||
\subsection{Project Listing}
|
||
Retrieve a list of the project names registered with the project index.
|
||
\subsection{Project Releases}
|
||
Retrieve a list of the releases registered for the given project name, ordered by version.
|
||
\subsection{Users}
|
||
Retrieve a list of role, user for a given project name.
|
||
\subsection{Release URLs}
|
||
Retrieve a list of download URLs for the given release version.
|
||
\subsection{Release Data}
|
||
retrieve metadata describing a specific release version.
|
||
\subsection{Classifiers}
|
||
Retrieve a list of name, version of all releases classified with all of the given classifiers, classifiers must be a list of Trove classifier strings.
|
||
\section{Conclusion}
|
||
|
||
\begin{thebibliography}{69}
|
||
\bibitem{xmlrpc}
|
||
The Python Packaging Authority.
|
||
\href{https://warehouse.readthedocs.io/api-reference/xml-rpc}
|
||
{\emph{PyPI’s XML-RPC methods}}.
|
||
Warehouse documentation.
|
||
\end{thebibliography}
|
||
\end{document}
|