b69bc771b9
but quite a few handy improvements nonetheless. Scrapy now supports anonymous FTP sessions with customizable user and password via the new :setting:`FTP_USER` and :setting:`FTP_PASSWORD` settings. And if you're using Twisted version 17.1.0 or above, FTP is now available with Python 3. There's a new :meth:`response.follow <scrapy.http.TextResponse.follow>` method for creating requests; **it is now a recommended way to create Requests in Scrapy spiders**. This method makes it easier to write correct spiders; ``response.follow`` has several advantages over creating ``scrapy.Request`` objects directly: * it handles relative URLs; * it works properly with non-ascii URLs on non-UTF8 pages; * in addition to absolute and relative URLs it supports Selectors; for ``<a>`` elements it can also extract their href values.
23 lines
778 B
Makefile
23 lines
778 B
Makefile
# $NetBSD: Makefile,v 1.3 2017/05/20 06:25:36 adam Exp $
|
|
|
|
DISTNAME= Scrapy-1.4.0
|
|
PKGNAME= ${PYPKGPREFIX}-${DISTNAME:tl}
|
|
CATEGORIES= www python
|
|
MASTER_SITES= ${MASTER_SITE_PYPI:=S/Scrapy/}
|
|
|
|
MAINTAINER= pkgsrc-users@NetBSD.org
|
|
HOMEPAGE= http://scrapy.org/
|
|
COMMENT= High-level Web Crawling and Web Scraping framework
|
|
LICENSE= modified-bsd
|
|
|
|
DEPENDS+= ${PYPKGPREFIX}-queuelib-[0-9]*:../../devel/py-queuelib
|
|
DEPENDS+= ${PYPKGPREFIX}-parsel>=1.1.0:../../www/py-parsel
|
|
DEPENDS+= ${PYPKGPREFIX}-pydispatcher>=2.0.5:../../devel/py-pydispatcher
|
|
DEPENDS+= ${PYPKGPREFIX}-twisted>=13.1.0:../../net/py-twisted
|
|
|
|
post-install:
|
|
${MV} ${DESTDIR}${PREFIX}/bin/scrapy \
|
|
${DESTDIR}${PREFIX}/bin/scrapy${PYVERSSUFFIX} || ${TRUE}
|
|
|
|
.include "../../lang/python/egg.mk"
|
|
.include "../../mk/bsd.pkg.mk"
|