pkgsrc

kpriv/pkgsrc

Fork 0

Commit graph

Author	SHA1	Message	Date
taca	d254c7cf3d	Update ruby-webrobots to 0.1.2. * Make the default `http_get` proc act more like Googlebot. - It only allows up to 5 redirect hops rather than 10. - It treats too many redirections as full allow. - It treats any 4xx error as full allow. * Fix context/should wording * Follow the deprecation warning from SimpleCov::Formatter::MultiFormatter * Update real world tests, introducing Webmock+VCR	2016-02-01 14:08:23 +00:00
taca	2902c78f8e	Update ruby-webrobots to 0.1.0. * Change the :ignore_crawl_delay option to a handler option :crawl_delay. * Merge back the previous change to the source file. * Remove Gemfile.lock. * Add the ability to externalize the enforcement of the robots.txt Craw jasonhowes authored a month ago. * Migrate from jeweler to bundle gem.	2013-03-14 16:27:28 +00:00
taca	52181f6970	Imporging www/ruby-webrobots package version 0.0.11. This library helps write robots.txt compliant web robots in Ruby.	2011-08-12 17:30:51 +00:00

Author

SHA1

Message

Date

taca

d254c7cf3d

Update ruby-webrobots to 0.1.2.

* Make the default `http_get` proc act more like Googlebot.

  - It only allows up to 5 redirect hops rather than 10.
  - It treats too many redirections as full allow.
  - It treats any 4xx error as full allow.

* Fix context/should wording

* Follow the deprecation warning from SimpleCov::Formatter::MultiFormatter

* Update real world tests, introducing Webmock+VCR

2016-02-01 14:08:23 +00:00

taca

2902c78f8e

Update ruby-webrobots to 0.1.0.

* Change the :ignore_crawl_delay option to a handler option :crawl_delay.
* Merge back the previous change to the source file.
* Remove Gemfile.lock.
* Add the ability to externalize the enforcement of the robots.txt Craw
 jasonhowes authored a month ago.
* Migrate from jeweler to bundle gem.

2013-03-14 16:27:28 +00:00

taca

52181f6970

Imporging www/ruby-webrobots package version 0.0.11.

This library helps write robots.txt compliant web robots in Ruby.

2011-08-12 17:30:51 +00:00

3 commits