Commits

Commit history

1. Commits on Jun 16, 2019

   • bpo-35922: Fix RobotFileParser when robots.txt has no relevant crawl delay or request rate (GH-11791)

     Show description for 8047e0e
     

     Rémi Lapeyre

     and

     taleinat

     committed
     8047e0eView commit details

     Copy full SHA for 8047e0e

     View code at this pointBrowse repository at this point
2. Commits on May 16, 2018

   • bpo-21475: Support the Sitemap extension in robotparser (GH-6883)

     

     mcscope

     authored and

     ned-deily

     committed
     5db5c06View commit details

     Copy full SHA for 5db5c06

     View code at this pointBrowse repository at this point
3. Commits on May 14, 2018

   • bpo-32861: urllib.robotparser fix incomplete __str__ methods. (GH-5711)

     Show description for bd08a0a
     

     michael-lazar

     authored and

     serhiy-storchaka

     committed
     bd08a0aView commit details

     Copy full SHA for bd08a0a

     View code at this pointBrowse repository at this point
4. Commits on Nov 23, 2017

   • bpo-31325: Fix usage of namedtuple in RobotFileParser.parse() (#4529)

     

     berkerpeksag

     authored and

     rhettinger

     committed
     3df02dbView commit details

     Copy full SHA for 3df02db

     View code at this pointBrowse repository at this point
5. Commits on Sep 18, 2016

   • Issue #25400: RobotFileParser now correctly returns default values for crawl_delay and request_rate

     Show description for 9a7bbb2
     berkerpeksag

     committed
     9a7bbb2View commit details

     Copy full SHA for 9a7bbb2

     View code at this pointBrowse repository at this point
6. Commits on May 8, 2016

   • Merge typo fixes from 3.5

     vadmium

     committed
     1ce738eView commit details

     Copy full SHA for 1ce738e

     View code at this pointBrowse repository at this point

   • Fix typos in comments, documentation and test method names

     vadmium

     committed
     f056416View commit details

     Copy full SHA for f056416

     View code at this pointBrowse repository at this point
7. Commits on Oct 8, 2015

   • Issue #16099: RobotFileParser now supports Crawl-delay and Request-rate

     Show description for 960e848
     berkerpeksag

     committed
     960e848View commit details

     Copy full SHA for 960e848

     View code at this pointBrowse repository at this point
8. Commits on May 13, 2014

   • Issue 21469: Minor code modernization (convert and/or expression to an if/else expression).

     Show description for 38acd4c
     rhettinger

     committed
     38acd4cView commit details

     Copy full SHA for 38acd4c

     View code at this pointBrowse repository at this point

   • Issue 21469: Mitigate risk of false positives with robotparser.

     Show description for 122541b
     rhettinger

     committed
     122541bView commit details

     Copy full SHA for 122541b

     View code at this pointBrowse repository at this point
9. Commits on May 29, 2013

   • #17403: urllib.parse.robotparser normalizes the urls before adding to ruleline.

     Show description for c70a6ae
     orsenthil

     committed
     c70a6aeView commit details

     Copy full SHA for c70a6ae

     View code at this pointBrowse repository at this point
10. Commits on Jul 29, 2010

    • #4108: the first default entry (User-agent: *) wins.

      birkenfeld

      committed
      0a0fc07View commit details

      Copy full SHA for 0a0fc07

      View code at this pointBrowse repository at this point
11. Commits on Jul 28, 2010

    • Fix Issue6325 - robotparse to honor urls with query strings.

      orsenthil

      committed
      3f8ab96View commit details

      Copy full SHA for 3f8ab96

      View code at this pointBrowse repository at this point
12. Commits on Jul 31, 2008

    • Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from

      Show description for d631371
      benjaminp

      committed
      d631371View commit details

      Copy full SHA for d631371

      View code at this pointBrowse repository at this point
13. Commits on Jul 18, 2008

    • Bug 3347: robotparser failed because it didn't convert bytes to string.

      Show description for 73fd46d
      jeremyhylton

      committed
      73fd46dView commit details

      Copy full SHA for 73fd46d

      View code at this pointBrowse repository at this point
14. Commits on Jun 18, 2008

    • Make a new urllib package .

      Show description for 1afc169
      jeremyhylton

      committed
      1afc169View commit details

      Copy full SHA for 1afc169

      View code at this pointBrowse repository at this point
15. Loading