cisticola.utils module

cisticola.utils.make_request(url, headers=None, max_retries=5, break_codes=None)

Retry request max_retries times, while catching arbitrary exceptions.

Parameters:

  • url (str) – URL of content that is being requested

  • headers (dict or None) – Dictionary of key-value pairs for request headers

  • max_retries (int) – Maximum number of times to retry the request

  • break_codes (list or None) – List of acceptable status codes that indicate that the request should not be retried further. Useful if, for example, a 404 is expected at some point to terminate a loop, and we don’t want to retry to get the 404-ed page multiple times.

Returns:

Reponse from the request, or None if all retries failed.

Return type:

requests.Response or None

cisticola.utils.request_until_200(url, headers=None, max_retries=5, break_codes=None)

Retry request max_retries times, or until the request is successful.