[Python-modules-commits] [python-urllib3] 02/08: Import python-urllib3_1.16.orig.tar.gz
Daniele Tricoli
eriol-guest at moszumanska.debian.org
Sun Sep 4 00:01:30 UTC 2016
This is an automated email from the git hooks/post-receive script.
eriol-guest pushed a commit to branch master
in repository python-urllib3.
commit a7f3c701b341965dfb3552fb223023dd5bd9e99b
Author: Daniele Tricoli <eriol at mornie.org>
Date: Sun Sep 4 01:23:18 2016 +0200
Import python-urllib3_1.16.orig.tar.gz
---
CHANGES.rst | 29 +-
CONTRIBUTORS.txt | 11 +-
PKG-INFO | 39 +-
README.rst | 6 +-
dev-requirements.txt | 1 +
docs/managers.rst | 44 +++
docs/security.rst | 2 +-
setup.cfg | 5 +-
setup.py | 2 +-
test/test_connectionpool.py | 77 ++++
test/test_poolmanager.py | 230 ++++++++++-
test/test_retry.py | 15 +
test/test_util.py | 32 +-
test/with_dummyserver/test_socketlevel.py | 62 ++-
urllib3.egg-info/PKG-INFO | 39 +-
urllib3/__init__.py | 2 +-
urllib3/connectionpool.py | 37 +-
urllib3/contrib/appengine.py | 2 +-
urllib3/contrib/socks.py | 2 +-
urllib3/packages/six.py | 635 ++++++++++++++++++++++++++----
urllib3/poolmanager.py | 93 ++++-
urllib3/response.py | 4 +
urllib3/util/connection.py | 47 ++-
urllib3/util/retry.py | 14 +-
urllib3/util/ssl_.py | 4 +-
25 files changed, 1304 insertions(+), 130 deletions(-)
diff --git a/CHANGES.rst b/CHANGES.rst
index 443379c..fdf6a6e 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -1,6 +1,31 @@
Changes
=======
+1.16 (2016-06-11)
+-----------------
+
+* Disable IPv6 DNS when IPv6 connections are not possible. (Issue #840)
+
+* Provide ``key_fn_by_scheme`` pool keying mechanism that can be
+ overridden. (Issue #830)
+
+* Normalize scheme and host to lowercase for pool keys, and include
+ ``source_address``. (Issue #830)
+
+* Cleaner exception chain in Python 3 for ``_make_request``.
+ (Issue #861)
+
+* Fixed installing ``urllib3[socks]`` extra. (Issue #864)
+
+* Fixed signature of ``ConnectionPool.close`` so it can actually safely be
+ called by subclasses. (Issue #873)
+
+* Retain ``release_conn`` state across retries. (Issues #651, #866)
+
+* Add customizable ``HTTPConnectionPool.ResponseCls``, which defaults to
+ ``HTTPResponse`` but can be replaced with a subclass. (Issue #879)
+
+
1.15.1 (2016-04-11)
-------------------
@@ -88,7 +113,7 @@ Changes
* Made ``HTTPHeaderDict`` usable as a ``headers`` input value
(Issues #632, #679)
-* Added `urllib3.contrib.appengine <https://urllib3.readthedocs.org/en/latest/contrib.html#google-app-engine>`_
+* Added `urllib3.contrib.appengine <https://urllib3.readthedocs.io/en/latest/contrib.html#google-app-engine>`_
which has an ``AppEngineManager`` for using ``URLFetch`` in a
Google AppEngine environment. (Issue #664)
@@ -245,7 +270,7 @@ Changes
* Unverified HTTPS requests will trigger a warning on the first request. See
our new `security documentation
- <https://urllib3.readthedocs.org/en/latest/security.html>`_ for details.
+ <https://urllib3.readthedocs.io/en/latest/security.html>`_ for details.
(Issue #426)
* New retry logic and ``urllib3.util.retry.Retry`` configuration object.
diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt
index d9a20ee..85df95a 100644
--- a/CONTRIBUTORS.txt
+++ b/CONTRIBUTORS.txt
@@ -1,4 +1,4 @@
-# Contributions to the urllib3 project
+# Contributions to the urllib3 project
## Creator & Maintainer
@@ -168,6 +168,8 @@ In chronological order:
* Jordan Moldow <https://github.com/jmoldow>
* Fix low-level exceptions leaking from ``HTTPResponse.stream()``.
+ * Bugfix for ``ConnectionPool.urlopen(release_conn=False)``.
+ * Creation of ``HTTPConnectionPool.ResponseCls``.
* Predrag Gruevski <https://github.com/obi1kenobi>
* Made cert digest comparison use a constant-time algorithm.
@@ -191,11 +193,14 @@ In chronological order:
* Started Recipes documentation and added a recipe about handling concatenated gzip data in HTTP response
* Jesse Shapiro <jesse at jesseshapiro.net>
- * Working on encoding unicode header parameter names
- * Making setup.py resilient to ASCII locales
+ * Various character-encoding fixes/tweaks
+ * Disabling IPv6 DNS when IPv6 connections not supported
* David Foster <http://dafoster.net/>
* Ensure order of request and response headers are preserved.
+* Jeremy Cline <jeremy at jcline.org>
+ * Added connection pool keys by scheme
+
* [Your name or handle] <[email or website]>
* [Brief summary of your changes]
diff --git a/PKG-INFO b/PKG-INFO
index 878f9c4..fc5d416 100644
--- a/PKG-INFO
+++ b/PKG-INFO
@@ -1,8 +1,8 @@
Metadata-Version: 1.1
Name: urllib3
-Version: 1.15.1
+Version: 1.16
Summary: HTTP library with thread-safe connection pooling, file post, and more.
-Home-page: http://urllib3.readthedocs.org/
+Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: andrey.petrov at shazow.net
License: MIT
@@ -82,7 +82,7 @@ Description: =======
Examples
========
- Go to `urllib3.readthedocs.org <http://urllib3.readthedocs.org>`_
+ Go to `urllib3.readthedocs.org <https://urllib3.readthedocs.io>`_
for more nice syntax-highlighted examples.
But, long story short::
@@ -98,7 +98,7 @@ Description: =======
The ``PoolManager`` will take care of reusing connections for you whenever
you request the same host. For more fine-grained control of your connection
pools, you should look at `ConnectionPool
- <http://urllib3.readthedocs.org/#connectionpool>`_.
+ <https://urllib3.readthedocs.io/#connectionpool>`_.
Run the tests
@@ -166,12 +166,37 @@ Description: =======
===========
If your company benefits from this library, please consider `sponsoring its
- development <http://urllib3.readthedocs.org/en/latest/#sponsorship>`_.
+ development <https://urllib3.readthedocs.io/en/latest/#sponsorship>`_.
Changes
=======
+ 1.16 (2016-06-11)
+ -----------------
+
+ * Disable IPv6 DNS when IPv6 connections are not possible. (Issue #840)
+
+ * Provide ``key_fn_by_scheme`` pool keying mechanism that can be
+ overridden. (Issue #830)
+
+ * Normalize scheme and host to lowercase for pool keys, and include
+ ``source_address``. (Issue #830)
+
+ * Cleaner exception chain in Python 3 for ``_make_request``.
+ (Issue #861)
+
+ * Fixed installing ``urllib3[socks]`` extra. (Issue #864)
+
+ * Fixed signature of ``ConnectionPool.close`` so it can actually safely be
+ called by subclasses. (Issue #873)
+
+ * Retain ``release_conn`` state across retries. (Issues #651, #866)
+
+ * Add customizable ``HTTPConnectionPool.ResponseCls``, which defaults to
+ ``HTTPResponse`` but can be replaced with a subclass. (Issue #879)
+
+
1.15.1 (2016-04-11)
-------------------
@@ -259,7 +284,7 @@ Description: =======
* Made ``HTTPHeaderDict`` usable as a ``headers`` input value
(Issues #632, #679)
- * Added `urllib3.contrib.appengine <https://urllib3.readthedocs.org/en/latest/contrib.html#google-app-engine>`_
+ * Added `urllib3.contrib.appengine <https://urllib3.readthedocs.io/en/latest/contrib.html#google-app-engine>`_
which has an ``AppEngineManager`` for using ``URLFetch`` in a
Google AppEngine environment. (Issue #664)
@@ -416,7 +441,7 @@ Description: =======
* Unverified HTTPS requests will trigger a warning on the first request. See
our new `security documentation
- <https://urllib3.readthedocs.org/en/latest/security.html>`_ for details.
+ <https://urllib3.readthedocs.io/en/latest/security.html>`_ for details.
(Issue #426)
* New retry logic and ``urllib3.util.retry.Retry`` configuration object.
diff --git a/README.rst b/README.rst
index 41e4cee..bf92abc 100644
--- a/README.rst
+++ b/README.rst
@@ -74,7 +74,7 @@ This library is perfect for:
Examples
========
-Go to `urllib3.readthedocs.org <http://urllib3.readthedocs.org>`_
+Go to `urllib3.readthedocs.org <https://urllib3.readthedocs.io>`_
for more nice syntax-highlighted examples.
But, long story short::
@@ -90,7 +90,7 @@ But, long story short::
The ``PoolManager`` will take care of reusing connections for you whenever
you request the same host. For more fine-grained control of your connection
pools, you should look at `ConnectionPool
-<http://urllib3.readthedocs.org/#connectionpool>`_.
+<https://urllib3.readthedocs.io/#connectionpool>`_.
Run the tests
@@ -158,4 +158,4 @@ Sponsorship
===========
If your company benefits from this library, please consider `sponsoring its
-development <http://urllib3.readthedocs.org/en/latest/#sponsorship>`_.
+development <https://urllib3.readthedocs.io/en/latest/#sponsorship>`_.
diff --git a/dev-requirements.txt b/dev-requirements.txt
index bc7cd7f..47af0d0 100644
--- a/dev-requirements.txt
+++ b/dev-requirements.txt
@@ -7,3 +7,4 @@ twine==1.5.0
wheel==0.24.0
tornado==4.2.1
PySocks==1.5.6
+pkginfo>=1.0,!=1.3.0
diff --git a/docs/managers.rst b/docs/managers.rst
index 825e2f4..e8e7e21 100644
--- a/docs/managers.rst
+++ b/docs/managers.rst
@@ -28,6 +28,44 @@ so you don't have to.
>>> conn.num_requests
3
+A :class:`.PoolManager` will create a new :doc:`ConnectionPool <pools>`
+when no :doc:`ConnectionPools <pools>` exist with a matching pool key.
+The pool key is derived using the requested URL and the current values
+of the ``connection_pool_kw`` instance variable on :class:`.PoolManager`.
+
+The keys in ``connection_pool_kw`` used when deriving the key are
+configurable. For example, by default the ``my_field`` key is not
+considered.
+
+.. doctest ::
+
+ >>> from urllib3.poolmanager import PoolManager
+ >>> manager = PoolManager(10, my_field='wheat')
+ >>> manager.connection_from_url('http://example.com')
+ >>> manager.connection_pool_kw['my_field'] = 'barley'
+ >>> manager.connection_from_url('http://example.com')
+ >>> len(manager.pools)
+ 1
+
+To make the pool manager create new pools when the value of
+``my_field`` changes, you can define a custom pool key and alter
+the ``key_fn_by_scheme`` instance variable on :class:`.PoolManager`.
+
+.. doctest ::
+
+ >>> import functools
+ >>> from collections import namedtuple
+ >>> from urllib3.poolmanager import PoolManager, HTTPPoolKey
+ >>> from urllib3.poolmanager import default_key_normalizer as normalizer
+ >>> CustomKey = namedtuple('CustomKey', HTTPPoolKey._fields + ('my_field',))
+ >>> manager = PoolManager(10, my_field='wheat')
+ >>> manager.key_fn_by_scheme['http'] = functools.partial(normalizer, CustomKey)
+ >>> manager.connection_from_url('http://example.com')
+ >>> manager.connection_pool_kw['my_field'] = 'barley'
+ >>> manager.connection_from_url('http://example.com')
+ >>> len(manager.pools)
+ 2
+
The API of a :class:`.PoolManager` object is similar to that of a
:doc:`ConnectionPool <pools>`, so they can be passed around interchangeably.
@@ -59,6 +97,12 @@ API
.. autoclass:: PoolManager
:inherited-members:
+ .. autoclass:: BasePoolKey
+ :inherited-members:
+ .. autoclass:: HTTPPoolKey
+ :inherited-members:
+ .. autoclass:: HTTPSPoolKey
+ :inherited-members:
ProxyManager
============
diff --git a/docs/security.rst b/docs/security.rst
index 59cbdc0..5117e32 100644
--- a/docs/security.rst
+++ b/docs/security.rst
@@ -180,7 +180,7 @@ Unverified HTTPS requests will trigger a warning via Python's ``warnings`` modul
urllib3/connectionpool.py:736: InsecureRequestWarning: Unverified HTTPS
request is being made. Adding certificate verification is strongly advised.
- See: https://urllib3.readthedocs.org/en/latest/security.html
+ See: https://urllib3.readthedocs.io/en/latest/security.html
This would be a great time to enable HTTPS verification:
:ref:`certifi-with-urllib3`.
diff --git a/setup.cfg b/setup.cfg
index ec26776..972e02c 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -13,12 +13,15 @@ max-line-length = 99
universal = 1
[metadata]
-provides-extra = secure
+provides-extra =
+ secure
+ socks
requires-dist =
pyOpenSSL>=0.13; python_version<="2.7" and extra == 'secure'
ndg-httpsclient; python_version<="2.7" and extra == 'secure'
pyasn1; python_version<="2.7" and extra == 'secure'
certifi; extra == 'secure'
+ PySocks>=1.5.6,<2.0; extra == 'socks'
[egg_info]
tag_svn_revision = 0
diff --git a/setup.py b/setup.py
index 994bd89..3ea3388 100644
--- a/setup.py
+++ b/setup.py
@@ -36,7 +36,7 @@ setup(name='urllib3',
keywords='urllib httplib threadsafe filepost http https ssl pooling',
author='Andrey Petrov',
author_email='andrey.petrov at shazow.net',
- url='http://urllib3.readthedocs.org/',
+ url='https://urllib3.readthedocs.io/',
license='MIT',
packages=['urllib3',
'urllib3.packages', 'urllib3.packages.ssl_match_hostname',
diff --git a/test/test_connectionpool.py b/test/test_connectionpool.py
index 55fa55a..4c9e37a 100644
--- a/test/test_connectionpool.py
+++ b/test/test_connectionpool.py
@@ -1,3 +1,5 @@
+from __future__ import absolute_import
+
import unittest
from urllib3.connectionpool import (
@@ -6,6 +8,7 @@ from urllib3.connectionpool import (
HTTPConnectionPool,
HTTPSConnectionPool,
)
+from urllib3.response import httplib, HTTPResponse
from urllib3.util.timeout import Timeout
from urllib3.packages.ssl_match_hostname import CertificateError
from urllib3.exceptions import (
@@ -16,7 +19,10 @@ from urllib3.exceptions import (
MaxRetryError,
ProtocolError,
SSLError,
+ TimeoutError,
)
+from urllib3._collections import HTTPHeaderDict
+from .test_response import MockChunkedEncodingResponse, MockSock
from socket import error as SocketError
from ssl import SSLError as BaseSSLError
@@ -309,6 +315,77 @@ class TestConnectionPool(unittest.TestCase):
new_pool_size = c.pool.qsize()
self.assertEqual(initial_pool_size, new_pool_size)
+ def test_release_conn_param_is_respected_after_http_error_retry(self):
+ """For successful ```urlopen(release_conn=False)```, the connection isn't released, even after a retry.
+
+ This is a regression test for issue #651 [1], where the connection
+ would be released if the initial request failed, even if a retry
+ succeeded.
+
+ [1] <https://github.com/shazow/urllib3/issues/651>
+ """
+
+ class _raise_once_make_request_function(object):
+ """Callable that can mimic `_make_request()`.
+
+ Raises the given exception on its first call, but returns a
+ successful response on subsequent calls.
+ """
+ def __init__(self, ex):
+ super(_raise_once_make_request_function, self).__init__()
+ self._ex = ex
+
+ def __call__(self, *args, **kwargs):
+ if self._ex:
+ ex, self._ex = self._ex, None
+ raise ex()
+ response = httplib.HTTPResponse(MockSock)
+ response.fp = MockChunkedEncodingResponse([b'f', b'o', b'o'])
+ response.headers = response.msg = HTTPHeaderDict()
+ return response
+
+ def _test(exception):
+ pool = HTTPConnectionPool(host='localhost', maxsize=1, block=True)
+
+ # Verify that the request succeeds after two attempts, and that the
+ # connection is left on the response object, instead of being
+ # released back into the pool.
+ pool._make_request = _raise_once_make_request_function(exception)
+ response = pool.urlopen('GET', '/', retries=1,
+ release_conn=False, preload_content=False,
+ chunked=True)
+ self.assertEqual(pool.pool.qsize(), 0)
+ self.assertEqual(pool.num_connections, 2)
+ self.assertTrue(response.connection is not None)
+
+ response.release_conn()
+ self.assertEqual(pool.pool.qsize(), 1)
+ self.assertTrue(response.connection is None)
+
+ # Run the test case for all the retriable exceptions.
+ _test(TimeoutError)
+ _test(HTTPException)
+ _test(SocketError)
+ _test(ProtocolError)
+
+ def test_custom_http_response_class(self):
+
+ class CustomHTTPResponse(HTTPResponse):
+ pass
+
+ class CustomConnectionPool(HTTPConnectionPool):
+ ResponseCls = CustomHTTPResponse
+
+ def _make_request(self, *args, **kwargs):
+ httplib_response = httplib.HTTPResponse(MockSock)
+ httplib_response.fp = MockChunkedEncodingResponse([b'f', b'o', b'o'])
+ httplib_response.headers = httplib_response.msg = HTTPHeaderDict()
+ return httplib_response
+
+ pool = CustomConnectionPool(host='localhost', maxsize=1, block=True)
+ response = pool.request('GET', '/', retries=False, chunked=True,
+ preload_content=False)
+ self.assertTrue(isinstance(response, CustomHTTPResponse))
if __name__ == '__main__':
diff --git a/test/test_poolmanager.py b/test/test_poolmanager.py
index 6195d51..fb134fb 100644
--- a/test/test_poolmanager.py
+++ b/test/test_poolmanager.py
@@ -1,11 +1,21 @@
+import functools
import unittest
+from collections import namedtuple
-from urllib3.poolmanager import PoolManager
+from urllib3.poolmanager import (
+ _default_key_normalizer,
+ HTTPPoolKey,
+ HTTPSPoolKey,
+ key_fn_by_scheme,
+ PoolManager,
+ SSL_KEYWORDS,
+)
from urllib3 import connection_from_url
from urllib3.exceptions import (
ClosedPoolError,
LocationValueError,
)
+from urllib3.util import retry, timeout
class TestPoolManager(unittest.TestCase):
@@ -87,6 +97,224 @@ class TestPoolManager(unittest.TestCase):
self.assertEqual(len(p.pools), 0)
+ def test_http_pool_key_fields(self):
+ """Assert the HTTPPoolKey fields are honored when selecting a pool."""
+ connection_pool_kw = {
+ 'timeout': timeout.Timeout(3.14),
+ 'retries': retry.Retry(total=6, connect=2),
+ 'block': True,
+ 'strict': True,
+ 'source_address': '127.0.0.1',
+ }
+ p = PoolManager()
+ conn_pools = [
+ p.connection_from_url('http://example.com/'),
+ p.connection_from_url('http://example.com:8000/'),
+ p.connection_from_url('http://other.example.com/'),
+ ]
+
+ for key, value in connection_pool_kw.items():
+ p.connection_pool_kw[key] = value
+ conn_pools.append(p.connection_from_url('http://example.com/'))
+
+ self.assertTrue(
+ all(
+ x is not y
+ for i, x in enumerate(conn_pools)
+ for j, y in enumerate(conn_pools)
+ if i != j
+ )
+ )
+ self.assertTrue(
+ all(
+ isinstance(key, HTTPPoolKey)
+ for key in p.pools.keys())
+ )
+
+ def test_http_pool_key_extra_kwargs(self):
+ """Assert non-HTTPPoolKey fields are ignored when selecting a pool."""
+ p = PoolManager()
+ conn_pool = p.connection_from_url('http://example.com/')
+ p.connection_pool_kw['some_kwarg'] = 'that should be ignored'
+ other_conn_pool = p.connection_from_url('http://example.com/')
+
+ self.assertTrue(conn_pool is other_conn_pool)
+
+ def test_http_pool_key_https_kwargs(self):
+ """Assert HTTPSPoolKey fields are ignored when selecting a HTTP pool."""
+ p = PoolManager()
+ conn_pool = p.connection_from_url('http://example.com/')
+ for key in SSL_KEYWORDS:
+ p.connection_pool_kw[key] = 'this should be ignored'
+ other_conn_pool = p.connection_from_url('http://example.com/')
+
+ self.assertTrue(conn_pool is other_conn_pool)
+
+ def test_https_pool_key_fields(self):
+ """Assert the HTTPSPoolKey fields are honored when selecting a pool."""
+ connection_pool_kw = {
+ 'timeout': timeout.Timeout(3.14),
+ 'retries': retry.Retry(total=6, connect=2),
+ 'block': True,
+ 'strict': True,
+ 'source_address': '127.0.0.1',
+ 'key_file': '/root/totally_legit.key',
+ 'cert_file': '/root/totally_legit.crt',
+ 'cert_reqs': 'CERT_REQUIRED',
+ 'ca_certs': '/root/path_to_pem',
+ 'ssl_version': 'SSLv23_METHOD',
+ }
+ p = PoolManager()
+ conn_pools = [
+ p.connection_from_url('https://example.com/'),
+ p.connection_from_url('https://example.com:4333/'),
+ p.connection_from_url('https://other.example.com/'),
+ ]
+ # Asking for a connection pool with the same key should give us an
+ # existing pool.
+ dup_pools = []
+
+ for key, value in connection_pool_kw.items():
+ p.connection_pool_kw[key] = value
+ conn_pools.append(p.connection_from_url('https://example.com/'))
+ dup_pools.append(p.connection_from_url('https://example.com/'))
+
+ self.assertTrue(
+ all(
+ x is not y
+ for i, x in enumerate(conn_pools)
+ for j, y in enumerate(conn_pools)
+ if i != j
+ )
+ )
+ self.assertTrue(all(pool in conn_pools for pool in dup_pools))
+ self.assertTrue(
+ all(
+ isinstance(key, HTTPSPoolKey)
+ for key in p.pools.keys())
+ )
+
+ def test_https_pool_key_extra_kwargs(self):
+ """Assert non-HTTPSPoolKey fields are ignored when selecting a pool."""
+ p = PoolManager()
+ conn_pool = p.connection_from_url('https://example.com/')
+ p.connection_pool_kw['some_kwarg'] = 'that should be ignored'
+ other_conn_pool = p.connection_from_url('https://example.com/')
+
+ self.assertTrue(conn_pool is other_conn_pool)
+
+ def test_default_pool_key_funcs_copy(self):
+ """Assert each PoolManager gets a copy of ``pool_keys_by_scheme``."""
+ p = PoolManager()
+ self.assertEqual(p.key_fn_by_scheme, p.key_fn_by_scheme)
+ self.assertFalse(p.key_fn_by_scheme is key_fn_by_scheme)
+
+ def test_pools_keyed_with_from_host(self):
+ """Assert pools are still keyed correctly with connection_from_host."""
+ ssl_kw = {
+ 'key_file': '/root/totally_legit.key',
+ 'cert_file': '/root/totally_legit.crt',
+ 'cert_reqs': 'CERT_REQUIRED',
+ 'ca_certs': '/root/path_to_pem',
+ 'ssl_version': 'SSLv23_METHOD',
+ }
+ p = PoolManager(5, **ssl_kw)
+ conns = []
+ conns.append(
+ p.connection_from_host('example.com', 443, scheme='https')
+ )
+
+ for k in ssl_kw:
+ p.connection_pool_kw[k] = 'newval'
+ conns.append(
+ p.connection_from_host('example.com', 443, scheme='https')
+ )
+
+ self.assertTrue(
+ all(
+ x is not y
+ for i, x in enumerate(conns)
+ for j, y in enumerate(conns)
+ if i != j
+ )
+ )
+
+ def test_https_connection_from_url_case_insensitive(self):
+ """Assert scheme case is ignored when pooling HTTPS connections."""
+ p = PoolManager()
+ pool = p.connection_from_url('https://example.com/')
+ other_pool = p.connection_from_url('HTTPS://EXAMPLE.COM/')
+
+ self.assertEqual(1, len(p.pools))
+ self.assertTrue(pool is other_pool)
+ self.assertTrue(all(isinstance(key, HTTPSPoolKey) for key in p.pools.keys()))
+
+ def test_https_connection_from_host_case_insensitive(self):
+ """Assert scheme case is ignored when getting the https key class."""
+ p = PoolManager()
+ pool = p.connection_from_host('example.com', scheme='https')
+ other_pool = p.connection_from_host('EXAMPLE.COM', scheme='HTTPS')
+
+ self.assertEqual(1, len(p.pools))
+ self.assertTrue(pool is other_pool)
+ self.assertTrue(all(isinstance(key, HTTPSPoolKey) for key in p.pools.keys()))
+
+ def test_https_connection_from_context_case_insensitive(self):
+ """Assert scheme case is ignored when getting the https key class."""
+ p = PoolManager()
+ context = {'scheme': 'https', 'host': 'example.com', 'port': '443'}
+ other_context = {'scheme': 'HTTPS', 'host': 'EXAMPLE.COM', 'port': '443'}
+ pool = p.connection_from_context(context)
+ other_pool = p.connection_from_context(other_context)
+
+ self.assertEqual(1, len(p.pools))
+ self.assertTrue(pool is other_pool)
+ self.assertTrue(all(isinstance(key, HTTPSPoolKey) for key in p.pools.keys()))
+
+ def test_http_connection_from_url_case_insensitive(self):
+ """Assert scheme case is ignored when pooling HTTP connections."""
+ p = PoolManager()
+ pool = p.connection_from_url('http://example.com/')
+ other_pool = p.connection_from_url('HTTP://EXAMPLE.COM/')
+
+ self.assertEqual(1, len(p.pools))
+ self.assertTrue(pool is other_pool)
+ self.assertTrue(all(isinstance(key, HTTPPoolKey) for key in p.pools.keys()))
+
+ def test_http_connection_from_host_case_insensitive(self):
+ """Assert scheme case is ignored when getting the https key class."""
+ p = PoolManager()
+ pool = p.connection_from_host('example.com', scheme='http')
+ other_pool = p.connection_from_host('EXAMPLE.COM', scheme='HTTP')
+
+ self.assertEqual(1, len(p.pools))
+ self.assertTrue(pool is other_pool)
+ self.assertTrue(all(isinstance(key, HTTPPoolKey) for key in p.pools.keys()))
+
+ def test_http_connection_from_context_case_insensitive(self):
+ """Assert scheme case is ignored when getting the https key class."""
+ p = PoolManager()
+ context = {'scheme': 'http', 'host': 'example.com', 'port': '8080'}
+ other_context = {'scheme': 'HTTP', 'host': 'EXAMPLE.COM', 'port': '8080'}
+ pool = p.connection_from_context(context)
+ other_pool = p.connection_from_context(other_context)
+
+ self.assertEqual(1, len(p.pools))
+ self.assertTrue(pool is other_pool)
+ self.assertTrue(all(isinstance(key, HTTPPoolKey) for key in p.pools.keys()))
+
+ def test_custom_pool_key(self):
+ """Assert it is possible to define addition pool key fields."""
+ custom_key = namedtuple('CustomKey', HTTPPoolKey._fields + ('my_field',))
+ p = PoolManager(10, my_field='barley')
+
+ p.key_fn_by_scheme['http'] = functools.partial(_default_key_normalizer, custom_key)
+ p.connection_from_url('http://example.com')
+ p.connection_pool_kw['my_field'] = 'wheat'
+ p.connection_from_url('http://example.com')
+
+ self.assertEqual(2, len(p.pools))
+
if __name__ == '__main__':
unittest.main()
diff --git a/test/test_retry.py b/test/test_retry.py
index 421e508..1e87585 100644
--- a/test/test_retry.py
+++ b/test/test_retry.py
@@ -148,6 +148,21 @@ class RetryTest(unittest.TestCase):
self.assertFalse(retry.is_forced_retry('GET', status_code=400))
self.assertTrue(retry.is_forced_retry('GET', status_code=418))
+ # String status codes are not matched.
+ retry = Retry(total=1, status_forcelist=['418'])
+ self.assertFalse(retry.is_forced_retry('GET', status_code=418))
+
+ def test_method_whitelist_with_status_forcelist(self):
+ # Falsey method_whitelist means to retry on any method.
+ retry = Retry(status_forcelist=[500], method_whitelist=None)
+ self.assertTrue(retry.is_forced_retry('GET', status_code=500))
+ self.assertTrue(retry.is_forced_retry('POST', status_code=500))
+
+ # Criteria of method_whitelist and status_forcelist are ANDed.
+ retry = Retry(status_forcelist=[500], method_whitelist=['POST'])
+ self.assertFalse(retry.is_forced_retry('GET', status_code=500))
+ self.assertTrue(retry.is_forced_retry('POST', status_code=500))
+
def test_exhausted(self):
self.assertFalse(Retry(0).is_exhausted())
self.assertTrue(Retry(-1).is_exhausted())
diff --git a/test/test_util.py b/test/test_util.py
index ef4caab..918b1fd 100644
--- a/test/test_util.py
+++ b/test/test_util.py
@@ -3,6 +3,7 @@ import warnings
import logging
import unittest
import ssl
+import socket
from itertools import chain
from mock import patch, Mock
@@ -28,7 +29,10 @@ from urllib3.exceptions import (
SSLError,
SNIMissingWarning,
)
-
+from urllib3.util.connection import (
+ allowed_gai_family,
+ _has_ipv6
+)
from urllib3.util import is_fp_closed, ssl_
from . import clear_warnings
@@ -442,3 +446,29 @@ class TestUtil(unittest.TestCase):
incorrect = hashlib.sha256(b'xyz').digest()
self.assertFalse(_const_compare_digest_backport(target, incorrect))
+
+ def test_has_ipv6_disabled_on_compile(self):
+ with patch('socket.has_ipv6', False):
+ self.assertFalse(_has_ipv6('::1'))
+
+ def test_has_ipv6_enabled_but_fails(self):
+ with patch('socket.has_ipv6', True):
+ with patch('socket.socket') as mock:
+ instance = mock.return_value
+ instance.bind = Mock(side_effect=Exception('No IPv6 here!'))
+ self.assertFalse(_has_ipv6('::1'))
+
+ def test_has_ipv6_enabled_and_working(self):
+ with patch('socket.has_ipv6', True):
+ with patch('socket.socket') as mock:
+ instance = mock.return_value
+ instance.bind.return_value = True
+ self.assertTrue(_has_ipv6('::1'))
+
+ def test_ip_family_ipv6_enabled(self):
+ with patch('urllib3.util.connection.HAS_IPV6', True):
+ self.assertEqual(allowed_gai_family(), socket.AF_UNSPEC)
+
+ def test_ip_family_ipv6_disabled(self):
+ with patch('urllib3.util.connection.HAS_IPV6', False):
+ self.assertEqual(allowed_gai_family(), socket.AF_INET)
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
index c6f3030..3048a34 100644
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -17,7 +17,7 @@ from urllib3.util.timeout import Timeout
from urllib3.util.retry import Retry
from urllib3._collections import HTTPHeaderDict, OrderedDict
-from dummyserver.testcase import SocketDummyServerTestCase
+from dummyserver.testcase import SocketDummyServerTestCase, consume_socket
from dummyserver.server import (
DEFAULT_CERTS, DEFAULT_CA, get_unreachable_address)
@@ -507,6 +507,66 @@ class TestSocketClosing(SocketDummyServerTestCase):
if not successful:
self.fail("Timed out waiting for connection close")
+ def test_release_conn_param_is_respected_after_timeout_retry(self):
+ """For successful ```urlopen(release_conn=False)```, the connection isn't released, even after a retry.
+
+ This test allows a retry: one request fails, the next request succeeds.
+
+ This is a regression test for issue #651 [1], where the connection
+ would be released if the initial request failed, even if a retry
+ succeeded.
+
+ [1] <https://github.com/shazow/urllib3/issues/651>
+ """
+ def socket_handler(listener):
+ sock = listener.accept()[0]
+ consume_socket(sock)
+
+ # Close the connection, without sending any response (not even the
+ # HTTP status line). This will trigger a `Timeout` on the client,
+ # inside `urlopen()`.
+ sock.close()
+
+ # Expect a new request. Because we don't want to hang this thread,
+ # we actually use select.select to confirm that a new request is
+ # coming in: this lets us time the thread out.
+ rlist, _, _ = select.select([listener], [], [], 5)
+ assert rlist
+ sock = listener.accept()[0]
+ consume_socket(sock)
+
+ # Send complete chunked response.
+ sock.send((
+ 'HTTP/1.1 200 OK\r\n'
+ 'Content-Type: text/plain\r\n'
+ 'Transfer-Encoding: chunked\r\n'
+ '\r\n'
+ '8\r\n'
+ '12345678\r\n'
+ '0\r\n\r\n').encode('utf-8')
+ )
+
+ sock.close()
+
+ self._start_server(socket_handler)
+ with HTTPConnectionPool(self.host, self.port, maxsize=1) as pool:
+ # First request should fail, but the timeout and `retries=1` should
+ # save it.
+ response = pool.urlopen('GET', '/', retries=1,
+ release_conn=False, preload_content=False,
+ timeout=Timeout(connect=1, read=0.001))
+
+ # The connection should still be on the response object, and none
+ # should be in the pool. We opened two though.
+ self.assertEqual(pool.num_connections, 2)
+ self.assertEqual(pool.pool.qsize(), 0)
+ self.assertTrue(response.connection is not None)
+
+ # Consume the data. This should put the connection back.
+ response.read()
+ self.assertEqual(pool.pool.qsize(), 1)
+ self.assertTrue(response.connection is None)
+
class TestProxyManager(SocketDummyServerTestCase):
diff --git a/urllib3.egg-info/PKG-INFO b/urllib3.egg-info/PKG-INFO
index 878f9c4..fc5d416 100644
--- a/urllib3.egg-info/PKG-INFO
+++ b/urllib3.egg-info/PKG-INFO
@@ -1,8 +1,8 @@
Metadata-Version: 1.1
Name: urllib3
-Version: 1.15.1
+Version: 1.16
Summary: HTTP library with thread-safe connection pooling, file post, and more.
-Home-page: http://urllib3.readthedocs.org/
+Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: andrey.petrov at shazow.net
License: MIT
@@ -82,7 +82,7 @@ Description: =======
Examples
========
- Go to `urllib3.readthedocs.org <http://urllib3.readthedocs.org>`_
+ Go to `urllib3.readthedocs.org <https://urllib3.readthedocs.io>`_
for more nice syntax-highlighted examples.
But, long story short::
@@ -98,7 +98,7 @@ Description: =======
The ``PoolManager`` will take care of reusing connections for you whenever
you request the same host. For more fine-grained control of your connection
pools, you should look at `ConnectionPool
- <http://urllib3.readthedocs.org/#connectionpool>`_.
+ <https://urllib3.readthedocs.io/#connectionpool>`_.
Run the tests
@@ -166,12 +166,37 @@ Description: =======
===========
If your company benefits from this library, please consider `sponsoring its
- development <http://urllib3.readthedocs.org/en/latest/#sponsorship>`_.
+ development <https://urllib3.readthedocs.io/en/latest/#sponsorship>`_.
Changes
=======
+ 1.16 (2016-06-11)
+ -----------------
+
+ * Disable IPv6 DNS when IPv6 connections are not possible. (Issue #840)
+
+ * Provide ``key_fn_by_scheme`` pool keying mechanism that can be
+ overridden. (Issue #830)
+
+ * Normalize scheme and host to lowercase for pool keys, and include
+ ``source_address``. (Issue #830)
+
+ * Cleaner exception chain in Python 3 for ``_make_request``.
+ (Issue #861)
+
+ * Fixed installing ``urllib3[socks]`` extra. (Issue #864)
+
+ * Fixed signature of ``ConnectionPool.close`` so it can actually safely be
+ called by subclasses. (Issue #873)
+
+ * Retain ``release_conn`` state across retries. (Issues #651, #866)
+
+ * Add customizable ``HTTPConnectionPool.ResponseCls``, which defaults to
+ ``HTTPResponse`` but can be replaced with a subclass. (Issue #879)
+
+
1.15.1 (2016-04-11)
-------------------
@@ -259,7 +284,7 @@ Description: =======
* Made ``HTTPHeaderDict`` usable as a ``headers`` input value
(Issues #632, #679)
- * Added `urllib3.contrib.appengine <https://urllib3.readthedocs.org/en/latest/contrib.html#google-app-engine>`_
+ * Added `urllib3.contrib.appengine <https://urllib3.readthedocs.io/en/latest/contrib.html#google-app-engine>`_
which has an ``AppEngineManager`` for using ``URLFetch`` in a
Google AppEngine environment. (Issue #664)
@@ -416,7 +441,7 @@ Description: =======
* Unverified HTTPS requests will trigger a warning on the first request. See
our new `security documentation
- <https://urllib3.readthedocs.org/en/latest/security.html>`_ for details.
+ <https://urllib3.readthedocs.io/en/latest/security.html>`_ for details.
(Issue #426)
* New retry logic and ``urllib3.util.retry.Retry`` configuration object.
diff --git a/urllib3/__init__.py b/urllib3/__init__.py
index 7366899..c353674 100644
--- a/urllib3/__init__.py
+++ b/urllib3/__init__.py
@@ -32,7 +32,7 @@ except ImportError:
__author__ = 'Andrey Petrov (andrey.petrov at shazow.net)'
__license__ = 'MIT'
-__version__ = '1.15.1'
+__version__ = '1.16'
__all__ = (
'HTTPConnectionPool',
diff --git a/urllib3/connectionpool.py b/urllib3/connectionpool.py
index 3fcfb12..ab634cb 100644
--- a/urllib3/connectionpool.py
+++ b/urllib3/connectionpool.py
@@ -90,7 +90,7 @@ class ConnectionPool(object):
# Return False to re-raise any potential exceptions
return False
- def close():
+ def close(self):
"""
Close all pooled connections and disable the pool.
"""
@@ -163,6 +163,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
scheme = 'http'
ConnectionCls = HTTPConnection
+ ResponseCls = HTTPResponse
... 1211 lines suppressed ...
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/python-modules/packages/python-urllib3.git
More information about the Python-modules-commits
mailing list