3
Z'ícJM ã @ s8 d dl mZ d dlZd dlZd dlZddlmZ ddlmZm Z m
Z
ddl m Z m
Z
mZmZmZ ddlmZ ddlmZ dd lmZ dd
lmZ dd lmZ dd lm Z d
ddgZ ej e Z d<Z!d=Z"ej#d3e"Z$d>Z%ej#d5e%Z&d6d7 Z'ej(e'e$ej(e'e$d8Z)ee d8Z*G d9d
d
eZ+G d:d de+Z,d;d Z-dS )?é )Úabsolute_importNé )ÚRecentlyUsedContainer)ÚHTTPConnectionPoolÚHTTPSConnectionPoolÚport_by_scheme)ÚLocationValueErrorÚ
MaxRetryErrorÚProxySchemeUnknownÚProxySchemeUnsupportedÚURLSchemeUnknown)Úsix)Úurljoin)ÚRequestMethods)Ú connection_requires_http_tunnel)ÚRetry)Ú parse_urlÚ PoolManagerÚ ProxyManagerÚproxy_from_urlÚkey_fileÚ cert_fileÚ cert_reqsÚca_certsÚ ssl_versionÚ ca_cert_dirÚ ssl_contextÚ key_passwordÚserver_hostnameÚ
key_schemeÚkey_hostÚkey_portÚ key_timeoutÚ key_retriesÚ
key_strictÚ key_blockÚkey_source_addressÚ key_key_fileÚkey_key_passwordÚ
key_cert_fileÚ
key_cert_reqsÚ key_ca_certsÚkey_ssl_versionÚkey_ca_cert_dirÚkey_ssl_contextÚ key_maxsizeÚ key_headersÚ
key__proxyÚkey__proxy_headersÚkey__proxy_configÚkey_socket_optionsÚkey__socks_optionsÚkey_assert_hostnameÚkey_assert_fingerprintÚkey_server_hostnameÚPoolKeyÚuse_forwarding_for_httpsÚ ProxyConfigc C sÎ |j }|d j |d< |d j |d< x4d D ],}||kr.|| dk r.t|| j ||< q.W |jd}|dk r|t||d< x&t|j D ]}|j||d| < qW x | j D ]}||kr¬d||< q¬W | f |S )
a÷
Create a pool key out of a request context dictionary.
According to RFC 3986, both the scheme and host are case-insensitive.
Therefore, this function normalizes both before constructing the pool
key for an HTTPS request. If you wish to change this behaviour, provide
alternate callables to ``key_fn_by_scheme``.
:param key_class:
The class to use when constructing the key. This should be a namedtuple
with the ``scheme`` and ``host`` keys at a minimum.
:type key_class: namedtuple
:param request_context:
A dictionary-like object that contain the context for a request.
:type request_context: dict
:return: A namedtuple that can be used as a connection pool key.
:rtype: PoolKey
ÚschemeÚhostÚheadersÚ_proxy_headersÚ_socks_optionsNÚsocket_optionsÚkey_)r> r? r@ )
ÚcopyÚlowerÚ frozensetÚitemsÚgetÚtupleÚlistÚkeysÚpopÚ_fields)Ú key_classÚrequest_contextÚcontextÚkeyÚ socket_optsÚfield© rS ú6/tmp/pip-build-gk9425m9/urllib3/urllib3/poolmanager.pyÚ_default_key_normalizerO s
rU )ÚhttpÚhttpsc @ s e Zd ZdZdZdZd ddZdd Zdd Zd!d
d Z d d
Z
d"ddZ dd Z d#ddZ
d$ddZdd Zdd Zdd Zd%d d ZdS )&r a$
Allows for arbitrary requests while transparently keeping track of
necessary connection pools for you.
:param num_pools:
Number of connection pools to cache before discarding the least
recently used pool.
:param headers:
Headers to include with all requests, unless other headers are given
explicitly.
:param \**connection_pool_kw:
Additional parameters are used to create fresh
:class:`urllib3.connectionpool.ConnectionPool` instances.
Example::
>>> manager = PoolManager(num_pools=2)
>>> r = manager.request('GET', 'http://google.com/')
>>> r = manager.request('GET', 'http://google.com/mail')
>>> r = manager.request('GET', 'http://yahoo.com/')
>>> len(manager.pools)
2
Né
c K s8 t j| | || _t|dd d| _t| _tj | _d S )Nc S s | j S )N)Úclose)ÚprS rS rT Ú<lambda>® s z&PoolManager.__init__.<locals>.<lambda>)Ú dispose_func)r Ú__init__Úconnection_pool_kwr ÚpoolsÚpool_classes_by_schemeÚkey_fn_by_schemerC )ÚselfÚ num_poolsr> r^ rS rS rT r] « s
zPoolManager.__init__c C s | S )NrS )rb rS rS rT Ú __enter__µ s zPoolManager.__enter__c C s | j dS )NF)Úclear)rb Úexc_typeÚexc_valÚexc_tbrS rS rT Ú__exit__¸ s zPoolManager.__exit__c C sf | j | }|dkr | jj }xdD ]}|j|d q"W |dkrXxtD ]}|j|d qDW |||f|S )aÎ
Create a new :class:`urllib3.connectionpool.ConnectionPool` based on host, port, scheme, and
any additional pool keyword arguments.
If ``request_context`` is provided, it is provided as keyword arguments
to the pool class used. This method is used to actually create the
connection pools handed out by :meth:`connection_from_url` and
companion methods. It is intended to be overridden for customization.
Nr< r= ÚportrV )r< r= rj )r` r^ rC rK Ú SSL_KEYWORDS)rb r< r= rj rN Úpool_clsrP ÚkwrS rS rT Ú _new_pool½ s
zPoolManager._new_poolc C s | j j dS )z´
Empty our store of pools and direct them all to close.
This will not affect in-flight connections, but they will not be
re-used after completion.
N)r_ re )rb rS rS rT re Ø s zPoolManager.clearrV c C sT |s t d| j|}|p d|d< |s:tj|d j d}||d< ||d< | j|S )a¤
Get a :class:`urllib3.connectionpool.ConnectionPool` based on the host, port, and scheme.
If ``port`` isn't given, it will be derived from the ``scheme`` using
``urllib3.connectionpool.port_by_scheme``. If ``pool_kwargs`` is
provided, it is merged with the instance's ``connection_pool_kw``
variable and used to create the new connection pool, if one is
needed.
zNo host specified.rV r< éP rj r= )r Ú_merge_pool_kwargsr rG rD Úconnection_from_context)rb r= rj r< Ú pool_kwargsrN rS rS rT Úconnection_from_hostá s
z PoolManager.connection_from_hostc C s: |d j }| jj|}|s$t|||}| j||dS )zù
Get a :class:`urllib3.connectionpool.ConnectionPool` based on the request context.
``request_context`` must at least contain the ``scheme`` key and its
value must be a key in ``key_fn_by_scheme`` instance variable.
r< )rN )rD ra rG r Úconnection_from_pool_key)rb rN r< Úpool_key_constructorÚpool_keyrS rS rT rq ø s z#PoolManager.connection_from_contextc
C s` | j jN | j j|}|r |S |d }|d }|d }| j||||d}|| j |< W dQ R X |S )a
Get a :class:`urllib3.connectionpool.ConnectionPool` based on the provided pool key.
``pool_key`` should be a namedtuple that only contains immutable
objects. At a minimum it must have the ``scheme``, ``host``, and
``port`` fields.
r< r= rj )rN N)r_ ÚlockrG rn )rb rv rN Úpoolr< r= rj rS rS rT rt s
z$PoolManager.connection_from_pool_keyc C s t |}| j|j|j|j|dS )aÖ
Similar to :func:`urllib3.connectionpool.connection_from_url`.
If ``pool_kwargs`` is not provided and a new pool needs to be
constructed, ``self.connection_pool_kw`` is used to initialize
the :class:`urllib3.connectionpool.ConnectionPool`. If ``pool_kwargs``
is provided, it is used instead. Note that if a new pool does not
need to be created for the request, the provided ``pool_kwargs`` are
not used.
)rj r< rr )r rs r= rj r< )rb Úurlrr ÚurS rS rT Úconnection_from_url s z PoolManager.connection_from_urlc C sZ | j j }|rVxF|j D ]:\}}|dkrJy
||= W qR tk
rF Y qRX q|||< qW |S )a
Merge a dictionary of override values for self.connection_pool_kw.
This does not modify self.connection_pool_kw and returns a new dict.
Any keys in the override dictionary with a value of ``None`` are
removed from the merged dictionary.
N)r^ rC rF ÚKeyError)rb ÚoverrideÚbase_pool_kwargsrP ÚvaluerS rS rT rp / s
z PoolManager._merge_pool_kwargsc C s" | j dkrdS t| j | j|j S )z³
Indicates if the proxy requires the complete destination URL in the
request. Normally this is only needed when not using an HTTP CONNECT
tunnel.
NF)Úproxyr Ú proxy_configr< )rb Ú
parsed_urlrS rS rT Ú!_proxy_requires_url_absolute_formC s
z-PoolManager._proxy_requires_url_absolute_formc C sB | j dks|dkrdS | j jdkr&dS tjr>| jj r>tddS )z
Validates that were not attempting to do TLS in TLS connections on
Python2 or with unsupported SSL implementations.
NrW zfContacting HTTPS destinations through HTTPS proxies 'via CONNECT tunnels' is not supported in Python 2)r r< r
ÚPY2r r: r )rb Z
url_schemerS rS rT Ú$_validate_proxy_scheme_url_selectionP s z0PoolManager._validate_proxy_scheme_url_selectionTc K s t |}| j|j | j|j|j|jd}d|d< d|d< d|krP| jj |d< | j|rl|j ||f|}n|j ||j
f|}|o|j }|s|S t ||}|j
dkrªd}|jd} t| tsÌtj| |d } | joÜ|j| r ttj|d }
x*|
D ]"} | j | jkrø|d j| d
qøW y| j||||d } W n( tk
r\ | jrX|j |S X | |d< ||d< t j d || |j | j ||f|S )
aN
Same as :meth:`urllib3.HTTPConnectionPool.urlopen`
with custom cross-host redirect logic and only sends the request-uri
portion of the ``url``.
The given ``url`` parameter must be absolute, such that an appropriate
:class:`urllib3.connectionpool.ConnectionPool` can be chosen for it.
)rj r< FÚassert_same_hostÚredirectr> i/ ÚGETÚretries)r N)ÚresponseÚ_poolzRedirecting %s -> %s) r r
r< rs r= rj r> rC r ÚurlopenÚ request_uriÚget_redirect_locationr ÚstatusrG Ú
isinstancer Úfrom_intÚremove_headers_on_redirectÚ is_same_hostrI r
ÚiterkeysrD rK Ú incrementr Úraise_on_redirectZ
drain_connÚlogÚinfo) rb Úmethodry r rm rz Úconnr Úredirect_locationr r> ÚheaderrS rS rT r a sJ
zPoolManager.urlopen)rX N)N)NrV N)N)N)T)Ú__name__Ú
__module__Ú __qualname__Ú__doc__r r r] rd ri rn re rs rq rt r{ rp r r
r rS rS rS rT r s
c sH e Zd ZdZd fdd Zd fdd Zdd
d Zd fd
d Z ZS )r a>
Behaves just like :class:`PoolManager`, but sends all requests through
the defined proxy, using the CONNECT method for HTTPS URLs.
:param proxy_url:
The URL of the proxy to be used.
:param proxy_headers:
A dictionary containing headers that will be sent to the proxy. In case
of HTTP they are being sent with each request, while in the
HTTPS/CONNECT case they are sent only once. Could be used for proxy
authentication.
:param proxy_ssl_context:
The proxy SSL context is used to establish the TLS connection to the
proxy when using HTTPS proxies.
:param use_forwarding_for_https:
(Defaults to False) If set to True will forward requests to the HTTPS
proxy to be made on behalf of the client instead of creating a TLS
tunnel via the CONNECT method. **Enabling this flag means that request
and response headers and content will be visible from the HTTPS proxy**
whereas tunneling keeps request and response headers and content
private. IP address, target hostname, SNI, and port are always visible
to an HTTPS proxy even when this flag is disabled.
Example:
>>> proxy = urllib3.ProxyManager('http://localhost:3128/')
>>> r1 = proxy.request('GET', 'http://google.com/')
>>> r2 = proxy.request('GET', 'http://httpbin.org/')
>>> len(proxy.pools)
1
>>> r3 = proxy.request('GET', 'https://httpbin.org/')
>>> r4 = proxy.request('GET', 'https://twitter.com/')
>>> len(proxy.pools)
3
rX NFc
s´ t |tr d|j|j|jf }t|}|jd kr:t|j|jsZtj|jd} |j | d}|| _
|pfi | _ || _ t
||| _| j
|d< | j |d< | j|d< tt| j||f| d S )
Nz
%s://%s:%irV rW ro )rj Ú_proxyr? Z
_proxy_config)rV rW )r r r< r= rj r r
r rG Ú_replacer Ú
proxy_headersÚproxy_ssl_contextr; r Úsuperr r] )
rb Ú proxy_urlrc r> r£ r¤ r: r^ r rj )Ú __class__rS rT r] Í s&
zProxyManager.__init__rV c sD |dkr t t| j||||dS t t| j| jj| jj| jj|dS )NrW )rr )r¥ r rs r r= rj r< )rb r= rj r< rr )r§ rS rT rs ò s
z!ProxyManager.connection_from_hostc C s0 ddi}t |j}|r ||d< |r,|j| |S )z
Sets headers needed by proxies: specifically, the Accept and Host
headers. Only sets headers not provided by the user.
ÚAcceptz*/*ÚHost)r ÚnetlocÚupdate)rb ry r> Úheaders_rª rS rS rT Ú_set_proxy_headersü s
z ProxyManager._set_proxy_headersTc sV t |}t| j| j|js8|jd| j}| j|||d< tt | j
||fd|i|S )z@Same as HTTP(S)ConnectionPool.urlopen, ``url`` must be absolute.r> r ) r r r r r< rG r> r r¥ r r )rb r ry r rm rz r> )r§ rS rT r s
zProxyManager.urlopen)rX NNNF)NrV N)N)T) r r r r r] rs r r Ú
__classcell__rS rS )r§ rT r ¥ s &
c K s t f d| i|S )Nr¦ )r )ry rm rS rS rT r s ) r r r r r r r r r )r r r! r" r# r$ r% r& r' r( r) r* r+ r, r- r. r/ r0 r1 r2 r3 r4 r5 r6 r7 r8 )r r: ).Ú
__future__r Ú collectionsÚ functoolsÚloggingÚ _collectionsr Úconnectionpoolr r r Ú
exceptionsr r r
r r Úpackagesr
Z packages.six.moves.urllib.parser Úrequestr Z
util.proxyr Z
util.retryr Zutil.urlr Ú__all__Ú getLoggerr r rk Ú _key_fieldsÚ
namedtupler9 Z_proxy_config_fieldsr; rU Úpartialra r` r r r rS rS rS rT Ú<module> sz
6
s