3
T'ícZp ã @ sÄ d Z ddlZddlZddlZddlZddlZddlmZmZ ddlm Z ddl
m Z ddl m
Z
ddlmZmZ ddlmZ dd lmZmZmZmZmZmZmZmZm Z m Z m Z dd
l m Z m!Z!m"Z" dd l#m$Z$ dd l%m&Z& dd
l'm(Z( ddl)m*Z*m+Z+m,Z, ddl-m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4 ddl5m6Z6 ddl7m8Z8 ddl9m:Z: ddl;m<Z<m=Z=m'Z' ddl>m?Z?m@Z@mAZAmBZBmCZC ddlDmEZE e=jFeGZHejIdZJG dd deZKG d d d eZLG d d d eZMG d d! d!eZNe eLe eOePePeQf f ZRd"d#iZSdZTd$ZUd%ZVe&eQd&d'd(ZWG d)d* d*e ZXe'j'j(ePeYd+d,d-ZZG d.d/ d/e0Z[G d0d1 d1Z\G d2d3 d3eZ]G d4d5 d5e:Z^e.ePeeP d6d7d8Z_e.e2dd9d:d;Z`e.eePef d<d=d>ZadS )?z$The CheckExternalLinksBuilder class.é N)ÚdatetimeÚtimezone)Úparsedate_to_datetime)Ú
HTMLParser)Úpath)Ú
PriorityQueueÚQueue)ÚThread) ÚAnyÚDictÚ GeneratorÚListÚ
NamedTupleÚOptionalÚPatternÚSetÚTupleÚUnionÚcast)ÚunquoteÚurlparseÚ
urlunparse)Únodes)ÚElement)ÚResponse)ÚConnectionErrorÚ HTTPErrorÚTooManyRedirects)ÚSphinx)Ú DummyBuilder)ÚConfig)ÚRemovedInSphinx50Warning)ÚBuildEnvironment)Ú__)ÚSphinxPostTransform)Ú
encode_uriÚloggingÚrequests)ÚdarkgrayÚ darkgreenÚpurpleÚredÚ turquoise)Ú
get_node_linez ([a-z]+:)?//c @ s e Zd ZU eeee dS )Ú HyperlinkN) Ú__name__Ú
__module__Ú __qualname__ÚstrÚuriÚdocnamer ÚintÚlineno© r7 r7 ú;/tmp/pip-build-gk9425m9/sphinx/sphinx/builders/linkcheck.pyr. ' s
r. c @ s e Zd ZU eee dS )Ú CheckRequestN)r/ r0 r1 ÚfloatÚ
next_checkr r. Ú hyperlinkr7 r7 r7 r8 r9 - s
r9 c @ s& e Zd ZU eeeee e
dS )Ú CheckResultN) r/ r0 r1 r2 r3 r4 r5 r6 ÚstatusÚmessageÚcoder7 r7 r7 r8 r= 2 s
r= c @ s e Zd ZU eedS )Ú RateLimitN)r/ r0 r1 r: Údelayr; r7 r7 r7 r8 rA ; s
rA ÚAcceptz/text/html,application/xhtml+xml;q=0.9,*/*;q=0.8é g N@)ÚnodeÚreturnc C s t jdtdd t| pdS )z
PriorityQueue items must be comparable. The line number is part of the
tuple used by the PriorityQueue, keep an homogeneous type for comparison.
z node_line_or_0() is deprecated.é )Ú
stacklevelr )ÚwarningsÚwarnr! r- )rE r7 r7 r8 Únode_line_or_0K s
rK c s: e Zd ZdZedd fdd ZeeddddZ ZS ) ÚAnchorCheckParserz9Specialized HTML parser that looks for a specific anchor.N)Ú
search_anchorrF c s t j || _d| _d S )NF)ÚsuperÚ__init__rM Úfound)ÚselfrM )Ú __class__r7 r8 rO X s
zAnchorCheckParser.__init__)ÚtagÚattrsrF c C s0 x*|D ]"\}}|dkr|| j krd| _P qW d S )NÚidÚnameT)rU rV )rM rP )rQ rS rT ÚkeyÚvaluer7 r7 r8 Úhandle_starttag^ s z!AnchorCheckParser.handle_starttag) r/ r0 r1 Ú__doc__r2 rO r
rY Ú
__classcell__r7 r7 )rR r8 rL U s rL )ÚresponseÚanchorrF c C sR t |}x:| jdddD ](}t|tr.|j }|j| |jrP qW |j |jS )z
Reads HTML data from a response object `response` searching for `anchor`.
Returns True if anchor was found, False otherwise.
i T)Ú
chunk_sizeÚdecode_unicode)rL Ú iter_contentÚ
isinstanceÚbytesÚdecodeÚfeedrP Úclose)r\ r] ÚparserÚchunkr7 r7 r8 Ú check_anchore s
rh c @ sb e Zd ZdZdZedZddddZee e
ddd Z ee e e
e
f dd
d Zee e
dd d
Zeee dddZeeeef dddZeeee eef f dddZddddZeee dddZee dddZ ee e ddd Z ee dd d Z e!dd d d!Z"eeeeedd"d#d$Z#e$dd%d&d'Z%ddd(d)Z&dS )*ÚCheckExternalLinksBuilderz+
Checks for broken external links.
Ú linkcheckzCLook for any errors in the above output or in %(outdir)s/output.txtN)rF c C s8 i | _ t | _i | _i | _tjd t | _t | _
d S )Ng @) Ú
hyperlinksÚsetÚ_goodÚ_brokenÚ _redirectedÚsocketÚsetdefaulttimeoutr Ú_wqueuer Ú_rqueue)rQ r7 r7 r8 Úinit s
z CheckExternalLinksBuilder.initc C s. t jd| jjdf tdd dd | jjD S )Nz%s.%s is deprecated.Úanchors_ignorerG )rH c S s g | ]}t j|qS r7 )ÚreÚcompile)Ú.0Úxr7 r7 r8 ú
<listcomp> s z<CheckExternalLinksBuilder.anchors_ignore.<locals>.<listcomp>)rI rJ rR r/ r! ÚconfigÚlinkcheck_anchors_ignore)rQ r7 r7 r8 ru s
z(CheckExternalLinksBuilder.anchors_ignorec C s. t jd| jjdf tdd dd | jjD S )Nz%s.%s is deprecated.ÚauthrG )rH c S s g | ]\}}t j||fqS r7 )rv rw )rx ÚpatternÚ auth_infor7 r7 r8 rz s z2CheckExternalLinksBuilder.auth.<locals>.<listcomp>)rI rJ rR r/ r! r{ Úlinkcheck_auth)rQ r7 r7 r8 r} s z CheckExternalLinksBuilder.authc C s. t jd| jjdf tdd dd | jjD S )Nz%s.%s is deprecated.Ú to_ignorerG )rH c S s g | ]}t j|qS r7 )rv rw )rx ry r7 r7 r8 rz ¥ s z7CheckExternalLinksBuilder.to_ignore.<locals>.<listcomp>)rI rJ rR r/ r! r{ Úlinkcheck_ignore)rQ r7 r7 r8 r s
z#CheckExternalLinksBuilder.to_ignorec C s" t jd| jjdf tdd | jS )Nz%s.%s is deprecated.ÚgoodrG )rH )rI rJ rR r/ r! rm )rQ r7 r7 r8 r § s
z CheckExternalLinksBuilder.goodc C s" t jd| jjdf tdd | jS )Nz%s.%s is deprecated.ÚbrokenrG )rH )rI rJ rR r/ r! rn )rQ r7 r7 r8 r ° s
z CheckExternalLinksBuilder.brokenc C s" t jd| jjdf tdd | jS )Nz%s.%s is deprecated.Ú
redirectedrG )rH )rI rJ rR r/ r! ro )rQ r7 r7 r8 r
¹ s
z$CheckExternalLinksBuilder.redirectedc C s t jd| jjdf tdd d S )Nz%s.%s is deprecated.Ú check_threadrG )rH )rI rJ rR r/ r! )rQ r7 r7 r8 r  s z&CheckExternalLinksBuilder.check_thread)r\ rF c C s: t jd| jjdf tdd t| j| jd d i }|j|S )Nz%s.%s is deprecated.Ú
limit_raterG )rH ) rI rJ rR r/ r! Ú HyperlinkAvailabilityCheckWorkerÚenvr{ r )rQ r\ Zworkerr7 r7 r8 r Ê s
z$CheckExternalLinksBuilder.limit_ratec C s" t jd| jjdf tdd | jS )Nz%s.%s is deprecated.ÚrqueuerG )rH )rI rJ rR r/ r! rs )rQ r\ r7 r7 r8 r Ô s
z CheckExternalLinksBuilder.rqueuec C s t jd| jjdf tdd g S )Nz%s.%s is deprecated.ÚworkersrG )rH )rI rJ rR r/ r! )rQ r\ r7 r7 r8 r Ü s
z!CheckExternalLinksBuilder.workersc C s" t jd| jjdf tdd | jS )Nz%s.%s is deprecated.ÚwqueuerG )rH )rI rJ rR r/ r! rr )rQ r\ r7 r7 r8 r ä s
z CheckExternalLinksBuilder.wqueue)ÚresultrF c C s¸ | j j|jd }t||j|j|j|j|jd}| j | |jdkrFd S |jdkr^|jdkr^d S |jrzt
j d|j|jdd |jdkrÂ|jrªt
j t d |j d
|j nt
j t d |j nò|jd krüt
j t d |j | j
d |j||j|j n¸|jdkr&t
j td
|j |j n|jdkr²| jjsF| jjrjt
jtd|j|j|j|jfd n"t
j td|j td|j | j
d|j||j|jd
|j n|jdkr¦y2dtfdtfdtfdtfdtfd|j \}}W n tk
r dt }}Y nX ||d< | jjrTt
jd|j d | d |j |j|jfd n*t
j |d|j |d| d |j | j
d | |j||j|jd |j ntd |j d S ) N)Úfilenamer6 r> r@ r3 ÚinfoÚ uncheckedÚworkingÚoldz(%16s: line %4d) T)ZnonlÚignoredz
-ignored- z: Úlocalz
-local- z
ok r zbroken link: %s (%s))Úlocationz
broken z - r
Z permanentlyz
with Foundzwith See OtherZ temporarily)i- i. i/ i3 i4 zwith unknown codeÚtextz
redirect z to z redirected zUnknown status %s.)r Údoc2pathr4 Údictr6 r> r@ r3 r? Úwrite_linkstatÚloggerr r( Ú write_entryr) ÚappÚquietZwarningiserrorÚwarningr# r+ r* r, ÚKeyErrorr{ Úlinkcheck_allowed_redirectsÚ
ValueError)rQ r r Zlinkstatr Úcolorr7 r7 r8 Úprocess_resultì s\
"
z(CheckExternalLinksBuilder.process_result)Úwhatr4 r Úliner3 rF c C s | j jd||||f d S )Nz%s:%s: [%s] %s
)Ú txt_outfileÚwrite)rQ r¤ r4 r r¥ r3 r7 r7 r8 r # s z%CheckExternalLinksBuilder.write_entry)ÚdatarF c C s" | j jtj| | j jd d S )NÚ
)Ú json_outfiler§ ÚjsonÚdumps)rQ r¨ r7 r7 r8 r ' s z(CheckExternalLinksBuilder.write_linkstatc C s t | j| j| }tjd ttj| jddJ| _ ttj| jdd(| _
x |j | j D ]}| j
| qXW W d Q R X W d Q R X | jrd| j_d S )NÚ z
output.txtÚwz output.jsonrD )Ú HyperlinkAvailabilityCheckerr r{ r r Úopenr ÚjoinZoutdirr¦ rª Úcheckrk r£ rn r Z
statuscode)rQ Úcheckerr r7 r7 r8 Úfinish+ s
"z CheckExternalLinksBuilder.finish)'r/ r0 r1 rZ rV r# Úepilogrt Úpropertyr
r ru r r
r} r r r2 r r r r5 r
r r r r: r r r r r r r= r£ r r r r´ r7 r7 r7 r8 ri w s4
7ri c @ sp e Zd ZdeeeddddZddddZdddd Ze e
e f e e
ddf d
d d Ze
ed
ddZdS )r¯ N)r r{ ÚbuilderrF c C s\ || _ || _|| _i | _g | _dd | jjD | _|rH|j| _|j | _
nt | _t | _
d S )Nc S s g | ]}t j|qS r7 )rv rw )rx ry r7 r7 r8 rz D s z9HyperlinkAvailabilityChecker.__init__.<locals>.<listcomp>)
r· r{ r Ú rate_limitsr r r rs r rr r r r )rQ r r{ r· r7 r7 r8 rO 9 s
z%HyperlinkAvailabilityChecker.__init__)rF c C sL xFt | jjD ]6}t| j| j| j| j| j| j}|j | j
j | qW d S )N) Úranger{ Úlinkcheck_workersr r r r r¸ r· Ústartr Úappend)rQ Z_iÚthreadr7 r7 r8 Úinvoke_threadsM s
z+HyperlinkAvailabilityChecker.invoke_threadsc C s2 | j j x"| jD ]}| j jttd d qW d S )NF)r r± r Úputr9 ÚCHECK_IMMEDIATELY)rQ Z_workerr7 r7 r8 Úshutdown_threadsU s
z-HyperlinkAvailabilityChecker.shutdown_threads)rk rF c c s | j d}xT|j D ]H}| j|jrBt|j|j|jdddV q| jjt t
|d |d7 }qW d}x ||k r| j j V |d7 }qhW | j
d S )Nr r r FrD )r¾ ÚvaluesÚis_ignored_urir3 r= r4 r6 r r¿ r9 rÀ r ÚgetrÁ )rQ rk Z total_linksr< Údoner7 r7 r8 r² Z s
z"HyperlinkAvailabilityChecker.check)r3 rF c s t fdd| jD S )Nc 3 s | ]}|j V qd S )N)Úmatch)rx Úpat)r3 r7 r8 ú <genexpr>n s z>HyperlinkAvailabilityChecker.is_ignored_uri.<locals>.<genexpr>)Úanyr )rQ r3 r7 )r3 r8 rà m s z+HyperlinkAvailabilityChecker.is_ignored_uri)N)r/ r0 r1 r" r ri rO r¾ rÁ r r2 r. r r= r² Úboolrà r7 r7 r7 r8 r¯ 8 s
"r¯ c s^ e Zd ZdZd eeeeeee f e
dd fdd
Z ddddZ e
ee d d
d Z ZS )
r z;A worker class for checking the availability of hyperlinks.N)r r{ r r r¸ r· rF c s || _ || _|| _|| _|| _dd | j jD | _dd | j jD | _dd | j j D | _
|rx|j | _ |j | _ |j
| _
nt | _ i | _ i | _
t jdd d S )Nc S s g | ]}t j|qS r7 )rv rw )rx ry r7 r7 r8 rz s z=HyperlinkAvailabilityCheckWorker.__init__.<locals>.<listcomp>c S s g | ]}t j|qS r7 )rv rw )rx Údocr7 r7 r8 rz s c S s g | ]\}}t j||fqS r7 )rv rw )rx r~ r r7 r7 r8 rz s T)Údaemon)r{ r r¸ r r r| ru Úlinkcheck_exclude_documentsÚdocuments_excluder r} rm rn ro rl rN rO )rQ r r{ r r r¸ r· )rR r7 r8 rO t s&
z)HyperlinkAvailabilityCheckWorker.__init__)rF c
s® i j jrj jd< tdfdd ttttf d fdd tttdfdd tttttf d
fd d }x jj }y |\}d kr¬P \}}W n t k
rØ |\}}}Y nX d kräP t
j }yj | j
}W n tk
r Y nX |tj krNtjt jjt|d
jj q||\}}} |dkrtjtd td njjt||||| jj qW d S )NÚtimeout)rF c sh t } d| j| jf d| j| jf dg}x6|D ].}| jjkr2tt}|j jj| |S q2W i S )Nz%s://%sz%s://%s/Ú*)r ÚschemeÚnetlocr{ Úlinkcheck_request_headersr ÚDEFAULT_REQUEST_HEADERSÚupdate)ÚurlÚ
candidatesÚuÚheaders)rQ r3 r7 r8 Úget_request_headers s
zAHyperlinkAvailabilityCheckWorker.run.<locals>.get_request_headersc ' sü dkr:j dd\} }x(jD ]}|j|r d }P q W n} d }y| jd W n tk
rl t| } Y nX x"jD ]\}}|jrvP qvW d } d< yì|ròjjròt j
| fdj|d}|j t |t
|}|sðttd| ny(t j| fdj|d}|j W nj tttfk
r } zFt|trP|jjd krP t j
| fdj|d}|j W Y d d }~X nX W nÔ tk
r2 } z|jjd
kr®dS |jjd krøj|j}|d k rêjjt|d dS dt|d
fS |jjdkrdt|d
fS dt|d
fS W Y d d }~X nZ tk
r\ } zdt|d
fS d }~X n0X t | j } y j | = W n t k
r Y nX |j j!d| j!dkr¨dS |j }
|rÀ|
d| 7 }
| |
rÐdS |j"rî|j"d j} d|
| fS d|
d
fS d S )Nú#rD ÚasciirÙ T)Ústreamr{ r} zAnchor '%s' not found)Úallow_redirectsr{ r} i i r ú - unauthorizedr Fú rate-limitedr r i÷ r ú/r
)r rß r )rà r r )r r r )r r r éÿÿÿÿ)#Úsplitru rÆ ÚencodeÚ UnicodeErrorr% r} r{ Úlinkcheck_anchorsr' rÄ Úraise_for_statusrh r Ú Exceptionr# Úheadr r r ra r\ Ú status_coder r r¿ r9 r2 r rÒ r¸ r rÖ ÚrstripÚhistory) Zreq_urlr] Zrexr~ r r\ rP Úerrr; rÒ Únew_urlr@ )Úallowed_redirectrÚ r< ÚkwargsrQ r3 r7 r8 Ú check_uri¨ s
z7HyperlinkAvailabilityCheckWorker.run.<locals>.check_uri)rÖ rî rF c s6 x0 j jj D ] \}}|j| r|j|rdS qW dS )NTF)r{ r ÚitemsrÆ )rÖ rî Zfrom_urlZto_url)rQ r7 r8 rï s z>HyperlinkAvailabilityCheckWorker.run.<locals>.allowed_redirect)r4 rF c sr x4j D ]*}|j| r| d|j d}d|dfS qW tdksLjdrPdS jds tjrhdS tjjj | }tj
tj |rdS d j < dS nRj
kr®dS j krÈd
j dfS jk ròdj d j d fS x.tjjD ] } \}}}|d
kr P q W |d kr:j
j n.|d
krP|j < n|dkrh||fj< |||fS )Nz matched z! from linkcheck_exclude_documentsr r rÛ úmailto:útel:r r úhttp:úhttps:r r r r
rD )rÛ ró rô )r r r )rõ rö )r r r )r r r )r r r )r r r )rÎ rÆ r~ ÚlenÚ
startswithÚuri_rer Údirnamer r Úexistsr± rn rm ro r¹ r{ Úlinkcheck_retriesÚadd)r4 Z doc_matcherr ÚsrcdirÚ_r> r@ )rñ rQ r3 r7 r8 r² s>
z3HyperlinkAvailabilityCheckWorker.run.<locals>.checkFz rate-limitedz-rate limited- z | sleeping...)r{ Úlinkcheck_timeoutr r r2 r5 rÊ r rÄ r¡ r rÒ r¸ r; r ÚtimeÚsleepÚQUEUE_POLL_SECSr¿ r9 Ú task_doner r r( r r= )
rQ r² Z
check_requestr; r4 r6 rÒ r> r r@ r7 )rï rñ rÚ r< rð rQ r3 r8 Úrun sB &`".
z$HyperlinkAvailabilityCheckWorker.run)r\ rF c
C s d }|j jd}|ry t|}W nX tk
rx y t|}W n ttfk
rT Y n X tj|}|tjt j
j }Y nX t j | }t
|jj}|d kr| jj}y| j| }W n tk
rÊ t}Y n$X |j} d| }||krî| |k rî|}||krúd S t j | }t||| j|< |S )Nz Retry-Afterg @)rÙ rÄ r: r¡ r Ú TypeErrorr Ú timestampÚnowr ÚutcÚ
total_secondsr r rÖ rÒ r{ Ú linkcheck_rate_limit_timeoutr¸ r Ú
DEFAULT_DELAYrB rA )
rQ r\ r; Ú retry_afterrB ZuntilrÒ Z max_delayZ
rate_limitZlast_wait_timer7 r7 r8 r b s:
z+HyperlinkAvailabilityCheckWorker.limit_rate)N)r/ r0 r1 rZ r" r r r r2 rA ri rO r r r r: r r[ r7 r7 )rR r8 r q s
( Or c @ s$ e Zd ZdZdZeddddZdS )ÚHyperlinkCollectorrj i N)rð rF c
K sð t t| jj}|j}xf| jjtjD ]T}d|kr2q$|d }| jj d|}|rP|}t
|}t || j j
|}||kr$|||< q$W xn| jjtjD ]\} | d jd}|rd|kr| jj d|}|rÀ|}t
| }t || j j
|}||kr|||< qW d S )NZrefurizlinkcheck-process-urir× ú?z://)r ri r r· rk ÚdocumentÚfindallr Ú referenceZemit_firstresultr- r. r r4 ÚimagerÄ )
rQ rð r· rk Zrefnoder3 Znewurir6 Zuri_infoZimgnoder7 r7 r8 r s. zHyperlinkCollector.run)rj )r/ r0 r1 ZbuildersZdefault_priorityr
r r7 r7 r7 r8 r s r )r r3 rF c C sH t |}|jdkrD|jrD|jjd}|sDd|j }t|j|dS dS )z¾Rewrite anchor name of the hyperlink to github.com
The hyperlink anchors in github.com are dynamically generated. This rewrites
them before checking and makes them comparable.
z
github.comz
user-content-)ÚfragmentN)r Úhostnamer rø r Ú_replace)r r3 ÚparsedÚprefixedr r7 r7 r8 Úrewrite_github_anchor s r )r r{ rF c C s xt | jjj D ]v\}}z\y tj|| jjtj|< W n: tjk
rr } z tjt d|j
|j W Y dd}~X nX W d| jjj | X qW dS )zFCompile patterns in linkcheck_allowed_redirects to the regexp objects.z=Failed to compile regex in linkcheck_allowed_redirects: %r %sN)
Úlistr{ r rò rv rw Úerrorr r r# r~ ÚmsgÚpop)r r{ rÖ r~ Úexcr7 r7 r8 Ú#compile_linkcheck_allowed_redirects¼ s
"r )r rF c C sÚ | j t | jt | jdg d | jdg d | jdi d | jdg d | jdi d | jddd | jdd d tg | jd d
d | jd d d | jd
dgd | jddd | jd | jdtdd dd d dS )Nr rÍ r r rÓ rü rD r rº é ræ Tr| z^!r g Àr@zlinkcheck-process-uriz
config-initedi )ÚpriorityÚbuiltin)ÚversionZparallel_read_safeZparallel_write_safe) Z add_builderri Zadd_post_transformr Zadd_config_valuer5 Z add_eventÚconnectr )r r7 r7 r8 ÚsetupÉ s$
r% )brZ r« rv rp r rI r r Ú email.utilsr Ú html.parserr Úosr Úqueuer r Ú threadingr Útypingr
r r r
r r r r r r r Ú urllib.parser r r Zdocutilsr Zdocutils.nodesr r' r Zrequests.exceptionsr r r Zsphinx.applicationr Zsphinx.builders.dummyr Z
sphinx.configr Zsphinx.deprecationr! Zsphinx.environmentr" Z
sphinx.localer# Z!sphinx.transforms.post_transformsr$ Z sphinx.utilr% r& Zsphinx.util.consoler( r) r* r+ r, Zsphinx.util.nodesr- Ú getLoggerr/ r rw rù r. r9 r= rA r: r2 r5 ZCheckRequestTyperÔ rÀ r r rK rL rÊ rh ri r¯ r r r r r% r7 r7 r7 r8 Ú<module> sf 4
B9 $