Issue2021

classification
Title: https with jython2.7 + trusting all certificates not working
Type: behaviour Severity: normal
Components: Core Versions: Jython 2.7
process
Status: closed Resolution: invalid
Dependencies: Superseder:
Assigned To: amak Nosy List: amak, erikive, fwierzbicki, kkurzweil, zyasoft
Priority: normal Keywords:

Created on 2013-02-28.11:45:03 by erikive, last changed 2014-05-10.05:21:04 by zyasoft.

Messages
msg7867 (view) Author: Erik Iveroth (erikive) Date: 2013-02-28.11:45:03
Hi

After changing to jython2.7beta1 from Jython2.5.3 I am no longer able to read content of webpages using SSL, http and using java "trusting all certificates". The response from the https-page is always an empty string, resulting in httplib.BadStatusLine exception from httplib.py in Jython.

I have described the problem on stack overflow: http://stackoverflow.com/questions/15133691/https-with-jython2-7-trusting-all-certificates-does-not-work-result-httplib
msg7870 (view) Author: Alan Kennedy (amak) Date: 2013-02-28.21:54:44
I think this is a duplicate of #2016

ssl sockets have broken recv() and makefile()
http://bugs.jython.org/issue2016

The recv() part of that issue is already fixed.

httplib switched to using makefile() for HTTPSConnections in python 2.7, which our SSL sockets do not yet support: I am working on a fix.

It is not related to the "trust all certificates" thing, because if certificates were an issue, you would have gotten a certificate verification exception.

I think I'm going to close this issue as a duplicate of #2016.

Please update your stackoverflow post with this information: I'm busy working on the fix :-)
msg7873 (view) Author: Alan Kennedy (amak) Date: 2013-02-28.23:09:35
I've checked i a temporary fix for the lack of makefile here

http://hg.python.org/jython/rev/baf84d8e91d0

Which should make httplib.HTTPSConnection start working.

This is just a temporary fix, so I have not added tests yet.

I have a more permanent fix in the works, which will be extensively tested.

Please retest with the latest code.

I will leave this bug open for a few days for fix verification.
msg7879 (view) Author: Erik Iveroth (erikive) Date: 2013-03-01.09:13:33
Hi Alan, thank you for your reply and fix. I have tested my code with your latest commit and the HTTPSConnection does indeed work!

But if I change to use urllib2 with HTTPBasicAuthHandler to request my https-url, I still receive socket problems. 

My urrlib2 test works fine with jython2.5.3, does not work at all with 2.7b1 (i.e. same result as prior of your commit), and with your latest commit (baf84d8e91d0) I get the following problem:

[...]
File "jytest.py", line 59, in urllib2test
    data = urllib2.urlopen(URL).read()
[...]
File "/Users/erikiveroth/Workspace/sandbox/jython/dist/Lib/socket.py", line 1669, in read
    data = self._sock.recv(recv_size)
File "/Users/erikiveroth/Workspace/sandbox/jython/dist/Lib/socket.py", line 174, in handle_exception
    raise _map_exception(jlx)
socket.error: [Errno 32] Socket closed

Full stack-trace and testcode: http://247bc66123446939.paste.se/
msg7884 (view) Author: Alan Kennedy (amak) Date: 2013-03-01.20:23:00
urllib2.urlopen("https://blah/").read() is working for me

Jython 2.7b1+ (default:e80a189574d0+, Feb 28 2013, 22:54:17)
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_41
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> f = urllib2.urlopen('https://httpbin.org/ip')
>>> f.read()
'{\n  "origin": "10.120.118.223"\n}'

I note that you re using basic authorization. On a private server, so I can get a look at that the server is returning.

I suspect that the server is returning a www-authenticate header without quoting the realm name.

Please can you paste the output of this command: I'm specifically interested in the response headers.

curl -v -u admin:test https://192.168.1.101:8888/rest/system
msg7898 (view) Author: Erik Iveroth (erikive) Date: 2013-03-04.09:53:46
Hi Alan, sorry for the delayed response.

It looks like my problem is with the Java SSLContext, which I use to accept all certificates. 
I have removed the basic authorization and simplified the code, still receiving same problem: http://49c386d0057a547c.paste.se/

And it is probably due to the response from the server, since my tests works with url 'https://httpbin.org/ip'.
But why does it work with earlier version of jython?


Response header looks ok to me (some parts masked): http://d795a35132c048bb.paste.se/
msg7901 (view) Author: Alan Kennedy (amak) Date: 2013-03-04.12:40:16
OK, I see you're Brandon Pedersens recipe for trusting all certificates.

http://tech.pedersen-live.com/2010/10/trusting-all-certificates-in-jython/

But you've commented out the key line from his context manager declaration

SSLContext.setDefault(TRUST_ALL_CONTEXT)

So the SSLContexts you're creating do not inherit your trust manager.
msg7902 (view) Author: Erik Iveroth (erikive) Date: 2013-03-04.12:46:51
Ah, no. I accidentally pasted the code I used on my first run. I executed my code without SSLContext.setDefault(TRUST_ALL_CONTEXT) (Lines 47-70) and with SSLContext.setDefault(TRUST_ALL_CONTEXT) (Lines 73-104) to see the difference.

Sorry for the confusion.
msg7904 (view) Author: Alan Kennedy (amak) Date: 2013-03-05.23:37:51
> "Response header looks ok to me (some parts masked): http://d795a35132c048bb.paste.se/"

OK, line 29 of that output looks interesting.

> 29: < HTTP/1.1 302 Found

Is your urlopener setup correctly to handle redirects?

(It's quite frustrating having to debug something that's behind a firewall that I can't access and cannot run code against. Is there anyway you can get this running somewhere I can access it and really see what's going on?)
msg8358 (view) Author: Jim Baker (zyasoft) Date: 2014-05-10.05:21:04
I have experience that this sort of trust all certs works well on Jython 2.7. Here's another example of how to set this up:
https://github.com/rackerlabs/romper/blob/master/romper/trust.py
History
Date User Action Args
2014-05-10 05:21:04zyasoftsetstatus: open -> closed
resolution: invalid
messages: + msg8358
nosy: + zyasoft
2013-03-25 23:02:11fwierzbickisetpriority: normal
2013-03-05 23:37:51amaksetmessages: + msg7904
2013-03-05 17:53:25kkurzweilsetnosy: + kkurzweil
2013-03-04 12:46:51erikivesetmessages: + msg7902
2013-03-04 12:40:17amaksetmessages: + msg7901
2013-03-04 09:53:47erikivesetmessages: + msg7898
2013-03-01 20:23:00amaksetmessages: + msg7884
2013-03-01 09:13:33erikivesetmessages: + msg7879
2013-02-28 23:09:35amaksetmessages: + msg7873
2013-02-28 21:54:45amaksetassignee: amak
messages: + msg7870
2013-02-28 15:47:41fwierzbickisetnosy: + fwierzbicki, amak
2013-02-28 11:45:03erikivecreate