python requests on Google App Engine not working for HTTPS

Question

I'm using python-request on Google App Engine and it's not working as expected for HTTPS. Let's see an example:

import requests
requests.get('https://www.digitalocean.com')

That line works perfectly if I execute it in a terminal. Response is 200 OK (without redirects).

However, if I execute it on GAE a TooManyRedirects error is raised. Trying to figure out what's the problem I execute with allow_redirects=False and I can see that the response is a redirect (301) which points to the same url!!! ('location' header value is 'https://www.digitalocean.com'). This obviously (when allow_redirect=True) happens over and over again until the TooManyRedirects error is raised.

So it seems that python-requests is not working on GAE for HTTPS (I've tested with several URL). However HTTP works perfectly.

Any idea about what's happening?

Thanks in advance.

If it helps, I recently upgraded to urllib3==1.71 and requests==2.2.1 — rattray
– rattray, Commented Feb 6, 2014 at 15:57
Filed an issue with AppEngine here: code.google.com/p/googleappengine/issues/detail?id=10562 — rattray
– rattray, Commented Feb 6, 2014 at 16:20
I did the same in python-requests repo: github.com/kennethreitz/requests/issues/1905 — Curro
– Curro, Commented Feb 6, 2014 at 16:22
Possible duplicate of Using the Requests python library in Google App Engine — Jon Wayne Parrott
– Jon Wayne Parrott, Commented Jun 8, 2016 at 19:54

rattray · Accepted Answer · 2014-02-07 06:22:28Z

17

Downgrading to requests==2.1.0 worked for me.

Having an up-to-date urllib3 is important for resolving an unrelated bug (import pwd, as I recall).

Hopefully App Engine fixes this soon, as requests won't.

EDIT:

I think you can also patch this in the latest requests by commenting lines 161-175 in sessions.py. Untested.

edited Feb 7, 2014 at 6:22

answered Feb 6, 2014 at 18:17

rattray

6,1771 gold badge40 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Connor Over a year ago

How do you downgrade to an earlier version of requests?

Curro Over a year ago

By now this seems to be the only way to make it works.... so it's a pretty valid response while we wait for the fix in GAE. Thanks rattray! good job!

jacob Over a year ago

Hi @rattay. The problem only seems to be to connect to Google IP addresses (e.g., Google oAuth), as they are blocked from the sockets API, as specified here: cloud.google.com/appengine/docs/python/sockets For that case, your patch does not seem to work anymore. In the latest version of requests (2.8.1), commenting our sessions.SessionRedirectMixin.rebuild_auth does not resolve the error: Failed to establish a new connection: [Errno 13] Permission denied

rattray Over a year ago

@jacob did you try downgrading to requests==2.1.0 ?

Josh Over a year ago

Here to confirm that 2.1.0 nor 2.3.0 work on the latest gcloud 104 -- app-engine-python 1.9.35

|

Alex · Accepted Answer · 2017-07-11 19:22:14Z

5

There is now a better solution than changing your requests version. As suggested in the official docs you can monkey patch requests to play nicely with Google App Engine.

First install requests-toolbelt:

pip install -t lib requests-toolbelt

Then in your main.py file (or equivalent):

import requests_toolbelt.adapters.appengine

requests_toolbelt.adapters.appengine.monkeypatch()

answered Jul 11, 2017 at 19:22

Alex

19.2k9 gold badges65 silver badges82 bronze badges

Collectives™ on Stack Overflow

python requests on Google App Engine not working for HTTPS

2 Answers 2

6 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Linked

Related