CLI: Allow to differentiate between 404 and connection timeout #681

Activity

1. igorbrigadir commented on Dec 30, 2022

   

   igorbrigadir

   on Dec 30, 2022

   Last edited by igorbrigadir

   Contributor

   More actions

   On another look, this may require some further digging. Technically, we already should be handling this - "twarc1" and "twarc2" should both handle retries for timeouts, and not retry for 404s. But it seems like 404s aren't being caught properly in twarc1 right now. So maybe there is a bug there:

   twarc users acthatnotexist1
   

   Should output a handled error, but instead

   Traceback (most recent call last):
     File "/python/bin/twarc", line 11, in <module>
       load_entry_point('twarc', 'console_scripts', 'twarc')()
     File "/python/twarc/twarc/command.py", line 293, in main
       for thing in things:
     File "/python/twarc/twarc/client.py", line 372, in user_lookup
       for u in do_lookup():
     File "/python/twarc/twarc/client.py", line 361, in do_lookup
       raise e
     File "/python/twarc/twarc/client.py", line 357, in do_lookup
       resp = self.get(url, params=params, allow_404=True)
     File "/python/twarc/twarc/decorators.py", line 88, in new_f
       resp.raise_for_status()
     File "/python/lib/python3.7/site-packages/requests/models.py", line 1021, in raise_for_status
       raise HTTPError(http_error_msg, response=self)
   requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.twitter.com/1.1/users/lookup.json?screen_name=acthatnotexist1&tweet_mode=extended
   

   A stack trace. I haven't fully replicated timeouts yet.

   As an alternative - does twarc2 work better here? When users are not found, it returns an error message you can parse instead of 404 errors:

   twarc2 user name accountthatdoesnotexist111
   

   [
     {
       "errors": [
         {
           "value": "acthatnotexist1",
           "detail": "Could not find user with usernames: [acthatnotexist1].",
           "title": "Not Found Error",
           "resource_type": "user",
           "parameter": "usernames",
           "resource_id": "acthatnotexist1",
           "type": "https://api.twitter.com/2/problems/resource-not-found"
         }
       ],
       "__twarc": {...
       }
     }
   ]
   
   
2. antibot4navalny commented on Dec 30, 2022

   

   antibot4navalny

   on Dec 30, 2022

   Author

   More actions

   If twarc2 implies having API2 access, that's a separate issue I haven't find a way to clarify: I don't have API2 access, and I'm afraid applying for it means risking my existing API1.1 access. Until then, I need to stick with 1.1.

   I am ready to provide stacktrace and any further diagnostics for request timeout cases if that can help us to have some progress.
3. igorbrigadir commented on Dec 31, 2022

   

   igorbrigadir

   on Dec 31, 2022

   Contributor

   More actions

   Oh, i see what you mean - i don't know about losing v1.1 Access, but as far as i know, brand new API accounts all get v2 access. Keeping this open for the v1.1 Errors bug tho - i'll double check stuff there for now.
4. antibot4navalny commented on Jan 2, 2023

   

   antibot4navalny

   on Jan 2, 2023

   Author

   More actions

   Similar problem with twarc {friends|followers} when account is protected: exit code for requests.exceptions.HTTPError: 401 is the same 1 as for requests.exceptions.ReadTimeout.