Resuming collection after interruption #589

Activity

1. igorbrigadir

   added

   enhancement

   on Jan 27, 2022
2. Michael-Gauthier commented on Jan 28, 2022

   

   Michael-Gauthier

   on Jan 28, 2022

   More actions

   Hello @igorbrigadir, and everyone else! This is pretty crazy, because I came here to post the exact same question because of disconnection problems I randomly have in my office... So I do not really understand: is it a feature you are planning to add, or is it already possible to resume collections after interruptions? Thanks a lot in advance, as usual! : )
3. edsu commented on Jan 28, 2022

   

   edsu

   on Jan 28, 2022

   Last edited by edsu

   Member

   More actions

   I agree it would be a nice feature to have (it doesn't exist yet), and shouldn't be too tricky given next_token is already persisted to the output?

   👍React with 👍1Reacted by Igor Brigadir
4. Michael-Gauthier commented on Jan 28, 2022

   

   Michael-Gauthier

   on Jan 28, 2022

   More actions

   Gods, when it is implemented, I'll send you guys chocolates or whatever, that will be so useful! ^^

   Thanks again for your hard work and efforts to keep improving the tool by the way! : )
5. igorbrigadir commented on Jan 28, 2022

   

   igorbrigadir

   on Jan 28, 2022

   ContributorAuthor

   More actions

   is it a feature you are planning to add, or is it already possible to resume collections after interruptions?

   Kinda both, it doesn't exist in the command line twarc yet, but it's possible to do this with the library, https://twittercommunity.com/t/pulling-a-large-data-set-with-twarc2-client/165685/6?u=igorbrigadir all you need is to extract a next_token and specify it as a parameter into the function and it should continue - but i need to double check that, and make sure that using old next_token values actually works like this.

   👍React with 👍1Reacted by Michael-Gauthier
6. Michael-Gauthier commented on Jan 28, 2022

   

   Michael-Gauthier

   on Jan 28, 2022

   Last edited by Michael-Gauthier

   More actions

   Thanks for the feedback, and for the link to the thread! So, if I understand correctly, I would need to find the next_token, add this to my "normal" twarc2 query program, and it should resume where it stopped?
7. igorbrigadir commented on Jan 28, 2022

   

   igorbrigadir

   on Jan 28, 2022

   ContributorAuthor

   More actions

   Yes, except there's no way to add the next_token to twarc2 yet - i'll have add that part in.

   👍React with 👍1Reacted by Michael-Gauthier
8. Michael-Gauthier commented on Jan 28, 2022

   

   Michael-Gauthier

   on Jan 28, 2022

   More actions

   Ok, thanks a lot for your reactivity and your clarification! : )
9. igorbrigadir

   added

   good first issueGood for newcomersGood for newcomers

   on Mar 20, 2022
10. SamHames

    removed

    good first issueGood for newcomersGood for newcomers

    on Sep 16, 2022
11. SamHames commented on Sep 16, 2022

    

    SamHames

    on Sep 16, 2022

    Contributor

    More actions

    Thinking about this because of #656 - I think there's two layers to this:

    1. The client methods need to consistently take the pagination token as a keyword argument everywhere (this is the easy bit)
    2. For the command line, it's actually a bit harder when we consider the bulk commands that are actually the biggest targets for resuming an interrupted operation.
       • At a minimum to resume in these cases we'd have to think about reading the last page of results, confirming whether we need to resume that last collected search based on the presence/absence of a next_page token, then find the appropriate point in the input file to resume from.
       • We'd also have to think about the actual file writing workflow, because we'd have to consider whether we resume from and continue appending to the same file, or do something else. My suggestion is that if the --resume argument is passed, then it makes sense to read from the output file to sync the state, then continue appending to that file.